Question about UDA server connections

[g2knizn]

Hi,

I have a question about UDA server connection and memory management for long running processes. Assume the following Access Layer code that uses UDA client:

import imas
while True:
    uri = input() # assume UDA backend
    with imas.DBEntry(uri, "r") as entry:
        summary = entry.get("summary")

With each subsequent uri that we pass, memory usage by UDA server (not UDA client) is increasing significantly until the UDA client process is finished or $UDA_TIMEOUT (default 10 mins) is reached. I guess that imas.close does not release memory.

So my question is - can I somehow release UDA server memory earlier or do you have any advice in this matter?

I would appreciate your help.

[stephen-dixon]

valgrind massif is a good tool for looking at this though you need to remove dl_close calls in the uda server code for it to report memory allocations in dynamically loaded libraries (like uda plugins). -- it shows how the memory footprint changes throughout the lifetime of the program e.g. if memory is freed at the end of the process (no leaks) but increases in unexpected ways while running. Otherwise valgrind will show memory leaks if they are an issue (maybe in the uda capnp? functions)

In general, I think memory allocated within the server for data it transports to the client is then freed once it has been sent. Nothing should accumulate ideally.
Things to note are:

1. that plugins may cache data. e.g. imas plugins holds a cache of ids requests/contexts (not data though) -- it looks like this is cleared in the close function but I don't work on imas myself so I'm not very familiar. Having said that, I believe (from memory only...) the imas plugin is set-up to accept batch requests (bundles of individual ids paths), it then processes the whole batch before passing all data back to the client in the imas lowlevel. so memory use will increase to the size of that batch before data is returned. I believe the batch size is a tuneable parameter somewhere in imas. If this were the only issue I think you'd see memory increase while processing requests but drop between them. Might need the opinion of someone more familiar with that system though to confirm what the expected behaviour is.
2. other libraries may reuse memory buffers instead of freeing them. probably not in use in your imas uda server, but if the server is reading (multiple) hd5 files directly then uda will cache multiple file handles and the hdf5 library generally holds onto a memory buffer the size of the maximum array you've read out of a file (which it reuses for each read), until you actually close that file. so each new file you open increases the memory footprint of the server for the lifetime of the server until you hit the file cache limit, depending on logic of your plugin.

Let me know what plugins are running on your server. That's probably what needs looking at and profiling (by someone who works on imas if that's where the issue is). A plugin not clearing a cache when expected (e.g. during a close function) is probably an easy fix. if it's memory leaks in the server library we can help fix those.

For short-term workarounds, API functions do exist in the client to close the current server connection (forcing a new one on the next data request) or to reduce the server lifetime to less than the default 10 minutes to limit memory growth, but I'm not sure either are exposed through imas, and it would probably be better to get to the root of the problem anyway.

[g2knizn]

Thanks for such a deep explanation. We only use imas UDA plugin. Indeed, decreasing value of$UDA_TIMEOUT env variable helped. And it is probably the easiest solution.

But it would be nice to find the root cause since we use UDA in a long running process with a large number of HDF5 files read. You are probably right that the problem lies with the UDA plugin. I run UDA server with valgrind and massif and received the following results:

valgrind.log
massif.log

Could you please take a look and decide should I create issue for UDA plugins?

[jholloc]

It looks like a lot of memory is ending up in capnp::MallocMessageBuilder::allocateSegment. This suggests that the capnp buffer is not being fully cleaned up after each request. This might be an issue with the plugin or internal to UDA itself - we'll have to investigate more.

The next biggest culprit is H5FL_reg_malloc which will be coming from the IMAS library as we aren't using HDF5 directly.

[jholloc]

There also seems to be 2 significant memory leaks with uda_capnp_get_root and uda_capnp_new_tree - I suspect we might be missing matching uda_capnp_free_root and uda_capnp_free_tree in the plugin.