I am investigating some sort of a memory leak in my rust application. The memory keeps growing at a steady pace (starts at around 30M and grows 50M per day). The application is written in rust and it pretty much just attaches an XDP/BPF program to a few interfaces and reads some data via BPF maps and publishes some metrics.
Interestingly, the "leak" is only occurring in our test environment (memory usage is very stable in production so far).
Below is some pmap output (I have filtered out *.so and anon_inode:[perf_event] as there are quite a few and they are not part of the issue).
Looking at this it seems that my leak is not on the heap (so no point in analyzing the heap). I want to understand what the unnamed mappings to other objects are as they seem to be growing (see >>>).
Any ideas or tools I can use to figure out what is taking up this memory?
Address Perm Offset Device Inode Size Rss Pss Referenced Anonymous LazyFree ShmemPmdMapped FilePmdMapped Shared_Hugetlb Private_Hugetlb Swap SwapPss Locked THPeligible Mapping
55f8470e2000 r-xp 00000000 fd:04 3145822 5820 4472 4472 4468 0 0 0 0 0 0 0 0 0 0 my-program-name
55f847891000 r--p 005af000 fd:04 3145822 368 368 368 368 368 0 0 0 0 0 0 0 0 0 my-program-name
55f8478ed000 rw-p 0060b000 fd:04 3145822 12 12 12 12 12 0 0 0 0 0 0 0 0 0 my-program-name
55f848a15000 rw-p 00000000 00:00 0 132 108 108 108 108 0 0 0 0 0 0 0 0 0 [heap]
>>> 7f14f8000000 rw-p 00000000 00:00 0 24140 20620 20620 20620 20620 0 0 0 0 0 0 0 0 0
7f14f9793000 ---p 00000000 00:00 0 41396 0 0 0 0 0 0 0 0 0 0 0 0 0
7f1500000000 rw-p 00000000 00:00 0 132 12 12 12 12 0 0 0 0 0 0 0 0 0
7f1500021000 ---p 00000000 00:00 0 65404 0 0 0 0 0 0 0 0 0 0 0 0 0
>>> 7f1504000000 rw-p 00000000 00:00 0 55792 47976 47976 47976 47976 0 0 0 0 0 0 0 0 0
7f150767c000 ---p 00000000 00:00 0 9744 0 0 0 0 0 0 0 0 0 0 0 0 0
7f1508000000 rw-p 00000000 00:00 0 65496 52136 52136 52080 52136 0 0 0 0 0 0 0 0 0
7f150bff6000 ---p 00000000 00:00 0 40 0 0 0 0 0 0 0 0 0 0 0 0 0
7f150c000000 rw-p 00000000 00:00 0 132 36 36 36 36 0 0 0 0 0 0 0 0 0
7f150c021000 ---p 00000000 00:00 0 65404 0 0 0 0 0 0 0 0 0 0 0 0 0
7f1510000000 rw-p 00000000 00:00 0 65484 62776 62776 62776 62776 0 0 0 0 0 0 0 0 0
7f1513ff3000 ---p 00000000 00:00 0 52 0 0 0 0 0 0 0 0 0 0 0 0 0
7f15172de000 ---p 00000000 00:00 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0
7f15172df000 rw-p 00000000 00:00 0 2048 20 20 20 20 0 0 0 0 0 0 0 0 0
7f15176f3000 rw-p 00000000 00:00 0 8 0 0 0 0 0 0 0 0 0 0 0 0 0
7f1517b0a000 rw-p 00000000 00:00 0 24 0 0 0 0 0 0 0 0 0 0 0 0 0
7f1517b13000 ---p 00000000 00:00 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0
7f1517b14000 rw-p 00000000 00:00 0 8 0 0 0 0 0 0 0 0 0 0 0 0 0
7f1517b16000 ---p 00000000 00:00 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0
7f1517b17000 rw-p 00000000 00:00 0 8 0 0 0 0 0 0 0 0 0 0 0 0 0
7f1517b19000 ---p 00000000 00:00 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0
7f1517b1a000 rw-p 00000000 00:00 0 2048 20 20 20 20 0 0 0 0 0 0 0 0 0
7f1517eeb000 ---p 00000000 00:00 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0
7f1517eec000 rw-p 00000000 00:00 0 2048 56 56 56 56 0 0 0 0 0 0 0 0 0
7f15180ec000 ---p 00000000 00:00 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0
7f15180ed000 rw-p 00000000 00:00 0 2048 56 56 56 56 0 0 0 0 0 0 0 0 0
7f1518696000 rw-p 00000000 00:00 0 16 16 16 16 16 0 0 0 0 0 0 0 0 0
7f1518df8000 rw-p 00000000 00:00 0 16 4 4 4 4 0 0 0 0 0 0 0 0 0
7f1519665000 rw-p 00000000 00:00 0 12 12 12 12 12 0 0 0 0 0 0 0 0 0
7f1519ae1000 ---p 00000000 00:00 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0
7f1519ae2000 rw-p 00000000 00:00 0 28 16 16 16 16 0 0 0 0 0 0 0 0 0
7f1519ae9000 ---p 00000000 00:00 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0
7f1519aea000 rw-p 00000000 00:00 0 8 0 0 0 0 0 0 0 0 0 0 0 0 0
7f1519aec000 ---p 00000000 00:00 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0
7f1519aed000 rw-p 00000000 00:00 0 16 8 8 8 8 0 0 0 0 0 0 0 0 0
7f1519af3000 rw-p 00000000 00:00 0 4 4 4 4 4 0 0 0 0 0 0 0 0 0
7ffd52d94000 rw-p 00000000 00:00 0 132 32 32 32 32 0 0 0 0 0 0 0 0 0 [stack]
7ffd52ddd000 r--p 00000000 00:00 0 12 0 0 0 0 0 0 0 0 0 0 0 0 0 [vvar]
7ffd52de0000 r-xp 00000000 00:00 0 8 4 0 4 0 0 0 0 0 0 0 0 0 0 [vdso]
ffffffffff600000 r-xp 00000000 00:00 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 [vsyscall]
====== ====== ====== ========== ========= ======== ============== ============= ============== =============== ==== ======= ====== ===========
455904 193740 191239 193668 184644 0 0 0 0 0 0 0 0 0 KB
