Puzzled by a potential memory leak

[Froyok]

Hi folks !

I'm puzzled by the behavior of Love, or more precisely lua, on my current game.
I have noticed a memory leak in my project, which can be fixed my running the garbage collector manually.

To describe the issue: when I load meshes for my custom 3D project, then run Dear ImGui (via cimgui) to render my debug UI, the result of collectgarbage( "count" ) gains 1mb every seconds, if not more.

Below are just love.graphics.print of the output of collectgarbage("count") every frame:

• Only running ImGui, no meshes:

  

  love_memory_noleak.gif (58.47 KiB) Viewed 4152 times

• Only loading my meshes, no ImGui:

  

  love_memory_noimgui.gif (6.09 KiB) Viewed 4152 times

• Loading meshes, then running ImGui:

  

  love_memory_leak.gif (44.21 KiB) Viewed 4152 times

Any suggestion on what could be happening or how to investigate it further ?
I'm not sure what to think about this issue, where it comes from or why it happens.

Details I can add:

• My mesh files are just lua file, with are a big table containing the vertices and a map to use with newMesh().
• My mesh file are loaded during onLoad() and cached in a local table in my main.lua file.
• My mesh file are loading by using love.filesystem.load() and not a require.
• ImGui Update() and NewFrame() run during the love love.update() function, then Render() and RenderDrawLists() during love.draw().
• I don't draw any UI with ImGui, just call newFrame().
• If I don't call imgui.newFrame() while the meshes are loaded, no leak (see gif above).
• The leak can be cleared by calling collectgarbage( "collect" ), but it needs to happen often and manually. Lua doesn't clean it by itself even after a while.
• If I only load a sub-set of the full list of meshes, I can see the memory going up still but it gets garbage collected after a while.
  The threshold seems to change depending on how much memory/meshes I load in first place. The memory still goes up quite high however for no specific reason.

[Froyok]

Okay so after playing a bit with more stuff, I think I have a better understanding of what is happening:

• The more memory I consume, the higher lua garbage collection threshold seems to be.
• ImGui generate garbage each frame, likely related to its drawing.
• On a small initial memory footprint imgui memory gets collected pretty quickly and doesn't accumulate.
• On a large initial memory footprint, imgui memory accumulate and balloon quite a lot. It takes a while to get cleared when quitting the game.

So I wonder now if there is a way to adjust the threshold, or to handle manually the garbage collection without impacting the cpu performance. I stumbled upon this, which could be helpful: https://gist.github.com/1bardesign/3ed0 ... da7fa3076d

Another solution would be to make ImGui memory usage more conservative, but given the nature of the tool I'm not sure.

[Froyok]

Alright so after reading a bit more about LUA garbage collector (here and here), I got confirmation that by default the GC runs when the memory double, which explains the behavior I saw.

I ended up going with Batteries manual_gc.lua code to solve my problem: https://github.com/1bardesign/batteries ... ual_gc.lua

[UnixRoot]

Hi, I think I have a similar problem. For my Pseudo3D engine I'm using a bunch of meshes. Each mesh has around 2.000 to 8.000 vertices that I have to stream new vertex positions while moving along the track.

I don't use tables, everything is set up with FFI, C structs and pointers. I even reuse the same meshes to draw every part of the landscape to a canvas.

It's fast, computation time is really low, even on my old office computer. But how can I get rid of the garbage, without introducing stuttering? The problem is, if I stop the garbage collector, it produces 500 MB of garbage per minute. If I start it, I get massive stuttering.

Does mesh:setVertices( data ) really produce that much garbage? What can I do to generate less garbage?

[Sasha264]

I don't use tables, everything is set up with FFI, C structs and pointers. I even reuse the same meshes to draw every part of the landscape to a canvas.

That sounds like a nice job

It's fast, computation time is really low, even on my old office computer. But how can I get rid of the garbage, without introducing stuttering? The problem is, if I stop the garbage collector, it produces 500 MB of garbage per minute. If I start it, I get massive stuttering.

I stepped on these same rakes twice (shame on me) in my life: with Lua (Love2d) and with C# (Unity). Spent a lot of days and months trying to solve this without satisfactory result. Steps to reproduce:

1. Use some engine based on a language with garbage collection.
2. Write some simple stuff, enjoy life.
3. Try to write some more complicated stuff and encounter some performance issues.
4. Do some light performance optimizations.
5. Encounter some serious performance problems caused by garbage collection or by reference types.
6. Do some heavy performance optimization with Lua C FFI and learn how to write Lua code that generates "less garbage". Try some "memory locality" optimizations.
7. Realize that now your code is insanely complex. A lot more complex than it would be if it was initially written in just C++ from the start. And still has a lot less performance.
8. Start to think about transferring all your existing codebase to C++. "Maybe I can do it in just 5 years, if I work hard"

So, when I do something in C++, it can be like that:

• "You want to parse 14 Gb of JSON files?"
• "Yes, I do some machine learning."
• "Do you have enough memory to store the result?"
• "Yes."
• "Well okay, it will work then. And will primarily depend on the read speed of your HDD/SSD (which is 100-1000 Mb/s)."

• 

  strong.png (8.14 KiB) Viewed 3528 times

And in Lua, it looks like this:

• "You want to load ~2 Mb JSON save game file which contains 15 000 buildings for your strategy game?"
• "Yes, is that a problem?"
• "Well, okay, I will do it in like 10 seconds, use 500 Mb of memory, generate tons of garbage in the process, and it will lag like crazy later thanks to garbage collections."

• 

  weak.png (3.65 KiB) Viewed 3528 times

For me, I found a solution on how to use garbage collection in Love2d without stuttering.
It helps to spread the GC evenly over the frames. It will be slower in total GC time, but will not be focused on rare "unlucky" frames.
I just do something simple like this every frame:

Code: Select all

local start = love.timer.getTime()
for i = 1, 100 do
    collectgarbage("step", 5)
    if love.timer.getTime() - start > 0.001 then
        break
    end
end

The more complex solution was mentioned earlier (Batteries manual_gc.lua). Thanks for that link, by the way! It somewhat works, stutters are gone, but still, you cannot have more than 100 Mb of Lua tables and live a good life at the same time.

[UnixRoot]

1. Use some engine based on a language with garbage collection.
2. Write some simple stuff, enjoy life.
3. Try to write some more complicated stuff and encounter some performance issues.
4. Do some light performance optimizations.
5. Encounter some serious performance problems caused by garbage collection or by reference types.
6. Do some heavy performance optimization with Lua C FFI and learn how to write Lua code that generates "less garbage". Try some "memory locality" optimizations.
7. Realize that now your code is insanely complex. A lot more complex than it would be if it was initially written in just C++ from the start. And still has a lot less performance.
8. Start to think about transferring all your existing codebase to C++. "Maybe I can do it in just 5 years, if I work hard"

Yep, that's me, but you forgot step 9. After everything fails, over optimize it to the point of no return

I'm just trying your approach and I have to say its A LOT smoother now, but the memory leak is still present. It just grows way slower. It seems like the garbage collector doesn't collect all the of garbage.

[Sasha264]

...but you forgot step 9...

Maybe I haven't reached that point yet...

I'm just trying your approach and I have to say its A LOT smoother now, but the memory leak is still present. It just grows way slower. It seems like the garbage collector doesn't collect all the of garbage.

Hmm, the second part sounds disturbing
Maybe there is an actual leak due to a logic error, rather than a garbage collector whim. You can try calling `collectgarbage()` every frame (let it lag for the test), to ensure that the memory consumption isn't increasing over time.

[UnixRoot]

If I collect every frame, the memory consumption stays at a stable level for hours, but it lags really bad. It's not a real leak, it's really a garbage problem.

But I don't know how to generate less garbage. I read a lot about optimization and there isn't much left I can do. I already use pretty efficient and optimized solutions for most of my stuff.

[dusoft]

If I collect every frame, the memory consumption stays at a stable level for hours, but it lags really bad. It's not a real leak, it's really a garbage problem.

But I don't know how to generate less garbage. I read a lot about optimization and there isn't much left I can do. I already use pretty efficient and optimized solutions for most of my stuff.

Just collect the garbage and the issue is solved, right?

[UnixRoot]

If you have a solution to collect the garbage every frame, that doesn't introduce massive lag, then yes, it's solved.