Huge map and memory use

[Spessman]

Hi everyone.

I create map with 300x300x10 tiles and i faced with big problems. My game crashes if i try exceed 250x250x10 limit. And I decided to check processes. ~1 GB memory used...
So there i really need help. Maybe split this map on chunks and save their in files? But how i can do this? And how i can reduce memory usage?

[zorg]

According to my calculations (300*300*10=900 000, 1 gigabyte / 900k ~= 1kb per tile), i'd like you to post some code since i don't know how you use that much space per tile. Just want to check for common errors, you know.

[Spessman]

It is not so important. You can run this code and receive the same result.

Code: Select all

map = {}
	for x = 1, 1000 do
			map[x] = {}
			for y = 1, 1000 do
				map[x][y] = {}
				for z = 1, 100 do
					map[x][y][z] = 1
				end
			end
	end

I think i must split map on chunks and save them to files. But i'm worry about this part may be bulky.
1000x1000x10 >> 100x100x2 chunk = ~500 chunks. And this is only tiles... I don't know how to store other objects and where?

Now i'm working with small map, but it needs to be increased.
Well, memory is smallest problem for now...

[HugoBDesigner]

If the map is procedurally-generated, I think you should load parts of the map as the player goes. Otherwise, save the chunks as files, then load each chunk as the player goes as well. Not much to do from that :T

[zorg]

Also, while there are many ways to chunk up the map, and many ways to load the chunks, there's a simple algorithm for top-down type games. Depending on where the character is in the world (its x and y position), calculate what tiles it stands on, then make sure that an area thrice the size of the screen is loaded around him at all times. You might forego the tile calculation if you store that with the character also. Loading the area outside on all sides means it can go around without any noticeable chunks not yet loaded. The 3x area is because you have 4 sides, 4 corners + the visible area (=9), so along each dimension, loading 3 chunks (offset by -screenWidthInTiles, 0, screenWidthInTiles and -screenHeightInTiles, 0, screenHeightInTiles) makes loading seamless.

[ivan]

Take a look at this post:
viewtopic.php?f=5&t=80725#p187848
It explains how to store maps in a 1D array/table.

1D indexes could remove the need to create intermediate tables (in your case: 1000*1000 extra tables)
The only downside being that you have to know/define the dimensions of the map ahead of time.

[bobbyjones]

Yup I would recommend using a 1D array and if you need to save even more space then use ffi structs because they have less overhead than a table and works just about the same. For each dimension you add to a table more and more memory would be used. My first link talks about lua's memory usage. Luajit would most likely use less than plain lua but im sure the memory allocation scheme is similar. My second link is an example of me hacking ffi structs into jumper to make it use much less memory. The third link is another example of ffi structs being used to represent vectors.

http://wowwiki.wikia.com/wiki/Lua_object_memory_sizes

https://github.com/Bobbyjoness/Jumper/c ... 4f8f2f9ae8

https://github.com/excessive/cpml/blob/ ... s/vec2.lua

[kikito]

First, make sure you are storing as little information as possible for each cell.

Also, if most of your cells hold the same kind of data (i.e. most of them are empty, or most of them are "full of the same thing" (i.e. earth) you will use less memory if you use a sparse model (you just store the cells which "have something" - in a list or however you want, and assume that the rest have "the default value" when drawing or otherwise interacting with them).

[Spessman]

Thanks, guys.
But i have another problems...

If the map is procedurally-generated...

Yes, but map have procedurally-generated structures and if generator started build room in one chunk, how continue building in another chunk? I mean there is not a single houses. There is massive continuous complex with pack of rooms. How i must generate map like this? Load all used chunks? If complex occupy 400x400 tiles...
This is my last serious problem. I can tell about this more if you want.

The 3x area is because you have 4 sides, 4 corners + the visible area (=9)

Yeah, thanks for explanation.

It explains how to store maps in a 1D array/table.

Thank you. Now i can store my map in one table. But your formula works a little wrong for me.
I correct this to: ( y - 1 ) * mapWidth + x and this works fine. Thanks, really smart formula and useful.

ffi structs

Wow... Thank you. I'm attentively study this feature. You did my life better.

First, make sure you are storing as little information as possible for each cell.

I keep in one cell one tile or string if this nil and i can't store less info. But thanks.


I never do something serious before(like split maps in chunks, advanced work with filesystem). I hope i can do this. Thanks, guys.
And now i have one problem left. With structs generation.

[bobbyjones]

You need help using structs? If so I can help. Just tell me what data you need for each point. If it's only 1 value then structs wouldn't help much. But if you are storing a value and a string and etc then list it all so that you would know what is the proper struct representation.