Silly strings...

[Chief]

Since this isn't related to the LOVE part of lua, i just dropped it here.

I don't know much about the stuff that goes on "backstage" in LUA, but i noticed something odd the other day...

I were making this "table/nested tables to file function", which basically made the table into a very long string. When i finished the function, I wanted to test the limits - so i made a humongous table! After about 4-5 seconds LOVE crashed, which was odd. I figured that it had to be the string that was getting too long, since the writing process hadn't started yet.

The question to you guys: Does LUA have a "max signs in string limit"... or was it gremlins?

PS. the function was running in a coroutine, and adding each index in the table to a string for each turn!

[Robin]

Can we see your code? I have a suspicion about the problem, which won't occur if you use table.concat, but usually does when concatenating strings in other ways.

[Chief]

The code is extremely braided and long, but i'll pick out the important parts:

Reading the table:

Code: Select all

-- Yada-yada lots of generation code

local str = "local map = {}\n"
		local function add( s )
			str = str .. s;
		end
		local num = 1

local function write(n, str)
			local path = "maps/" .. name .. "/"
			if love.filesystem.isDirectory( path ) then
				local success = love.filesystem.write( path .. tostring(n) .. ".lua", str )
				return success;
			else
				local ok = love.filesystem.mkdir( path )
				if ok then
					local success = love.filesystem.write( path .. tostring(n) .. ".lua", str )
					return success;
				else
					return false, "could not save table"
				end
			end
end

for x, t in pairs(map) do
		
			add( "map["..x.."] = {}\n" )
			
			for y, g in pairs(map[x]) do
			
				add( "map["..x.."]["..y.."] = {}\n" )
				
				add( table.getString( "map["..x.."]["..y.."]", g ) )
			end
			add( "return map;" )
			
			write(num, str)
			num = num + 1
			str = "local map = {}\n"
			percent = 25 + (x/#map)*75
			-- this is where i reran the thread.
		end
end

The table-to-string code:

Code: Select all

function table.getString( base, tab )
	local str = ""
	local function add( s )
		str = str .. s;
	end
	for id, info in pairs(tab) do
		local text = base
		if type(id) == "string" then
			text = text .. "['" .. id .. "'] = "
		elseif type(id) == "number" then
			text = text .. "[" .. id .. "] = "
		else
			text = text .. "." .. id .. " = "
		end
		
		if type(info) == "string" then
			add( text .. "'" .. tostring(info) .. "'" .. "\n" )
		elseif type(info) == "bool" or type(info) == "number" then
			add( text .. tostring(info) .. "\n" )
		elseif type(info) == "table" then
			add( text .. tostring(table.nestedTableString( info )) .. "\n" )
		else
			add( text .. tostring(info) .. "\n" )
		end
	end
	return str;
end

function table.nestedTableString( tab )
	local str = "{ "
	local function add( s )
		str = str .. s;
	end
	for k,v in pairs( tab ) do
		if type(v) == "string" then
			add( "'" .. tostring(v) .. "'" .. ", " )
		elseif type(v) == "bool" or type(v) == "number" then
			add( tostring(v) .. ", " )
		elseif type(v) == "table" then
			add( tostring(table.nestedTableString( v )) .. ", " )
		else
			add( tostring(v) .. ", " )
		end
	end
	
	add( "}" )
	
	return str
end

I "fixed" it by making a separate file for each index which contains another table...

The important part is that it crashes when the "str" is too long.

By the way - you may all use the table functions if you want to.

[Robin]

It seems I was right.

The problem is, concatenating strings makes a new copy. Thus a = b .. c has three strings: b, c and b .. c. If you concatenate yet another string, you get a = b .. c .. d. Here you get five strings: b, c, d, b .. c and b .. c .. d. When you have one million strings in a table, you will need to create one million (minus one) new strings. What's worse, making strings doesn't take constant time: the longer the strings get, the more bytes need to be copied. So in the end, that explodes (usually because there is no memory left for the program to use).

[Chief]

Ah right! Thanks!

[kikito]

It's much better to use table.concat:

Code: Select all

a = a .. b
a = a .. c
a = a .. d
return a

This creates 7 strings in memory: the initial a, b, c & d, plus a .. b, a .. c and finally a .. d. It gets worse with every bit added to a.

Using table.concat, you only create a,b,c,d & the final concatenation, plus a table:

Code: Select all

local buffer = {}
table.insert(buffer, a)
table.insert(buffer, b)
table.insert(buffer, c)
table.insert(buffer, d)
return table.concat(buffer)

With a big number of strings, tables perform much better, memory-wise.

[tentus]

It seems I was right.

The problem is, concatenating strings makes a new copy. Thus a = b .. c has three strings: b, c and b .. c. If you concatenate yet another string, you get a = b .. c .. d. Here you get five strings: b, c, d, b .. c and b .. c .. d. When you have one million strings in a table, you will need to create one million (minus one) new strings. What's worse, making strings doesn't take constant time: the longer the strings get, the more bytes need to be copied. So in the end, that explodes (usually because there is no memory left for the program to use).

Whuh... WHY would it be set up this way? Whats the advantage?

[bartbes]

Advantage? That's just the way it is done. Note that this is because of the way C strings work.

EDIT: Oh right, as kikito's link says, the string objects stay around until the next garbage collection cycle, so yeah, that too.
As this is not a single operation, a .. b .. c is actually a .. (b .. c) (the order I don't know exactly, but you get the point), and each has a result that is then discarded and then there's a wait for the garbage collector, it takes a lot of memory for a short period of time. Due to the way the GC works, in such a loop, it probably won't kick in, or at least not often enough.

[kikito]

I think this explains the issue better than I could:

http://www.lua.org/pil/11.6.html

[Robin]

As this is not a single operation, a .. b .. c is actually a .. (b .. c) (the order I don't know exactly, but you get the point)

You're right.

The concatenation ('..') and exponentiation ('^') operators are right associative.

Which actually makes it slightly better for repeatedly adding short strings to long strings, but it's still awful of course.

Whuh... WHY would it be set up this way? Whats the advantage?

Thing is, the options are really this or mutable strings. And mutable strings just don't work so well in languages like Lua, since it probably means you'll be changing the same string in other places unexpectedly as well.