Page 1 of 1
Silly strings...
Posted: Wed Jun 15, 2011 1:52 pm
by Chief
Since this isn't related to the LOVE part of lua, i just dropped it here.
I don't know much about the stuff that goes on "backstage" in LUA, but i noticed something odd the other day...
I were making this "table/nested tables to file function", which basically made the table into a very long string. When i finished the function, I wanted to test the limits - so i made a humongous table! After about 4-5 seconds LOVE crashed, which was odd. I figured that it had to be the string that was getting too long, since the writing process hadn't started yet.
The question to you guys: Does LUA have a "max signs in string limit"... or was it gremlins?
PS. the function was running in a coroutine, and adding each index in the table to a string for each turn!
Re: Silly strings...
Posted: Wed Jun 15, 2011 2:09 pm
by Robin
Can we see your code? I have a suspicion about the problem, which won't occur if you use table.concat, but usually does when concatenating strings in other ways.
Re: Silly strings...
Posted: Wed Jun 15, 2011 2:21 pm
by Chief
The code is extremely braided and long, but i'll pick out the important parts:
Reading the table:
Code: Select all
-- Yada-yada lots of generation code
local str = "local map = {}\n"
local function add( s )
str = str .. s;
end
local num = 1
local function write(n, str)
local path = "maps/" .. name .. "/"
if love.filesystem.isDirectory( path ) then
local success = love.filesystem.write( path .. tostring(n) .. ".lua", str )
return success;
else
local ok = love.filesystem.mkdir( path )
if ok then
local success = love.filesystem.write( path .. tostring(n) .. ".lua", str )
return success;
else
return false, "could not save table"
end
end
end
for x, t in pairs(map) do
add( "map["..x.."] = {}\n" )
for y, g in pairs(map[x]) do
add( "map["..x.."]["..y.."] = {}\n" )
add( table.getString( "map["..x.."]["..y.."]", g ) )
end
add( "return map;" )
write(num, str)
num = num + 1
str = "local map = {}\n"
percent = 25 + (x/#map)*75
-- this is where i reran the thread.
end
end
The table-to-string code:
Code: Select all
function table.getString( base, tab )
local str = ""
local function add( s )
str = str .. s;
end
for id, info in pairs(tab) do
local text = base
if type(id) == "string" then
text = text .. "['" .. id .. "'] = "
elseif type(id) == "number" then
text = text .. "[" .. id .. "] = "
else
text = text .. "." .. id .. " = "
end
if type(info) == "string" then
add( text .. "'" .. tostring(info) .. "'" .. "\n" )
elseif type(info) == "bool" or type(info) == "number" then
add( text .. tostring(info) .. "\n" )
elseif type(info) == "table" then
add( text .. tostring(table.nestedTableString( info )) .. "\n" )
else
add( text .. tostring(info) .. "\n" )
end
end
return str;
end
function table.nestedTableString( tab )
local str = "{ "
local function add( s )
str = str .. s;
end
for k,v in pairs( tab ) do
if type(v) == "string" then
add( "'" .. tostring(v) .. "'" .. ", " )
elseif type(v) == "bool" or type(v) == "number" then
add( tostring(v) .. ", " )
elseif type(v) == "table" then
add( tostring(table.nestedTableString( v )) .. ", " )
else
add( tostring(v) .. ", " )
end
end
add( "}" )
return str
end
I "fixed" it by making a separate file for each index which contains another table...
The important part is that it crashes when the "str" is too long.
By the way - you may all use the table functions if you want to.
Re: Silly strings...
Posted: Wed Jun 15, 2011 2:31 pm
by Robin
It seems I was right.
The problem is, concatenating strings makes a new copy. Thus a = b .. c has three strings: b, c and b .. c. If you concatenate yet another string, you get a = b .. c .. d. Here you get five strings: b, c, d, b .. c and b .. c .. d. When you have one million strings in a table, you will need to create one million (minus one) new strings. What's worse, making strings doesn't take constant time: the longer the strings get, the more bytes need to be copied. So in the end, that explodes (usually because there is no memory left for the program to use).
Re: Silly strings...
Posted: Wed Jun 15, 2011 3:17 pm
by Chief
Ah right! Thanks!
Re: Silly strings...
Posted: Wed Jun 15, 2011 3:32 pm
by kikito
It's much better to use table.concat:
Code: Select all
a = a .. b
a = a .. c
a = a .. d
return a
This creates 7 strings in memory: the initial a, b, c & d, plus a .. b, a .. c and finally a .. d. It gets worse with every bit added to a.
Using table.concat, you only create a,b,c,d & the final concatenation, plus a table:
Code: Select all
local buffer = {}
table.insert(buffer, a)
table.insert(buffer, b)
table.insert(buffer, c)
table.insert(buffer, d)
return table.concat(buffer)
With a big number of strings, tables perform much better, memory-wise.
Re: Silly strings...
Posted: Thu Jun 16, 2011 5:16 pm
by tentus
Robin wrote:It seems I was right.
The problem is, concatenating strings makes a new copy. Thus a = b .. c has three strings: b, c and b .. c. If you concatenate yet another string, you get a = b .. c .. d. Here you get five strings: b, c, d, b .. c and b .. c .. d. When you have one million strings in a table, you will need to create one million (minus one) new strings. What's worse, making strings doesn't take constant time: the longer the strings get, the more bytes need to be copied. So in the end, that explodes (usually because there is no memory left for the program to use).
Whuh... WHY would it be set up this way? Whats the advantage?
Re: Silly strings...
Posted: Thu Jun 16, 2011 5:24 pm
by bartbes
Advantage? That's just the way it is done. Note that this is because of the way C strings work.
EDIT: Oh right, as kikito's link says, the string objects stay around until the next garbage collection cycle, so yeah, that too.
As this is not a single operation, a .. b .. c is actually a .. (b .. c) (the order I don't know exactly, but you get the point), and each has a result that is then discarded and then there's a wait for the garbage collector, it takes a lot of memory for a short period of time. Due to the way the GC works, in such a loop, it probably won't kick in, or at least not often enough.
Re: Silly strings...
Posted: Thu Jun 16, 2011 5:28 pm
by kikito
I think this explains the issue better than I could:
http://www.lua.org/pil/11.6.html
Re: Silly strings...
Posted: Thu Jun 16, 2011 6:14 pm
by Robin
bartbes wrote:As this is not a single operation, a .. b .. c is actually a .. (b .. c) (the order I don't know exactly, but you get the point)
You're right.
Lua 5.1 Reference Manual wrote:The concatenation ('..') and exponentiation ('^') operators are right associative.
Which actually makes it slightly better for repeatedly adding short strings to long strings, but it's still awful of course.
tentus wrote:Whuh... WHY would it be set up this way? Whats the advantage?
Thing is, the options are really this or mutable strings. And mutable strings just don't work so well in languages like Lua, since it probably means you'll be changing the same string in other places unexpectedly as well.