Objective Lua - additional syntax to original Lua!

[pgimeno]

YACC and Flex also use regular experessions in the lexing phase so yes it is common practice.

Agreed.

Regex is probaby slower than a hand written lexer like pgimeno's but regex is much easier to read and modify.

Yeah, my hand written lexer basically mimics what Flex would generate.

Also, you still need a parser to handle the actual grammar, lexing is just the first phase.

Yes but in this case, the OP's grammar is so simple that there's very little to do after lexing, grammar wise. He doesn't need to identify the full Lua syntax, just the functions contained between 'class' and the matching 'end', and some keywords like 'inherits'.

[pgimeno]

It's quite fast because it uses string.byte, not string.sub

If you microbenchmark string.byte(i) vs. string.sub(i, i), you'll find that they're basically identical in performance. Garbage isn't an issue either for 256 interned strings.

http://www.formauri.es/personal/pgimeno ... g.byte.lua
http://www.formauri.es/personal/pgimeno ... ng.sub.lua

Code: Select all

$ luajit benchmark-string.byte.lua 
1.0272738933563
$ luajit benchmark-string.sub.lua 
2.4919781684875

That's almost 2.5 times slower.

[grump]

Code: Select all

$ luajit benchmark-string.byte.lua 
1.0272738933563
$ luajit benchmark-string.sub.lua 
2.4919781684875

That's almost 2.5 times slower.

Code: Select all

local str = love.filesystem.read('main.lua')

local s = love.timer.getTime()
for j = 1, 1e7 do
	for i = 1, #str do
		local a = str:byte(i)
	end
end
print("byte", love.timer.getTime() - s)

local s = love.timer.getTime()
for j = 1, 1e7 do
	for i = 1, #str do
		local a = str:sub(i, i)
	end
end
print("sub", love.timer.getTime() - s)

Code: Select all

byte	2.087397079
sub	2.096616101

[pgimeno]

Not sure where the difference lies, but it's clear that the "lab test" gives quite different results than my "field test".

[grump]

Not sure where the difference lies, but it's clear that the "lab test" gives quite different results than my "field test".

Obviously. Seems to be a JIT(?) quirk with the strip_strings_and_comments function, not generally slower performance of string.sub.

Out of curiosity, I removed code from your test until the performance difference vanished.

This code is slow:

Code: Select all

local function strip_strings_and_comments(code)
  local i = 1
  local len = #code
  while i <= len do
    local c = code:sub(i, i)
    if c == '"' or c == "'" then
    elseif c == "[" then
    end
    i = i + 1
  end
  return code
end

This code is fast (elseif line removed):

Code: Select all

local function strip_strings_and_comments(code)
  local i = 1
  local len = #code
  while i <= len do
    local c = code:sub(i, i)
    if c == '"' or c == "'" then
    end
    i = i + 1
  end
  return code
end

In this slightly modified test from before, the code using :sub is actually 15% faster than :byte:

Code: Select all

local str = love.filesystem.read('main.lua')

local n = 0
local s = love.timer.getTime()
for j = 1, 1e7 do
	for i = 1, #str do
		local a = str:byte(i)
		if i == 1 then n = n + 1 end
		-- if i == 1 then n = n + 1 end
	end
end
print("byte", love.timer.getTime() - s)

local n = 0
local s = love.timer.getTime()
for j = 1, 1e7 do
	for i = 1, #str do
		local a = str:sub(i, i)
		if i == 1 then n = n + 1 end
		-- if i == 1 then n = n + 1 end
	end
end
print("sub", love.timer.getTime() - s)

Code: Select all

byte	4.760761467
sub	4.056498461

Uncommenting the second condition in the inner loop makes the runtime explode to almost 9 times slower for the :sub code, while the :byte variant only gets marginally slower. LuaJIT is weird.

[lauriszz123]

Objective Lua Version 2.0 Released
Thank you all for your suggestions and comments. I've remade the whole parser and added test case support so that I can find any potential bugs! Right now everything should work as intended, even the Lua's:

Code: Select all

local str = [=[
--[[
some comment inside
--]]
]=]

Also as comments:

Code: Select all

--[=====[
  --[[
    some comment inside
  --]]
]=====]

[grump]

Here are some basic and some obscure things that shoudln't fail.
Helpful reference for parser authors: Lexical conventions

[lauriszz123]

Thanks, grump for your comment. I've fixed the needed Lexical Conventions for Lua, I will continue to improve the parser with that page in mind.

Objective Lua Version 2.2 Released
Thanks to grump for pointing out the issues.

In addition 2.1 version is also released with support for math operations such as:

Code: Select all

i += value
i -= value
i *= value
i /= value
i ^= value
i %= value

Thanks for the feedback everybody!

[pgimeno]

Generally, var *= expr does not translate to var = var * expr, it translates to var = var * (expr).

For example, most people would expect this:

Code: Select all

i = 1
j = 2
k = 5
k *= i + j
print(k)

to print 15, as it's the result of 5 * (1 + 2), and not 7, which is the result of 5 * 1 + 2.

However, I can't get that to work at all, so maybe it's a moot point.

[lauriszz123]

Thanks for the feedback pgimeno! I've fixed the issue in 2.3 release, but for precedence, I suppose you are right. I will look into that and will publish a patch soon, for now, I've fixed the issue that these math operations for some reason were not detected in classes.

Regards,
Laurynas.