LuaPreprocess - straightforward preprocessor with simple syntax

[ReFreezed]

LuaPreprocess is a library that adds a code preprocessing phase before your program runs. It allows you to execute code that then generates the final program.

Example:

Code: Select all

-- input.lua

-- Lines starting with "!" are part of the metaprogram.
!local DEV_MODE = true

-- Here "!()" outputs a value as a literal.
local gameData = !(parseJsonFile("game.json"))

function love.load()
	initGame()

	!if DEV_MODE then
		initLevelEditor()
	!else
		setupTelemetry()
	!end
end

Code: Select all

-- output.lua

-- This table is what parseJsonFile() returned.
local gameData = {classes={"Warrior","Archer","Wizard"},stats={"Health","Mana"}}

function love.load()
	initGame()

	initLevelEditor() -- We got this line because DEV_MODE was true.
end

See example with better highlighting
More examples at GitHub
Another example relevant to LÖVE

Usage:

Code: Select all

local pp = require("preprocess")

local info, err = pp.processFile{
	pathIn   = "game.lua2p",    -- This is the file we want to process.
	pathMeta = "game.meta.lua", -- Temporary output file for the metaprogram.
	pathOut  = "game.lua",      -- The output path.
}

if not info then
	error(err)
end

print("Lines of code processed: "..info.lineCount)

One benefit of all this is that certain tasks can be performed only once at build time instead of every time you run the program. For example, you may load and parse a bunch of data files only to finally convert the data to Lua tables or something. With preprocessing you can do that once at build time and enjoy a program that starts up and runs faster in the end.

Another benefit is that you can make choices in your metaprogram about what code the final program should have. For example, during development you may have a bunch of developer-only functionality that should be excluded from a public build. (Compare this to #if/#ifdef directives in C/C++.)

Yet another benefit is that some kinds of code are simply easier to write and look nicer if the code can run in multiple steps. Some boilerplate code can be reduced, for example.

Some key points about LuaPreprocess:

• It's all Lua - metaprograms are just normal Lua.
• Simplicity - use "!" to specify what code is part of the metaprogram.
• Robustness - processed files are parsed properly.
• Pure Lua - no external dependencies.
• Can be used as a library or command line program.

LuaPreprocess on GitHub
Direct link to library file
Direct link to command line program
Releases
Documentation

~~~

Why did I do this?

After having published a couple of games on Steam and itch.io I knew that having a "build phase" when making games was very helpful. I could automate tasks and generally reduce a bunch of unpleasant repetitive work. A while ago I started experimenting with taking things a step further. I began to process all Lua files (and eventually images and shaders etc. too) before ever running the game, even locally. This allowed me to easily include or exclude functionality that only certain versions of the game should have/not have. For example, my game may have a developer version, a normal public version and a demo version. All I have to do now is to change a boolean and my build system spits out a different program, or set another boolean and the system spits out a Zip file for every platform and OS, ready to be published. I think any serious LÖVE game could benefit from this kind of workflow.

Also, I'm aware of other preprocessing tools for Lua but none caught my interest. *shrugs*

[yetneverdone]

Hi, awesome work. How does one use this to preprocess many lua files?

Does it support preprocessing bunch of lua files and then output the those preprocessed files in another folder like in c/c++ turning files into obj files? What will be the command then?

Thanks

[ReFreezed]

In my case I have a build script that looks something like this:

Code: Select all

local pp = require("preprocess")

-- getFilesInFolderRecursively() may use for example LuaFileSystem.
local pathsRelativeToSrcFolder = getFilesInFolderRecursively("src")

for pathRelativeToSrcFolder in ipairs(pathsRelativeToSrcFolder) do
	local pathIn   = "src/"..pathRelativeToSrcFolder
	local pathMeta = "temp/"..pathRelativeToSrcFolder:gsub("/", ";")
	local pathOut  = "srcGenerated/"..pathRelativeToSrcFolder -- The game require()s files from this folder.

	local dirOut = pathOut:gsub("/[^/]+$", "")
	createDirectory(dirOut)

	pp.processFile{
		pathIn   = pathIn,
		pathMeta = pathMeta,
		pathOut  = pathOut,

		onError = function(err)
			os.exit(1)
		end,
	}
end

-- Start the game.
os.execute("love .")

I'm also using a separate Lua installation to run this, though you could theoretically use LÖVE.

If you use the library as a command line program then the output folder is currently the same folder as where the specified files are (you send it a list of files as arguments). I should maybe add a way to specify an output path for each file if the library is used this way. The library won't be able to create any folders for the output though as that's impossible with just Lua's standard libraries and I don't want to add any external dependencies.

[yetneverdone]

Thanks.

Yes, that should be helpful. Making the folder is the user's job.

Looking forward for this

[yetneverdone]

Hi, just want to say thank you for the amazing lib! Ive made a blog post discussing about it (next blog post will be more focused about how to use it) https://flamendless.github.io/2019/06/1 ... nt-blog-1/

[ReFreezed]

Awesome! I'll have a look at that blog post.

[yetneverdone]

Hi, ive made a new post in my website about a simple setup of a love game project using your library. Please check it. Comments and feedbacks are appreciated and needed.

https://flamendless.github.io/2019/07/1 ... nt-blog-2/

[ReFreezed]

Cool! I read through it. There's one thing in step #4 I think is a bit confusing/misleading:

Code: Select all

//this is how to use the luapreprocess as a code block, remember the syntax ! to denote the start and end of the code block to be preprocessed

An ! without a parenthesis simply means the rest of the line is part of the metaprogram, so it's just used to denote the start of the metaprogram code block. I think this comment makes it seem like maybe !if... is the start and !end is the end when it's actually two separate code blocks.

Also, I'd say that the whole file is preprocessed, not just a part of it (as in, the whole file is read and parsed by the library, then it generates a metaprogram which in turn generates the final output).

A couple of other minor things: you're using "//" for comments in the Lua code instead of "--", and the Makefile code in step #2 reaches outside the page.

[yetneverdone]

Cool! I read through it. There's one thing in step #4 I think is a bit confusing/misleading:

Code: Select all

//this is how to use the luapreprocess as a code block, remember the syntax ! to denote the start and end of the code block to be preprocessed

An ! without a parenthesis simply means the rest of the line is part of the metaprogram, so it's just used to denote the start of the metaprogram code block. I think this comment makes it seem like maybe !if... is the start and !end is the end when it's actually two separate code blocks.

Also, I'd say that the whole file is preprocessed, not just a part of it (as in, the whole file is read and parsed by the library, then it generates a metaprogram which in turn generates the final output).

A couple of other minor things: you're using "//" for comments in the Lua code instead of "--", and the Makefile code in step #2 reaches outside the page.

Oh okay. Thanks a lot! I will update my post

[yetneverdone]

I've set up a project template with this library as one of the main tools https://github.com/flamendless/love2d-template