LÖVE

I recently played else heart.break(), a unity game where you can program inside the game in a fictional programming language interpreted by the game.
One can "hack" the game.
You can do stuff like opening closed doors by writing code which tries every combination and so on.
How would someone start to make such a thing in a game?
What are the precautions you have to apply to make sure you don't execute that code in the "main" game?
I mean, you need a sand box or something.
Has someone ever tried to do that?
Do you know any other games like that?

Lua's pcall can evaluate any string into a function.
But if you want your game to execute a 'custom' language then you're looking at compiler theory.
The basic principle is: you start with an input string, break it into tokens (lexical analysis) and associate the tokens into some sort of (tree-like) structure.
The old King's Quest games are based on the same principle, except that the 'language' used there has the simple grammar: verb-noun (go north, pickup object, etc)

Take a look at Bussard project.

I recommend using Lua (or a simplified version of it) for this purpose. This way you don't have to write your own compiler.
You can start by looking at Lua's loadstring function. You can pass it code as a string (which the user has typed) and it will get interpreted. Then you can run that chunk of code.
I did something similar in trAInsported - players write Lua code and then the game interprets it and lets the trains act accordingly. You can check out the sourcecode for it if you want to - I do a lot of sandboxing - but it's a little old and messy.

Yes, trAInsported and Bussard are definetely stuff to check, also I suggest you look at Sandbox a library by Kikito.

There is also LIKO12 which allows you to code entire games, and sandmas which allows you to modify your LÖVE game at runtime

Someone reacently asked something similar and I gave a similar answer

Making you own programming language is hard (not impossible), it will take lots of time, in addition you will need to write documentation, modules and what not, you'll end up coding a new language but never coding your game, so be careful with that. I haven't even talked about programming a code editor in LÖVE which doesn't provide any form of textinput element... that would take time too so I recommend Pollywell which is what Bussard uses