Machine learning: Q table

[togFox]

So been reading up about machine learning and Q tables:

https://en.wikipedia.org/wiki/Q-learning#Algorithm

I want to have an AI in my current project that, when faced with a combination of states, it will learn how to react for a positive outcome. To avoid confusion, I will be defining a known set of finite states and a known set of finite actions. I intend to build a Q table.

The learning initially will be random and AI behavior will be terrible but, arguably, no worse than any other random AI bot. As actions are taken and evaluated, the Q table will populate and the Q table can give the bot meaningful instructions under a set of finite states.

Here is my question:

It's easy to think of a Q table as a series of states (singular) in the sense that there is only one thing to compute. For example, when navigating a maze, the tile next the bot can have a range of states:
- tile is blocked
- tile is closer to the goal
- tile is further from the goal
- tile is neither closer or further from the goal

Whilst their are four possible outcomes, there is only one concept being tested: a tile, or specifically, the tile next to the bot.

I want to make my Q table more complex and meaningful by testing more than one concept. I want to test the tile next to me AND the proximity of some monster that needs to be avoided (I'm making up an example ).

So how to do a Q table? Now, with more information, this state:

- neighboring tile is closer to the goal

may not be the best option if

- neighboring tile is closer to the monster.

With this new criteria, i would change the scoring so that moving closer to the monster is always bad (this is theoretical - lets ignore that the bot may never reach the goal this way).

Getting to the point: how to construct a Q table when there are multiple 'concepts' a bot needs to check before taking an action?

[Gunroar:Cannon()]

Don't know much about Q tables (though I really should) you could use an average value for all the things to take into consideration. I used this a while back in my early days,
Like

Code: Select all

 preference = m/l 

where the higher preference of tile is the better, "l" is the average value of all the negative things and "m" is the average value of all the positive things.(Might not even need average )

Code: Select all

m = (closensess of tile + comfort of tile)/2
l =  danger level of tile(enemies/fire)
preference = m/l

Does that work?

[dcuny]

Getting to the point: how to construct a Q table when there are multiple 'concepts' a bot needs to check before taking an action?

The Q Table doesn't know what any of the states or actions actually mean. A state can represent any concept you want it to - as long as you can test for it. Similarly, any action can mean anything you want it to - as long as you can evaluate the results.

From what I see, a "state" in a Q Table is a boolean. That means you have to be able to convert concepts into boolean values, not numeric values.

So instead of using distanceToMonster - a numeric value - you'd have to convert it into a boolean state.

For example, you might break down the concept of "Monster Nearby" into three arbitrary distances, "next to me", "near", and "out of sight":

Code: Select all

local distance = distanceToMonster( bot.pos, monster.pos )
-- only one can be true, so initially set them all to false
monsterNextToMe, monsterNear, monsterFar = false, false, false
if distance > 10 then
   monsterFar = true
elseif distance > 3 then
   monsterNear = true
else
   monster = true
end

The code evaluating the Q Table doesn't know what any of these concepts are - only that there are states that are true or false.

As long as you can evaluate for the presence of a state, or trigger an action, that's enough.

Similarly, the Q Table doesn't know how to perform an action, only how to choose one.

For example, let's say that you have a concept of "move in the direction of treasure". The information about how to move in the direction of treasure can be as complex a routine as you want it to be - as long as you can trigger that action and evaluate the results.

The Q Table doesn't know any of the details of the world - that is, how to decide if a state is true. Nor does it need to know how to perform an action, like "fight a monster". It only needs to know how to look at states, choose an action, and measure the total cost.