<jupyter_start><jupyter_text>Reinforcement Learning env: _taxi-v2_<jupyter_code>import gym<jupyter_output><empty_output><jupyter_text>Loading and initializing an environment<jupyter_code>env = gym.make('Taxi-v2')
env.reset()<jupyter_output>[2017-11-04 01:33:57,794] Making new env: Taxi-v2<jupyter_text>Observation states<jupyter_code>print('Total number of states = {:,}'.format(env.observation_space.n))<jupyter_output>Total number of states = 500<jupyter_text>Visualizing the stateIn this environment the yellow square represents the taxi, the (“|”) represents a wall, the blue letter represents the pick-up location, and the purple letter is the drop-off location. The taxi will turn green when it has a passenger aboard. While we see colors and shapes that represent the environment, the algorithm does not think like us and only understands a flattened state, in this case an integer.<jupyter_code>env.render()<jupyter_output>+---------+
|R: | : :G|
| : : : : |
| : : : : |
| |[43m [0m: | : |
|[34;1mY[0m| : |[35mB[0m: |
+---------+<jupyter_text>Action spaceThis shows us there are a total of six actions available. Gym will not always tell you what these actions mean, but in this case, the six possible actions are: down (0), up (1), right (2), left (3), pick-up (4), and drop-off (5).<jupyter_code>print('Total number of action the agent can carry out = {:,}'.format(env.action_space.n))<jupyter_output>Total number of action the agent can carry out = 6<jupyter_text>Overriding and moving the agent stateThese four variables are: the new state (St+1 = 14), reward (Rt+1 = -1), a boolean stating whether the environment is terminated or done, and extra info for debugging. Every Gym environment will return these same four variables after an action is taken, as they are the core variables of a reinforcement learning problem.Take a look at the rendered environment. What do you expect the environment would return if you were to move left? It would, of course, give the exact same return as before. The environment always gives a -1 reward for each step in order for the agent to try and find the quickest solution possible. If you were measuring your total accumulated reward, constantly running into a wall would heavily penalize your final reward. The environment will also give a -10 reward every time you incorrectly pick up or drop off a passenger.<jupyter_code>env.env.s = 114  # reset the current state of the agent
env.render()
step = env.step(1)  # move up (1)
print(step)
env.render()
step = env.step(0)  # move down (0)
print(step)
env.render()
step = env.step(2)  # move right (2)
print(step)
env.render()
step = env.step(3)  # move left (3)
print(step)
env.render()
step = env.step(1)  # move up (1)
print(step)
env.render()<jupyter_output>(14, -1, False, {'prob': 1.0})
+---------+
|[43mR[0m: | : :G|
| : : : : |
| : : : : |
| | : | : |
|[35mY[0m| : |[34;1mB[0m: |
+---------+
  (North)<jupyter_text>Random Actions One surprising way you could solve this environment is to choose randomly among the six possible actions. The environment is considered solved when you successfully pick up a passenger and drop them off at their desired location. Upon doing this, you will receive a reward of 20 and done will equal True. The odds are small, but it’s still possible, and given enough random actions you will eventually luck out. A core part of evaluating any agent’s performance is to compare it to a completely random agent. In a Gym environment, you can choose a random action using **`env.action_space.sample()`**. You can create a loop that will do random actions until the environment is solved. We will put a counter in there to see how many steps it takes to solve the environment.<jupyter_code>state = env.reset()
total_reward, reward = 0, None
counter = 0

while state != 20:
    random_action = env.action_space.sample()
    state, reward, done, info = env.step(random_action)
    total_reward += reward
    counter += 1
    # Let's finish this after 10,000 attempts
    if counter > 10000:
        break
print(counter)
print('Okay, I give up! After {:,} attempts we still haven\'t found a solution :('.format(counter))<jupyter_output>Okay, I give up! After 10,001 attempts we still haven't found a solution :(<jupyter_text>Q-action value tableYou may luck out and solve the environment fairly quickly, but on average, a completely random policy will solve this environment in about 2000+ steps, so in order to maximize our reward, we will have to have the algorithm remember its actions and their associated rewards. In this case, the algorithm’s memory is going to be a Q action value table.To manage this Q table, we will use a NumPy array. The size of this table will be the number of states (500) by the number of possible actions (6).<jupyter_code>import numpy as np
Q = np.zeros(shape=[env.observation_space.n, env.action_space.n])<jupyter_output><empty_output><jupyter_text>Over multiple episodes of trying to solve the problem, we will be updating our Q values, slowly improving our algorithm’s efficiency and performance. We will also want to track our total accumulated reward for each episode, which we will define as G.<jupyter_code>G = 0  # total accumulated reward for each episode<jupyter_output><empty_output><jupyter_text>Similar to most machine learning problems, we will need a learning rate as well. I will use my personal favorite of 0.618, also known as the mathematical constant phi or the _Golden ratio_.<jupyter_code>alpha = 0.618  # learning rate<jupyter_output><empty_output><jupyter_text>Basic Q Learning algorithm<jupyter_code>episodes = 1000
# go through each episodes
for episode in range(1, episodes+1):
    done = False
    G, reward = 0, None
    state = env.reset()
    while not done:
        # First (#1)
        action = np.argmax(Q[state])  # select an action from the current state
        # Second (#2)
        state2, reward, done, info = env.step(action)  # exec the action
        # Third (#3)
        Q[state, action] += alpha * (reward + np.max(Q[state2]) - Q[state, action])
        G += reward
        state = state2
    if episode % 100 == 0:
        print('Episode {:,}\tAccumulated reward: {}'.format(episode, G))

env.render()<jupyter_output>Episode 100	Accumulated reward: -124
Episode 200	Accumulated reward: -9
Episode 300	Accumulated reward: 15
Episode 400	Accumulated reward: 8
Episode 500	Accumulated reward: 10
Episode 600	Accumulated reward: 4
Episode 700	Accumulated reward: 6
Episode 800	Accumulated reward: 10
Episode 900	Accumulated reward: 8
Episode 1,000	Accumulated reward: 11
+---------+
|R: | : :G|
| : : : : |
| : : : : |
| | : | : |
|[35m[42mY[0m[0m| : |B: |
+---------+
  (Dropoff)<jupyter_text>**First (1):** The agent starts by choosing an _action_ with the highest Q value for the current state using argmax. Argmax will return the index/action with the highest value for that state. Initially, our Q table will be all zeros. But, after every step, the Q values for _state-action pairs_ will be updated.**Second (2):** The agent then takes action and we store the future state as state2 `(St+1)`. This will allow the agent to compare the previous state to the new state.**Third (3):** We update the _state-action pair_ `(St, At)` for Q using the reward, and the max Q value for state2 `(St+1)`. This update is done using the action value formula (based upon the Bellman equation) and allows state-action pairs to be updated in a recursive fashion (based on future values).<jupyter_code>print(state, Q)<jupyter_output>418 [[  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00]
 [ -5.44432654e+00  -4.94400000e+00  -5.02631716e+00  -4.94400000e+00
    2.67020271e+03  -6.18000000e+00]
 [ -3.81634701e+00  -3.70800000e+00  -3.93903048e+00  -3.70800000e+00
    2.41760594e+03  -6.18000000e+00]
 ..., 
 [ -3.09000000e+00  -2.79825703e+00  -3.09000000e+00  -3.62212608e+00
   -6.18000000e+00  -6.18000000e+00]
 [ -5.56200000e+00  -5.13721801e+00  -5.56200000e+00  -5.92070325e+00
   -6.18000000e+00  -6.18000000e+00]
 [ -1.23600000e+00  -1.23600000e+00  -1.23600000e+00   6.78440400e+00
   -6.18000000e+00  -6.18000000e+00]]
StarcoderdataJupyterStructuredCleanDedup