<jupyter_start><jupyter_text>Collaboration and Competition---You are welcome to use this coding environment to train your agent for the project.  Follow the instructions below to get started! 1. Start the EnvironmentRun the next code cell to initialize the environment. If there's any error, please refer to the instruction in the `README.md`.<jupyter_code>from unityagents import UnityEnvironment
import numpy as np
from collections import namedtuple, deque
import matplotlib.pyplot as plt
%matplotlib inline

env = UnityEnvironment(file_name="Tennis.app")<jupyter_output>INFO:unityagents:
'Academy' started successfully!
Unity Academy name: Academy
        Number of Brains: 1
        Number of External Brains : 1
        Lesson number : 0
        Reset Parameters :
		
Unity brain name: TennisBrain
        Number of Visual Observations (per agent): 0
        Vector Observation space type: continuous
        Vector Observation space size (per agent): 8
        Number of stacked Vector Observation: 3
        Vector Action space type: continuous
        Vector Action space size (per agent): 2
        Vector Action descriptions: ,<jupyter_text>Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python.<jupyter_code># get the default brain
brain_name = env.brain_names[0]
brain = env.brains[brain_name]<jupyter_output><empty_output><jupyter_text>2. Examine the State and Action SpacesRun the code cell below to print some information about the environment.<jupyter_code># reset the environment
env_info = env.reset(train_mode=True)[brain_name]

# number of agents 
num_agents = len(env_info.agents)
print('Number of agents:', num_agents)

# size of each action
action_size = brain.vector_action_space_size
print('Size of each action:', action_size)

# examine the state space 
states = env_info.vector_observations
state_size = states.shape[1]
print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))
print('The state for the first agent looks like:', states[0])<jupyter_output>Number of agents: 2
Size of each action: 2
There are 2 agents. Each observes a state with length: 24
The state for the first agent looks like: [ 0.          0.          0.          0.          0.          0.
  0.          0.          0.          0.          0.          0.
  0.          0.          0.          0.         -6.65278625 -1.5
 -0.          0.          6.83172083  6.         -0.          0.        ]<jupyter_text>3. Take Random Actions in the EnvironmentIn the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.Note that **in this coding environment, you will not be able to watch the agents while they are training**, and you should set `train_mode=True` to restart the environment.<jupyter_code>for i in range(5):                                         # play game for 5 episodes
    env_info = env.reset(train_mode=False)[brain_name]     # reset the environment    
    states = env_info.vector_observations                  # get the current state (for each agent)
    scores = np.zeros(num_agents)                          # initialize the score (for each agent)
    while True:
        actions = np.random.randn(num_agents, action_size) # select an action (for each agent)
        actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1
        env_info = env.step(actions)[brain_name]           # send all actions to tne environment
        next_states = env_info.vector_observations         # get next state (for each agent)
        rewards = env_info.rewards                         # get reward (for each agent)
        dones = env_info.local_done                        # see if episode finished
        scores += env_info.rewards                         # update the score (for each agent)
        states = next_states                               # roll over states to next time step
        if np.any(dones):                                  # exit loop if episode finished
            break
    print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))<jupyter_output>Total score (averaged over agents) this episode: -0.004999999888241291
Total score (averaged over agents) this episode: 0.04500000085681677
Total score (averaged over agents) this episode: -0.004999999888241291
Total score (averaged over agents) this episode: -0.004999999888241291
Total score (averaged over agents) this episode: 0.04500000085681677<jupyter_text>When finished, you can close the environment.<jupyter_code>env.close()<jupyter_output><empty_output><jupyter_text>4. It's Your Turn!Now it's your turn to train your own agent to solve the environment!  A few **important notes**:- When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:```pythonenv_info = env.reset(train_mode=True)[brain_name]```- To structure your work, you're welcome to work directly in this Jupyter notebook, or you might like to start over with a new file!  You can see the list of files in the workspace by clicking on **_Jupyter_** in the top left corner of the notebook.- In this coding environment, you will not be able to watch the agents while they are training.  However, **_after training the agents_**, you can download the saved model weights to watch the agents on your own machine!   4-1 Multi-agent DDPG<jupyter_code>config = {\
          'BUFFER_SIZE': int(1e5),  # replay buffer size
          'BATCH_SIZE': 128,        # minibatch size
          'GAMMA': 0.99,            # discount factor
          'TAU': 1e-3,              # for soft update of target parameters
          'LR_ACTOR': 1e-4,         # learning rate of the actor
          'LR_CRITIC': 1e-3,        # learning rate of the critic
          'WEIGHT_DECAY': 0,        # L2 weight decay
          'RANDOM_SEED': 403,       # random seed
          'REPLAY_START_EPISODE': 50,          # episode to start training with replay
          'NUM_LEARN_PER_STEP': 4              # number of trainings per step (replay)
         }

from maddpg_agent import MADDPG

agent_MADDPG = MADDPG(state_size=state_size, action_size=action_size, num_agents=num_agents, config=config)
!pip install progressbar
import progressbar as pb
def MADDPG_Training(n_episodes=2500, t_max=1000):
    
    widget = ['training loop: ', pb.Percentage(), ' ', 
          pb.Bar(), ' ', pb.ETA() ]
    timer = pb.ProgressBar(widgets=widget, maxval=n_episodes).start()

    scores_deque = deque(maxlen=100)
    scores_list = []
    scores_list_100_avg = []
    for i_episode in range(n_episodes):
        env_info = env.reset(train_mode=True)[brain_name]     # reset the environment
        states = env_info.vector_observations                   # get the current states (for all agents)
        agent_MADDPG.reset() #reset the MADDPG_obj OU Noise
        scores = np.zeros(num_agents)                          # initialize the score (for each agent in MADDPG)
        num_steps = 0
        for _ in range(t_max):
            actions = agent_MADDPG.act(states, i_episode)
            env_info = env.step(actions)[brain_name]           # send all actions to the environment
            next_states = env_info.vector_observations         # get next state (for each agent in MADDPG)
            rewards = env_info.rewards                         # get rewards (for each agent in MADDPG)
            dones = env_info.local_done                        # see if episode finished
            scores += rewards                                  # update the score (for each agent in MADDPG)
            agent_MADDPG.step(i_episode+1, states, actions, rewards, next_states, dones) #train the MADDPG_obj           
            states = next_states                               # roll over states to next time step
            num_steps += 1
            if np.any(dones):                                  # exit loop if episode finished
                break
            #print('Total score (averaged over agents) this episode: {}'.format(np.mean(score)))
        
        scores_deque.append(np.max(scores))
        scores_list.append(np.max(scores))
        scores_list_100_avg.append(np.mean(scores_deque))
        
        #print('\rEpisode {}\tAverage Score: {:.2f}\tScore: {}'.format(i_episode, np.mean(scores_deque), score), end="")
        #print('\rEpisode {}\tAverage Score: {:.4f}\tCurrent Total Score: {:.4f}'.format(i_episode+1, np.mean(scores_deque), np.max(scores)), end="")
         
        if (i_episode+1) % 50 ==0:
            print('\rEpisode {}\tAverage Score: {:.2f}'.format(i_episode+1, np.mean(scores_deque)))  
        if (i_episode+1) % 500 == 0:
            agent_MADDPG.save_maddpg()
            print('Saved Model: Episode {}\tAverage Score: {:.2f}'.format(i_episode+1, np.mean(scores_deque))) 
        if np.mean(scores_deque) > 1.0 and len(scores_deque) >= 100:
            agent_MADDPG.save_maddpg()
            print('Saved Model: Episode {}\tAverage Score: {:.2f}'.format(i_episode+1, np.mean(scores_deque)))
            break
            
        timer.update(i_episode+1)
            
    return scores_list, scores_list_100_avg
scores, scores_100 = MADDPG_Training()
fig = plt.figure()
ax = fig.add_subplot(111)
plt.plot(np.arange(1, len(scores)+1), scores)
plt.plot(np.arange(1, len(scores)+1), scores_100)
plt.ylabel('Score')
plt.xlabel('Episode #')
plt.show()<jupyter_output><empty_output><jupyter_text>4-2 Adding noise decay and use normal distributed noise<jupyter_code>config = {\
          'BUFFER_SIZE': int(1e5),  # replay buffer size
          'BATCH_SIZE': 256,        # minibatch size
          'GAMMA': 0.99,            # discount factor
          'TAU': 1e-3,              # for soft update of target parameters
          'LR_ACTOR': 1e-4,         # learning rate of the actor
          'LR_CRITIC': 1e-3,        # learning rate of the critic
          'WEIGHT_DECAY': 0,        # L2 weight decay
          'NOISE_DECAY': .999,      # action noise decay
          'NOISE_MIN': .05,         # minimum noise scale
          'RANDOM_SEED': 403,       # random seed
          'REPLAY_START_EPISODE': 50,          # episode to start training with replay
          'NUM_LEARN_PER_STEP': 4              # number of trainings per step (replay)
         }

from maddpg_agent_noisedecay import MADDPG

agent_MADDPG = MADDPG(state_size=state_size, action_size=action_size, num_agents=num_agents, config=config)
def MADDPG_Training(n_episodes=2500, t_max=1000):
    
    widget = ['training loop: ', pb.Percentage(), ' ', 
          pb.Bar(), ' ', pb.ETA() ]
    timer = pb.ProgressBar(widgets=widget, maxval=n_episodes).start()

    scores_deque = deque(maxlen=100)
    scores_list = []
    scores_list_100_avg = []
    for i_episode in range(n_episodes):
        env_info = env.reset(train_mode=True)[brain_name]     # reset the environment
        states = env_info.vector_observations                   # get the current states (for all agents)
        agent_MADDPG.reset() #reset the MADDPG_obj OU Noise
        scores = np.zeros(num_agents)                          # initialize the score (for each agent in MADDPG)
        num_steps = 0
        for _ in range(t_max):
            actions = agent_MADDPG.act(states, i_episode)
            env_info = env.step(actions)[brain_name]           # send all actions to the environment
            next_states = env_info.vector_observations         # get next state (for each agent in MADDPG)
            rewards = env_info.rewards                         # get rewards (for each agent in MADDPG)
            dones = env_info.local_done                        # see if episode finished
            scores += rewards                                  # update the score (for each agent in MADDPG)
            agent_MADDPG.step(i_episode+1, states, actions, rewards, next_states, dones) #train the MADDPG_obj           
            states = next_states                               # roll over states to next time step
            num_steps += 1
            if np.any(dones):                                  # exit loop if episode finished
                break
            #print('Total score (averaged over agents) this episode: {}'.format(np.mean(score)))
        
        scores_deque.append(np.max(scores))
        scores_list.append(np.max(scores))
        scores_list_100_avg.append(np.mean(scores_deque))
        
        #print('\rEpisode {}\tAverage Score: {:.2f}\tScore: {}'.format(i_episode, np.mean(scores_deque), score), end="")
        #print('\rEpisode {}\tAverage Score: {:.4f}\tCurrent Total Score: {:.4f}'.format(i_episode+1, np.mean(scores_deque), np.max(scores)), end="")
         
        if (i_episode+1) % 50 ==0:
            print('\rEpisode {}\tAverage Score: {:.4f}\tCurrent Scores: {:.4f}'.format(i_episode+1, np.mean(scores_deque), np.max(scores)))  
        if (i_episode+1) % 500 == 0:
            agent_MADDPG.save_maddpg()
            print('Saved Model: Episode {}\tAverage Score: {:.4f}'.format(i_episode+1, np.mean(scores_deque))) 
            np.savez('final_scores.npz',scores=np.asarray(scores), scores_100=np.asarray(scores_list_100_avg))
        if np.mean(scores_deque) > 0.75 and len(scores_deque) >= 100:
            agent_MADDPG.save_maddpg()
            print('Saved Model: Episode {}\tAverage Score: {:.4f}'.format(i_episode+1, np.mean(scores_deque)))
            break
            
        timer.update(i_episode+1)
            
    return scores_list, scores_list_100_avg
import progressbar as pb
scores, scores_100 = MADDPG_Training()
fig = plt.figure()
ax = fig.add_subplot(111)
plt.plot(np.arange(1, len(scores)+1), scores)
plt.plot(np.arange(1, len(scores)+1), scores_100)
plt.ylabel('Score')
plt.xlabel('Episode #')
plt.show()
np.savez('final_scores.npz',scores=np.asarray(scores), scores_100=np.asarray(scores_100))
print(np.where(np.asarray(scores_100)>0.5)[0][0])<jupyter_output>1330
StarcoderdataJupyterStructuredCleanDedup