#
Reinforcement Learning Training

For training our model, we will rely on already implemented RL algorithms, leveraging state-of-the-art Reinforcement Learning libraries. There are multiple advantages in doing so: these libraries provide high quality algorithms, efficiently implemented and continuously tested, they allow to focus efforts on higher level aspects such as policy network architecture, features selection, and hyper-parameters tuning, they provide native solutions to parallelize environment execution, and, in some cases, they even support distributed training using multiple GPUs in a single workstation or in cluster contexts.

DIAMBRA Arena natively supports three of the most important RL libraries: Stable Baselines 3, Ray RLlib and SheepRL.

##
Creating Scripted Agents

The classical way to create an agent able to play a game is to hand-code the rules governing its behavior. These rules can vary from very simple heuristics to very complex behavioral trees, but they all have in common the need of an expert coder that knows the game and is able to distill the key elements of it to craft the scripted bot.

The following are two examples of (very simple) scripted agents interfaced with our environments, and they are available here: DIAMBRA Agents - Basic.

###
No-Action Agent

This agent simply performs the “No-Action” action at every step. By convention it is the action with index 0, and it needs to be a single value for `Discrete` action spaces, and a tuple of 0s for `MultiDiscrete` ones, as shown in the snippet below.

    import diambra.arena
    from diambra.arena import SpaceTypes, Roles, EnvironmentSettings

    # Settings
    settings = EnvironmentSettings()
    settings.step_ratio = 6
    settings.frame_shape = (128, 128, 1)
    settings.role = Roles.P2
    settings.difficulty = 4
    settings.action_space = SpaceTypes.MULTI_DISCRETE

    env = diambra.arena.make("sfiii3n", settings)
    observation, info = env.reset()

    while True:
        action = env.get_no_op_action()
        observation, reward, terminated, truncated, info = env.step(action)

        if terminated or truncated:
            break

    # Close the environment
    env.close()

###
Random Agent

This agent simply performs a random action at every step. In this case, the sampling method takes care of generating an action that is consistent with the environment action space.

    import diambra.arena
    from diambra.arena import SpaceTypes, Roles, EnvironmentSettings

    # Settings
    settings = EnvironmentSettings()
    settings.step_ratio = 6
    settings.frame_shape = (128, 128, 1)
    settings.role = Roles.P2
    settings.difficulty = 4
    settings.action_space = SpaceTypes.MULTI_DISCRETE

    env = diambra.arena.make("sfiii3n", settings)
    observation, info = env.reset()

    while True:
        action = env.action_space.sample()
        observation, reward, terminated, truncated, info = env.step(action)

        if terminated or truncated:
            break

    # Close the environment
    env.close()

You can run these simple agents on the environment to start building confidence with respect to how it works, and to start understanding the different moving parts, for example inspecting the observation space returned in the different phases.

More complex scripts can be built in similar ways, for example continuously performing user-defined combos moves, or adding some more complex choice mechanics. But this would still require to decide the tactics in advance, properly translating knowledge into code. A different approach would be to leverage Reinforcement Learning, so that the agent will improve leveraging its own experience.

##
Training DeepRL Agents

An alternative approach to scripted agents is to leverage Reinforcement Learning to create a model that improves over time by leveraging its experience.

In this tutorial we will show step-by-step how to training a deep neural network using A2C and PPO algorithms from Stable Baselines 3.

###
Getting Ready

We highly recommend using virtual environments to isolate your python installs, especially to avoid conflicts in dependencies. In what follows we use Conda but any other tool should work too.

Create and activate a new dedicated virtual environment:

    conda create -n diambra-arena-sb3 python=3.8
    conda activate diambra-arena-sb3

Install DIAMBRA Arena with Stable Baselines 3 interface:

    pip install diambra-arena[stable-baselines3]

This should be enough to prepare your system to execute the scripts presented here below. You can refer to the official Stable Baselines 3 documentation or reach out on DIAMBRA Discord Server for specific needs.

All the examples presented below are available here: DIAMBRA Agents - Stable Baselines 3. They have been created following the high level approach found on Stable Baselines 3 examples page, thus allowing to easily extend them and to understand how they interface with the different components.

These examples aims at demonstrating the core functionalities and high level aspects, providing a base for the user to build upon, exploring aspects like policy network architecture, algorithm hyperparameter tuning, observation space tweaking, rewards wrapping and other similar ones.

###
Basic

For all the examples there are two main things to note about the observation space.

First, the normalization wrapper is applied on all elements but the image frame, as Stable Baselines 3 automatically normalizes images and expects their pixels to be in the range [0 - 255].

Second, the library also has a specific constraint on dictionary observation spaces: they cannot be nested. For this reason we provide a flattening wrapper that creates a shallow, not nested, dictionary from the original observation space, allowing in addition to filter it by keys.

Stable Baselines 3 automatically defines the network architecture, properly matching the input type. In some of the examples the architecture is printed to the console output, allowing to clearly identify all the different contributions.

####
Basic Example

This example demonstrates how to:

- Leverage DIAMBRA Arena native Stable Baselines 3 interface to create the environment
- Interface the environment with one of Stable Baselines 3’s algorithms
- Train the algorithm
- Run the trained agent in the environment for one episode

It uses the A2C algorithm, with a `MultiInputPolicy` policy network to properly process the dictionary observation space as input. For demonstration purposes, the algorithm is trained for only 200 steps, so the resulting agent will be far from optimal.

    from diambra.arena.stable_baselines3.make_sb3_env import make_sb3_env, EnvironmentSettings, WrappersSettings
    from stable_baselines3 import A2C

    def main():
        # Settings
        settings = EnvironmentSettings()
        settings.frame_shape = (128, 128, 1)
        settings.characters = ("Ryu")

        # Wrappers Settings
        wrappers_settings = WrappersSettings()
        wrappers_settings.normalize_reward = True
        wrappers_settings.stack_frames = 5
        wrappers_settings.add_last_action = True
        wrappers_settings.stack_actions = 12
        wrappers_settings.scale = True
        wrappers_settings.exclude_image_scaling = True
        wrappers_settings.role_relative = True
        wrappers_settings.flatten = True
        wrappers_settings.filter_keys = ["action", "own_health", "opp_health", "own_side", "opp_side", "opp_character", "stage", "timer"]

        # Create environment
        env, num_envs = make_sb3_env("sfiii3n", settings, wrappers_settings)
        print("Activated {} environment(s)".format(num_envs))

        print("\nStarting training ...\n")
        agent = A2C("MultiInputPolicy", env, verbose=1)
        agent.learn(total_timesteps=200)
        print("\n .. training completed.")

        print("\nStarting trained agent execution ...\n")
        observation = env.reset()
        while True:
            env.render()

            action, _state = agent.predict(observation, deterministic=True)
            observation, reward, done, info = env.step(action)

            if done:
                observation = env.reset()
                break
        print("\n... trained agent execution completed.\n")

        # Close the environment
        env.close()

        # Return success
        return 0

    if __name__ == "__main__":
        main()

How to run it:

    diambra run python basic.py

####
Saving, loading and evaluating

In addition to what seen in the previous example, this one demonstrates how to:

- Save a trained agent
- Load a saved agent
- Evaluate an agent on a given number of episodes

The same conditions of the previous example for algorithm, policy and training steps are used in this one too.

    from diambra.arena.stable_baselines3.make_sb3_env import make_sb3_env, EnvironmentSettings, WrappersSettings
    from stable_baselines3 import A2C
    from stable_baselines3.common.evaluation import evaluate_policy

    def main():
        # Settings
        settings = EnvironmentSettings()
        settings.frame_shape = (128, 128, 1)
        settings.characters = ("Ryu")

        # Wrappers Settings
        wrappers_settings = WrappersSettings()
        wrappers_settings.normalize_reward = True
        wrappers_settings.stack_frames = 5
        wrappers_settings.add_last_action = True
        wrappers_settings.stack_actions = 12
        wrappers_settings.scale = True
        wrappers_settings.exclude_image_scaling = True
        wrappers_settings.role_relative = True
        wrappers_settings.flatten = True
        wrappers_settings.filter_keys = ["action", "own_health", "opp_health", "own_side", "opp_side", "opp_character", "stage", "timer"]

        # Create environment
        env, num_envs = make_sb3_env("sfiii3n", settings, wrappers_settings)
        print("Activated {} environment(s)".format(num_envs))

        # Instantiate the agent
        agent = A2C("MultiInputPolicy", env, verbose=1)
        # Train the agent
        agent.learn(total_timesteps=200)
        # Save the agent
        agent.save("a2c_sfiii3n")
        del agent  # delete trained agent to demonstrate loading

        # Load the trained agent
        # NOTE: if you have loading issue, you can pass `print_system_info=True`
        # to compare the system on which the agent was trained vs the current one
        # agent = A2C.load("a2c_sfiii3n", env=env, print_system_info=True)
        agent = A2C.load("a2c_sfiii3n", env=env)

        # Evaluate the agent
        # NOTE: If you use wrappers with your environment that modify rewards,
        #       this will be reflected here. To evaluate with original rewards,
        #       wrap environment in a "Monitor" wrapper before other wrappers.
        mean_reward, std_reward = evaluate_policy(agent, agent.get_env(), n_eval_episodes=3)
        print("Reward: {} (avg) ± {} (std)".format(mean_reward, std_reward))

        # Run trained agent
        observation = env.reset()
        cumulative_reward = 0
        while True:
            env.render()

            action, _state = agent.predict(observation, deterministic=True)
            observation, reward, done, info = env.step(action)

            cumulative_reward += reward
            if (reward != 0):
                print("Cumulative reward =", cumulative_reward)

            if done:
                observation = env.reset()
                break

        # Close the environment
        env.close()

        # Return success
        return 0

    if __name__ == "__main__":
        main()

How to run it:

    diambra run python saving_loading_evaluating.py

####
Parallel Environments

In addition to what seen in previous examples, this one demonstrates how to:

- Run training using parallel environments
- Print out the policy network architecture

In this example, the PPO algorithm is used, with the same `MultiInputPolicy` seen before. The policy architecture is also printed to the console output, allowing to visualize how inputs are processed and “translated” to actions probabilities.

This example also runs multiple environments, automatically detecting the number of instances created by DIAMBRA CLI when running the script.

    from diambra.arena.stable_baselines3.make_sb3_env import make_sb3_env, EnvironmentSettings, WrappersSettings
    from stable_baselines3 import PPO

    def main():
        # Settings
        settings = EnvironmentSettings()
        settings.frame_shape = (128, 128, 1)
        settings.characters = ("Ryu")

        # Wrappers Settings
        wrappers_settings = WrappersSettings()
        wrappers_settings.normalize_reward = True
        wrappers_settings.stack_frames = 5
        wrappers_settings.add_last_action = True
        wrappers_settings.stack_actions = 12
        wrappers_settings.scale = True
        wrappers_settings.exclude_image_scaling = True
        wrappers_settings.role_relative = True
        wrappers_settings.flatten = True
        wrappers_settings.filter_keys = ["action", "own_health", "opp_health", "own_side", "opp_side", "opp_character", "stage", "timer"]

        # Create environment
        env, num_envs = make_sb3_env("sfiii3n", settings, wrappers_settings)
        print("Activated {} environment(s)".format(num_envs))

        # Instantiate the agent
        agent = PPO("MultiInputPolicy", env, verbose=1)

        # Print policy network architecture
        print("Policy architecture:")
        print(agent.policy)

        # Train the agent
        agent.learn(total_timesteps=200)

        # Run trained agent
        observation = env.reset()
        cumulative_reward = [0.0 for _ in range(num_envs)]
        while True:
            action, _state = agent.predict(observation, deterministic=True)

            observation, reward, done, info = env.step(action)
            cumulative_reward += reward
            if any(x != 0 for x in reward):
                print("Cumulative reward(s) =", cumulative_reward)

            if done.any():
                observation = env.reset()
                break

        # Close the environment
        env.close()

        # Return success
        return 0

    if __name__ == "__main__":
        main()

How to run it:

    diambra run -s=2 python parallel_envs.py

###
Advanced

####
Complete Training Script

In addition to what seen in previous examples, this one demonstrates how to:

- Build a complete training script to be used with Stable Baselines via a config file
- How to properly handle hyper-parameters scheduling via callbacks
- How to use callbacks for auto-saving
- How to control some policy network models and optimizer parameters

This example show exactly how we trained our own models on these environments. It should be considered a starting point from where to explore and experiment, the following are just a few options among the most obvious ones:

- Tweak hyper-parameters for the chosen algorithm
- Evolve the policy network architecture
- Test different algorithms, both on and off-policy
- Try to leverage behavioral cloning / imitation learning
- Modify the reward function to guide learning in other directions

    import os
    import yaml
    import json
    import argparse
    from diambra.arena import load_settings_flat_dict, SpaceTypes
    from diambra.arena.stable_baselines3.make_sb3_env import make_sb3_env, EnvironmentSettings, WrappersSettings
    from diambra.arena.stable_baselines3.sb3_utils import linear_schedule, AutoSave
    from stable_baselines3 import PPO

    def main(cfg_file):
        # Read the cfg file
        yaml_file = open(cfg_file)
        params = yaml.load(yaml_file, Loader=yaml.FullLoader)
        print("Config parameters = ", json.dumps(params, sort_keys=True, indent=4))
        yaml_file.close()

        base_path = os.path.dirname(os.path.abspath(__file__))
        model_folder = os.path.join(base_path, params["folders"]["parent_dir"], params["settings"]["game_id"],
                                    params["folders"]["model_name"], "model")
        tensor_board_folder = os.path.join(base_path, params["folders"]["parent_dir"], params["settings"]["game_id"],
                                            params["folders"]["model_name"], "tb")

        os.makedirs(model_folder, exist_ok=True)

        # Settings
        params["settings"]["action_space"] = SpaceTypes.DISCRETE if params["settings"]["action_space"] == "discrete" else SpaceTypes.MULTI_DISCRETE
        settings = load_settings_flat_dict(EnvironmentSettings, params["settings"])

        # Wrappers Settings
        wrappers_settings = load_settings_flat_dict(WrappersSettings, params["wrappers_settings"])

        # Create environment
        env, num_envs = make_sb3_env(settings.game_id, settings, wrappers_settings)
        print("Activated {} environment(s)".format(num_envs))

        # Policy param
        policy_kwargs = params["policy_kwargs"]

        # PPO settings
        ppo_settings = params["ppo_settings"]
        gamma = ppo_settings["gamma"]
        model_checkpoint = ppo_settings["model_checkpoint"]

        learning_rate = linear_schedule(ppo_settings["learning_rate"][0], ppo_settings["learning_rate"][1])
        clip_range = linear_schedule(ppo_settings["clip_range"][0], ppo_settings["clip_range"][1])
        clip_range_vf = clip_range
        batch_size = ppo_settings["batch_size"]
        n_epochs = ppo_settings["n_epochs"]
        n_steps = ppo_settings["n_steps"]

        if model_checkpoint == "0":
            # Initialize the agent
            agent = PPO("MultiInputPolicy", env, verbose=1,
                        gamma=gamma, batch_size=batch_size,
                        n_epochs=n_epochs, n_steps=n_steps,
                        learning_rate=learning_rate, clip_range=clip_range,
                        clip_range_vf=clip_range_vf, policy_kwargs=policy_kwargs,
                        tensorboard_log=tensor_board_folder)
        else:
            # Load the trained agent
            agent = PPO.load(os.path.join(model_folder, model_checkpoint), env=env,
                             gamma=gamma, learning_rate=learning_rate, clip_range=clip_range,
                             clip_range_vf=clip_range_vf, policy_kwargs=policy_kwargs,
                             tensorboard_log=tensor_board_folder)

        # Print policy network architecture
        print("Policy architecture:")
        print(agent.policy)

        # Create the callback: autosave every USER DEF steps
        autosave_freq = ppo_settings["autosave_freq"]
        auto_save_callback = AutoSave(check_freq=autosave_freq, num_envs=num_envs,
                                      save_path=model_folder, filename_prefix=model_checkpoint + "_")

        # Train the agent
        time_steps = ppo_settings["time_steps"]
        agent.learn(total_timesteps=time_steps, callback=auto_save_callback)

        # Save the agent
        new_model_checkpoint = str(int(model_checkpoint) + time_steps)
        model_path = os.path.join(model_folder, new_model_checkpoint)
        agent.save(model_path)

        # Close the environment
        env.close()

        # Return success
        return 0

    if __name__ == "__main__":
        parser = argparse.ArgumentParser()
        parser.add_argument("--cfgFile", type=str, required=True, help="Configuration file")
        opt = parser.parse_args()
        print(opt)

        main(opt.cfgFile)

How to run it:

    diambra run python training.py --cfgFile /absolute/path/to/config.yaml

and the configuration file to be used with this training script is reported below:

    folders:
      parent_dir: "./results/"
      model_name: "sr6_128x4_das_nc"

    settings:
      game_id: "sfiii3n"
      step_ratio: 6
      frame_shape: !!python/tuple [128, 128, 1]
      continue_game: 0.0
      action_space: "discrete"
      characters: "Ryu"
      difficulty: 6
      outfits: 2

    wrappers_settings:
      normalize_reward: true
      no_attack_buttons_combinations: true
      stack_frames: 4
      dilation: 1
      add_last_action: true
      stack_actions: 12
      scale: true
      exclude_image_scaling: true
      role_relative: true
      flatten: true
      filter_keys: ["action", "own_health", "opp_health", "own_side", "opp_side", "opp_character", "stage", "timer"]

    policy_kwargs:
      #net_arch: [{ pi: [64, 64], vf: [32, 32] }]
      net_arch: [64, 64]

    ppo_settings:
      gamma: 0.94
      model_checkpoint: "0"
      learning_rate: [2.5e-4, 2.5e-6] # To start
      clip_range: [0.15, 0.025] # To start
      #learning_rate: [5.0e-5, 2.5e-6] # Fine Tuning
      #clip_range: [0.075, 0.025] # Fine Tuning
      batch_size: 256 #8 #nminibatches gave different batch size depending on the number of environments: batch_size = (n_steps * n_envs) // nminibatches
      n_epochs: 4
      n_steps: 128
      autosave_freq: 256
      time_steps: 512

Finally, after the agent training is completed, besides running it locally in your own machine, you may want to submit it to DIAMBRA Competition Platform to compete with other coders around the globe in epic video games tournaments! The next section provides a step by step guide to do exactly that.

< > Update on GitHub