<jupyter_start><jupyter_text>Pruebas iniciales con RLLIB y RAY <NAME><jupyter_code>#imports necesarios
import ray
import ray.rllib.agents.ppo as ppo
import json, os, shutil, sys
import gym
import pprint
import time
import shelve
from tensorflow import keras
from ray import tune<jupyter_output>WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term<jupyter_text>Al inicializar ray podemos configurar varios parámetros.Entre ellos:- `local_mode=True` para no distribuir el trabajo entre workers paralelos.- `_metrics_export_port` el puerto donde se van a exportar las métricas que podemos ver en el Dashboard. Las podemos visualizar y graficar con prometheus https://docs.ray.io/en/master/ray-metrics.html- `num_cpus`, `num_gpus` para establecer los recursos sobre los que queremos trabajar. En este caso, tenemos 4 CPUs (los 4 cores de mi laptop) y ninguna GPU.-`ignore_reinit_error=True` para que no nos de error al hacer `ray.init` si ray ya estaba inicializado. En ese caso no se vuelve a inicializar ray.Enlace a la documentación de etsa función: https://docs.ray.io/en/master/package-ref.html<jupyter_code>#inicializamos ray
ray.shutdown()

# Si no ponemos un puerto para exportar las métricas se pone uno aleatorio.
ray.init(ignore_reinit_error=True, _metrics_export_port=63419)<jupyter_output>2020-11-19 18:24:26,266	INFO services.py:1090 -- View the Ray dashboard at [1m[32mhttp://127.0.0.1:8265[39m[22m<jupyter_text>Caso Básico: Problema basado en texto: TaxiEjemplo de https://medium.com/distributed-computing-with-ray/intro-to-rllib-example-environments-3a113f532c70<jupyter_code>#Directorio donde guardaremos checkpoints
CHECKPOINT_ROOT = "/tmp/ppo/taxi"
shutil.rmtree(CHECKPOINT_ROOT, ignore_errors=True, onerror=None)

#Directorio donde se guardan los resultados de cada sesión de ray (los podemos ver con tensorboard)
ray_results =os.getenv("HOME")+ "/ray_results"
shutil.rmtree(ray_results, ignore_errors=True, onerror=None)

#Entorno: problema del taxi (https://gym.openai.com/envs/Taxi-v3/)
SELECT_ENV = "Taxi-v3"

config = ppo.DEFAULT_CONFIG.copy()
config["log_level"] = "WARN"
print("Configuración del agente:\n\n" + str(config))
print("\nConfiguración del modelo del agente:\n\n" + str(config["model"]))
#Configuración del agente
agent = ppo.PPOTrainer(config, env=SELECT_ENV)<jupyter_output>[2m[36m(pid=1466)[0m WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=1466)[0m Instructions for updating:
[2m[36m(pid=1466)[0m non-resource variables are not supported in the long term
[2m[36m(pid=1468)[0m WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=1468)[0m Instructions for updating:
[2m[36m(pid=1468)[0m non-resource variables are not supported in the long term
2020-11-19 18:25:40,306	WARNING util.py:40 -- Install gputil for GPU system monitoring.<jupyter_text>Una ve creado el agente, cada etapa de su entrenamiento genera unos resultados entre los que podemos estraerinformación muy útil, como estaísiticos sobre las recompensas, la longitud de los episodios o distintas mediciones de tiempos. En la siguiente celda se puede ver un ejemplo de ello para el agente del taxi, donde vemos los valores de los timers, que nos permiten ver, entre otros, el tiempo de aprendizaje de cada iteración de train(), por ejemplo.<jupyter_code>result = agent.train()
pprint.pprint(result)

#Observar los valores de timers
#Entrenamos el modelo con 30 iteraciones, llamando al método train sobre agent
N_ITER = 30
s = "{:3d} reward {:6.2f}/{:6.2f}/{:6.2f} len {:6.2f} learn_time(ms) {:6.2f} saved {}"

results = []
episode_data = []
episode_json = []

total_learn_time = 0
for n in range(N_ITER):
    result = agent.train()
    results.append(result)
    episode = {'n': n,
               'episode_reward_min': result['episode_reward_min'],
               'episode_reward_mean': result['episode_reward_mean'],
               'episode_reward_max': result['episode_reward_max'],
               'episode_len_mean': result['episode_len_mean'],
               'learn_time_ms': result['timers']['learn_time_ms']}
    episode_data.append(episode)
    episode_json.append(json.dumps(episode))
    file_name = agent.save(CHECKPOINT_ROOT)
    print(s.format(
    n + 1,
    result["episode_reward_min"],
    result["episode_reward_mean"],
    result["episode_reward_max"],
    result["episode_len_mean"],
    result["timers"]["learn_time_ms"],
    file_name
   ))
    total_learn_time+= result["timers"]["learn_time_ms"]
    
print("Total learn time: " + str(total_learn_time))
print("Average learn time per iteration: " + str(total_learn_time/N_ITER))
    
#Visualizamos algunos datos del modelo entrenado

policy = agent.get_policy()
model = policy.model
print(model.base_model.summary())
keras.utils.plot_model(model.base_model, "taxi_model.png", show_shapes=True)

print("Variables")
pprint.pprint(model.variables())
print("Value function")
pprint.pprint(model.value_function())
 
#Linea de tiempo que se puede abrir en chrome://tracing y se ve el reparto de tareas y las tareas ejecutasdas por los workers
# así como medir tiempos
ray.timeline("/mnt/c/Users/javig/timelines/time_taxi.json")
ray.shutdown()<jupyter_output><empty_output><jupyter_text>Hemos entrenado un modelo con una capa input con una dimensión concreta, en este caso de tamaño 50, por lo que para probar el modelo necesitamos un entorno que nos proporcione entradas de estas características. Al ser entornos tan concretos no tenemos ninguno que reúna estos requisitos, por lo que lo vamos a probar con el mismo prolema con el que le hemos entrenado.Ejecutamos un rollout de 10 episodios (esto es, tomamos acciones en los modelos hasta llegaral estado de Done=True 10 veces). Cada episodio tiene una longitud arbitraria. Guardamos toda la info posible en el fichero taxi.pkl.Medimos el tiempo del rollout en conjunto pues al ser un comando predefinido (en verdad estamos ejecutando el script rollout.py) de momento no vamos a hacer más. Más adelante veremos como precisar esto un poco más. este tiempo es bastante improciso pues cuenta también el tiempo de inicializar ray o escribir los datos, entre otrosEl problema del taxi es un entorno denominado 'ToyText', es decir que su salida es únicamente "un texto". Sus estados se codifican como un entero del 0 al 499 y sus acciones con otro entero del 0 al 5 (moverse en las 4 direcciones, dejar o coger un pasajero).Las recompensas son -1 en cualquier acción salvo si es un intento de coger o soltar al pasajero ilegalmente (porque el pasajero no está donde está el taxi o porque lo intenta soltra cuando no lleva pasajero o en un destino al que no quiere ir) y 20 si sulta al pasajero correctamente (en su posición)<jupyter_code>t0=time.time()
!rllib rollout /tmp/ppo/taxi/checkpoint_30/checkpoint-30 --run PPO --env=Taxi-v3 --episodes 10  --out 'taxi.pkl' --save-info --use-shelve --track-progress --video-dir /mnt/c/Users/javig/videoGym
t1=time.time()-t0
print("El rollout ha tardado {} segundos".format(t1))<jupyter_output><empty_output><jupyter_text>El fichero de salida está en formato shelve pickle y con un shleve para cada rollout (para cada episodio). En el propio ficherofuente de rollout.py se nos indica como procesar esta infromación.Lo que obtenemos es, para cada episodio, el número del mismo y una lista con tantos elementos (que son a su vez listas) como pasos hayamos dado en ese episodio con iformación acerca de los mismos. Esta información es:- `obs`: estado observado antes de tomar la acción.- `action`: acción tomada en base al estado observado, la última acción tomada y la última recompensa obtenida- `next_obs`: el estado observado tras tomar la acción y que será el estado incial del siguiente elemento de la lista- `done`: booleano que inidca si estamos en un estado 'Done': esto es si hemos conseguido el objetivo del juego o si hemos agotado algún temporizador.-`info`: este campo sólo se incluye si hemos activado el flag `--save-info`e incluye información sobre el entorno y el problema concreto. En este caso, cuando hemos llegado a un estado 'Done' por agotra un temporizador se nos informa en este diccionario de ello.<jupyter_code># El fichero está en formato shelve pickle y con un shleve para cada rollout (para cada episodio). En el propio fichero
# fuente de rollout.py se nos indica como procesar esta infromación
with shelve.open('taxi.pkl') as rollouts:
    for episode_index in range(rollouts["num_episodes"]):
        rollout = rollouts[str(episode_index)]
        print(str(episode_index))
        pprint.pprint(rollout)<jupyter_output><empty_output><jupyter_text>Función genérica para el entrenamiento.A continuación, basada en el ejemplo anterior, esta función ejecuta iteraciones de entrenamiento sobre un entorno indicado guardando checkpoints en una ruta especificada y con la configuración elegida por nosotros también. Los parámetros que recibe son:- `checkpoint_root`: directorio en el que queremos guardar nuestros checkpoints (uno por cada iteración)- `env`: entorno gym con el que crearemos el agente- `config`: configuración para nuestro agente: aquí podemos indicar las capas del modelo- `n_iter`: número de iteraciones a realizar<jupyter_code>def full_train(checkpoint_root, env, config, n_iter):
    shutil.rmtree(checkpoint_root, ignore_errors=True, onerror=None)
    agent = ppo.PPOTrainer(config=config, env=env)
    s = "{:3d} reward {:6.2f}/{:6.2f}/{:6.2f} len {:6.2f} learn_time(ms) {:6.2f} saved {}"
    
    

    results = []
    episode_data = []
    episode_json = []

    total_learn_time = 0
    for n in range(n_iter):
        result = agent.train()
        results.append(result)
        episode = {'n': n,
                   'episode_reward_min': result['episode_reward_min'],
                   'episode_reward_mean': result['episode_reward_mean'],
                   'episode_reward_max': result['episode_reward_max'],
                   'episode_len_mean': result['episode_len_mean'],
                   'learn_time_ms': result['timers']['learn_time_ms']}
        episode_data.append(episode)
        episode_json.append(json.dumps(episode))
        file_name = agent.save(checkpoint_root)
        print(s.format(
        n + 1,
        result["episode_reward_min"],
        result["episode_reward_mean"],
        result["episode_reward_max"],
        result["episode_len_mean"],
        result["timers"]["learn_time_ms"],
        file_name
       ))
        total_learn_time+= result["timers"]["learn_time_ms"]

    print("Total learn time: " + str(total_learn_time))
    print("Average learn time per iteration: " + str(total_learn_time/n_iter))
    policy = agent.get_policy()
    model = policy.model
    print(model.base_model.summary())

    print("Variables")
    pprint.pprint(model.variables())
    print("Value function")
    pprint.pprint(model.value_function())
    keras.utils.plot_model(model.base_model, "taxi_model.png", show_shapes=True)<jupyter_output><empty_output><jupyter_text>Modelo con dos capas internas de 512 neuronas cada una<jupyter_code>ray.shutdown()
ray.init(ignore_reinit_error=True, _metrics_export_port=63419)
checkpoint_root = '/tmp/ppo/taxi_2'
env = 'Taxi-v3'
config = ppo.DEFAULT_CONFIG.copy()
config['model']['fcnet_hiddens'] = [512,512]
n_iter = 30
full_train(checkpoint_root, env, config, n_iter)
ray.timeline("/mnt/c/Users/javig/timelines/time_taxi.json")<jupyter_output>2020-11-19 19:18:23,380	INFO services.py:1090 -- View the Ray dashboard at [1m[32mhttp://127.0.0.1:8265[39m[22m
[2m[36m(pid=2954)[0m WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=2954)[0m Instructions for updating:
[2m[36m(pid=2954)[0m non-resource variables are not supported in the long term
[2m[36m(pid=2960)[0m WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=2960)[0m Instructions for updating:
[2m[36m(pid=2960)[0m non-resource variables are not supported in the long term
2020-11-19 19:18:34,030	WARNING util.py:40 -- Install gputil for GPU system monitoring.
[2m[36m(pid=2954)[...]<jupyter_text>Modelo con dos capas internas de 1024 neurnas<jupyter_code>ray.shutdown()
ray.init(ignore_reinit_error=True, _metrics_export_port=63419)
checkpoint_root = '/tmp/ppo/taxi_3'
env = 'Taxi-v3'
config = ppo.DEFAULT_CONFIG.copy()
config['model']['fcnet_hiddens'] = [1024,1024]
n_iter = 30
full_train(checkpoint_root, env, config, n_iter)
ray.timeline("/mnt/c/Users/javig/timelines/time_taxi.json")<jupyter_output>2020-11-19 19:25:17,303	INFO services.py:1090 -- View the Ray dashboard at [1m[32mhttp://127.0.0.1:8265[39m[22m
[2m[36m(pid=3130)[0m WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=3130)[0m Instructions for updating:
[2m[36m(pid=3130)[0m non-resource variables are not supported in the long term
[2m[36m(pid=3132)[0m WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=3132)[0m Instructions for updating:
[2m[36m(pid=3132)[0m non-resource variables are not supported in the long term
2020-11-19 19:25:28,224	WARNING util.py:40 -- Install gputil for GPU system monitoring.
[2m[36m(pid=3130)[...]<jupyter_text>Modelo con tres capas internas de 256 neuronas<jupyter_code>ray.shutdown()
ray.init(ignore_reinit_error=True, _metrics_export_port=63419)
checkpoint_root = '/tmp/ppo/taxi_2'
env = 'Taxi-v3'
config = ppo.DEFAULT_CONFIG.copy()
config['model']['fcnet_hiddens'] = [256,256,256]
n_iter = 30
full_train(checkpoint_root, env, config, n_iter)
ray.timeline("/mnt/c/Users/javig/timelines/time_taxi.json")<jupyter_output>2020-11-19 19:41:08,392	INFO services.py:1090 -- View the Ray dashboard at [1m[32mhttp://127.0.0.1:8265[39m[22m
[2m[36m(pid=4074)[0m WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=4074)[0m Instructions for updating:
[2m[36m(pid=4074)[0m non-resource variables are not supported in the long term
[2m[36m(pid=4076)[0m WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=4076)[0m Instructions for updating:
[2m[36m(pid=4076)[0m non-resource variables are not supported in the long term
2020-11-19 19:41:19,373	WARNING util.py:40 -- Install gputil for GPU system monitoring.
[2m[36m(pid=4076)[...]<jupyter_text>Modelo con tres capas internas de 512 neuronas<jupyter_code>ray.shutdown()
ray.init(ignore_reinit_error=True, _metrics_export_port=63419)
checkpoint_root = '/tmp/ppo/taxi_4'
env = 'Taxi-v3'
config = ppo.DEFAULT_CONFIG.copy()
config['model']['fcnet_hiddens'] = [512,512,512]
n_iter = 30
full_train(checkpoint_root, env, config, n_iter)
ray.timeline("/mnt/c/Users/javig/timelines/time_taxi.json")<jupyter_output>2020-11-19 19:45:29,985	INFO services.py:1090 -- View the Ray dashboard at [1m[32mhttp://127.0.0.1:8265[39m[22m
[2m[36m(pid=4272)[0m WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=4272)[0m Instructions for updating:
[2m[36m(pid=4272)[0m non-resource variables are not supported in the long term
[2m[36m(pid=4274)[0m WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=4274)[0m Instructions for updating:
[2m[36m(pid=4274)[0m non-resource variables are not supported in the long term
2020-11-19 19:45:40,494	WARNING util.py:40 -- Install gputil for GPU system monitoring.
[2m[36m(pid=4272)[...]<jupyter_text>Modelo con tres capas internas de 1024 neuronas<jupyter_code>ray.shutdown()
ray.init(ignore_reinit_error=True, _metrics_export_port=63419)
checkpoint_root = '/tmp/ppo/taxi_5'
env = 'Taxi-v3'
config = ppo.DEFAULT_CONFIG.copy()
config['model']['fcnet_hiddens'] = [1024,1024,1024]
n_iter = 30
full_train(checkpoint_root, env, config, n_iter)
ray.timeline("/mnt/c/Users/javig/timelines/time_taxi.json")<jupyter_output>2020-11-19 21:01:12,454	INFO services.py:1090 -- View the Ray dashboard at [1m[32mhttp://127.0.0.1:8265[39m[22m
2020-11-19 21:01:14,973	INFO trainer.py:592 -- Tip: set framework=tfe or the --eager flag to enable TensorFlow eager execution
2020-11-19 21:01:14,974	INFO trainer.py:1064 -- `_use_trajectory_view_api` only supported for PyTorch so far! Will run w/o.
2020-11-19 21:01:14,975	INFO trainer.py:617 -- Current log_level is WARN. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.
[2m[36m(pid=7592)[0m WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=7592)[0m Instructions for updating:
[2m[36m(pid=7592)[0m non-resource variables are not supported in the long term
[2m[36m(pid=7596)[0m WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages[...]<jupyter_text>Modelo con tres capas internas con la estructura \[256, 512, 256\]<jupyter_code>ray.shutdown()
ray.init(ignore_reinit_error=True, _metrics_export_port=63419)
checkpoint_root = '/tmp/ppo/taxi_6'
env = 'Taxi-v3'
config = ppo.DEFAULT_CONFIG.copy()
config['model']['fcnet_hiddens'] = [256,512,256]
n_iter = 30
full_train(checkpoint_root, env, config, n_iter)
ray.timeline("/mnt/c/Users/javig/timelines/time_taxi.json")<jupyter_output>2020-11-19 21:26:51,364	INFO services.py:1090 -- View the Ray dashboard at [1m[32mhttp://127.0.0.1:8265[39m[22m
[2m[36m(pid=8969)[0m WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=8969)[0m Instructions for updating:
[2m[36m(pid=8969)[0m non-resource variables are not supported in the long term
[2m[36m(pid=8971)[0m WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=8971)[0m Instructions for updating:
[2m[36m(pid=8971)[0m non-resource variables are not supported in the long term
2020-11-19 21:27:01,476	WARNING util.py:40 -- Install gputil for GPU system monitoring.
[2m[36m(pid=8971)[...]<jupyter_text>Modelo con cuatro capas internas de 256 neuronas<jupyter_code>ray.shutdown()
ray.init(ignore_reinit_error=True, _metrics_export_port=63419)
checkpoint_root = '/tmp/ppo/taxi_7'
env = 'Taxi-v3'
config = ppo.DEFAULT_CONFIG.copy()
config['model']['fcnet_hiddens'] = [256,256,256,256]
n_iter = 30
full_train(checkpoint_root, env, config, n_iter)
ray.timeline("/mnt/c/Users/javig/timelines/time_taxi.json")<jupyter_output>2020-11-19 23:15:30,826	INFO services.py:1090 -- View the Ray dashboard at [1m[32mhttp://127.0.0.1:8265[39m[22m
[2m[36m(pid=9544)[0m WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=9544)[0m Instructions for updating:
[2m[36m(pid=9544)[0m non-resource variables are not supported in the long term
[2m[36m(pid=9547)[0m WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=9547)[0m Instructions for updating:
[2m[36m(pid=9547)[0m non-resource variables are not supported in the long term
2020-11-19 23:15:41,826	WARNING util.py:40 -- Install gputil for GPU system monitoring.
[2m[36m(pid=9547)[...]<jupyter_text>Midiendo tiempos específicos en el rolloutComo hemos visto antes, el script de rollout no nos aporta información acerca de los tiempos específicos. Para ellos podemos acceder al código fuente y modificar esto manualmente.Indicamos en el código donde hemos añadido el temporizador y guardamos en un array los distintos tiempos por episodio. Vamos a modificar el script `rollout.py` y ejecutarlo directamente con los parámetros deseados.<jupyter_code>def rollout(agent,
            env_name,
            num_steps,
            num_episodes=0,
            saver=None,
            no_render=True,
            video_dir=None):
    policy_agent_mapping = default_policy_agent_mapping

    if saver is None:
        saver = RolloutSaver()

    if hasattr(agent, "workers") and isinstance(agent.workers, WorkerSet):
        env = agent.workers.local_worker().env
        multiagent = isinstance(env, MultiAgentEnv)
        if agent.workers.local_worker().multiagent:
            policy_agent_mapping = agent.config["multiagent"][
                "policy_mapping_fn"]

        policy_map = agent.workers.local_worker().policy_map
        state_init = {p: m.get_initial_state() for p, m in policy_map.items()}
        use_lstm = {p: len(s) > 0 for p, s in state_init.items()}
    else:
        env = gym.make(env_name)
        multiagent = False
        try:
            policy_map = {DEFAULT_POLICY_ID: agent.policy}
        except AttributeError:
            raise AttributeError(
                "Agent ({}) does not have a `policy` property! This is needed "
                "for performing (trained) agent rollouts.".format(agent))
        use_lstm = {DEFAULT_POLICY_ID: False}

    action_init = {
        p: flatten_to_single_ndarray(m.action_space.sample())
        for p, m in policy_map.items()
    }

    # If monitoring has been requested, manually wrap our environment with a
    # gym monitor, which is set to record every episode.
    if video_dir:
        env = gym_wrappers.Monitor(
            env=env,
            directory=video_dir,
            video_callable=lambda x: True,
            force=True)

    steps = 0
    episodes = 0
    times = []
    while keep_going(steps, num_steps, episodes, num_episodes): # Este bucle es para el número de episodio o pasos
        mapping_cache = {}  # in case policy_agent_mapping is stochastic
        saver.begin_rollout()
        obs = env.reset()
        agent_states = DefaultMapping(
            lambda agent_id: state_init[mapping_cache[agent_id]])
        prev_actions = DefaultMapping(
            lambda agent_id: action_init[mapping_cache[agent_id]])
        prev_rewards = collections.defaultdict(lambda: 0.)
        done = False
        reward_total = 0.0
        episode_time = 0.0
        while not done and keep_going(steps, num_steps, episodes,
                                      num_episodes):  #este bucle es para cada episodio
            multi_obs = obs if multiagent else {_DUMMY_AGENT_ID: obs}
            action_dict = {}
            for agent_id, a_obs in multi_obs.items(): 
                if a_obs is not None:
                    policy_id = mapping_cache.setdefault(
                        agent_id, policy_agent_mapping(agent_id))
                    p_use_lstm = use_lstm[policy_id]
                    # Aquí es donde empezamos a aplicar el modelo para ver que acción tomar.
                    
                    ################
                    t0 = time.time()
                    ################
                    
                    if p_use_lstm:
                        a_action, p_state, _ = agent.compute_action(
                            a_obs,
                            state=agent_states[agent_id],
                            prev_action=prev_actions[agent_id],
                            prev_reward=prev_rewards[agent_id],
                            policy_id=policy_id)
                        agent_states[agent_id] = p_state
                    else:
                        a_action = agent.compute_action(
                            a_obs,
                            prev_action=prev_actions[agent_id],
                            prev_reward=prev_rewards[agent_id],
                            policy_id=policy_id)
                        
                    ########################
                    t1 = time.time()
                    episode_time += (t1-t0)
                    ########################
                    
                    a_action = flatten_to_single_ndarray(a_action)
                    action_dict[agent_id] = a_action
                    prev_actions[agent_id] = a_action
            action = action_dict

            action = action if multiagent else action[_DUMMY_AGENT_ID]
            next_obs, reward, done, info = env.step(action)
            if multiagent:
                for agent_id, r in reward.items():
                    prev_rewards[agent_id] = r
            else:
                prev_rewards[_DUMMY_AGENT_ID] = reward

            if multiagent:
                done = done["__all__"]
                reward_total += sum(reward.values())
            else:
                reward_total += reward
            if not no_render:
                env.render()
            saver.append_step(obs, action, next_obs, reward, done, info)
            steps += 1
            obs = next_obs
        saver.end_rollout()
        print("Episode #{}: reward: {}".format(episodes, reward_total))
        
        ####################################################################
        print("Episode #{}: model_time: {}".format(episodes, episode_time))
        times.append[episode_time]
        ####################################################################
        
        if done:
            episodes += 1
            
    ########################
    print("Episodes times:")
    print(times)
    ########################
ray.shutdown()
t0=time.time()
!python3 rollout.py /tmp/ppo/taxi/checkpoint_30/checkpoint-30 --env=Taxi-v3 --run PPO --episodes 10 --out='taxi_time.pkl' --save-info --use-shelve
t1 = time.time()-t0
print("Rollout total time: " + str(t1))<jupyter_output><empty_output><jupyter_text>Juegos basados en imágenes: categoría AtariEstos juegos aprenden a partir de imágenes de videojuegos. Hemos elgido como ejemplo el entorno `Pong-v0`.La red neuronal que se crea es una VisionNet de TF, definida para RLLIB y que cuenta con tantas como filtros de convolución se especifiquen en la configuración.Las observaciones tienen un tamaño de (210,160,3), que se preprocesa para convertirla en una de la forma (84,84,4), dimesiones que sí son compatibles con los modelos preconfigurados de tensorflow.<jupyter_code>ray.shutdown()
ray.init()
config = ppo.DEFAULT_CONFIG.copy()
agent = ppo.PPOTrainer(config, env='Pong-v0')
policy=agent.get_policy()
print(policy.model.model_config)
print(policy.model.base_model.summary())<jupyter_output>2020-11-20 01:59:16,010	INFO services.py:1090 -- View the Ray dashboard at [1m[32mhttp://127.0.0.1:8265[39m[22m
[2m[36m(pid=11851)[0m WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=11851)[0m Instructions for updating:
[2m[36m(pid=11851)[0m non-resource variables are not supported in the long term
[2m[36m(pid=11853)[0m WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=11853)[0m Instructions for updating:
[2m[36m(pid=11853)[0m non-resource variables are not supported in the long term
2020-11-20 01:59:27,246	WARNING util.py:40 -- Install gputil for GPU system monitoring.<jupyter_text>Inspeccionando el código de `visionnet.py` encontramos que se nos crea, por defecto, una red con tres capas de convolución con los parámetros especificados.- Tamaño del espacio de salida (número de filtros en la convolución)- Tamaño de la ventana de convolución- Strides: desplazamiento de la ventana por ancho y por largo Ahora ejecutamos un entrenamiento de 5 episodios (tarda casi una hora en completarse) con el entorno `Pong-v0`<jupyter_code>ray.shutdown()
ray.init(ignore_reinit_error=True, _metrics_export_port=63419)
checkpoint_root = '/tmp/ppo/pong'
env = 'Pong-v0'
config = ppo.DEFAULT_CONFIG.copy()
n_iter = 5
full_train(checkpoint_root, env, config, n_iter)
#ray.timeline("/mnt/c/Users/javig/timelines/time_taxi.json")<jupyter_output>2020-11-19 00:25:17,174	INFO services.py:1090 -- View the Ray dashboard at [1m[32mhttp://127.0.0.1:8265[39m[22m
[2m[36m(pid=3859)[0m WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=3859)[0m Instructions for updating:
[2m[36m(pid=3859)[0m non-resource variables are not supported in the long term
[2m[36m(pid=3857)[0m WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=3857)[0m Instructions for updating:
[2m[36m(pid=3857)[0m non-resource variables are not supported in the long term
2020-11-19 00:25:30,829	INFO trainable.py:252 -- Trainable.setup took 10.082 seconds. If your trainable is s[...]<jupyter_text>Ejecutamos rollout con el comando de consola, sin poder medir los tiempos<jupyter_code>ray.shutdown()
t0=time.time()
!rllib rollout /tmp/ppo/pong/checkpoint_5/checkpoint-5 --env=AirRaid-v0 --run PPO --episodes=10 --out='airRaid.pkl' --save-info --use-shelve --no-render
t1 = time.time()-t0
print("Rollout time: " + str(t1))
with shelve.open('airRaid.pkl') as rollouts:
    for episode_index in range(rollouts["num_episodes"]):
        rollout = rollouts[str(episode_index)]
        print(str(episode_index))
        pprint.pprint(rollout)
ray.shutdown()
t0=time.time()
!python3 rollout.py /tmp/ppo/pong/checkpoint_5/checkpoint-5 --env='AirRaid-v0' --run PPO --episodes 10 --out='airRaid_time.pkl' --save-info --use-shelve
t1 = time.time()-t0
print("Rollout total time: " + str(t1))
ray.shutdown()
t0=time.time()
!python3 rollout.py /tmp/ppo/pong/checkpoint_5/checkpoint-5 --env='Bowling-v0' --run PPO --episodes 10 --out='bowling.pkl' --save-info --use-shelve
t1 = time.time()-t0
print("Rollout total time: " + str(t1))
ray.shutdown()
t0=time.time()
!python3 rollout.py /tmp/ppo/pong/checkpoint_5/checkpoint-5 --env='Carnival-v0' --run PPO --episodes 10 --out='carnival.pkl' --save-info --use-shelve
t1 = time.time()-t0
print("Rollout total time: " + str(t1))
ray.shutdown()
t0=time.time()
!python3 rollout.py /tmp/ppo/pong/checkpoint_5/checkpoint-5 --env='DemonAttack-v0' --run PPO --episodes 10 --out='demonattack.pkl' --save-info --use-shelve
t1 = time.time()-t0
print("Rollout total time: " + str(t1))
ray.shutdown()
t0=time.time()
!python3 rollout.py /tmp/ppo/pong/checkpoint_5/checkpoint-5 --env='Pong-v0' --run PPO --episodes 10 --out='pong.pkl' --save-info --use-shelve
t1 = time.time()-t0
print("Rollout total time: " + str(t1))
ray.shutdown()
ray.init()
config = ppo.DEFAULT_CONFIG.copy()
agent = ppo.PPOTrainer(config, env='Pong-v0')
agent.get_policy()
policy=agent.get_policy()
print(policy.model.model_config)
print(policy.model.base_model.summary())
print(dir(policy.model))
print(policy.model.inference_view_requirements)
env = gym.make("CarRacing-v0")
print(env.reset().shape)
env.close()
from ray.rllib.models.preprocessors import get_preprocessor
prep = get_preprocessor(env.observation_space)(env.observation_space)
prep
prep.transform(env.reset()).shape
env = gym.make("Pong-v0")
print(type(env.reset()))
print(env.reset()[25:-25, :, :])
print(env.reset())
import gym
env = gym.make('CartPole-v0')
env.reset()
for _ in range(1000):
    env.render()
    env.step(env.action_space.sample()) # take a random action
env.close()
ray.shutdown()
ray.init(ignore_reinit_error=True, _metrics_export_port=63419)
#checkpoint_root = '/tmp/ppo/taxi_3'
env = 'Pong-v0'
config = ppo.DEFAULT_CONFIG.copy()
config['model']['dim'] = 168
config['model']['conv_filters'] = [[16, [16, 16], 8],[32, [8, 8], 2],[256, [11, 11], 1]]
agent = ppo.PPOTrainer(config, env=env)
print(agent.get_policy().model.base_model.summary())
ray.shutdown()
ray.init(ignore_reinit_error=True, _metrics_export_port=63419)
#checkpoint_root = '/tmp/ppo/taxi_3'
env = 'Pong-v0'
config = ppo.DEFAULT_CONFIG.copy()
#config['model']['dim'] = 84
#config['preprocessor_pref'] = 'deepmind'
agent = ppo.PPOTrainer(config, env=env)
print(agent.get_policy().model.base_model.summary())<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup