| --- |
| language: en |
| license: apache-2.0 |
| library_name: pytorch |
| tags: |
| - deep-reinforcement-learning |
| - reinforcement-learning |
| - DI-engine |
| - TicTacToe-play-with-bot |
| benchmark_name: OpenAI/Gym/Atari |
| task_name: TicTacToe-play-with-bot |
| pipeline_tag: reinforcement-learning |
| model-index: |
| - name: SampledAlphaZero |
| results: |
| - task: |
| type: reinforcement-learning |
| name: reinforcement-learning |
| dataset: |
| name: TicTacToe-play-with-bot |
| type: TicTacToe-play-with-bot |
| metrics: |
| - type: mean_reward |
| value: 0.3 +/- 0.64 |
| name: mean_reward |
| --- |
| |
| # Play **TicTacToe-play-with-bot** with **SampledAlphaZero** Policy |
|
|
| ## Model Description |
| <!-- Provide a longer summary of what this model is. --> |
|
|
| This implementation applies **SampledAlphaZero** to the OpenAI/Gym/Atari **TicTacToe-play-with-bot** environment using [LightZero](https://github.com/opendilab/LightZero) and [DI-engine](https://github.com/opendilab/di-engine). |
|
|
| **LightZero** is an efficient, easy-to-understand open-source toolkit that merges Monte Carlo Tree Search (MCTS) with Deep Reinforcement Learning (RL), simplifying their integration for developers and researchers. More details are in paper [LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios](https://huggingface.co/papers/2310.08348). |
|
|
| ## Model Usage |
| ### Install the Dependencies |
| <details close> |
| <summary>(Click for Details)</summary> |
|
|
| ```shell |
| # install huggingface_ding |
| git clone https://github.com/opendilab/huggingface_ding.git |
| pip3 install -e ./huggingface_ding/ |
| # install environment dependencies if needed |
| |
| pip3 install DI-engine[common_env,video] |
| pip3 install LightZero |
| |
| ``` |
| </details> |
|
|
| ### Git Clone from Huggingface and Run the Model |
|
|
| <details close> |
| <summary>(Click for Details)</summary> |
|
|
| ```shell |
| # running with trained model |
| python3 -u run.py |
| ``` |
| **run.py** |
| ```python |
| from lzero.agent import SampledAlphaZeroAgent |
| from ding.config import Config |
| from easydict import EasyDict |
| import torch |
| |
| # Pull model from files which are git cloned from huggingface |
| policy_state_dict = torch.load("pytorch_model.bin", map_location=torch.device("cpu")) |
| cfg = EasyDict(Config.file_to_dict("policy_config.py").cfg_dict) |
| # Instantiate the agent |
| agent = SampledAlphaZeroAgent( |
| env_id="TicTacToe-play-with-bot", exp_name="TicTacToe-play-with-bot-SampledAlphaZero", cfg=cfg.exp_config, policy_state_dict=policy_state_dict |
| ) |
| # Continue training |
| agent.train(step=5000) |
| # Render the new agent performance |
| agent.deploy(enable_save_replay=True) |
| |
| ``` |
| </details> |
|
|
| ### Run Model by Using Huggingface_ding |
| |
| <details close> |
| <summary>(Click for Details)</summary> |
| |
| ```shell |
| # running with trained model |
| python3 -u run.py |
| ``` |
| **run.py** |
| ```python |
| from lzero.agent import SampledAlphaZeroAgent |
| from huggingface_ding import pull_model_from_hub |
| |
| # Pull model from Hugggingface hub |
| policy_state_dict, cfg = pull_model_from_hub(repo_id="OpenDILabCommunity/TicTacToe-play-with-bot-SampledAlphaZero") |
| # Instantiate the agent |
| agent = SampledAlphaZeroAgent( |
| env_id="TicTacToe-play-with-bot", exp_name="TicTacToe-play-with-bot-SampledAlphaZero", cfg=cfg.exp_config, policy_state_dict=policy_state_dict |
| ) |
| # Continue training |
| agent.train(step=5000) |
| # Render the new agent performance |
| agent.deploy(enable_save_replay=True) |
|
|
| ``` |
| </details> |
| |
| ## Model Training |
| |
| ### Train the Model and Push to Huggingface_hub |
| |
| <details close> |
| <summary>(Click for Details)</summary> |
| |
| ```shell |
| #Training Your Own Agent |
| python3 -u train.py |
| ``` |
| **train.py** |
| ```python |
| from lzero.agent import SampledAlphaZeroAgent |
| from huggingface_ding import push_model_to_hub |
|
|
| # Instantiate the agent |
| agent = SampledAlphaZeroAgent(env_id="TicTacToe-play-with-bot", exp_name="TicTacToe-play-with-bot-SampledAlphaZero") |
| # Train the agent |
| return_ = agent.train(step=int(500000)) |
| # Push model to huggingface hub |
| push_model_to_hub( |
| agent=agent.best, |
| env_name="OpenAI/Gym/Atari", |
| task_name="TicTacToe-play-with-bot", |
| algo_name="SampledAlphaZero", |
| github_repo_url="https://github.com/opendilab/LightZero", |
| github_doc_model_url=None, |
| github_doc_env_url=None, |
| installation_guide=''' |
| pip3 install DI-engine[common_env,video] |
| pip3 install LightZero |
| ''', |
| usage_file_by_git_clone="./sampled_alphazero/tictactoe_play_with_bot_sampled_alphazero_deploy.py", |
| usage_file_by_huggingface_ding="./sampled_alphazero/tictactoe_play_with_bot_sampled_alphazero_download.py", |
| train_file="./sampled_alphazero/tictactoe_play_with_bot_sampled_alphazero.py", |
| repo_id="OpenDILabCommunity/TicTacToe-play-with-bot-SampledAlphaZero", |
| platform_info="[LightZero](https://github.com/opendilab/LightZero) and [DI-engine](https://github.com/opendilab/di-engine)", |
| model_description="**LightZero** is an efficient, easy-to-understand open-source toolkit that merges Monte Carlo Tree Search (MCTS) with Deep Reinforcement Learning (RL), simplifying their integration for developers and researchers. More details are in paper [LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios](https://huggingface.co/papers/2310.08348).", |
| create_repo=True |
| ) |
| |
| ``` |
| </details> |
| |
| **Configuration** |
| <details close> |
| <summary>(Click for Details)</summary> |
| |
| |
| ```python |
| exp_config = { |
| 'main_config': { |
| 'exp_name': 'TicTacToe-play-with-bot-SampledAlphaZero', |
| 'seed': 0, |
| 'env': { |
| 'env_id': 'TicTacToe-play-with-bot', |
| 'board_size': 3, |
| 'battle_mode': 'play_with_bot_mode', |
| 'bot_action_type': 'v0', |
| 'channel_last': False, |
| 'collector_env_num': 8, |
| 'evaluator_env_num': 5, |
| 'n_evaluator_episode': 5, |
| 'manager': { |
| 'shared_memory': False |
| }, |
| 'agent_vs_human': False, |
| 'prob_random_agent': 0, |
| 'prob_expert_agent': 0, |
| 'scale': True, |
| 'alphazero_mcts_ctree': False, |
| 'save_replay_gif': False, |
| 'replay_path_gif': './replay_gif' |
| }, |
| 'policy': { |
| 'on_policy': False, |
| 'cuda': True, |
| 'multi_gpu': False, |
| 'bp_update_sync': True, |
| 'traj_len_inf': False, |
| 'model': { |
| 'observation_shape': [3, 3, 3], |
| 'action_space_size': 9, |
| 'num_res_blocks': 1, |
| 'num_channels': 16, |
| 'fc_value_layers': [8], |
| 'fc_policy_layers': [8] |
| }, |
| 'torch_compile': False, |
| 'tensor_float_32': False, |
| 'sampled_algo': False, |
| 'gumbel_algo': False, |
| 'update_per_collect': 50, |
| 'model_update_ratio': 0.1, |
| 'batch_size': 256, |
| 'optim_type': 'Adam', |
| 'learning_rate': 0.003, |
| 'weight_decay': 0.0001, |
| 'momentum': 0.9, |
| 'grad_clip_value': 0.5, |
| 'value_weight': 1.0, |
| 'collector_env_num': 8, |
| 'evaluator_env_num': 5, |
| 'lr_piecewise_constant_decay': False, |
| 'threshold_training_steps_for_final_lr': 500000, |
| 'manual_temperature_decay': False, |
| 'threshold_training_steps_for_final_temperature': 100000, |
| 'fixed_temperature_value': 0.25, |
| 'mcts': { |
| 'num_simulations': 25 |
| }, |
| 'other': { |
| 'replay_buffer': { |
| 'replay_buffer_size': 1000000, |
| 'save_episode': False |
| } |
| }, |
| 'cfg_type': 'AlphaZeroPolicyDict', |
| 'mcts_ctree': False, |
| 'simulation_env_name': 'tictactoe', |
| 'simulation_env_config_type': 'play_with_bot', |
| 'board_size': 3, |
| 'entropy_weight': 0.0, |
| 'n_episode': 8, |
| 'eval_freq': 2000 |
| }, |
| 'wandb_logger': { |
| 'gradient_logger': False, |
| 'video_logger': False, |
| 'plot_logger': False, |
| 'action_logger': False, |
| 'return_logger': False |
| } |
| }, |
| 'create_config': { |
| 'env': { |
| 'type': 'tictactoe', |
| 'import_names': ['zoo.board_games.tictactoe.envs.tictactoe_env'] |
| }, |
| 'env_manager': { |
| 'type': 'subprocess' |
| }, |
| 'policy': { |
| 'type': 'alphazero', |
| 'import_names': ['lzero.policy.alphazero'] |
| }, |
| 'collector': { |
| 'type': 'episode_alphazero', |
| 'import_names': ['lzero.worker.alphazero_collector'] |
| }, |
| 'evaluator': { |
| 'type': 'alphazero', |
| 'import_names': ['lzero.worker.alphazero_evaluator'] |
| } |
| } |
| } |
| |
| ``` |
| </details> |
| |
| **Training Procedure** |
| <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. --> |
| - **Weights & Biases (wandb):** [monitor link](<TODO>) |
| |
| ## Model Information |
| <!-- Provide the basic links for the model. --> |
| - **Github Repository:** [repo link](https://github.com/opendilab/LightZero) |
| - **Doc**: [Algorithm link](<TODO>) |
| - **Configuration:** [config link](https://huggingface.co/OpenDILabCommunity/TicTacToe-play-with-bot-SampledAlphaZero/blob/main/policy_config.py) |
| - **Demo:** [video](https://huggingface.co/OpenDILabCommunity/TicTacToe-play-with-bot-SampledAlphaZero/blob/main/replay.mp4) |
| <!-- Provide the size information for the model. --> |
| - **Parameters total size:** 51.13 KB |
| - **Last Update Date:** 2024-02-01 |
| |
| ## Environments |
| <!-- Address questions around what environment the model is intended to be trained and deployed at, including the necessary information needed to be provided for future users. --> |
| - **Benchmark:** OpenAI/Gym/Atari |
| - **Task:** TicTacToe-play-with-bot |
| - **Gym version:** 0.25.1 |
| - **DI-engine version:** v0.5.0 |
| - **PyTorch version:** 2.0.1+cu117 |
| - **Doc**: [Environments link](<TODO>) |
| |
