Homework 6: Actor-Critic for Buckshot Roulette
Train, evaluate, and play against a reinforcement learning agent
hw6submission.pdf
Graphical game, shared rules engine, unfinished agent, and training driver.
1. Play the starter game
Extract the ZIP and run these commands from the extracted hw6 folder.
python -m pip install -r requirements.txt
python check_environment.py
python main.pyIn Settings, choose Human and Random, then click Save Settings. The shotgun rests beside the active player, and each player’s HP appears above their portrait. Click a portrait to shoot that player, or click an item in your tray to use it. Hover over an item to see its effect. The shell tray shows the remaining live and blank counts, and the move log records public outcomes. Use New Game to change players.
Game overview
This educational adaptation is inspired by Buckshot Roulette, developed by Mike Klubnika and published by CRITICAL REFLEX. Original game on Steam
Two players take turns using items and shooting themselves or the opponent. Each player starts with 7 HP and can hold up to four items. On each reload, each player draws up to two items, subject to the inventory limit. The starting player is random.
The shotgun holds 3-7 randomly ordered live and blank shells. A live shell deals one damage (two with an active handsaw). A blank self-shot keeps your turn. Other shots pass control unless the opponent is handcuffed. The game reloads automatically when no live shells remain. A player loses when their HP reaches zero.
| Item | Effect |
|---|---|
| Handsaw | The next shot deals two damage if live, zero if blank. Every fired shot consumes the effect, even if you keep your turn. |
| Magnifying glass | Privately reveals the next shell to the player using it. |
| Cigarette | Restores one HP, up to the maximum. |
| Beer | Ejects the next shell without firing. |
| Handcuffs | Makes the opponent skip their next turn. |
2. Implement the agent
Complete ActorNetwork, CriticNetwork, and RLAgent in ai.py. The actor returns seven action logits, and the critic estimates a scalar state value. Keep the networks on self.actor and self.critic so the supplied driver can save their weights and the GUI can load them.
The shared rules engine gives the agent an 18-value observation from its own perspective. The seven actions are handsaw (0), magnifying glass (1), cigarette (2), beer (3), handcuffs (4), shoot self (5), and shoot opponent (6). Use the supplied action mask when sampling so the actor only selects legal actions.
The game calls choose_action(info) when the agent acts and see_result(result, info, training_flag) to report outcomes. A win gives +1, a loss gives -1, and nonterminal actions give 0. Both agents receive the terminal result. Apply learning updates only in training and clear any pending state at the end of each game.
You may use the supporting code as-is, modify it, or write your own solution.
3. Train, evaluate, and try your agent
Run the training driver from the same folder.
python train.pyThe default run trains for 1,000 games against the random agent. It measures win rate before training and every 100 training games, using 200 evaluation games per measurement and alternating seats. Use learning_curve.csv or learning_curve.png for your report. Describe the opponent and evaluation protocol you used.
The driver also saves the final network weights to agent.pt. From the same folder, run python main.py, choose Actor-Critic for one player and Random for the other, and watch a match to check that your trained agent plays legal moves. The GUI loads the weights for play. Use the headless evaluation games for your reported win rates.
Submission
Submit one PDF named hw6submission.pdf through Canvas. Late submissions follow the syllabus policy. Scoring is 80% completeness and 20% correctness. This homework contributes 10% of the course grade.
Your PDF should include:
- A brief explanation of your Actor-Critic implementation covering network structure, action selection, and training. You may use PyTorch, but avoid built-in Actor-Critic implementations.
- A plot or table showing how your agent’s winning rate against a reference agent (such as a random agent) changes during training.
- A code appendix containing only your
ai.pyand any other source files you wrote or modified.
You may consult the textbook, online materials, or large language models to understand the algorithm, including how to handle terminal states.