Homework 6: Actor-Critic for Buckshot Roulette

Train, evaluate, and play against a reinforcement learning agent

Implement Actor-Critic, measure its learning curve, and try the trained agent in the graphical game.
Reinforcement learningPython + PyTorchPySide6 game
Your goal: implement an Actor-Critic agent, measure how its win rate changes during training, and try the trained agent in the supplied graphical game.
Submithw6submission.pdf
AnnouncedWed, Nov 18, 2026after Lecture 33: Policy-Based Model-Free RL
DueWed, Dec 2, 202611:59 PM Eastern · before Prep Days
StarterDownload below
Starter packageBuckshot Actor-Critic

Graphical game, shared rules engine, unfinished agent, and training driver.

1. Play the starter game

Extract the ZIP and run these commands from the extracted hw6 folder.

python -m pip install -r requirements.txt
python check_environment.py
python main.py

In Settings, choose Human and Random, then click Save Settings. The shotgun rests beside the active player, and each player’s HP appears above their portrait. Click a portrait to shoot that player, or click an item in your tray to use it. Hover over an item to see its effect. The shell tray shows the remaining live and blank counts, and the move log records public outcomes. Use New Game to change players.

Game overview

This educational adaptation is inspired by Buckshot Roulette, developed by Mike Klubnika and published by CRITICAL REFLEX. Original game on Steam

Two players take turns using items and shooting themselves or the opponent. Each player starts with 7 HP and can hold up to four items. On each reload, each player draws up to two items, subject to the inventory limit. The starting player is random.

The shotgun holds 3-7 randomly ordered live and blank shells. A live shell deals one damage (two with an active handsaw). A blank self-shot keeps your turn. Other shots pass control unless the opponent is handcuffed. The game reloads automatically when no live shells remain. A player loses when their HP reaches zero.

Item Effect
Handsaw The next shot deals two damage if live, zero if blank. Every fired shot consumes the effect, even if you keep your turn.
Magnifying glass Privately reveals the next shell to the player using it.
Cigarette Restores one HP, up to the maximum.
Beer Ejects the next shell without firing.
Handcuffs Makes the opponent skip their next turn.

2. Implement the agent

Complete ActorNetwork, CriticNetwork, and RLAgent in ai.py. The actor returns seven action logits, and the critic estimates a scalar state value. Keep the networks on self.actor and self.critic so the supplied driver can save their weights and the GUI can load them.

The shared rules engine gives the agent an 18-value observation from its own perspective. The seven actions are handsaw (0), magnifying glass (1), cigarette (2), beer (3), handcuffs (4), shoot self (5), and shoot opponent (6). Use the supplied action mask when sampling so the actor only selects legal actions.

The game calls choose_action(info) when the agent acts and see_result(result, info, training_flag) to report outcomes. A win gives +1, a loss gives -1, and nonterminal actions give 0. Both agents receive the terminal result. Apply learning updates only in training and clear any pending state at the end of each game.

You may use the supporting code as-is, modify it, or write your own solution.

3. Train, evaluate, and try your agent

Run the training driver from the same folder.

python train.py

The default run trains for 1,000 games against the random agent. It measures win rate before training and every 100 training games, using 200 evaluation games per measurement and alternating seats. Use learning_curve.csv or learning_curve.png for your report. Describe the opponent and evaluation protocol you used.

The driver also saves the final network weights to agent.pt. From the same folder, run python main.py, choose Actor-Critic for one player and Random for the other, and watch a match to check that your trained agent plays legal moves. The GUI loads the weights for play. Use the headless evaluation games for your reported win rates.

Submission

Submit one PDF named hw6submission.pdf through Canvas. Late submissions follow the syllabus policy. Scoring is 80% completeness and 20% correctness. This homework contributes 10% of the course grade.

Your PDF should include:

  • A brief explanation of your Actor-Critic implementation covering network structure, action selection, and training. You may use PyTorch, but avoid built-in Actor-Critic implementations.
  • A plot or table showing how your agent’s winning rate against a reference agent (such as a random agent) changes during training.
  • A code appendix containing only your ai.py and any other source files you wrote or modified.

You may consult the textbook, online materials, or large language models to understand the algorithm, including how to handle terminal states.