RL for Blackjack Part 4
Doubling Down and More
This is the fourth article in the ML for Blackjack strategy series. In Part 3 we worked to reproduce the basic strategy presented in Beat the Dealer (Thorp 1966) by extending the example problem described in Reinforcement Learning (Sutton and Barto 2020) using Monte Carlo with Exploring Starts. To model Thorp’s single deck basic strategy, we expanded the state space to account for hands that contain pairs as well as multi-card hands. In this way we were able to generate a policy that reproduces the basic strategy for standing and hitting. However, casino blackjack usually allows for more actions than just standing and hitting. In many blackjack games the player can also choose to double down, take insurance, split pairs, or surrender.
Thorp’s basic strategy incorporates both pair splitting and doubling. In blackjack, when the player splits a pair, he makes an additional bet of the same size as the original, splits his two cards into two hands, and receives an additional card for each hand. The player then plays each of the split hands as if they were a normal 2-card hand. Doubling down involves the player doubling the size of the original bet on a single hand and receiving one and only one additional card.
The expanded state space from Part 3 is sufficient to implement the splitting and doubling mechanics. It tracks whether the first two cards are a pair (required for splitting) and the number of cards (doubling is only allowed on the first two cards). For this study we opted to only look at the doubling action for a couple reasons. First, modifying the simulation engine to include the game mechanics for doubling is relatively straightforward. The player simply receives one more card and the reward (or loss) is doubled. Pair splitting is a bit more complicated since we may have to track multiple trajectories in the same episode. Second, with the increased size of the state space as well as more actions to track, the state-action space is starting to get large. This means more iterations to converge and longer compute times. The 100 million iterations needed to converge took about 45 minutes without adding the actions for doubling and splitting. We would prefer to rewrite the simulation code with performance and multiprocessing in mind before undertaking a larger study.
Thorp’s Doubling Strategy
The following tables represent doubling strategy as reproduced from Beat the Dealer (Thorp 1966).
Hard Hands
| You have | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | A |
|---|---|---|---|---|---|---|---|---|---|---|
| 11 | D | D | D | D | D | D | D | D | D | D |
| 10 | D | D | D | D | D | D | D | D | ||
| 9 | D | D | D | D | D | |||||
| 8 | \(\ast\) | \(\ast\) |
- \(\ast\) Double down except with (6,2).
Soft Hands
| You have | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | A |
|---|---|---|---|---|---|---|---|---|---|---|
| 18 | D | D | D | D | ||||||
| 17 | D | D | D | D | D | |||||
| 16 | D | D | D | |||||||
| 15 | D | D | D | |||||||
| 14 | D | D | D | |||||||
| 13 | D | D | D | |||||||
| 12 | D | D |
Doubling in MC ES
Simulating doubling with MC ES involves adding a third action D in addition to S and H. The reward for winning a hand after doubling and the cost of losing is twice the original bet. We must also include the D action in the exploring starts for any 2-card hand. However, we must exclude the option to double from any 3 or more card hands. This means that the state-action space for 2-card hands grows while the state-action space for 3 or more card hands is the same as in the prior optimization.
Derived Policies
Unlike the Stand and Hit policies we derived in the previous study, we were unable to match up the doubling strategy exactly with Thorp’s work. The MC ES policies seem to favor doubling in a few key instances where Thorp does not.
For 2-card hard hands, the derived policies are the same as Thorp’s for player counts of 9, 10, and 11. Thorp’s policy says to double on a hard 8 against a dealer 5 or 6 unless the player’s cards are \([2,6]\). The state representation we used does not track \([2,6]\) as a unique state. Instead, the policy recommends doubling on any hard 8 hand against a dealer 4, 5, or 6. Doubling a hard 8 against a 4 is definitely more aggressive. In addition, the MC ES strategy recommends doubling a 2-card 5, 6, or 7 against a dealer 5 and a 2-card 7 against a dealer 6. In Thorp’s policy, the player would hit those hands instead of doubling.
For 2-card soft hands, the MC ES policy is again very similar to Thorp’s strategy buy a bit more aggressive. In addition to the doubling actions indicated by Thorp, the MC ES policy also recommends doubling a soft 19 against a dealer 4, 5, or 6 and a soft 13 or 14 against a dealer 2 or 3.
Understanding Policy Differences
There are several possible reasons for the difference in Thorp’s basic strategy and the policies derived here. The first is that the policy iteration simply has not reached final convergence. In it’s apparent that the policy is still actively changing even after 100 million episodes. A second possible reason is a difference in the specific rules used for the simulated blackjack game. Our simulation did not include the action for splitting, which may influence the policy for certain hands. For example the policy recommends doubling a soft 12 in many cases. A soft 12 is actually an \([A,A]\) hand which is one of the best hands to split. Other rules like whether the dealer hits a soft 17 or if doubling after a split is allowed will also impact the policies. Third, the difference in expected value between two actions may be extremely small as we saw in the regret analysis from the previous study. Thorp may have simplified his tables to make them easier to remember in cases where the difference between two actions was very small.
Future Directions
I enjoyed this extended application of MC ES and was very pleased to see how well the results lined up with Thorp’s combinatorial calculations. I continue to be impressed by the level of detail Thorp was able to achieve with his analysis given the limited compute resources he had available to him. I’m eager to continue this analysis to include mechanics like splitting, surrender, and variable numbers of decks. In addition, it would be fascinating to evaluate some of the many card counting strategies that have been published over the years. Currently, 100 million episodes takes about an hour on my M5 with my simple Python blackjack code. So to practically approach some of these large simulations, I would like to write a speed-optimized blackjack simulation before I return to this study.