RL for Blackjack Part 1
Monte Carlo Control for Blackjack Strategy
I recently experienced a wonderful coincidence while reviewing some examples in Chapter 5 of Reinforcement Learning (Sutton and Barto 2020) after just finishing A Man For All Markets (Thorp 2017). In example Y, Sutton & Barto showed how policy iteration could be used to generate blackjack strategy tables similar to the ones that Thorp had produced with a different approach in the 1960s. This article describes both Thorp’s original methodology and how the massive increase in computational power over the last 60 years changes the dynamics of the problem.
Edward Thorp Finds an Edge in Blackjack
Prior to Thorp’s work, it was a widely held belief that no strategy existed to beat casino Blackjack (aka Twenty-One). In “The Optimum Strategy in Blackjack” (Baldwin et al. 1956) the authors were able to use hand calculators to devise a strategy that reduced the house’s edge to 0.62%. This is a very small edge, but still losing for the player in the long run. For contrast, American Roulette has a house edge of 5.26%.
What Thorp realized is that Blackjack was inherently different from memoryless games like Roulette or Craps. In Roulette every spin of the wheel is an independent trial. Previous spins have zero impact on the next spin. Blackjack on the other hand is a dependent trials process. As cards are removed from the deck from one hand to the next, the state of the deck changes. The state is not reset until the deck is thoroughly shuffled.
In his seminal paper “A Favorable Strategy for Twenty-One” (Thorp 1961) and his later book Beat the Dealer (Thorp 1966), Edward Thorp showed how a player could get a small edge by using a specific strategy that he calculated using combinatorial analysis. Improving on the work of Baldwin et al, Thorp performed recursive calculations to find the exact Expected Value for every possible hand composition.
a high speed computer was programmed to find the player’s best possible strategy and the corresponding expectation.
– A Favorable Strategy for Twenty-One
This high speed computer was an IBM 704, a significant speed improvement over using hand calculators. Thorp was able to calculate exact EV’s for many different deck states, such as when certain cards were removed. From these simulations, Thorp determined that there were deck states where the conditions were actually favorable to the player. In other words, buy using some strategy where the player monitored the cards that had already been dealt from the deck, the player could have positive EV while betting.
Sutton & Barto Apply Learning through Experience
Decades later, Richard Sutton and Andrew Barto pioneered a different paradigm called reinforcement learning (RL). While I don’t believe they were thinking much about casino games at the time, it’s definitely possible to use RL methods to generate some of the same results that Thorp did.
Inspired by example Y (Sutton and Barto 2020), I tried applying Monte Carlo Policy Iteration. This approach requires an initial policy that dictates what action (e.g. hit, stand, or double) a player should take given their cards and the visible dealer cards. It also requires a simulation engine that can complete large numbers of blackjack games using the player policy.
The policy iteration approach cycles through two phases:
- Policy Evaluation: The agent plays a large number of games using its current strategy to estimate the action-value function.
- Policy Improvement: The agent updates its strategy by becoming “greedy” with respect to its estimates.
For a far more complete explanation of policy iteration, please refer to the excellent textbook by Sutton & Barto.
Combinatorial Analysis v. Policy Iteration
It’s important to note that Thorp’s approach was quite different from using Monte Carlo simulations. Thorp’s program calculated the exact probability distribution of the dealer’s final hand for every possible upcard. He then did the same for the player. To find the optimal strategy, he worked backward. He first determined the expectations for “standing” (which are fixed once the player has a total), then calculated the value of “hitting” by looking at the expected values of the resulting hands. He then repeated this analysis with certain combinations of cards removed from the deck.
In other words, Thorp calculated the EV’s exactly using a model of the game. On the other hand, Monte Carlo simulations run a large number of trials to estimate the EV’s. With sufficiently many trials, the estimates should converge to the values Thorp calculated.
Monte Carlo in 1960?
It is tempting to wonder why Thorp didn’t simply “simulate” his way to the answer. Thorp’s approach was very elegant but writing and running the Fortran code for all the possible combinations must have been painstaking work. Writing a blackjack simulation and running it many times is relatively easy. I was able to do it in an afternoon. The reason is purely hardware-bound. Even though it was fast for it’s time, Thorp’s IBM 704 could only perform roughly 12,000 instructions per second.
A robust Monte Carlo simulation that converges to a high degree of precision requires a massive number of iterations to converge. In 1960, the compute time required for an RL agent to “learn” basic strategy would likely have been prohibitively expensive and time consuming to run. Recursive math, while complex to code, was computationally cheaper because it reached an exact answer in a single, deterministic pass.
In 2025, the compute landscape is massively different. On my 2025 M5 Macbook Pro, a decent estimate is 9 instructions per cycle at 4.6 GHz per performance core. So this comes to about 160 billion instructions per second across 4 performance cores, or about 13.3 million times what Thorp had. To put that in perspective, it would have taken Thorp about half a year of compute time in 1960 to do what I can do in one second on my laptop today.
Limitations of Modern Simulation
While our modern PCs can brute-force the answer in seconds, the Monte Carlo approach has inherent flaws that Thorp’s exact methods avoided:
- Stochastic Noise: Because we sample randomly, our results have a variance (). We never get the exact value, only a value that is “likely” correct within a margin of error.
- The Rare State Problem: In a standard simulation, states like “Player 12 vs. Dealer 2” appear much less frequently than “Player 20 vs. Dealer 10.” Without “Exploring Starts,” the agent may take a very long time to learn the optimal move for these rare edge cases.
- Model-Free Blindness: RL is “model-free,” meaning it doesn’t know it’s playing with a deck of 52 cards; it just sees numbers. It effectively ignores the rich combinatorial structure that Thorp exploited to create card counting.
Thorp’s method was incredibly labor-intensive to program. Every rule variation (e.g., “Dealer hits Soft 17”) required a complete re-run of the recursive engine. His calculations were also “static”—they provided a strategy for a specific deck state but didn’t inherently “learn” to adapt to new rules without human intervention.