Yahtzee

My second video, on Yahtzee strategy, is out. You can find it here. This post covers some more details about strategy that I couldn’t fit in the video, as well as some interesting problems in Yahtzee strategy that I think are still open. I also made a post about the response to my Battleship video, and some of the changes I made for the new video, which you can find here.

In Battleship, a lot of my time was spent trying to salvage something from the increasingly complex placement strategy, ultimately delaying the video by quite a bit. For Yahtzee, I decided to avoid this by focusing on what worked: leaning into the single-player point maximization and relegating multiplayer approaches some small segments at the end. For what it’s worth, I do think a vast majority of the advantage in Yahtzee comes from point maximization and not from multiplayer adjustments. However, this post will mostly just be providing some more insight and data on single player cases, although there are some interesting open questions on multiplayer strategy at the end.

The Math

Especially compared to Battleship, the Yahtzee math was relatively straightforward. We just do some dynamic programming over all possible scorecard states that you can encounter in a game. The 342 billion state version doesn’t actually appear anywhere in my code, other than a standalone script designed specifically to calculate that number for the video.

My original code centered around the version in which you just ignore which boxes gave which numbers of points, focusing on boxes filled in, top and bottom section scores, and number of Yahtzees. I believe this is the most simplified version of our scorecard that still allows us to make all possible strategic decisions, with 105,285,166 non-terminal scorecard states. If you’re trying to maximize your chances of reaching a specific number or of beating a specific opponent, you might need to know all of this information.

However, the video requires us to do a lot of computation on these states. We need to know probabilities and expected points for each state, as well as various other calculations like “expected points for a box given that we haven’t filled that box by each turn number” for the line graph scene. In most cases, we can’t calculate these numbers for the start state without calculating them for every other state, so it became very important to shrink the state space to do these computations.

This is part of the reason why I focused almost the entire video on the point-maximizing strategy. If the only thing we care about is maximizing expected points, then you can ignore pieces of information that don’t directly affect future points, like the bottom section score. This brings us down to 536,320 scorecard states, which actually allows for efficient computation of any statistic I may need in just a few minuts.

The Endgame

The endgame is fun to think about because it essentially gets down to what most people think about when they play Yahtzee: how do you get one box in particular? The interesting ones here are the straights, 3 and 4 of a kind, and chance.

Chance is fun because it’s basically just a quant interview question showing up in the middle of a board game. 3 and 4 of a kind do a good job of illustrating this tradeoff between probability and raw numbers that make up an expected value, which I hope I got across in the video. 3 and 4 of a kind had a bunch of weird exceptions where you give up probability of success to maximize points. I briefly showed most of these examples in the video, but I’ll put the full tables with these exceptions at the bottom of this article (notice how eg a single 5 appears ahead of 2-2 in the 3 of a kind table, and how 1-1 doesn’t appear at all. There’s always something in your roll that you’d rather have than a pair of ones).

Getting the straight strategies into the video was one of the more fun challenges. It’s an interesting puzzle to try to come up with rules like “reroll everything if you have no 3s or 4s” and “only keep a 1 if you have 1 and 2 but no 5” from a table like this:

This is a table of all of the 252 possible dice rolls prior to the last reroll of a game that is only missing a small straight (and doesn’t have Yahtzee bonuses available). Notice that the expected points depend only on the dice that you choose to keep (this seems obvious in hindsight but took me a minute to realize). Here’s a simplified version, grouped by the kept dice. There are surprisingly few cases.

The way to read this chart is that you find go through the list form top to bottom until you find a subset of the dice you currently have, and then you choose those to keep.

One thing that adds weirdness to these tables is the strange rules around Yahtzee bonuses. If it’s the last turn, and the 100 point bonus for an extra Yahtzee isn’t available, then getting a Yahtzee is equivalent to getting whatever bottom-section box you’re looking for. In a vast majority of cases, this doesn’t help you - Yahtzees are rare. If you happen to have 4 of a kind and no 3s or 4s, though, then your 1 in 6 chance of finishing up the Yahtzee is a safer bet for filling your small straight than actually trying for a small straight. I tried to use language that avoided these cases in the video, but for both straights and 3/4 of a kind there are cases where it’s better to go for a Yahtzee (full house has several cases where you’re indifferent on which one you go for).

If the 100 point bonus is available, though, then things get a lot weirder. In any actual game, your strategy should basically be “go for a Yahtzee bonus if I need 100 points to win, and otherwise ignore it.” If you’re trying to maximize expectation, though, then it leads to all kinds of weird changes. Here’s that same small straight table, but with the Yahtzee bonus available:

4 of a kind becomes one of the best things to save, ahead of having 3/4 of the small straight. Large straight is even weirder:

If you have, say, 12333, the strategy says to keep the threes. It’s a 1/36 chance of 100 points versus a 2/36 chance of 40 points, so it makes sense, but it still seems strange to ignore the box you’re actually going for when you’re that close to getting it.

Having a Yahtzee bonus available messes with pretty much every box. Even in the top section, there are plenty of cases where you should just ignore the number you’re supposed to be going for to try to get a Yahtzee instead because of how valuable that 100 point bonus is.

The Middlegame

One of my regrets from the video is that I spent a lot of time talking about option value at various times, but don’t think I did a great job of communicating what that looks like in Yahtzee.

Yahtzee has

• 9 boxes where you try to get a lot of the same number (top section, 3kind, 4kind, Yahtzee)

• 2 boxes where you try to get a lot of different numbers (small straight, large straight)

• 1 box in between these two (full house)

• 1 box that doesn’t care (chance)

The goal is to keep around boxes, or combinations of boxes, that can go well with a wide variety of rolls. Chance is the most extreme example of this, since it can get points for anything, but 3 and 4 of a kind are also good examples.

If you roll a 3 or 4 of a kind, it’s usually best to fill in a top section box instead. This is partially because of the value of the top bonus, but also because 3 and 4 of a kind are easier to get later. By filling in 3 5’s instead of a 3 of a kind, you get rid of a box that can only be filled in a few specific ways, and replace it with a box that can be filled in with 3 of any of the 6 numbers (even if you usually don’t want to fill it with small numbers). The probability of getting 3 of a top box on the last turn is about 35%. For 3 of a kind, it’s 71%.

While there are individual boxes that provide option value, there are also combinations of boxes that provide option value. Both straights are good to keep around because of how easy it is to score points with them, but my line graph in the video actually undersells the importance of keeping these boxes open. Once those boxes and chance are gone, every single box left on the board benefits from getting many of the same number. If you get a roll with 4 or 5 distinct values, then that makes it difficult to do well in 10 of the 13 boxes in the game, so keeping around straights and chance for most of the game help to provide a nice balance where most rolls can do something for you.

The Opening

I left a lot out of the video, but opening theory for Yahtzee (at least for turn 1) turns out to not be that complicated. Across both rerolls and the final box choice, there are only 89 cases to think about, and they can be simplified further for the purposes of trying to remember them if you’re so inclined.

End of Turn 1

Below is the full table from the video. There are 32 distinct outcomes for turn 1 using the optimal strategy, which is actually fairly small compared to the 252 possible outcomes for the dice. Note that some dice combinations can be represented by multiple rows on this table. In those cases, the strategy is to choose the highest relevant row on the table. For example, the roll 11335 could go under 2 1s or 2 3s, but 2 1s appears higher on the table, so we choose that box.

Some highlights that I didn’t mention in the video include that one place where you gain a small fraction of a point by choosing a 3 of a kind instead of a full house if you get 55666. Each additional die you put in a top section box is worth about 3 times its face value in expected game score (eg going from 3 5s to 4 5s is worth almost 15 points, going from 1 1 to 2 1s is worth almost 3 points). There are chance values ranging from 19 to 26. The distribution of outcomes is fairly skewed: only 6 outcomes, totaling less than 30% of the probability, actually increase your expected score.

Second Reroll of Turn 1

Even though I cut them from the video for time, the strategy on rerolls for the first turn may actually be more instructive than the final decisions at the end of the turn. The following tables have outcomes grouped by the dice that you’re supposed to keep, since that’s the only thing that matters for your expected score. In cases where you keep all 5 dice, I just kept one representative roll (eg 11111 for Yahtzee, even though all Yahtzees are worth the same amount). Note again that some dice rolls could fall under multiple rows in this table, so we should always keep the dice in the highest corresponding row.

Some highlights here: we sometimes keep a full house, but not always. If we have 3 4s, 5s or 6s, we just keep those three dice rather than taking the full house. 2345 is noticeably better than the other small straights, since it doubles your chance at getting a large straight (this comes up a lot throughout the game). You always keep at least 2 dice, and the only time you don’t keep a pair or 3 in a row is the roll 12356.

First Reroll of Turn 1

On the first roll, we care less about straights and focus more on groups of identical numbers. The only time we keep multiple distinct numbers without a straight is in the specific roll 11345, since the model really doesn’t like keeping a pair of ones. That reasoning also covers the roll 11236, which is the lone case where we keep just a 6 at the bottom of the chart. Note that this also means that we never keep a full house on the first roll. We’d always prefer to just keep 3 of a number, with a 52% chance that we’ll roll at least a fourth later on.

2-Player Strategy

I had originally planned to focus more of this video on multiplayer strategy, but ultimately decided against it. The Battleship video took a really long time to make. That was partially due to having to figure everything out from scratch, but partially due to a bunch of my attempts at the placement strategy not working. I wanted to see if I could get videos done on a monthly timeline, so I decided to stick with things that I knew would work for the Yahtzee video. I did miss the more open-ended problems that I worked on in Battleship, but they ultimately turned out to take up a vast majority of my time on what turned out to be a small portion at the end of the video.

Single player point maximizing mostly just worked. Other than the simplifications that had to be made to the state space, everything basically just went as intended. Just adding one more player makes things much harder. The single player version could be reduced to 500k states, but that doesn’t mean that the 2 player version has (500k)² states. The thing that made the single player reduction work was that we only had to think about our ability to get future points. When you add a second player, you need to know not only the distribution of future points for each player, but also the difference between the two players’ points.

The number of reduced states peaks at turn 7 with 126,219. Squaring that gives almost 16 billion, and when you factor in all possible differences in points between the two players and the 756 possible dice states, we’re potentially looking at quadrillions of positions, just for player 1 on turn 7.

Reducing the State Space

Now, there are definitely simplifications to be made here. I’m sure there are plenty of score differences that are not possible, and we can filter out large numbers of states where one player is guaranteed to win the game, but we still lose the thing that was nice about the single player version. We can’t just go through every state anymore.

Here are some thoughts I’ve had on ways around this, but I’d love to hear more if people have ideas. You can probably eliminate a lot of states where one player is overwhelmingly likely, if not guaranteed, to win. While the probability of two players’ scores being close at any given time is quite high, the fraction of states in which the scores are close is much lower, so most states can be eliminated this way.

I also think there’s some reduction that can be done using the reduced scoring system I mentioned in the video. Using the point-maximizing strategy, if player 1 beats player 2 by 1 point in the simplified scoring, there’s a 97% chance they also win the game. If they win by 2 reduced points, the win probability is 99.98%. I think a version where we just try to maximize the probability of beating our opponent on the reduced scoring scale (which, in a vast majority of games, caps out at 10) would get us most of the benefit, and would extend well to much larger numbers of players. If we want to get a bit more involved, we can try to maximize P(more reduced points) + P(same reduced points) * P(more points | same reduced points), so that we only have to consider actual points in the case where reduced points are the same.

This also seems like a great place for a neural net or a tree search, but I’m not sure how much more can be squeezed out of 2-player strategy at this point. For what it’s worth, I think a vast majority of the advantage in Yahtzee just comes from maximizing expected points, although I’m curious to seem how much the strategy changes in practice from trying to target an opponent.

Multiplayer Strategy

There are a bunch of different directions that you can go with multiplayer Yahtzee. How many opponents do you have? Are you trying to optimize your average placement, or specifically trying to win? Do you have access to your opponents’ scores? Are you assuming that your opponents are using an optimal strategy, or a typical human one? How do we even create a “typical human strategy?” (Battleship had the hunt-and-target, which was a sort of decent approximation, and appeared a few times in other analyses. Coming up with a decent approximation of a human Yahtzee player seems like a really daunting task.)

My brief analysis in the video focused on one of the simplest versions of this. Everyone else is using an optimal point-maximizing strategy. Only you are trying to win, although you don’t have access to your opponents’ scores. I mostly just wanted to make the cool moving histograms and talk about the ridiculously large numbers of players, and it was also probably one of the easiest cases to analyze.

Realistic Opponents

We can at least talk about some of the other versions of this, though. I mentioned in the video that more, worse players could be approximated as fewer, better players. I don’t really know how accurate this is in practice. I don’t have a great sense of what a “typical player” looks like, or what their chances would be of getting a lucky good score in a random game. I would assume that if you’re thinking deeply about this, then you’re hopefully playing against players that are at least somewhat good at the game. I do think that with some practice, it’s possible for a human to get pretty close to optimal. In my 45 online games played while working on this video, my average score is actually slightly above 255, although there’s definitely some luck involved there.

Maximizing Placement

In the case where you’re just trying to maximize average placement, I mentioned that you should be more aggressive if you’re doing badly and more conservative if you’re doing well. This is roughly true, but there’s a slightly more accurate version. You should be more aggressive if your expected future points put you in a part of the overall distribution where the derivative of the probability density is positive, but conservative if the derivative is negative.

This version is still not completely accurate, but gets at a more general idea. If the derivative of the pdf is positive, then you’re more likely to pass someone by gaining points than you are to get passed by losing points. In most games, where point distributions are more bell-shaped, this is the same as just “be aggressive when you’re behind.” Because Yahtzee has this weird multimodal distribution, it actually is worth thinking about this weird derivative effect, especially later in the game, if you are trying to be truly optimal about maximizing expected placement.

Blind Multiplayer

I think the version of multiplayer that I thought the most about, but didn’t actively work on, was this: everyone is trying to win, and playing optimally to do so, but you don’t know each others’ scores. Basically, you’re just trying to maximize your chances of having the highest score out of n good players. This is sort of iterating on the version I put in the video. Not only are you trying to beat all of these optimal players, but they are as well.

My approach to solving this problem is something like this: start by assuming a final distribution for the max of your opponents (maybe just use the distribution of the max of n-1 point maximizers like I did in the video). Then come up with a strategy to maximize your probability of beating a point drawn from that distribution. This requires using the larger state space with the 100 million states instead of the 500k, but this is still feasible to do.

Once you do that, replace everyone’s strategy with the new strategy you just computed, and repeat. This should converge to an optimal strategy for this version of the game. I would expect this to lead to a riskier strategy for everyone, resulting in higher chances of large totals than the ones shown in the video. We’re still not properly reacting to everyone’s performance throughout the game, but we can (somewhat) efficiently calculate a strategy that is good in practice.

Actual Optimal Multiplayer Strategy

That leaves open the purer question of true optimal play. I think the reduced point system can get you pretty far, but I don’t have anything particularly promising for how to beat that. I’m imagining some sort of model that can take in everyone’s current state and spit out a distribution for the winning score of all of your opponents, but presumably that model should also take your state as an input, making it a bit circular. Endgames can probably be analyzed pretty thoroughly, since there are hopefully only a few people with a realistic chance of winning by the end of the game, but earlier strategies would require a lot of things that I don’t have great thoughts on. I’d love to hear what people think.

Miscellaneous Tables