The Call of Duty community seems to be up in arms about Skill-Based Matchmaking and how it’s supposedly ruining the new Modern Warfare. Unfortunately, most people seem to have a lot more feelings about the issue than books. I’ve read countless reddit threads about it and watched a few Youtube videos from people who are against skill-based matchmaking. Long story short it would basically be impossible to prove with a reasonable degree of certainty that SBMM is even in the game without querying so much player data you’d be looking at a tresspass to chattels lawsuit. Skill-based matchmaking also isn’t hurting the game even if it’s in the game and the arguments against it are pathetically weak.
Is SBMM is Modern Warfare?
Probably… maybe… I guess but also maybe not… kinda? Anyone that tells you that there is for certain or that it has very strict parameters is full of shit. The top video I got to when searching “SBMM modern warfare” is a video by King Plays who claims he “tested SBMM and its clear that it is in Modern warfare and it is strong”. To do his test he:
started on a level 1 account, played well for 10 games, my games were getting harder the better I done, I then reverse boosted for 10 games, after 3 games, its certain that SBMM is strong in MW, I was playing much easier players and this is why people are quitting Modern Warfare, if you are a good player, playing against only good players every game, it becomes tiring.
So he determined that SBMM was in the game based upon his personal opinion of how hard the games were, his subjective evaluation of how he performed in those games, and a sample size of 23 games. That’s not how science works. It would be very difficult to prove that SBMM is in the game, let alone how it works with any reasonable degree of certainty. I’ve seen a few other threads where players collected their own personal data, which when trying to determine how the system works is basically worthless. The sample size is so small it may as well not even exist. The only real evidence that SBMM is in the game is from an interview with the multiplayer design director of Infinity Ward, Brian Shea. Shea says, in short, that balanced matchmaking games was a priority for them moreso than in the past. Ultimately this is the strongest evidence that suggests that SBMM is in the game. At the same time, game developers have also said stuff like “yeah you can climb those mountains” or “we want to make the next Better Devils just a little more interesting”. At best, based on that statement you could there probably is some degree of skill-based matchmaking. How it compares to the SBMM if applicable in other Call of Duty games is undetermined.
To determine that SBMM is even in the game you would need a huge sample, consisting of thousands of players and the results of them playing a huge number of games starting with all new accounts. How many exactly? I just want to be a psychiatrist, okay? Math is hard. The point is nce you have your huge series of games, you would split the series of the games they played in half. Then you would analyze the results of the matches in the first half in the order they were played and crunch a skill number for each player based on the results of each match. Once you have a skill rating crunched for each player, you could iterate through each match in the series in order for the second half of the set. You would parse the second half of the matches and adjust players skill rating for each match and look at the differences in skill-rating between teams in each match. If the average difference of in rating betwen each team was pretty big, that means there’s probably no SBMM. If it was really small, then there’s probably some sort of skill-based matchmaking. Once again, that’s all theoretical and even then it’s kind of a jank methodology. But that is the best way you could determine if SBMM was in a video game or not without outside information. Nobody has done that, and nobody is probably going to do that if it’s even possible. Once again it would require querying so much data from Activision you would get sued. Absent further clarification from Infinity Ward about the presence of SBMM in Modern Warfare or some sort of concrete-ish evidence, it would be hard to prove that SBMM is in the game at all outside the comments from Brian Shea.
But for the sake of argument, let’s just assume that SBMM is in the game.
How SBMM works, and how it doesn’t.
I’ve been seeing a lot of misinformation about what SBMM is and how it might work. While I’ve stated that it’s impossible to know how it works without a developer explaining it or looking at the code, the stuff I’ve seen in the COD community about how it might work is just plain wrong. There are two ways you could pseudo-reverse engineer how a system might work, you can take a top-down or a bottom-up approach. With the top-down approach you just come up you figure out how it should work just theoretically and see how it matches the behavior of the game. The bottom-up approach involves looking at how the system behaves based on the data that the game gives you and figuring out how it works based on that. That data would include some sort of ELO/MMR stat or whatever you want to call it. So you simply don’t have the data that you need where you could get anywhere close. So let’s take the top-down approach, here’s how SBMM in Modern Warfare might work, or at the very least, how it doesn’t work.
SBMM in Modern Warfare almost certainly works off of something similar to Microsoft’s TrueSkill Matchmaking System to rate players. If you want to build a matchmaking system for a casual video game, it’s basically the gold standard. It’s quite clever, and also well documented. You’re not allowed to use the system for licensed non-Xbox Live games, so it’s not what is being used, but it’s probably close. True Skill basically assigns each player an average skill rating, and a standard deviation of skill. Unlike in other systems, TrueSkill keeps track of your average skill and your standard deviation of skill. So a newer player will have a much wider standard deviation than someone who has been playing a while. As you play more games your skill deviation gets lower and you get more consistent matches. This takes longer based upon the number of players on a team in a match, but the skill rating is generalizable on a per-playlist basis. When the system does its job, you can statistically approximate how likely a team or player is to win a game. That doesn’t tell you about how the game actually decides to match players only how it rates them. You can allow totally lopsided matches, it really just depends on the matchmaking criteria. Skill rankings and matchmaking are totally different things.
I’ve heard multiple Youtubers and reddit commenters talk about K/D with regards to how skill is approximated, this simply isn’t how it works. I don’t think I’ve ever heard of a ranking system that adjusts player ratings based upon in-game performance alone or even weights performance heavily. After reading through a white paper on True Skill 2, it seems MS utilizes game performance to give players who perform well more credit for a win. But not substantially more. I really, really, really doubt Modern Warfare bases player skill ratings based upon in-game performance like Kill/Death Ratio, score-per-minute, or anything else. Why? It would be really, really dumb. I know this is going to be a tough pill for the COD community to swallow, but K/D doesn’t mean anything. K/D/SPM can vary wildly based upon your playstyle, maps, mode, playstyle of your opponent, and all kinds of stuff. You can have a great K/D, contribute nothing to the team, and lose the game. Even the writers in the MS white paper acknowledge this fact. The outcome of the match is the most valid metric to determine player skill, so much so that in-game statistics are almost irrelevant. You don’t get a 5000 K/D in a 10th percentile skill bracket game and then get moved to the top 99th percentile skill bracket the next game. That’s not how it works.
Bad arguments against SBMM.
One argument I see against SBMM is that it throws people in servers in other countries or whatever. This is a hard question to answer, because computer networks are sorcery. I don’t know of any game that doesn’t prioritize networking first, and I don’t know why Modern Warfare would be any different. Will you occassionally get games with really terrible pings? Of course, because once again: computer networks are sorcery. You can just have bad routing to a random game server that the MM server throws you on. An individual server can have problems that lead to high pings. I really doubt the matchmaking system pings every individual server it could place you onto before matching, it would be really ineffecient. Sometimes you get on a crappy server, shit happens.
Another bad argument I see is that it stops you from playing with friends. Any skill-based matchmaking system worth a damn can build a reasonably fair match. I’m sorry, it’s just not a thing. New players are going to have a rough time when playing with friends if their skill level is highly uncertain but this is such a minute edge case it’s just not worth considering. It’s just how the ranking systems work.
The biggest argument I see against SBMM is that players in the high skill bracket are forced to use the meta-loadout of M4/725/Claymore or whatever the FOTM loadout may be to do well. Alright anti-SBMM people, I’m going to level with you. You just want SBMM gone so you can pubstomp low-skilled players with your ultra-pro FOTM loadout to inflate your ego. If you were using a non-meta loadout and not playing sweaty, and using a meta-loadout and playing sweaty is a requirement to win games; then you you’re in a high skill lobby because you use meta-loadouts and play ultra sweaty. If you don’t play sweaty and don’t use meta-loadouts but are still in a high-skill lobby, then playing sweaty and using meta-loadouts isn’t a requirement for doing well and I don’t see what your problem is. It’s really one or the other.
If Infinity Ward really chose to put SBMM in the game there’s probably a pretty good reason. That reason being player retention. Most of the research I’ve seen has shown that poor MM game balancing contributes to reduced player retention. I’m surprised how much research there is on this topic actually, it’s quite interesting. Less player retention means less sales, less sales means less money. It’s really not complicated. So obviously companies want to implement matchmaking systems that lead to more balanced games. As it turns out, more balanced games are more fun for people. If you want your games to stop being so sweaty, you’re going to need to meet the system halfway and play less sweaty.