Riot Games prepares to close its last data center as it completes global migration to AWS

At AWS re:Invent, Riot Games’ Head of Global Infrastructure and Operations, Brent Rich, outlined how the company is in the final stages of its multi-year global datacenter decommissioning effort, which will be completed in early 2024. Through this effort, Riot is bringing servers closer than ever to players of Riot titles, including League of Legends, VALORANT, League of Legends: Wild Rift, Teamfight Tactics and Legends of Runeterra.

Starting in 2017, Riot Games made the decision to decommission its physical data centers and move entirely to the Amazon Web Services (AWS) cloud. Since then, 14 data centers have been closed, with data centers in Las Vegas and Chile shutting down in the past month. Riot plans to shut down its remaining data centers in Brazil and Turkey in the coming months.

In addition to Riot’s official Cloud Services provider, AWS is Riot’s official Cloud Artificial Intelligence (AI), Cloud Machine Learning (ML) and Cloud Deep Learning (DL) provider.

As Riot charts the company’s path forward and continues to explore new ways to serve players, including TV shows, music and esports broadcasts, Rich encourages his team to adopt a cloud-first mindset. “When people run into limits, their first instinct used to be ‘I can do it better.’ Now it’s ‘let’s talk to AWS’ and see what they can do for us.”

Let’s walk through the history of Riot’s work with AWS and how Riot’s step-by-step approach to adopting the cloud won over even the biggest skeptics within the company.

Where things began

“Back around 2015, Riot was a rocket ship that wouldn’t slow down,” Rich said. “League of Legends (LoL) had exploded. Everything Riot did erred on the side of performance and player experience.”

In the years between 2015 and 2018, Riot focused on making LoL a game that would continue to enthrall players, updating the game every two weeks. Technology in Riot’s data centers was now about a decade old, and although the company had deployed lifecycle upgrades and virtualized an older tech stack of software services with AWS, it remained tethered to on-premises infrastructure.

As 2019 came into focus, Riot was looking toward the launch of a standalone mobile game Teamfight Tactics (TFT) and its next major release in 2020, VALORANT. The original plan for VALORANT was to have 40 global data center locations to service players worldwide. A low-latency solution was absolutely critical for VALORANT’s success. Early on in the game’s development, it was identified that eliminating peeker’s advantage (the slight advantage a player may get due in part to latency discrepancies and the server registering their actions) was going to be a key value add for players.

“At that time the belief was that metal was the way to go for performance reasons, but there’s so much complexity in owning and operating data centers, even without bringing automation into the equation. We wanted to figure out how we could get the same level of performance out of the cloud,” Rich said.

Riot Games’ David Press, Senior Principal Software Engineer for League of Legends and Tech Lead, noted, “We needed a lot more flexibility to spin capacity up or down. In the on-premises world we needed months of planning, which forces projects into a waterfall mentality. We wanted to be much more agile.”

The company began exploring the idea of the cloud as an extension of the data center, aiming to simplify operations and gain efficiency that would allow it to iterate quicker and automate load testing. With this goal, Rich and his team committed to collaborating with AWS and set a plan in motion.

“AWS was and is the premier player in the cloud space, and having worked with them at that point for several years, we had experienced their customer obsession first-hand and felt confident that they would be a great strategic partner,” Rich said.

With aggressive latency goals for VALORANT, Riot worked with the Amazon Elastic Kubernetes Service (Amazon EKS) team on a roadmap that would deliver the features, support, and experience Riot and its players would require.

An evolutionary approach

Riot started to move its game development approach to the cloud with TFT in June 2019, which, Rich says, “was born in AWS.” But VALORANT was the big test. The Riot team decided on 18 global deployments for VALORANT’s launch, with 14 on AWS and four on Riot’s data centers. Early in 2020, the company started a closed beta that was nearly at full launch scale, with close to three million players per day between April and May.

“From March forward everything relied on the cloud,” Rich said. “We erred on the side of hyperscaling on the cloud.”

The full public launch of VALORANT was cloud-based on day one, and it quickly became another multi-billion-dollar franchise for Riot. It has since launched a number of smaller games in the cloud as well. After those successful launches, the company made the call to begin migrating the rest of its servers to AWS.

Getting internal buy-in

Rich explained that taking the approach of gradually proving the concept was what earned the trust of C-level executives who were initially wary. “We had to demonstrate that we could make new things work in the cloud. The biggest proof point was that User Datagram Protocol (UDP) latency and packet loss was at an acceptable level. When you lose packets, they’re gone and that gap while playing a game makes it look like you’re teleporting in position.”

Rich started the project by asking his team for all the reasons they believed the cloud was not going to work. He then set out to confirm or disprove each point, one by one. “Their reasons were valid, and we investigated and proved that we could get over each hurdle,” he said. “When we got TFT running a direct game experience in the cloud, it showed that there was no compute issue. Cloud quality was the same we were getting with data centers.”

Rich’s step-by-step approach was supported by Riot CTO Derek DeFields, who redirected those who proposed Riot continue to build new data centers. “Not everyone was on board and wanted to buy gear as backup. We didn’t make any declarations of being ‘all-in’ when we put TFT and even VALORANT on AWS. But our relationship with AWS and how they worked with us went a long, long way.”

Press said, “When we were on premises, any time we had a hardware failure we had a 90-minute outage. When we moved to AWS and using RDS, hardware failures can still happen, but now it’s a 30-second outage.”

Rich said that he knew that the project had succeeded when the lead engineering team in charge of migrating legacy games to AWS asked to pick up the reins from Rich and his team. “It took us two years of work to get them to the point of picking up the project. We checked off every item on that list.”

A new mentality

When asked what moving to the cloud opened up for Riot, Rich reframed the thought. “It’s more about what it closes off,” he said. “It’s a whole chapter. Everything born in the data center is almost gone. The data center mindset has changed now that we’ve shown cloud is viable for our goals,” he said.

Riot has regular feature planning sessions with the Amazon EKS team to continue developing tools and new features to support the company’s many projects. “There are some things that are very hard to do without a strategic partner. Working with AWS and our integration partner, Slalom, for example, we now have a shared automation runbook for LoL that is incredibly detailed. If we have to stand something up in a new region, we can now do that in a matter of weeks. Those partnerships are invaluable.”