Electronic Arts streamlines game patching with AWS

This post was co-authored by Bryan Chiu, Site Reliability Engineer, EA.

Electronic Arts (EA) engages players around the globe in immersive games across PC, console, and mobile. Known for popular game franchises including EA Sports FC, Madden NFL, Battlefield, and Apex Legends, EA regularly releases new game versions to mitigate issues, make improvements, and add new in-game content. Quickly deploying these patches to player devices is crucial for providing a positive player experience.

This blog post discusses how EA has streamlined its game patching process for players using AWS services. By leveraging Amazon Elastic Kubernetes Service (Amazon EKS) and Amazon Elastic File System (Amazon EFS), EA delivers light and fast game patches, reducing patch installation times for their players.

The challenge: Inefficient game patching process

Every game update requires players to download and install a large file on their machines, from tens to hundreds of gigabytes. Game patch computations previously took place on the client side and required several minutes to complete. The algorithm only compared files with the same name as the updated version, missing cases where data had moved between files. This resulted in inflated patch sizes, as any new or changed data had to be downloaded. Additionally, the block size for comparing data was set to a high value of 64KB on the client-side to minimize metadata, leading to redownloading of data that could have been copied from the local version. These issues yielded unnecessarily large patch sizes, which led to longer download times.

The solution: Known Version Patching (KVP) on AWS

To address these challenges, EA implemented a new game patch deployment approach in August 2023 called Known Version Patching (KVP). KVP uses the knowledge of the installed version and language of the game to generate smaller patches with only the changed content. By moving the patch computation to the server side, KVP eliminates the time spent computing patches on the players’ machines and enables players to play the latest versions of their games more quickly. The solution downloads and unpacks the game on AWS, generates a hash of the game version, and creates a patch by comparing the hashes between two game versions. This automated process takes less than an hour to complete for a AAA title, outputting pre-computed, incremental patches for previous versions of the game.

Figure 1: EA KVP architecture diagram.

As shown in the KVP architecture diagram above, the solution uses an always-on coordinator microservice to manage the patch computation pipeline and notify downstream EA systems when a new patch is available. Upon receiving a new patch notification from an Amazon Managed Streaming for Apache Kafka (Amazon MSK) stream, the KVP coordinator uses the Kubernetes API to trigger additional EKS jobs in parallel to download and unpack each game version onto Amazon EFS, then compute hashes of each build. KVP creates metadata of the game builds and uses the metadata to compute the differences between the latest and all prior versions. The coordinator microservice then submits prepare jobs to the Kubernetes cluster, gathers job completion status, and when successful, submits the plan jobs. Upon completion of the pipeline, the computed patches are uploaded to the content delivery network for players to download through the EA app.

Parallelization with EKS

Figure 2: Patch pipeline flow diagram.

The patch pipeline flow diagram shows the Kubernetes jobs in the KVP patch pipeline and demonstrates the parallelized workflow of the pipeline. For new titles initially onboarding to the KVP solution, the pipeline must download and unpack all game build versions onto Amazon EFS to generate the incremental patches. For subsequent patches of existing game releases, KVP reuses the already downloaded and unpacked game builds in Amazon EFS, removing the need to redownload each game version and recompute the hash for every new patch.

To optimize performance and scalability, the solution uses Kubernetes requests and limits in combination with static CPU affinities and the cluster autoscaler. This approach enables EA to maximize throughput per node and dynamically scale resources up or down as needed. By adopting this parallelized, Kubernetes-based solution on EKS, EA can efficiently compute game patches, reducing the time and resources required compared to previous methods.

Shared file system for containers with EFS

When the next game patch is released, the previous patch becomes one of the source versions that needs to be cached for future patch computations. Amazon EFS and Amazon EKS add-ons allow EA to easily manage volumes and infrastructure within the Amazon EKS cluster using the Amazon EFS Container Storage Interface (CSI) driver. By configuring Amazon EFS with elastic throughput for automatically scaling throughput performance based on the workload activity, EA caches previous game builds and saves 10-15 minutes per patch computation while minimizing storage costs when the patch pipeline is not actively using the volumes. Optimizing the caching mechanism using Amazon EFS improved the overall performance and scalability of EA’s game patching process.

Alternative considerations for storage solutions included Amazon Simple Storage Service (Amazon S3), Amazon Elastic Block Store (Amazon EBS) and Amazon FSx. However, the data access patterns for the game patching pipeline were not frequent or repeated, making Amazon S3 a poor fit. Amazon EBS was also not suitable as it did not allow data sharing between pods and nodes, which would have complicated the coordination of the pipeline. Additionally, the EBS storage solution did not scale down when the patch computation workload decreased. Lastly, Amazon EFS provided better price to performance than Amazon FSx while meeting the system requirements for the solution.

The Amazon EFS solution successfully computes all patches within the desired timeframe and yields a net performance improvement while minimizing costs despite its lower read/write speeds in comparison to Amazon FSx. During a game patching pipeline, the underlying pods read data and write their results to the same disk, which limits the number of simultaneous patch computations, limiting the number of total concurrent patch computation jobs. As game files continue to grow over time and throughput starts to become a bottleneck, EA has the option to upgrade to Amazon FSx as the storage solution to further parallelize and expedite the patch pipeline.

Key benefits

• Faster patch computation: By moving the patch computation to EA’s servers, the time spent computing patches on the players’ machines was removed.
• Reduced patch size: KVP looks at all files in the installed version, including those with the same name, allowing it to reuse more data and reducing patch sizes by up to 80%.
• Improved player experience: With smaller, faster patches, players can get back into the game more quickly with up to 3.6 times the speed to complete the download and installation, improving the overall experience.
• Cost optimization: The use of AWS services, such as Amazon EKS and Amazon EFS, allowed EA to scale resources efficiently and reduce content delivery costs.

Conclusion

EA’s KVP solution supports the company’s mission to improve player experience. Players using the EA app can now access the latest game versions faster than ever before. The KVP approach also expedites EA’s game development cycles by reducing download times for testing new versions. The flexibility and scalability of Amazon EKS in combination with Amazon EFS has enabled EA to deliver a better gaming experience while also optimizing costs.

To learn more about how AWS customers build, deploy, and operate games on AWS, visit the AWS for Games Blog.