Launching Ghostbusters: Rise of the Ghost Lord – How nDreams scale tested with AWS Fargate

This blog was co-authored by Tim Edmunds, Technical Director, nDreams; Jamie Holding, Senior Principal Programmer, nDreams; and Nick Robinson, Technical Director, Near Light.

Leading up to the launch of the multiplayer VR game Ghostbusters: Rise of the Ghost Lord, the talented team at nDreams knew they needed to put the game through its paces at scale. With Amazon GameLift providing the reliable and scalable game server hosting and matchmaking, nDreams set out to create a new serverless scale testing framework using AWS Fargate. Their goal? Ensuring a silky-smooth launch experience for players.

Jokingly reiterating the mantra that “a boring launch is a good launch,” the nDreams team worked in partnership with AWS to create a serverless test framework using AWS Fargate and make sure they reviewed every aspect of the game using the AWS Well-Architected Framework.

About nDreams

nDreams is one of the world’s biggest virtual reality game developer and publisher at the forefront of XR innovation. Founded by CEO Patrick O’Luanaigh in 2006, the company has grown from its base in Farnborough, UK, to over 250 people across a mix of hybrid and fully-remote studios.

Since 2013, nDreams has been focused entirely on virtual reality and—starting with early access to Oculus Rift and PlayStation VR prototypes—was among the first to step into this immersive new medium. With a decade of VR gaming specialism, nDreams has unparallelled experience and a multi-award-winning team across all major VR platforms.

About Amazon GameLift

Amazon GameLift is a managed service for deploying, operating, and scaling dedicated game servers for session-based multiplayer games. Built on the proven AWS computing environment, Amazon GameLift lets you scale high-performance game servers up and down to meet player demand. You pay only for the capacity you use, so you can get started whether you’re working on a new game idea or running a game with millions of players. Amazon GameLift FlexMatch makes it easier for game developers to use FlexMatch standalone from Amazon GameLift game servers. In standalone mode, game developers use FlexMatch to form player matches and then place them into any game server solution they want.

Preparing for launch

Leading up to the launch, nDreams wanted to ensure the best possible player experience from day one—and the team saw scale testing as an essential part of the preparations.

The scale testing process let nDreams load test individual components of their architecture and observe how those interconnected systems would behave at scale. It also provided a chance for nDreams to work closely with their AWS Enterprise Support Technical Account Manager (TAM) to ensure all applicable service limits and quotas were raised to the appropriate level.

Additionally, the nDreams team collaborated with their AWS account team to go through the AWS Well-Architected Pillars, as well as the extended Games Industry Lens, and created a report in the AWS Well-Architected Tool. This report was then used to surface and communicate trade-offs and launch preparedness from an architecture perspective with business and technical stakeholders.

Scale testing architecture

The following diagram illustrates the architecture used to perform the scale testing across two AWS Regions. It shows a combination of manual Quality Assurance (QA) and the automated Load Testing Tool.

Load testing tool

This in-house-developed custom scaling test tool was essential to the scale testing. Developed in .Net and containerised with Docker, the tool managed the build of the headless game test client image and deployed it to AWS Fargate. Scheduling and management of test runs and test configurations was also managed through this application.

AWS Fargate

The headless game test client used a stripped-down version of the game built using Unreal Engine. The client included scripted gameplay to skip any intro sequences, log in to the backend, join random missions, and walk around the game maps to simulate basic player behavior.

Amazon GameLift

The team used Amazon GameLift for hosting the game servers that all game clients connect to for updates on game state and coordination of game actions with other players in the same game session. A game server runs a version of the game code that does not render any game graphics, but contains core game logic and maps.

Amazon API Gateway

Amazon API Gateway was used to present an API for game clients to request login, matchmaking, and game session action, as well as to accept user data updates and gameplay data relevant for downstream analytics.

AWS Lambda

AWS Lambda functions exposed by the API Gateway implemented login functionality, user management, and requests for player matchmaking.

Amazon GameLift FlexMatch

Amazon GameLift FlexMatch is a customizable matchmaking service for multiplayer games. With FlexMatch, nDreams built a custom set of rules that defined multiplayer match composition and to select compatible players for each match.

Amazon ElastiCache for Redis

ElastiCache for Redis is a fast in-memory database. nDreams used Redis to store active player and game sessions. Session data did not need to be stored persistently, making ElastiCache for Redis an ideal solution for speed and scaling to player demand as needed.

Amazon CloudWatch

nDreams uses a Grafana setup that pulls data collected in Amazon CloudWatch Metrics to create snapshots that can be shared internally. Metrics include Redis memory usage (rightsizing), game server memory and CPU usage (rightsizing number of players per instance), matchmaking queue depth, and more.

Setting up a headless game client with Unreal Engine

To create a headless game client, nDreams built a Docker container of the game client running on Linux with some bespoke commands to let the game play by itself.

Bundling a complex video game into a Docker container can be challenging. There are many libraries required to start a video game that are usually not wanted in container images. nDreams chose to base their image on Ubuntu, which comes with all of the required dependencies.

# Ubuntu is setup for launching Unreal games without having to install 
# individual dependencies.
# Using a smaller base distro and install all dependencies individually will
# result in a smaller image and less risk of security vulnerabilities 
# being introduced.
FROM ubuntu:mantic

ARG PROJECT_NAME="Ghosts"

# Directory to copy our game files to
WORKDIR /work

# Temporarily use root
USER root
# Install any required libs etc.
RUN apt update
# update SSL certs so our websockets etc. connect successfully
RUN apt install -y ca-certificates libcurl4

# Copy over each bit separately so don't accidentally 
# invalidate the whole Docker layer.
# Copy in order of least likely to change
COPY --chown=1000:1000 ./Engine/ ./Engine/
# Copy over game content (see PROJECT_NAME arg)
COPY --chown=1000:1000 ./${PROJECT_NAME}/ ./${PROJECT_NAME}/
# Copy all *.sh files
COPY --chown=1000:1000 ./*.sh ./

# Revert to a non-root user
USER 1000:1000

# Set our headless bash script as our entrypoint. This will be executed when the container starts
ENTRYPOINT ["./Headless.sh"]

Dockerfile to create a headless Linux Unreal Engine game client

Unreal Engine has various headless modes available, allowing control over whether the client renders graphics, or plays sounds. To avoid requiring container hosts that can render graphics, and to keep resource usage as low as possible, the container start-up script starts the container in this headless state and overrides some game configurations required for the test.

#!/bin/bash
# call Ghosts.sh, passing through any arguments from cmdline

# if no arguments are passed, default to -autoconnect=quick_play -autoconnect_travel=90
if [ $# -eq 0 ]; then
    set -- -autoconnect=quick_play -autoconnect_travel=90
fi

# Start the game forcing no VR Headset, no graphics, no sound, unattended mode (no user prompts)
# We also override some build defaults to configure the backend service to use, and some log verbosity tweaks to get the data we need from our test
#  Note, this file is generated automatically by Unreal Engine
#  Change this to match your Project Name
./Ghosts.sh -nohmd -nullrhi -nosound -unattended -ExecCmds="ND.ServerBackend prod" -ini:Engine:[Core.Log]:global=Off -ini:Engine:[Core.Log]:LogAutoConnect=Verbose -LogCmds="LogStreamableManager off, LogEOS off" "$@"

Bash shell script to start the game client in a headless state

To automate game behavior, nDreams updated the game’s GameInstance::Init function in Unreal Engine to respond to parameters passed in from the command line. nDreams used the new -autoconnect parameter to tell the game client how to automate its behavior. If a value is detected, an Actor is spawned into the world that automates the game’s login flow and starts a game. Any system could query the global auto connect settings to change behavior if needed.

#if !UE_BUILD_SHIPPING
    FString AutoConnectCommand;
    if (FParse::Value(FCommandLine::Get(), TEXT("-autoconnect="), AutoConnectCommand))
    {
        AutoConnectMode = FName(AutoConnectCommand);
    }
#endif

Detecting parameters on the command line

Keeping the Game Client Container small

When running thousands of game clients, it’s important to keep the container as small as possible. This will reduce the time each container host needs to spend downloading large game images.

nDreams made two major optimizations to reduce the file size of the headless container.

1. Reduced the texture sizes on Linux
   • Update the Device Profiles for Linux to cook textures to a maximum of 128×128
   • The headless game client is not intended to be rendered, so reducing texture sizes will not impact the game’s logic
2. Skipped adding debug symbols to the container
   • Use .dockerignore to remove any debug symbols
   • For example: Ghosts/Binaries/Linux/Ghosts.debug

Learnings

The team ran multiple scale tests with progressively more clients and servers to build out the load and learn system behaviors along the way gradually. nDreams also collaborated with Epic Online Services, who provided key identity and user services, to make sure all integrations were optimized and make any required tweaks.

The scale testing also provided valuable insights for the nDreams team on how to customize GameLift scaling behaviors to fit their particular needs, such as tuning auto-scaling thresholds. The conclusion was that documentation gets you a long way, but being able to test real-life scenarios at scale provides insights that are otherwise difficult to attain.

The built-in randomness of some of the client behaviors, coupled with the scale of matchmaking requests, led the team to discover a number of matchmaking edge cases. These edge cases could have risked negatively impacting the player experience.

Ultimately, coordination across all stakeholders during the testing phase proved valuable to all parties. It meant that the game launch on Amazon GameLift was precisely as predictable and uneventful as everyone had hoped for.

Next steps

The nDreams team has done an incredible job leveraging AWS services to put their game through rigorous scale testing.

Looking ahead, nDreams is excited to explore exciting new integrations, such as an AI-driven headless game client to create a more realistic, dynamic NPC behavior. Containerizing their game servers is another focus, which will further streamline the packaging and deployment of their infrastructure.

The team is also keen to leverage the power and efficiency of AWS Graviton instances, powered by Arm-based processors, to drive even greater performance and cost savings.

These are just a few of the ways nDreams is looking to build on their already impressive game scale testing implementation. The team is excited to create more VR game experience with the power and scalability of AWS.