Insecure by Design, Epic Games Peer-to-Peer Multiplayer Service

The opinions expressed in this publication are those of the authors. They do not reflect the opinions or views of my employer. All research was conducted independently.

As someone who enjoys games that involve logistics, a recently-released game caught my eye. That game was Satisfactory, which is all about building up a massive factory on an alien planet from nothing. Since I loved my time with Factorio, another logistics-oriented game, Satisfactory seemed like a perfect fit!

Satisfactory had a unique feature that I rarely see in games today, peer-to-peer multiplayer sessions! Generally speaking, most games take the client-server approach of having dedicated servers serving multiple clients instead of the peer-to-peer model where clients serve each other.

I was curious to see how it worked under the hood. As a security researcher, one of my number one priorities is to always "stay curious" with anything I interact with. This was a perfect opportunity to exercise that curiosity!

Introduction

When analyzing the communication of any application, a great place to start is to attempt to intercept the HTTP traffic. Often times applications won't use an overly complicated protocol if they don't need to and web requests happen to be one of the most common ways of communicating. For intercepting not only plaintext traffic, I used Fiddler by Telerik which features a simple-to-use interface for intercepting HTTP(S) traffic. Fiddler took care of installing the root certificate it would use to issue certificates for websites automatically, but the question was if Satisfactory used a platform that had certificate pinning. Only one way to find out!

Lucky for us, the game did not have any certificate pinning mechanism. The first two requests appeared to query the latest game news from Coffee Stain Games, the creators of Satisfactory. I was sad to see that these requests were performed over HTTP, but fortunately did not contain sensitive information.

Taking a look at the other requests revealed that Satisfactory was authenticating with Epic Games. Perhaps they were using an Epic Games service for their multiplayer platform?

In the first request to Epic Games' API servers, there are a few interesting headers that indicate what service this could be in connection to. Specifically the User-Agent and Miscellaneous headers referred to something called "EOS". A quick Google search revealed that this stood for Epic Online Services!

Discovery

Epic Online Services is intended to assist game developers in creating multiplayer-compatible games by offering a platform that supports many common multiplayer features. Whether it be matchmaking, lobbies, peer-to-peer functionality, etc; there are many attractive features the service offers to game developers.

Before we continue with any security review, it's important to have a basic conceptual understanding of the product you're reviewing. This context can be especially helpful in later stages when trying to understand the inner workings of the product.

First, let's take a look at how you authenticate with the Epic Online Services API (EOS-API). According to documentation, the EOS Client uses OAuth Client Credentials to authenticate with the EOS-API. Assuming you have already set up a project in the EOS Developer Portal, you can generate credentials for either the GameServer or GameClient role , which are expected to be hardcoded into the server/client binary:

The EOS SDK requires a program to provide valid Client Credentials, including a Client ID and Client Secret. This information is passed in the the EOS_Platform_ClientCredentials structure, which is a parameter in the function EOS_Platform_Create. This enables the SDK to recognize the program as a valid EOS Client, authorize the Client with EOS, and grant access to the features enabled in the corresponding Client Role.

I found it peculiar that the only "client roles" that existed were GameServer and GameClient. The GameClient role "can access EOS information, but typically can't modify it", whereas the GameServer role "is a server or secure backend intended for administrative purposes". If you're writing a peer-to-peer game, what role do you give clients?

GameClient credentials won't work for hosting sessions, given that it's meant for read-only access, but GameServer credentials are only meant for "a server or secure backend". A peer-to-peer client is neither a server nor anything I'd call "secure", but Epic Games effectively forces developers to embed GameServer credentials, because otherwise how can peer-to-peer clients host sessions?

The real danger here is that the documentation falsely assumes that if a client has GameServer role, it should be trusted, when in fact the client may be an untrusted P2P client. I was amused by the fact that Epic Games even gives an example of the problem with giving untrusted clients the GameServer role:

The Client is a server or secure backend intended for administrative purposes. This type of Client can directly modify information on the EOS backend. For example, an EOS Client with a role of GameServer could unlock Achievements for a player.

Going back to those requests we intercepted earlier, the client credentials are pretty easy to extract.

In the authentication request above, the client credentials are simply embedded as a Base64-encoded string in the Authorization header. Decoding this string provides a username:password combination, which represents the client credentials. With such little effort, we are able to obtain credentials for the GameServer role giving significant access to the backend used for Satisfactory. We'll take a look at what we can do with GameServer credentials in a later section.

Since we're interested in peer-to-peer sessions/matchmaking, the next place to look is the "Sessions interface", which "gives players the ability to host, find, and interact with online gaming sessions". This Sessions interface can be obtained through the Platform interface function EOS_Platform_GetSessionsInterface. The core components of session creation that are high value targets include session creation, session discovery, and joining a session. Problems in these processes could have significant security impact.

Discovery: Session Creation

The first process to look at is session creation. Creating a session with the Sessions interface is relatively simple.

First, you create a EOS_Sessions_CreateSessionModificationOptions structure that contains the following information:

Finally, you need to pass this structure to the EOS_Sessions_CreateSessionModification function to create the session. Although this function will end up creating your session, there is a significant amount you can configure about a session given that the initial structure passed only contains required information to create a barebone session.

Discovery: Finding Sessions

For example, let's talk about how matchmaking works with these sessions. A major component of Sessions is the ability to add "custom attributes":

Sessions can contain user-defined data, called attributes. Each attribute has a name, which acts as a string key, a value, an enumerated variable identifying the value's type, and a visibility setting.

These "attributes" are what allows developers to add custom information about a session, such as the map the session is for or what version the host is running at.

EOS makes session searching simple. Similar to session creation, to create a barebone "session search", you simply call EOS_Sessions_CreateSessionSearch with a EOS_Sessions_CreateSessionSearchOptions structure. This structure has the minimum amount of information needed to perform a search, containing only the maximum number of search results to return.

Before performing a search, you can update the session search object to filter for specific properties. EOS allows you to search for session based on:

A known unique session identifier (at least 16 characters in length).
A known unique user identifier.
Session Attributes.

Although you can use a session identifier or a user identifier if you're joining a known specific session, for matchmaking purposes, attributes are the only way to "discover" sessions based on user-defined data.

Discovery: Sensitive Information Disclosure

Satisfactory isn't a matchmaking game and you'd assume they'd use the unique session identifier provided by the EOS-API. Unfortunately, until a month ago, EOS did not allow you to set a custom session identifier. Instead they forced game developers to use their randomly generated 32 character session ID.

When hosting a session in Satisfactory, hosts are given the option to set a custom session identifier. When joining a multiplayer session, besides joining via a friend, Satisfactory gives the option of using this session identifier to join sessions. How does this work in the background?

Although the EOS-API assigns a random session identifier to every newly created session, many implementations ignore it and choose to use attributes to store a session identifier. Being honest, who wants to share a random 32 character string with friends?

I decided the best way to figure out how Satisfactory handled unique session identifiers was to see what happens on the network when I attempt to join a custom session ID.

Here is the request performed when I attempted to join a session with the identifier "test123":

POST https://api.epicgames.dev/matchmaking/v1/[deployment id]/filter HTTP/1.1
Host: api.epicgames.dev
...headers removed...

{
  "criteria": [
    {
      "key": "attributes.NUMPUBLICCONNECTIONS_l",
      "op": "GREATER_THAN_OR_EQUAL",
      "value": 1
    },
    {
      "key": "bucket",
      "op": "EQUAL",
      "value": "Satisfactory_1.0.0"
    },
    {
      "key": "attributes.FOSS=SES_CSS_SESSIONID_s",
      "op": "EQUAL",
      "value": "test123"
    }
  ],
  "maxResults": 2
}

Of course, this returned no valid sessions, but this request interestingly revealed that the mechanism used for finding sessions was based on a set of criteria. I was curious, how much flexibility did the EOS-API give the client when it comes to this criteria?

Exploitation: Sensitive Information Disclosure

The first idea I had when I saw that filtering was based on an array of "criteria" was what happens when you specify no criteria? To my surprise, the EOS-API was quite accommodating:

Although sessions in Satisfactory are advertised to be "Friends-Only", I was able to enumerate all sessions that weren't set to "Private". Along with each session, I am given the IP Address for the host and their user identifier. On a large-scale basis, an attacker could easily use this information to create a map of most players.

To be clear, this isn't just a Satisfactory issue. You can enumerate the sessions of any game you have at least GameClient credentials for. Obviously in a peer-to-peer model, other players have the potential to learn your IP Address, but the problem here is that it is very simple to enumerate the thousands of sessions active given that besides the initial authentication, there are literally no access controls (not even rate-limiting). Furthermore, to connect to another client through the EOS-API, you don't even need to have the IP Address of the host!

Discovery: Session Hijacking

Going back to what we're really interested in, the peer-to-peer functionality of EOS, I was curious to learn what connecting to other clients actually looks like and what problems might exist with its design. Reading the "Sending and Receiving Data Through P2P Connections" section of the NAT P2P Interface documentation reveals that to connect to another player, we need their:

Product user ID - This "identifies player data for an EOS user within the scope of a single product".
Optional socket ID - This is a unique identifier for the specific socket to connect to. In common implementations of EOS, this is typically the randomly generated 32 character "session identifier".

Now that we know what data we need, the next step is understanding how the game itself shares these details between players.

One of the key features of the EOS Matchmaking/Session API we took a look at in a previous section is the existence of "Attributes", described by documentation to be a critical part of session discovery:

The most robust way to find sessions is to search based on a set of search parameters, which act as filters. Some parameters could be exposed to the user, such as enabling the user to select a certain game type or map, while others might be hidden, such as using the player's estimated skill level to find matches with appropriate opponents.

For peer-to-peer sessions, attributes are even more important, because they are the only way of carrying information about a session to other players. For a player to join another's peer-to-peer session, they need to retrieve the host's product user ID and an optional socket ID. Most implementations of Epic Online Services store this product user ID in the session's attributes. Of course, only clients with the GameServer role are allowed to create sessions and/or modify its attributes.

Exploitation: Session Hijacking

Recalling the first section, the core vulnerability and fundamental design flaw with EOS is that P2P games are required to embed GameServer credentials into their application. This means that theoretically speaking, an attacker can create a fake session with any attribute values they'd like. This got me thinking: if attributes are the primary way clients find sessions to join, then with GameServer credentials we can effectively "duplicate" existing sessions and potentially hijack the session clients find when searching for the original session.

Sound confusing? It is! Let's talk through a real-world example.

One widely used implementation of Epic Online Services is the "OnlineSubsystemEOS" (EOS-OSS) included with the Unreal Engine. This plugin is a very popular implementation widely used by games such as Satisfactory.

In Satisfactory's use of EOS-OSS, they use an attribute named SES_CSS_SESSIONID to track sessions. For example, if a player wanted their friend to join directly, they could give their friend a session ID from their client which the friend would be able to use to join. When the session ID search is executed, all that's happening is a filter query against all sessions for that session ID attribute value. Once the session has been found, EOS-OSS joins the session by retrieving the required product user ID of the host through another session attribute named OWNINGPRODUCTID.

Since Satisfactory is a peer-to-peer game exclusively using the Epic Online Services API, an attacker can use the credentials embedded in the binary to get access to the GameServer role. With a GameServer token, an attacker can hijack existing sessions by creating several "duplicate" sessions that have the same session ID attribute, however, have the OWNINGPRODUCTID attribute set to their own product user ID.

When a victim executes a search for the session with the right session ID, more likely than not, the query will return one of the duplicated sessions that has the attacker's product user ID (ordering of sessions is random). Thus, when the victim attempts to join the game, they will join the attacker's game instead!

This attack is quite more severe than it may seem, because it is trivial to script this attack to hijack all sessions at once and cause all players joining any session to join the attacker's session instead. To summarize, this fundamental design flaw allows attackers to:

Hijack any or all "matchmaking sessions" and have any player that joins a session join an attacker's session instead.
Prevent players from joining any "matchmaking session".

Remediation

When it comes to communication, Epic Games was one of the best vendors I have ever worked with. Epic Games consistently responded promptly, showed a high level of respect for my time, and was willing to extensively discuss the vulnerability. I can confidently say that the security team there is very competent and that I have no doubt of their skill. When it comes to actually "getting things done" and fixing these vulnerabilities, the response was questionable.

To my knowledge:

Epic Games has partially* remediated the sensitive information disclosure vulnerability, opting to use a custom addressing format for peer-to-peer hosts instead of exposing their real IP Address in the ADDRESS session attribute.
Epic Games has not remediated the session hijacking issue, however, they have released a new "Client Policy" mechanism, including a Peer2Peer policy intended for "untrusted client applications that want to host multiplayer matches". In practice, the only impact to the vulnerability is that there is an extra layer of authentication to perform it, requiring that the attacker be a user of the game (in contrast to only having client credentials). Epic Games has noted that they plan to "release additional tools to developers in future updates to the SDK, allowing them to further secure their products".

* If a new session does not set its own ADDRESS attribute, the EOS-API will automatically set their public IP Address as the ADDRESS attribute.

Instead of fixing the severe design flaw, Epic Games has opted to update their documentation with several warnings about how to use EOS correctly and add trivial obstacles for an attacker (i.e the user authentication required for session hijacking). Regardless of their warnings, the vulnerability is still in full effect. Here are a list of various notices Epic Games has placed in their documentation:

In the documentation for the Session interface, Epic Games has placed several warnings all stating that attackers can leak the IP Address of a peer-to-peer host or server.
Epic Games has created a brand new Matchmaking Security page which is a sub-page of the Session interface outlining:

Attackers may be able to access the IP Address and product user ID of a session host.
Attackers may be able to access any custom attributes set by game developers for a session.
Some "best practices" such as only expose information you need to expose.

Epic Games has transitioned to a new Client Policy model for authentication, including policies for untrusted clients.

Can this really be fixed?

I'd like to give Epic Games every chance to explain themselves with these questionable remediation choices. Here is the statement they made after reviewing this article.

Regarding P2P matchmaking, the issues you’ve outlined are faced by nearly all service providers in the gaming industry - at its core P2P is inherently insecure. The EOS matchmaking implementation follows industry standard practices. It's ultimately up to developers to implement appropriate checks to best suit their product. As previously mentioned, we are always working to improve the security around our P2P policy and all of our other matchmaking policies.

Unfortunately, I have a lot of problems with this statement:

Epic Games erroneously included the IP Address of peers in the session attributes, even though this information was never required in the first place. Exposing PII unnecessarily is certainly not an industry standard. At the end of the day, an attacker can likely figure out the IP of a client if they have their product user ID and socket ID, but Epic Games made it trivial for an attacker to enumerate and map an entire games' matchmaking system by including that unnecessary info.
"The EOS matchmaking implementation follows industry standard practices": The only access control present is a single layer of authentication...
Epic Games has yet to implement common mitigations such as rate-limiting.
There are an insane number of ways to insecurely implement Epic Online Services and Epic Games has documented none of these cases or how to defend against them. A great example is when games like Satisfactory store sensitive information like a session ID in the session's attributes.
The claim that the Session Hijacking issue cannot be remediated because "P2P is inherently insecure" is simply not true. It's not like you can't trust the central EOS-API server. Here is what Epic could do to remediate the Session Hijacking vulnerability:

A large requirement about the session hijacking attack is that an attacker must create multiple duplicate sessions that match the given attributes. An effective remediation is to limit the number of concurrent sessions a single user can have at once.
To prevent attackers from using multiple accounts to hijack a session to a singular session, Epic Games could enforce that the product user ID stored in the session must match the product user ID of the token the client is authenticating with.
To be clear, if Epic Games applied my suggested path of remediation, an attacker could still buy 10-20 copies of the game on different accounts to perform the attack on a small-scale. A large-scale attack would require thousands of copies to create enough fake hijacking sessions.
The point with this path of remediation is that it would significantly reduce the exploitability of this vulnerability, much more than a single layer of authentication provides.

Epic Games has yet to comment on these suggestions.

But... why?

When I first came across the design flaw that peer-to-peer games are required to embed GameServer credentials, I was curious to how such a fundamental flaw could have gotten past the initial design stage and decided to investigate.

After comparing the EOS landing page from its initial release in 2019 to the current page, it turns out that peer-to-peer functionality was not in the original release of Epic Online Services, but the GameClient and GameServer authentication model was!

It appears as though Epic Games simply didn't consider adding a new role dedicated for peer-to-peer hosts when designing the peer-to-peer functionality of the EOS-API.

Timeline

The timeline for this vulnerability is the following:

07/23/2020 - The vulnerability is reported to Epic Games.
07/28/2020 - The vulnerability is reproduced by HackerOne's triage team.
08/11/2020 - The vulnerability is officially triaged by Epic Games.
08/20/2020 - A disclosure timeline of 135 days is negotiated (for release in December 2020).
11/08/2020 - New impact/attack scenarios that are a result of this vulnerability are shared with Epic Games.
12/02/2020 - The vulnerability severity is escalated to Critical.
12/03/2020 - This publication is shared with Epic Games for review.
12/17/2020 - This publication is released to the public.

Frequently Asked Questions

Am I vulnerable?
If your game uses Epic Online Services, specifically its peer-to-peer or session/matchmaking functionality, you are likely vulnerable to some of the issues discussed in this article.

Epic Games has not remediated several issues discussed; opting to provide warnings in documentation instead of fixing the underlying problems in their platform.

What do I do if I am vulnerable?
If you're a player of the game, reach out to the game developers.

If you're a game or library developer, reach out to Epic Games for assistance. Make sure to review the documentation warnings Epic Games has added due to this vulnerability.

What is the potential impact of these vulnerabilities?
The vulnerabilities discussed in this article could be used to:

Hijack any or all "matchmaking sessions" and have any player that joins a session join an attacker's session instead.
Prevent players from joining any "matchmaking session".
Leak the user identifier and IP Address of any player hosting a session.