Building the RC Fortress: Protecting your payments against outages

At RevenueCat, we’re fully aware that we are a critical component in our clients’ infrastructure. Our platform’s dependability directly affects our clients’ reputation and customer experience. To combat the ever-present potential for service disruptions, we embarked on a project to bolster the resilience of our system. We call it “RC Fortress”. The aim is simple, yet ambitious: guarantee a seamless purchasing experience for end-users, even when our primary services are temporarily unavailable.

The catalyst

Every project begins with an idea; for us, the seed was sown when we faced a series of service disruptions. Some were due to our cloud providers, while others were internal system hiccups and, while we have always designed the SDK in a way it could tolerate connectivity issues (caching, retries, etc) the purchase flow had always required our services to be up and reachable to unlock entitlements. These unfortunate incidents brought home the stark reality of our situation: We needed to do more to safeguard ourselves, and by extension, our clients, from such disruptions. This realization was the catalyst for RC Fortress.

Designing the RC Fortress: Ensuring seamless service

We recently released a new feature in our SDK, Offline Entitlements, that does a great job at handling situations where our servers are not reachable. But there were still a couple of scenarios that weren’t covered, which RC Fortress now solves:

1. Apps using an older version of our SDK still had issues during RevenueCat API outages.
2. Apps with the SDK still needed to communicate with our servers to know which products should be shown in the paywall and which entitlements should be unlocked after the purchase was completed. If the end customer installed the app before the outage, that should have been fine because the information would be cached. But if they happened to install the app during an outage, the SDK wouldn’t be able to get this information.

Service disruptions happen; we must deal with them. Our goal was to create a version of RevenueCat that, while limited, preserves key functionality during service disruptions (even if those services are the App Store or Play Store). With RC Fortress, paywalls can still be shown, and purchases can be processed after disruptions are resolved — all without losing data. The system revolves around two primary sets of functionality:

1. Displaying the paywall: Under normal circumstances, RevenueCat defines the products presented in our clients’ app paywalls. If our service goes down, the paywall fails to load. But with RC Fortress, we circumvent this issue by returning the products to be displayed on the paywall from a cached file storage. This storage is updated daily and matches the data returned by our actual API.
2. Handling purchases: Typically, when an end-user completes a transaction, the app sends a request to RevenueCat. We verify the transaction, identify the entitlements unlocked by the purchase, and relay this information back to the app. With RC Fortress, we automatically unlock these entitlements for a limited period, even when our main service is down. We log the actual HTTP requests and “replay” them against our servers once the primary service has recovered, so there’s no data lost.

Navigating the current limitations and planning for the future

Like any complex system, RC Fortress has a few limitations. We believe in transparency and think it’s important for our clients to understand these limitations:

• Experiments and other Offering customization for specific customers are ignored while RC Fortress is enabled.
• While RC Fortress is enabled, we don’t send webhooks and events to third-party integrations. However, we store these and send them out when the logged requests are replayed after service restoration.
• RC Fortress doesn’t validate purchases with the app stores.
• At present, RC Fortress doesn’t fully support Stripe and Amazon.

We’re actively working on addressing these limitations and aim to resolve them in the near future.

Assessing the results: A resilient service

Despite these limitations, RC Fortress has brought about a significant improvement in the resilience of our services. Even during disruptions, we’ve maintained the continuity of the purchasing process for end-users. 

This protects our clients from the potential fallout of such disruptions and ensures their customers’ experiences remain seamless. 

Some examples:

• On March 29, about 450,000 purchases succeeded during a service disruption that affected one of our storage layers.
• On April 13, RC Fortress helped 13,000 people to use the app they had just purchased, despite Apple itself having issues processing new purchases.

The road ahead

Our journey to create RC Fortress has been enlightening, reinforcing the critical importance of proactive measures in ensuring service reliability.

In designing RC Fortress, we sought to make RevenueCat more resilient in the face of outages, preserving the seamless purchasing experience for end-users, and, by extension, safeguarding our customers’ reputations. We’re actively enhancing our system, incorporating lessons learned, and working diligently to address current limitations.

This is more than just a technical upgrade — it’s a testament to our dedication to serving our customers in the best possible way. Every stride in this long road of improvements brings us closer to our mission of helping developers make more money, while we handle the complexities of in-app subscriptions.

As we continue to innovate and refine RC Fortress, our commitment remains the same: to provide the most reliable, trustworthy services possible, and to uphold our promise of acting as an extension of your team, fully invested in your success. Our work on RC Fortress is one more step on this ongoing journey. With every stride, we grow stronger, and so do the developers we support.