How we test SDKs at RevenueCat

At RevenueCat, we provide critical infrastructure as a service. Our SDKs process millions of transactions and run on many types of devices running different operating systems.

If something goes wrong with a purchase, users get understandably frustrated, so we do absolutely everything in our power to ensure that our SDKs work seamlessly.

In this blog post, we’re going to take a look at how we test and update our iOS SDK to keep it robust. We have quite a few test suites – each of them serves a different purpose, and together they help protect us from shipping a buggy version.

Unit tests

‍What they do: Identify bugs as soon as they happen, pointing to very specific sections of the code.

When they run: Continuously while we’re developing. In CI, they also run in multiple OS versions and platforms for every commit.We use XCTest and Nimble for unit testing and our general philosophy for unit tests is that every unit of code should be tested independently. So for each class, we have a unit test class that checks all of the internal and public methods of the class. We don’t test private methods directly, since their logic should be testable through the interface of the class.

‍
If there’s logic in the private methods that’s too complex to test through the class interface, that’s a sign that the logic should be extracted into a new, separate class.

Within unit tests, we use snapshot testing for big, string-based data structures that are awkward to test otherwise. For example, to make sure we’re sending the correct JSON values to the backend, we make a snapshot of the baseline in a JSON file and compare against that.

Since we want these tests to call out bugs as soon as they happen, they need to run fast so we can run them all the time while we’re developing. Our current test suite, which includes 925 tests, takes 1-2 seconds to run on an M1 Pro. This allows us to quickly verify changes and easily clean up and refactor code, while remaining confident that we’re not breaking anything.

Our unit tests also run multiple times in CI, for iOS, tvOS, macOS, and watchOS — with versions ranging from iOS 12-15. This helps ensure that we don’t accidentally break an implementation on iOS 12 devices, for example.

StoreKit unit tests

What they do: Ensure that we’re using StoreKit correctly and provide a way to test scenarios that are otherwise impossible to test.

When they run: Continuously while we’re developing. In CI, they also run in multiple OS versions and platforms for every commit.

During WWDC 2020, Apple announced the StoreKit Test Framework. This was a huge deal for us — until StoreKit Test came out, there was no way to create automated tests that use StoreKit to make purchases, and we had to create mocks for all cases. And some features were entirely left out.

With StoreKit Test, we write new unit tests that use specific StoreKit configuration files to test features of the SDK. This is particularly important for StoreKit 2, since StoreKit 2’s types unfortunately cannot be mocked. But with StoreKit Test, we can use the real types in a test.

Backend integration tests

Unit tests pass, no integration tests pic.twitter.com/8geAsHgSBY

— Ryan Stortz (@withzombies) February 9, 2017

What they do: Ensure that the SDK works with the backend in a real app.

When they run: In CI, for every commit that isn’t from a fork.

These tests also use the StoreKit Test framework, but instead of mocking classes or even network calls, they test against a real RevenueCat backend server. This lets us verify that each purchase unlocks the correct entitlement and that renewing/expiring purchases are reflected correctly in the SDK.

While backend integration tests are easy to write and great at catching bugs, they can’t pinpoint errors as accurately as unit tests. For example, a backend integration test could tell us that an entitlement wasn’t unlocked after a purchase, but it wouldn’t tell us why. The corresponding unit test would point us to the specific code that isn’t working as expected (in this case, the logic that parses entitlements from a JSON HTTP response).

SDK installation tests

What they do: Ensure that our releases work on all package managers and can be installed successfully.

When they run: In CI, for release branches only.

While backend integration tests verify that our backend can be integrated, SDK installation tests check that the SDK can be installed into an app successfully.

Our SDK can be installed through:

• The Swift Package Manager
• CocoaPods
• Carthage (using XCFrameworks)
• Carthage (using regular frameworks)
• Directly adding the .xcodeproj file to your project
• Copy/pasting the XCFramework from our releases page
• Copy/pasting the framework from our releases page

Each of these integration paths works a little differently. The Swift Package Manager uses a Package.swift file to specify how to build the SDK, whereas CocoaPods uses a Podspec. Carthage uses the Xcode project and relies on a few specific build settings to get dSYMs included in the XCFramework.

If the folder structure is updated or a new feature is added that requires a specific build configuration, the SDK installation process can break. And it’s hard to catch when this happens, since during development, we don’t typically install the SDK over and over through all of those methods.

To address this, we built our SDK installation tests, which automatically install the SDK in a sample app through every installation method. After installing the SDK, they automatically build and run a simple app that makes a call to our SDK. If the call is successful and returns the expected output, we know the SDK can be installed through this path.

These tests don’t make extensive calls to the SDK because that’s not what they’re for (we check that with other test suites). Instead, they focus on building the SDK itself and installing it with every build system.

API tests

‍What they do: Prevent unwanted changes to our public API.

When they run: Continuously while we’re developing. In CI, they also run in multiple OS versions and platforms for every commit.

As SDK developers, we live and die by our public API. We want to make sure that we never make any unintended changes to it because doing so could accidentally break compilation for thousands of apps.

To ensure we don’t break our public API, we created our API testers: two small apps (one in Objective-C and one in Swift) that call every single method and use every public property in our SDK. If our public API changes, those API testers will fail to build, so we’ll know about it right away.

Having the API testers in place was absolutely vital when we migrated our codebase to Swift. We were able to confirm that, although the interface was written in a different language, it would still translate into the same public interface.

Docs reference tests

‍What they do: Ensure that our public types have appropriate and up-to-date docs.

When they run: Continuously while we’re developing. In CI, they also run in multiple OS versions and platforms for every commit.Our public docs are an important part of our system. We want our docs to stay up-to-date, as well as help you understand all the types and features that are related to each method. For example, the docs for the CustomerInfo type should point to you the methods for fetching CustomerInfo, and vice versa. Those two docs should also link you to EntitlementInfo, since you might use those types together.

To make sure we get the names right and provide links to related content, we use Xcode’s “Build documentation during Build” setting. This makes generating DocC docs part of the build process and helps us catch naming errors in our docs before they happen.

For example, if a method name is incorrect, it will generate a build warning in Xcode.

Testing app

‍What it does: Provide an easy way to test UI-based features.

When it runs: While developing and prototyping, and during ad-hoc testing.

We originally created our testing app to test things that StoreKit wouldn’t let us automate, like making purchases (before StoreKitTest was introduced). However, it’s still useful for us! We use it to test all of our features, but especially features that require a UI.

For example, our new APIs for opening up a subscriptions management page in an app or starting a refund request can only be tested with an actual UI.

Our testing app is also useful when we’re prototyping a new feature, since it lets us try it out just like our customers would — in a real app. We can also upload the app to TestFlight when we need to.

Our testing app supports 2 modes: regular Sandbox and StoreKit Configuration Files. Together, they allow us to test our SDK’s behavior realistically and for almost every possible use case.

The clear disadvantage of the testing app is that running tests with it is a largely manual process — it can’t be automated as easily as some of our other test suites. But being able to test with a UI is really valuable for some scenarios.

Testing parties

‍What they’re for: Making sure we have a top-notch user experience.

When we have them: Before releasing big or impactful features.

Before we release any big feature, we host one or more testing parties — we get lots of team members together to test out the feature and provide feedback. We ask people from every part of the company to participate, not just the SDK team. This helps us find our blindspots, especially when it comes to the UX and documentation.

The feedback we receive during testing parties can be anything from “this method’s name doesn’t feel intuitive” to “the docs for this feature are confusing” — sometimes even catching bugs that we missed.

Testing parties are incredibly helpful for us. We almost always come out of them with a big to-do list of things we can improve before releasing the feature.

Real apps from team members

‍What they’re for: Making sure everything works and the experience is great in a real-life use case.

When we use them: Before releasing big or impactful features.

A lot of RevenueCat employees are also RevenueCat customers — indie developers who’ve built real apps that are published on the App Store and are making money. This is fantastic for us because it means we can get feedback from customers as we build.

When we’re almost ready to launch a new feature, we ask employees to test it out in their own apps and give us feedback — which gives us a first view of what things look like in production.

Real apps from customers who opt in to betas

‍What we use them for: Gathering feedback from customers and finding ways to improve our SDK.

When we test with them: Before releasing big or impactful features.

I’m consistently humbled by the amazing developers who use RevenueCat and their willingness to help us improve our SDK.

When we let customers know we’re working on feature requests they’ve submitted, they frequently offer to test out the betas for us. Similarly, when we reach out to our developer community and ask if anyone is interested in testing out new features (like we did for v4 of our iOS SDK), we always get a fantastic response.

The way forward

At RevenueCat, we take the responsibility of building critical infrastructure for our customers very seriously. We go above and beyond to ensure that our SDKs are as robust and battle-tested as possible and that the user experience is as seamless as it can be.

While setting up all these test suites took a lot of work, it’s really paid off. Now most of our testing is automated, so we get to spend most of our time focusing on developing new features instead of worrying about testing. We’re able to ship big features quickly and remain confident that our SDK will work for all apps — regardless of what OS version or platform they’re running on.

Of course, we’re always open to suggestions on how to improve our test suites! All of our SDK repos are open-source and we ❤️ our external contributors.

‍

We’re hiring at RevenueCat engineering! Click here to check out the open roles that we have available at the moment.