Updating our Subscription Retention chart for better accuracy

What’s changed

Since we released the Subscription Retention chart in late 2021, it’s measured the portion of paid subscriptions that remained active in a given period after they started. Typically a subscription remains active after its initial period due to a paid renewal, but stores also support grace periods, which (when enabled) allow a subscription to remain active after its initial period as long as it has not been canceled if the store’s first attempt to renew that subscription failed. These grace periods allow the store to make additional attempts to renew the paid subscription before expiring it.

We recently discovered that the Subscription Retention chart was including grace periods for Play Store and Stripe subscriptions when calculating the retention of subscriptions for a given period, even if the subscription does not successfully convert to paid during that grace period, which differs from the expected behavior of other stores like the App Store where these grace periods do not influence retention. We’ve now made the calculation consistent across stores, effectively lowering subscription retention for some Play Store and Stripe subscriptions that expired due to the billing issues not being resolved.

In addition, you may also see lower counts of subscriptions started through Stripe, since this billing issue behavior would have additionally caused trials through Stripe that entered a grace period to be incorrectly marked as trial conversions (i.e. new paid subscriptions).

At the same time, we’ve audited our grace period handling behavior across all of our charts to ensure consistent handling across stores. More on this below.

Why we made the change

Though a subscription in a grace period in a given period is technically “retained” as active for part of that period due to their grace period, we don’t think that is useful information when measuring the retention rate of a cohort of subscriptions for a given period. Instead, what we think you as an app developer should be focused on is the number (or portion) of subscriptions who made a renewal payment to continue their subscription in future periods, and it’s important that our Subscription Retention chart allows you to accurately measure that.

We don’t take definition changes like this lightly, but in this case we believe the change far more closely aligns with the intention and use cases of the chart, and brings the behavior of Play Store and Stripe subscriptions into consistency with App Store subscriptions.

One of the primary benefits we hope you get out of RevenueCat is a reliable, normalized view of your subscription business performance; and we failed to provide that by missing this inconsistency in the Subscription Retention chart, which we regret and apologize for. We know what it’s like to operate subscription app businesses on a daily basis and try to make important decisions by estimating lifetime value, active subscriber base, and other metrics based on retention data; which is why it’s critically important to us that the data we’re providing is reliable and normalized.

How we’ll continue to improve

Beyond fixing inconsistencies that we discover, we’re also committed to continually improving the reliability and transparency of the data we provide going forward.

What we do today to ensure data reliability and transparency

Today, when any change is made to our charts, or other data features like Experiment Results and our Overview stats which rely on the same infrastructure, we generate a full data set based on the changes and run tests on it against production charts data and the underlying tables that the charts derive from to ensure the results are in line with our expectations.

Of course, as anyone whose written tests knows, there can always be cases you did not previously account for, and in this case the inconsistency between stores has existed in this chart since its inception. Since we’ve not had a test explicitly checking reported retention rates when grace periods are excluded, or checking grace period handling between stores, it’s gone undetected until now.

Regarding transparency, we’ve recently published a sample query for the Subscription Retention chart which can be accessed from our docs to reproduce the results of the Subscription Retention using raw data from our Scheduled Data Exports. Many customers use these exports to either fully understand the data RevenueCat is providing, or to build their own custom reporting on top of our base definitions.

In addition, we’ve completed a full audit of how grace periods are represented across our data features to ensure they are handled consistently, and made a handful of other changes as a result of that:

1. Stripe trials that enter a grace period but do not convert to paid successfully were previously considered new active (paid) subscriptions, and they’ve now been reclassified as trials.
2. Transaction counts in the Refund chart and Revenue chart have been updated to ensure grace period transactions are excluded from them, since they never generated any revenue to begin with.
3. A small number of Play Store transactions with an expiration date that’s not greater than their purchase date were included in paying customer counts for the LTV charts and Conversion to Paying chart, and they’ve now been excluded.

Please note that revenue, MRR, and related metrics are not affected by these changes, since grace period transactions never have any revenue associated with them.

What we’ll continue to do going forward to ensure data reliability and transparency

We’ve since added appropriate test cases for expected grace period behavior across all charts, and will continue working to improve data reliability and flexibility through investments like:

1. Monitoring for and addressing cases of non-normalized data across stores
2. Building a new underlying data infrastructure that can serve all RevenueCat data features and increase consistency across those surfaces
3. Increasing visibility into subscription statuses like grace periods or auto renew status so that you can better understand the granular state of your subscriber base

Conclusion

As always, if you have any questions about the data we’re providing or the definition change we made, we would be happy to answer them at support@revenuecat.com. In the meantime, thanks for trusting us as a critical provider in your subscription app business. We’ll keep working on our mission to help you make more money so that you can accomplish your mission.