[Part 4] Shopify App Store: Guide to revenue – Comparison of app versions

Let’s say you’ve got 150 new installs. We get a certain conversion to paying customers, and this conversion describes those 150 customers. 

At the same time, if we get 150 more new installs, then the conversion of these two groups will differ with a probability of 99%.

For example, in the first case, we got a conversion of 40%, and in the second, 46%. This interval between conversions is called the confidence interval. The confidence interval contains the real conversion.

An important property of the confidence interval is that it becomes smaller with the growth of the audience in the selection.

• If there are 5 people in your selection, then the confidence interval will be wide.
• If there are 10,000 people in the selection, then the confidence interval will be narrow.

The confidence interval is a convenient way to estimate conversion leaps. This range contains the real conversion value. The main parameter of the confidence interval is the level of confidence. A 95% confidence level means that there is a 95% probability that the true conversion value will be contained within the interval.

Assessing the real situation vs. the real situation itself

Let’s say that after the onboarding update, 150 people participated in the considered cohort, and we got an estimate of the real conversion. In fact, this estimate describes exactly these 150 users. We cannot approve and pass this estimated conversion on to the rest of the users who come to the application.

Most product managers assume they have found and know the actual conversion based on a selection of a specific cohort of people, but this is not the case. This is not a real conversion, but an estimated one. Having received the estimated conversion, we use the confidence interval to calculate in what range the real conversion is.

Therefore, if your conversion drops, and this drop is within the acceptable range, you should not panic. But, at the same time, it is important not to make a mistake and not check anything if there were few users in the cohort and because of which we got a wide range of confidence intervals.

How to calculate a confidence interval

To calculate the confidence interval, you need to know the:

1. Number of users = N
2. Conversion score = C
3. Confidence level = 1.96 (this ratio means a confidence level of 95%)

There are various levels of confidence. You can find more coefficients here.

How to interpret the resulting range

For example, we got the interval [1.72%; 2.27%] using a coefficient of 1.96 (95% confidence level). This means that the real conversion with a probability of 95% lies in the obtained interval, or in 95% of cases we will get a conversion in the interval [1.72%; 2.27%].

Use this Confidence interval calculator to simplify the task.

• Fill in the sample size (1,000) and percentage of responses (this is the percentage of conversion – let it be 27) (you don’t need to fill in the main entity).
• Click Calculate and get the result 2.75.
• Now we take our conversion of 27, add 2.75 (27 + 2.75), and get the upper threshold of the interval. Then we get the difference (27 – 2.75) and get the lower threshold of the interval.
• The interval in which the real conversion is [24.15%; 29.75%]

Let’s say we have 2 versions of the app:

Version A received 35% conversion in payment with 400 users.

Version B received 39% conversion in payment with 450 users.

In order to compare the versions of the app, we use an additional formula, where:

A = conversion of the app’s first version,

B = conversion of the second version,

C1 = the number of users in the cohort in version A.

C2 = the number of users in the cohort in version b.

If the resulting interval lies in 0, then the changes with the probability of 95% were random.

If the interval does not lie in 0, then we can assert with the probability of 95% that the changes were not accidental, but due to the app update.

For example, the formula returned the result as a range [from -1.66%; up to 2.34%]. The range lies through 0, therefore, with a probability of 95%. This change is accidental.

Confidence interval with specified precision

where:

a is a hypothetical conversion that should be, 

confidence interval is a confidence interval in the range of which the real conversion lies, 

C is the number of users you will need to achieve the conversion a.

What this formula is used for:

For example, you want to understand how much traffic you need to buy in order to get a hypothetical conversion. The formula will show you exactly how many users you need to acquire to get a range with real conversions.