Experiment Readout: Google OAuth Launch

March 18, 2024

john@thorntale.com

Show Source Notes

Parameters

Google OAuth Launch

Dori Wilson

Agenda

TL;DR

Experiment Metrics and Results

Next Steps

TL;DR

•

Google auth access increased new user signups 15%

•

On average google auth signups churn off platform 2 days faster than other signup types

•

New signup method rolled out to 100% of traffic

Experiment Setup

Treatment

Control

Experiment Details

Dates:

January 1st 2024 - Feb 1st 2024

Context:

We've had flat signups from June 2024-Dec 2023, despite having increasing number of website visitors. We want to increase the conversion of website visitor to new signup. Potential new users often drop-off at the signup page, where they have to create a username and password. By adding a Google auth option for account creation, we hope to increase the number of new signups to the platform.

Goal:

Increase new user signups by 5%+

Hypothesis

Adding Google auth as a signup method will increase the number of new user signups.

Audience:

The ~100k monthly visitors to marketing website that do not have existing accounts with Thorntale.

Experimentation Methodology

We'll be running a simple A/B test with 50/50 treatment control split between our experiment audience. We'll track our progress in PostHog using their built in features. We'll do supplementary qualitative and robustness checking analyses in juypter notebooks.

We'll start with a slow experiment rollout starting with 5% of website traffic in our treatment, increasing to 50% of traffic over a 24 hour period.

Expected Issues/Concerns

We are changing the login page entirely to support the new signup method. This should be fine, but we don't want to make it harder for existing users to sign into the platform.

Experiment Metrics

Primary Success Metrics

The lowest bound of our confidence interval is above our targeted goal of 5% increase in new user signups compared to control. Our hypothesis that decreasing friction increases new signups is validated.

Metrics	Goal	Results: Confidence Interval	Results: Status	Results: Actual Values
New users signups	>= 5%	[ 5.25, 7.25 ]	Success	Treatment: 1280 Control: 1205

Why these metrics

# New user signups: our main success metric and what we hope to drive

New user signups: treatment vs. control

Secondary or Product Metrics

Metrics	Goal	Results: Confidence Interval	Results: Status	Results: Actual Values
% new signups from Google Auth	>= treatment impact	N/A. This is a robust checking metric. Tracking only.	Confirms experiment results	N/A
% google auth signups convert to active	tracking only	N/A tracking only	Google Auth signups convert in lower rates to active than other signup types	N/A

Why these metrics

•

% new signups from Google Auth

•

We want to ensure that our results are robust. If we see an increase in signups, but they aren't due to google auth signups, then our results would not be robust. In this experiment we do see an equivalent increase in google auth signups that would explain the increased number of signups.

Breakglass/Guardrail Metrics

With this experiment we need to make sure we don't negatively impact our existing users. The login page is being overhauled to reflect the new signup method, so we want to make sure we don't have increased churn of existing users being blocked from signing in.

Metrics	Stop Experiment if
% existing users drop off login page	> 5% drop-off

3% is our baseline from the last 6 months of existing users dropping off the login page. There is no need for stat sig here, this is an absolute bar.

Rollback procedures

The platform engineering team will immediately scale back the website traffic to 0% treatment. We will start with a slow staggered roll-out to our audience, so we should catch any immediate bugs or issues.

Monitoring Chart

Experiment results are robust

Google Auth signups both increase the overall numbers and take away from some users that would have signed up with username and password previous.

Next Steps

•

We have fully rolled out the Google Auth option to our signup/login pages

•

Platform team will sync with marketing team to ensure that we improve the overall quality of our traffic

•

Consider rolling out other types of auth sign-in