Agenda

TL;DR

Experiment Details

Dates:

January 1st 2024 - Feb 1st 2024

Context:

We've had flat signups from June 2024-Dec 2023, despite having increasing number of website visitors. We want to increase the conversion of website visitor to new signup. Potential new users often drop-off at the signup page, where they have to create a username and password. By adding a Google auth option for account creation, we hope to increase the number of new signups to the platform.

Goal:

Increase new user signups by 5%+

Hypothesis

Adding Google auth as a signup method will increase the number of new user signups.

Audience:

The ~100k monthly visitors to marketing website that do not have existing accounts with Thorntale.

Experimentation Methodology

We'll be running a simple A/B test with 50/50 treatment control split between our experiment audience. We'll track our progress in PostHog using their built in features. We'll do supplementary qualitative and robustness checking analyses in juypter notebooks.

We'll start with a slow experiment rollout starting with 5% of website traffic in our treatment, increasing to 50% of traffic over a 24 hour period.

Expected Issues/Concerns

We are changing the login page entirely to support the new signup method. This should be fine, but we don't want to make it harder for existing users to sign into the platform.

Primary Success Metrics

The lowest bound of our confidence interval is above our targeted goal of 5% increase in new user signups compared to control. Our hypothesis that decreasing friction increases new signups is validated.

Why these metrics

# New user signups: our main success metric and what we hope to drive

New user signups: treatment vs. control

Secondary or Product Metrics

Why these metrics

Breakglass/Guardrail Metrics

With this experiment we need to make sure we don't negatively impact our existing users. The login page is being overhauled to reflect the new signup method, so we want to make sure we don't have increased churn of existing users being blocked from signing in.

3% is our baseline from the last 6 months of existing users dropping off the login page. There is no need for stat sig here, this is an absolute bar.

Rollback procedures

The platform engineering team will immediately scale back the website traffic to 0% treatment. We will start with a slow staggered roll-out to our audience, so we should catch any immediate bugs or issues.

Monitoring Chart

Experiment results are robust

Google Auth signups both increase the overall numbers and take away from some users that would have signed up with username and password previous.

Next Steps