Dori Wilson
January 1st 2024 - Feb 1st 2024
We've had flat signups from June 2024-Dec 2023, despite having increasing number of website visitors. We want to increase the conversion of website visitor to new signup. Potential new users often drop-off at the signup page, where they have to create a username and password. By adding a Google auth option for account creation, we hope to increase the number of new signups to the platform.
Increase new user signups by 5%+
Adding Google auth as a signup method will increase the number of new user signups.
The ~100k monthly visitors to marketing website that do not have existing accounts with Thorntale.
We'll be running a simple A/B test with 50/50 treatment control split between our experiment audience. We'll track our progress in PostHog using their built in features. We'll do supplementary qualitative and robustness checking analyses in juypter notebooks.
We'll start with a slow experiment rollout starting with 5% of website traffic in our treatment, increasing to 50% of traffic over a 24 hour period.
We are changing the login page entirely to support the new signup method. This should be fine, but we don't want to make it harder for existing users to sign into the platform.
The lowest bound of our confidence interval is above our targeted goal of 5% increase in new user signups compared to control. Our hypothesis that decreasing friction increases new signups is validated.
| Metrics | Goal | Results: Confidence Interval | Results: Status | Results: Actual Values |
| New users signups | >= 5% | [ 5.25, 7.25 ] | Success | Treatment: 1280 Control: 1205 |
# New user signups: our main success metric and what we hope to drive
| Metrics | Goal | Results: Confidence Interval | Results: Status | Results: Actual Values |
| % new signups from Google Auth | >= treatment impact | N/A. This is a robust checking metric. Tracking only. | Confirms experiment results | N/A |
| % google auth signups convert to active | tracking only | N/A tracking only | Google Auth signups convert in lower rates to active than other signup types | N/A |
With this experiment we need to make sure we don't negatively impact our existing users. The login page is being overhauled to reflect the new signup method, so we want to make sure we don't have increased churn of existing users being blocked from signing in.
| Metrics | Stop Experiment if |
| % existing users drop off login page | > 5% drop-off |
3% is our baseline from the last 6 months of existing users dropping off the login page. There is no need for stat sig here, this is an absolute bar.
The platform engineering team will immediately scale back the website traffic to 0% treatment. We will start with a slow staggered roll-out to our audience, so we should catch any immediate bugs or issues.
Google Auth signups both increase the overall numbers and take away from some users that would have signed up with username and password previous.