# Overview: Free Trial Screener¶

### Background¶

• Start free trial to allow paid version

### experiment¶

• users see this popup after click
• if they're not committed, warning them.

We have two initial hypothesis.

• “..this might set clearer expectations for students upfront, thus reducing the number of frustrated students who left the free trial because they didn't have enough time”
• ”..without significantly reducing the number of students to continue past the free trial and eventually complete the course. “

• users in free trial tracked by user-id
• same user-id can't enroll in free trial twice
• users not enroll can't tracked.

# Experiment Design¶

## Metric Choice¶

### Invariant metrics¶

• Number of clicks
• Click through probability

### Evaluation metrics¶

• Gross Conversion
• Net Conversion

We have two initial hypothesis.

• “..this might set clearer expectations for students upfront, thus reducing the number of frustrated students who left the free trial because they didn't have enough time”
• ”..without significantly reducing the number of students to continue past the free trial and eventually complete the course. “

## Measuring Standard Deviation¶

• Gross Conversion: 0.02
• Net Conversion 0.0156

Expect analytical variance match empirical variance because unit of analysis and unit of diversion is same.

# Sizing¶

## Number of Samples vs. Power¶

• Gross Conversion. Baseline: 0.20625 dmin: 0.01 = 25.839 cookies who clicks.
• Net Conversion. Baseline: 0.1093125 dmin: 0.0075 = 27,411 cookies who clicks.
• Not using Bonferroni correction.
• Using alpha = 0.05 and beta 0.2

The pageviews needed then will be: 685275 impression.

## Duration vs. Exposure¶

• Fraction: 0.8 (Low risk)
• Duration: 22 days (40000 pageviews/day)

# Experiment Analysis¶

## Sanity Checks¶

• Bounds = (0.4988,0.5012)
• Observed = 0.5006
• Passes? Yes
• Number of clicks on “Start free trial”:

• Bounds = (0.4959,0.5041)
• Observed = 0.5005
• Passes? Yes
• Click-through-probability on “Start free trial”:

• Bounds = (0.0812,0.0830)
• Observed = 0.0821
• Passes? Yes

Since we have passed all of the sanity checks, we can continue to analyze the experiment.

# Effect Size Test¶

• Did not use Bonferroni correction
• Gross Conversion
• Bounds = (-0.0291, -0.0120)
• Statistical Significance? Yes
• Practical Significance? Yes
• Net Conversion
• Bounds = (-0.0116,0.0019)
• Statistical Significance? No
• Practical Significance? No

# Sign Test¶

• Did not use Bonferroni correction
• Gross Conversion
• p-value = 0.0026
• Statistical Significance? Yes
• Net Conversion
• p-value = 0.6776
• Statistical Significance? No

### Conclusion¶

• Not use Benferroni correction.
• Gross Conversion need significant but Net Conversion doesn't.

## Recommendation¶

• Gross Conversion: pass
• Net Conversion: somehow pass, can loss potential money

• decision: risky. delay for further experiment or cancel the launch.

# Follow-Up Experiment¶

• Not necessary to show warning
• Start Debt Program
• Risky, users break agreement
• Not become Udacity Code Reviewer
• Cancel in midway program
• Hypothesis
• Non-serious users become more committed after incentive
• Number of users who cancel early is reduced
• Boost compared to already committed

We can use Invariant metrics for this experiment for the follow-up:

• Number of cookies: That is, number of unique cookies to view the course overview page.
• Number of clicks: That is, number of unique cookies to click the "Start free trial" button (which happens before the free trial screener is trigger).
• Click-through-probability: That is, number of unique cookies to click the "Start free trial" button divided by number of unique cookies to view the course overview page
• Gross conversion: That is, number of user-ids to complete checkout and enroll in the free trial divided by number of unique cookies to click the "Start free trial" button.

And the evaluation metric:

• Debt Conversion: That is, number of user-ids to click “Start Debt Program” divided by number of user-ids that enroll in the free trial.
• Debt-Net conversion: That is, number of user-ids to click “Start Debt Program” divided by number of user-ids to remain enrolled past the 14-day boundary (and thus make at least one payment)
• Net conversion: That is, number of user-ids to remain enrolled past the 14-day boundary (and thus make at least one payment) divided by the number of unique cookies to click the "Start free trial" button.

We use user-ids as unit of diversion, expect all of the evaluation metrics to be practically significant.