Subject of Experiment

When designing an A/B Testing experiment, there's four main considerations:

  • Choosing subject
  • Choosing population
  • Size
  • Duration

In this blog, we want to choose the subject of our experiment. That is unit to measure for our control and experiment group. This unit will be called unit of diversion.

Read more…

Variability of Metrics

When characterizing our experiment, we're talking about how the metric can used in particular situation. We want to know range of possible condition that metrics can be used, and that's variability. Usually, when we talking about probability (e.g. click-through-probability) or count of users, we can see nice normal distribution by plot it in histogram. This would make it easy for us to use theorically-compute variability, in form of Confidence interval. We could also use our normal practival significance level. However, when using video latencies like in previous blog gives us sort of lumpy shape, we want to compute the variability empirically. This blog discuss about how we characterize metrics for our experiment, specifically by measures of spread.

Read more…

High to Low Level Metrics A/B Testing

Earlier when choosing metrics, we go through 3 steps: Defining the metrics, build intuition about our data and our system, and finally characterizing the metrics. We have talked about defining turn high level concept into metrics that we want to measure. Also talking about filtering any potential bias. This blog discuss how to turn high level concept of metrics into lower more technical metrics. Then we create summary metrics in form of statistics to build intuition. Specifically, we discuss about measures of center of summary statistic in the metric for experiments.

Read more…

Difficult Metrics in A/B Testing

Sometimes when performing A/B testing, you faced with a difficult metrics. Audacity MOOC example, has trouble to find whether the students successfully landed a job. This blog discuss how to deal with difficult metrics. It's not it's impossible, but we can vary our validation technique to create such that.


Screenshot taken from Udacity, A/B Testing, Difficult Metrics

Read more…

A/B Testing Metrics

When choosing a metric, there's a couple of thing that we need to know. It's the definition of the metris, build the intuition from it by performing sanity check and sensitivity test, and then characterize the metric by evaluation

Read more…

Policy and Ethics in A/B Testing

When talking about gathering data and experiments, we potentially leaking any information that may considered private for some of the participants. There are 4 main principles to observed:

  • Risk
  • Benefits
  • alternatives
  • data sensitivity

Read more…

A/B Testing Overview

A/B Testing is the test that we want to test for particular product. Usually A/B testing works for testing changes in elements in the web page. A/B testing framework is following sequence:

  • Design a research question.
  • Choose test statistics method or metrics to evaluate experiment.
  • Designing the control group and experiment group.
  • Analyzing results, and draw valid conclusions.

A/B testing is used to validate whether the changes that we have applied in our product is significantly affected our users instead of relying solely on the expert opinion.


Screenshot taken from Udacity 0:03

Read more…

Why 2016 U.S. presidential candidates should care at where they're running their campaign


Research question : Does there appear to be a relationship between 2012 U.S. presidential candidates with their financial contributors’ neighborhood?

Candidates for presidential election need donations from contributors to run a campaign.You have seen result of votes for New York and its cities.But state and city are too general to make a conclusion. This research will dive deep into the neighborhood level and analyze whether or not there is a correlation between neighborhood and the candidates.If it’s indeed there’s a correlation, then it will provide meaningful factors for the next U.S. presidential candidates.

Read more…

Frequentist vs. Bayesian Approach

In this blog we're going to discuss about frequentist approach that use p-value, vs bayesian approach that use posterior.


Screenshot taken from Coursera 01:04

Inference and Diagnostics for MLR

In this blog we will discuss about the inference for MLR. How to choose significant predictor by Hypothesis Test and Confidence Interval, as well as doing interpretations for the slope.


Screenshot taken from Coursera 00:48