Subject of Experiment
When designing an A/B Testing experiment, there's four main considerations:
- Choosing subject
- Choosing population
- Size
- Duration
In this blog, we want to choose the subject of our experiment. That is unit to measure for our control and experiment group. This unit will be called unit of diversion.
Variability of Metrics
When characterizing our experiment, we're talking about how the metric can used in particular situation. We want to know range of possible condition that metrics can be used, and that's variability. Usually, when we talking about probability (e.g. click-through-probability) or count of users, we can see nice normal distribution by plot it in histogram. This would make it easy for us to use theorically-compute variability, in form of Confidence interval. We could also use our normal practival significance level. However, when using video latencies like in previous blog gives us sort of lumpy shape, we want to compute the variability empirically. This blog discuss about how we characterize metrics for our experiment, specifically by measures of spread.
High to Low Level Metrics A/B Testing
Earlier when choosing metrics, we go through 3 steps: Defining the metrics, build intuition about our data and our system, and finally characterizing the metrics. We have talked about defining turn high level concept into metrics that we want to measure. Also talking about filtering any potential bias. This blog discuss how to turn high level concept of metrics into lower more technical metrics. Then we create summary metrics in form of statistics to build intuition. Specifically, we discuss about measures of center of summary statistic in the metric for experiments.
Difficult Metrics in A/B Testing
Sometimes when performing A/B testing, you faced with a difficult metrics. Audacity MOOC example, has trouble to find whether the students successfully landed a job. This blog discuss how to deal with difficult metrics. It's not it's impossible, but we can vary our validation technique to create such that.
A/B Testing Metrics
When choosing a metric, there's a couple of thing that we need to know. It's the definition of the metris, build the intuition from it by performing sanity check and sensitivity test, and then characterize the metric by evaluation
Policy and Ethics in A/B Testing
When talking about gathering data and experiments, we potentially leaking any information that may considered private for some of the participants. There are 4 main principles to observed:
- Risk
- Benefits
- alternatives
- data sensitivity
A/B Testing Overview
A/B Testing is the test that we want to test for particular product. Usually A/B testing works for testing changes in elements in the web page. A/B testing framework is following sequence:
- Design a research question.
- Choose test statistics method or metrics to evaluate experiment.
- Designing the control group and experiment group.
- Analyzing results, and draw valid conclusions.
A/B testing is used to validate whether the changes that we have applied in our product is significantly affected our users instead of relying solely on the expert opinion.
Why 2016 U.S. presidential candidates should care at where they're running their campaign
Introduction:
Research question : Does there appear to be a relationship between 2012 U.S. presidential candidates with their financial contributors’ neighborhood?
Candidates for presidential election need donations from contributors to run a campaign.You have seen result of votes for New York and its cities.But state and city are too general to make a conclusion. This research will dive deep into the neighborhood level and analyze whether or not there is a correlation between neighborhood and the candidates.If it’s indeed there’s a correlation, then it will provide meaningful factors for the next U.S. presidential candidates.
Frequentist vs. Bayesian Approach
In this blog we're going to discuss about frequentist approach that use p-value, vs bayesian approach that use posterior.
Screenshot taken from Coursera 01:04
Inference and Diagnostics for MLR
In this blog we will discuss about the inference for MLR. How to choose significant predictor by Hypothesis Test and Confidence Interval, as well as doing interpretations for the slope.
Screenshot taken from Coursera 00:48