Population of Experiment

When choosing population, we left with two kind of user experiment, inter and intra user. Intra user is like event-based experiments, where it could be same user to the same group. Intra user let same user in control and experiment group. The thing to keep in mind is that you don't want to run the experiments before and after big event, like Christmas. This could be vary greatly.

The other alternative is inter-user experiments. This would means that different user in both groups. And we also want to keep something like Cohort, or lurking variables. It's the variable that potentially makes bias if we don't divide equal features for both group. Medical experiment usually more sensitive, randomly assign equal of both groups, variables like demographic, gender, age, etc. A/B testing on internet experiments lack of such things. We don't event know whether the participants is real people.

Read more…