When talking about gathering data and experiments, we potentially leaking any information that may considered private for some of the participants. There are 4 main principles to observed:

• Risk
• Benefits
• alternatives
• data sensitivity

There are experiments where it's gone wrong because of the policy.

There is experiments that didn't told to the participants in healthcare. Some participants receive placebo, where others receive treatments. This untold experiments will caused high risk to the placebo participants because they thought they receive medicine. Milgram experiments can be high risk physcologically. Or maybe gray area of Facebook experiments as stated in this link: http://www.wsj.com/articles/furor-erupts-over-facebook-experiment-on-users-1404085840

Risk

Is it minimal? if it is, then require consent.

1. harmless
2. harmful, if provide wrong information

Benefits

Benefits for participants. Sometimes tradefoff. Higher the risk, higher the nbenefit.

Alternatives

balanced against risk and benefit, is the alternative make participant willing to go through changes? is it because they limited by other options?

Data sensitivity

https://www.udacity.com/course/viewer#!/c-ud257/l-3998098714/m-3999708754

• For new data being collected and stored, how sensitive is the data and what are the internal safeguards for handling that data? E.g., what access controls are there, how are breaches to that security caught and managed, etc.?

• Then, for that data, how will it be used and how will participants’ data be protected? How are participants guaranteed that their data, which was collected for use in the study, will not be used for some other purpose? This becomes more important as the sensitivity of the data increases.

• Finally, what data may be published more broadly, and does that introduce any additional risk to the participants?

There are examples of data sensitivity.

Census data or shopping stats by zipcode is too general for re-identification for each user, so it's not sensitive. Daily traffic to specific sites can be signed annonimously as cookie, so we can't identify the user. Same goes to online game stats. Glucose levels by some health app is subject to data sensitivity however, since HIPPA regulation stated that timestamp can be tracked to the user though sound impossible. Credit card information is of course very sensitive.

4 main principles can be written in form of concent. We can handed to them to decide whether they want to participate. It also given to IRB to asses whether such a form is required.

Term of Service or Privacy Policy is usually included in form of concent. It doesn't include company information, or even list of experiment that you are planning, which latter makes your experiment bias as participants are not blinded

So is these information necessary for internal training who runs A/B tests? Of course! They should know which questions to choose when evaluating metrics that uphold these principles. Trainee should have some understanding about these policies so they know and keep it in the experiments.

Disclaimer:

This blog is originally created as an online personal notebook, and the materials are not mine. Take the material as is. If you like this blog and or you think any of the material is a bit misleading and want to learn more, please visit the original sources at reference link below.

References:

• Diane Tang and Carrie Grimes. Udacity