Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
- Downloads：3788
- Type：Epub+TxT+PDF+Mobi
- Create Date：2021-06-05 06:54:26
- Update Date：2025-09-07
- Status：finish
- Author：Ron Kohavi
- ISBN：1108724264
- Environment：PC/Android/iPhone/iPad/Kindle
- Download
Summary
Getting numbers is easy; getting numbers you can trust is hard。 This practical guide by experimentation leaders at Google， LinkedIn， and Microsoft will teach you how to accelerate innovation using trustworthy online controlled experiments， or A/B tests。 Based on practical experiences at companies that each run more than 20，000 controlled experiments a year， the authors share examples， pitfalls， and advice for students and industry professionals getting started with experiments， plus deeper dives into advanced topics for practitioners who want to improve the way they make data-driven decisions。 Learn how to - Use the scientific method to evaluate hypotheses using controlled experiments - Define key metrics and ideally an Overall Evaluation Criterion - Test for trustworthiness of the results and alert experimenters to violated assumptions - Build a scalable platform that lowers the marginal cost of experiments close to zero - Avoid pitfalls like carryover effects and Twyman's law - Understand how statistical issues play out in practice。
Reviews
Jacob
，
This book is excellent。 While plenty has been written around RCTs in general， there are fewer resources specific to online experimentation。 I also thought they provided the right amount of technical depth。 They weren’t afraid to discuss concepts like variance and power but it did not feel like I was reading a statistics textbook。 They also covered a lot of ground in a short amount of space， including instrumentation， web performance， sample ratio mismatch （SRM）， observational causal studies， spi This book is excellent。 While plenty has been written around RCTs in general， there are fewer resources specific to online experimentation。 I also thought they provided the right amount of technical depth。 They weren’t afraid to discuss concepts like variance and power but it did not feel like I was reading a statistics textbook。 They also covered a lot of ground in a short amount of space， including instrumentation， web performance， sample ratio mismatch （SRM）， observational causal studies， spillover， and long-term effects。 Arguably the most valuable contribution is the guidance in selecting the right metric or combination of metrics， which they call the OEC （p。 103）。 The metric must be movable enough that a change could be detected， but also have a causal impact on the desired outcome。 Your company’s total revenue is not movable， because it’s highly unlikely that a small usability change would produce a significant difference， but has a direct causal link to the desired outcome （or in this case， *is* the desired outcome）。 Stock price is an even less movable metric。 On the flipside， “number of listing photos viewed” is very movable based on one UX change but is unlikely to have a large causal impact on revenue。 It’s important to use a metric like booking conversion rate that is somewhere in the sweet spot。 As an example of an OEC， you could use a weighted combination of engagement （e。g。， # of listing views）， revenue， and an ease-of-use metric like time-to-checkout。 Few books distill so much information and experience into 245 pages。 I will definitely be keeping this on my desk as a reference in future roles。 。。。more
Srinidhi Melkote
，
Awesome book on large scale experimentation。 Covers a lot of ground starting with the basics of conducting a standard A/B test to measuring long term impacts such as learning effects。 Authors are applied statisticians who led experimentation at Microsoft， Google and LinkedIn。 There’s no other book like this and it’s a must for anyone involved in digital product experimentation- product managers， analysts and data scientists。 The writing style is terse and to the point。 There’s a ton of info pack Awesome book on large scale experimentation。 Covers a lot of ground starting with the basics of conducting a standard A/B test to measuring long term impacts such as learning effects。 Authors are applied statisticians who led experimentation at Microsoft， Google and LinkedIn。 There’s no other book like this and it’s a must for anyone involved in digital product experimentation- product managers， analysts and data scientists。 The writing style is terse and to the point。 There’s a ton of info packed into the 250 pages and has a large references section。 Some sections are pretty technical but only the most important results are highlighted。 I can see myself returning to this book and references within often。 。。。more
Daniel Walton
，
As far as I can tell， this is the only decent source for implementing experiments at technology firms。 Somewhat of a disorganized and nonrigorous approach to testing is taken， though there are plenty of helpful tips， language， and best practices given along the way。
Tuan Doan Nguyen
，
This is a very good and informative book。 I usually encounter some of the difficulties and pitfalls when running online experiments so it's definitely good reference for what the industry practices are。 However， it is a very technical and academic research fact-heavy book。 I wouldn't say that the book is particularly inspiring or engrossing to read。 This is a very good and informative book。 I usually encounter some of the difficulties and pitfalls when running online experiments so it's definitely good reference for what the industry practices are。 However， it is a very technical and academic research fact-heavy book。 I wouldn't say that the book is particularly inspiring or engrossing to read。 。。。more
Evan
，
A great comprehensive resource for designing and analyzing online experiments。
Viktor Lototskyi
，
So much of well structured， up-to-date， and dense information on a tricky subject。 Truly a gem in a printing IT literature。
Anastasiia Kornilova
，
Awesome book。 Distilled overview of various aspects in experimentation。
Emily Fay
，
Thorough and highly applicableAs a data scientist， I greatly enjoyed this book and learned a lot that I can apply in my work。 The authors do a great job combining theory with real world examples。 The content is approachable for a wide audience！
Jun Zhao
，
Excellent guidance for practionersExcellent summary of all important aspects running A/B test。 My team are running more than hundred experiments per quarter， we hit pretty very corners of A/B test scenerios。 The book covers almost all of them and provide practical insight on what can be done to improve it。 Strongly recommend to anyone running amount experiments， the valuable book can save you a lot of time to learn all the tricks， pitfalls。
Lee Richardson
，
# ReviewEasily the best book I've read this year， and I can't imagine anything topping it。 The book distills ~ 20 years of hard earned knowledge on running large experiments at internet companies。The author's are pioneers of the internet age who were instrumental in building the large scale experimentation platforms at Microsoft， Google and LinkedIn。 A big reason why the book is so valuable is that the authors worked in industry， which constrains their approach to be practical。The practicality i # ReviewEasily the best book I've read this year， and I can't imagine anything topping it。 The book distills ~ 20 years of hard earned knowledge on running large experiments at internet companies。The author's are pioneers of the internet age who were instrumental in building the large scale experimentation platforms at Microsoft， Google and LinkedIn。 A big reason why the book is so valuable is that the authors worked in industry， which constrains their approach to be practical。The practicality is really nice， because the author's hit the main points， telling you things like:- What's the deal with online experiments？ - What are the major components？- What are the big problems in practice？ - How can I approach solving them？ - What are some references if I want to learn more？Much appreciated！## Other notes- Prediction gets more emphasis than experiments， "especially now"。 Both have value， but this book helped me conceptualize the differences。 I now see machine learning as more of a treatment， or a new product， or even a technique to enhance existing products。 Getting trustworthy inferences is a separate endeavor。 - The skill it takes to build a large scale experiment system， get it working properly， and scale it across a large company， is phenomenal。 As a statistician， I'm so happy companies like Google and Microsoft brought in in talented folks like this to carry the torch of statistical thinking into the 21st century。 。。。more
Richa
，
In depth guide to experimentation- covers topics such as - why run experiments， common analysis pitfalls， building metrics and my fav part of the book: how to build a robust experimentation system with a focus on trustworthy analysis。
Eddie Wharton
，