Innovate with Experimentation

Innovate with Experimentation Incorporate A/B Testing Into Your Product Development

John Provine Product Manager Optimizely

Matt Auerbach Dev Evangelist Optimizely

Formerly: Google

Formerly: AWS, Facebook

Agenda ● ● ● ● ●

Why experimentation matters How to build an experimentation platform Key challenges and learnings Demo Q&A

A B

A

B

9,800 Google search experiments in 2016

www.google.com/search/howsearchworks/

1,653 Google search launches in 2016

www.google.com/search/howsearchworks/

“I haven’t failed.

I’ve just found 10,000 ways that won’t work.” Thomas Edison

medium.com/netflix-techblog

uber.com/blog/colorado

Experimentation-Driven Product Development

Bad Product Development (high risk) Build

Launch

Better Product Development (minimize risk) Build

Rollout

Cleanup

Experiment-Driven Product Development (maximize impact and minimize risk) Build

Experiment

Analyze

Rollout

Cleanup

Fail as soon as possible

How to build an experimentation platform that is fast and scales

from planout.experiment import SimpleExperiment from planout.ops.random import *

class FirstExperiment(SimpleExperiment): def assign(self, params, userid): params.button_color = UniformChoice(choices=['#ff0000', '#00ff00'], unit=userid) params.button_text = WeightedChoice( choices=['Join now!', 'Sign up.'], weights=[0.3, 0.7], unit=userid)

my_exp = FirstExperiment(userid=12) # parameters may be accessed via the . operator print my_exp.get('button_text'), my_exp.get('button_color')

medium.com/netflix-techblog

medium.com/netflix-techblog

Analytics

medium.com/netflix-techblog

medium.com/netflix-techblog

eng.uber.com/data-viz-intel/

3 Key Ingredients for Robust Experimentation

Traffic Splitting

Remote Configuration

Analytics

Traffic Sampling Audience Targeting Mutual Exclusion State Persistence Multivariate Treatment Global Holdout QA Tool Automation Whitelisting Logging

Permissions Feature Toggles Feature Rollouts Kill Switch Scheduling Staging Environments Audience Definitions Webhooks REST API Audit Trail

Event Dispatching Event Storage Sessionization Statistical Methods Metric Definitions User Aliasing Segmentation DW Integration Notifications Visualizations

3 Key Challenges with Experimentation 1. Trust “How can I trust the experiment was set up and tracked correctly?” 2.

Process “What are the best instrumentation practices to minimize technical debt?”

3.

Maintenance “The engineer who built this left the company… now what?”

Client-side A/B testing for websites

Deep experimentation anywhere in the stack

2010

2017

Demo

Q&A