Online Controlled Experimentation at Scale : An Empirical Survey on the Current State of A/B Testing

DSpace Repository

Online Controlled Experimentation at Scale : An Empirical Survey on the Current State of A/B Testing

Details

Files for download

Find Full text There are no files associated with this item..

Overview of item record
Publication Paper in proceeding
Title Online Controlled Experimentation at Scale : An Empirical Survey on the Current State of A/B Testing
Author Fabijan, Aleksander ; Dmitriev, Pavel ; Olsson Holmström, Helena ; Bosch, Jan
Date 2018
English abstract
Online Controlled Experiments (OCEs, aka A/B tests) are one of the most powerful methods for measuring how much value new features and changes deployed to software products bring to users. Companies like Microsoft, Amazon, and Booking.com report the ability to conduct thousands of OCEs every year. However, the competences of the remainder of the online software industry remain unknown. The main objective of this paper is to reveal the current state of A/B testing maturity in the software industry based on a maturity model from our previous research. We base our findings on 44 responses from an online empirical survey. Our main contribution of this paper is the current state of experimentation maturity as operationalized by the ExG model for a convenience sample of companies doing online controlled experiments. Our findings show that, among others, companies typically develop in-house experimentation platforms, that these platforms are of various levels of maturity, and that designing key metrics - Overall Evaluation Criteria - remains the key challenge for successful experimentation.
Conference
44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) (29-31 Aug. 2018 : Prague, Czech Republic)
DOI https://doi.org/10.1109/SEAA.2018.00021 (link to publisher's fulltext.)
Publisher IEEE
Host/Issue Proceedings of the EUROMICRO Conference;
Series/Issue Proceedings of the Euromicro Conference;
ISSN 1089-6503
ISBN 978-1-5386-7383-6
Language eng (iso)
Subject controlled experimentation
A/B testing
empirical survey
Experimentation Growth Model
Technology
Research Subject Categories::TECHNOLOGY
Handle http://hdl.handle.net/2043/27273 Permalink to this page
Link http://dsd-seaa2018.fit.cvut.cz/seaa/... (external link to related web page)
Link to publication in DiVA Find this research publication in DiVA (n/a for student publ.)
Facebook

This item appears in the following Collection(s)

Details

Search


Browse

My Account

Statistics