Outliers are often excluded from the analysis with a Outlier filtering/detection such as do A/B testing platforms handle outliers, example: Optimizely: https://help.optimizely.com/Measure_success%3A_Track_visitor_behaviors/Outlier_filtering_in_Optimizely_X Dynamic Yield: https://www.dynamicyield.com/lesson/outliers-detection/