Project: Launching a New Menu

In this case, we will use A/B Analysis to evaluate if the store RoundRoasters should launch the new menu in its all stores or not. This is project from Udacity - Predictive Analytics for Business Program.

A/B testing allows companies to make careful changes to their user experiences while collecting data on the results. Here, we will use A/B Analysis to evaluate the results of my test due to the small samples. In the results, we will get the average lift of Weekly Store Gross Margin, Expected impect on Sum Gross Margin, and significance level, which will give us a clear insight if the store should launch the new menu in its all stores or not. The testing period is 12 weeks, and the data will be aggregated at weekly level.

Step One: Clean Up Data

We aggregate the transaction data to the appropriate level and filter on the appropriate data ranges. The three raw datasets were used to create three new file for our analysis, Weekly Store Traffic data for A/B Trend Tool, Store List data for A/B control tool, and Store Weekly Sales Analysis data for A/B Analysis tool.

Step Two: A/B Testing

Based on correlation analysis between numeric variable and performance metric Sum Gross Margin, in addition to Trend and Seasonality, we also use variable AvgMonthSales as another numeric measure to match treatment and control stores. The matching workflow is as below.

Here is the matching result generated by Alteryx.

We conduct our A/B analysis with the following workflow for Central Region and West Region separately.

From the visualization report, the average lift for Central Region is 45.3%, which means a comparison of treatment and control pairs indicates an average lift in Gross Margin for the treatment units over the control units of 45.3%. Significance level is 99.6%, which indicates a high statistical significance.

Recommendations: Yes, we recommend the company roll out the updated menu to all stores due to a significant lift on Gross Margin at a high statistical significance level of 99.6%.