Wayfair A/B Testing Campaign Analysis

"Ships in Time" Evaluation

Conducted A/B test campaign analysis using exploratory data analytics (EDA) and key performance indicator (KPI) metrics to evaluate the impact of Wayfair’s “Ships in Time” delivery guarantee on customer behavior, uncovering strategic insights into spending and returns.

My responsibilities included validating the experimental setup using EDA and uncovering behavioral consequences of unmet delivery guarantees via return rate analysis (Question 2).

Project Summary

As part of an analytics case sponsored by Wayfair, our team analyzed the impact of the 2016 holiday “Ships in Time” A/B test campaign. The campaign aimed to reassure customers (Group B) that their orders would arrive before Christmas, a potentially costly but high-reward promise. Using a large dataset including clickstream and order records, we examined whether this campaign influenced purchasing behavior and post-purchase outcomes.

The analysis was designed to mirror a real-world business analytics project, requiring both exploratory insight and business-relevant interpretation.

Project Overview

Objective

Evaluate whether the "Ships in Time" guarantee increased key customer performance metrics, such as Average Order Value (AOV)
Investigate if missed delivery expectations led to higher return rates
Validate the fairness of the experimental setup using exploratory data analysis (EDA)
Derive actionable insights to guide future campaign design and operational planning

Approach & Structure

Part 1: Exploratory Data Analysis (EDA)

Question:

Are the visitor type distributions between Group A and Group B consistent, ensuring that both groups are balanced for an accurate evaluation of the ‘Ships in Time’ guarantee?

As the team member responsible for the EDA section, I validated the experimental design by analyzing whether visitor type (e.g., new, returning, activated) distributions were balanced between Group A (control) and Group B (treatment). This ensured that any differences observed later could be attributed to the campaign itself, not sample bias.

Result: Visitor type proportions were nearly identical, confirming the test’s fairness.

🔗 Visualization Link:

Counts and Proportions Each Visitor Type

Part 2: Business Impact Analysis

Question 1:

Campaign's Impact on Average Order Value (AOV)

Did the “Ships in Time” campaign lead to an increase in Average Order Value (AOV) during key periods such as weekends and regular days and across several types of visitors?

Yes - AOV increased in most segments, notably by +7.28% for Activated Customers on weekdays
An exception was Returning Visitors on weekdays, who showed a slight decline of 1.40%, suggesting campaign fatigue or different expectations

🔗 Visualization Link: Campaign's Impact on AOV

Question 2: Late Delivery Impact on Return Rates

How does missing the 'Ships in Time' guarantee impact the return rate of Group B customers, especially for orders delivered after Christmas, compared to those delivered on time?

Yes - When deliveries missed Christmas, Group B’s return rate surged to 18.94%, compared to Group A’s 6.19%
This reveals that unmet promises severely undermine customer satisfaction and retention

🔗 Visualization Link:

Late Delivery Impact on Return Rate

Recommendations

Improve Delivery Reliability During Peak Seasons

To minimize the sharp increase in return rates caused by unmet delivery promises, especially after Christmas, Wayfair should enhance coordination with logistics partners and build more conservative delivery estimates during high-demand periods, such a holiday periods.

Segment Marketing Strategies by Visitor Type

While the campaign increased Average Order Value (AOV) for most customer types, Returning Visitors showed a negative or flat response. Future campaigns should adopt personalized incentives (e.g., loyalty rewards, curated bundles, exclusive discounts) for this segment instead of emphasizing delivery speed.

Set Clearer Expectations Around Guarantees

The strong negative reaction to unmet expectations in Group B suggests the importance of transparent messaging. Use clearer disclaimers or proactive notifications when delivery targets are at risk to reduce dissatisfaction and avoid preventable returns.

Run Post-Campaign Surveys to Measure Perceived Trust

To assess the long-term effect of such delivery promises on brand trust, consider implementing customer satisfaction surveys post-purchase, especially for customers who experienced delays. This can inform messaging tone and operational improvements.

Continue Leveraging Data-Driven A/B Testing

This case demonstrates the power of A/B testing in isolating campaign effectiveness. Wayfair should continue embedding data experimentation into its marketing strategy and broaden it to test messaging formats, timing, and audience segmentation.

Key Takeaways

The "Ships in Time" campaign increased spending for most customer types, particularly New and Activated Visitors.
Returning Visitors responded less positively and were more prone to dissatisfaction when promises were unmet.
Unmet expectations had a significant cost, as seen in the 18.94% return rate for late deliveries in Group B.
Expectation management is critical in time-sensitive marketing campaigns; delivery guarantees should be backed by operational reliability.
Effective campaign design requires audience segmentation and differentiated messaging to maximize impact while minimizing risks.