Shipping and tax costs are a critical piece of information that shoppers use to evaluate their online purchases. Shipping/tax estimators are a site feature designed to leverage that information desire. Because of a shopper’s perceived importance of that cost info, sometimes they are willing to offer up something to get it, specifically their e-mail address. Thus, estimators are often hidden behind an e-mail address collection input, but can be also placed on pages higher in the conversion funnel (particularly the cart page). The risk for sites placing the estimator behind an e-mail address collection field is that shoppers may abandon earlier in the funnel due to how they react to the estimator. However, that risk can be mitigated because the e-mails collected might be utilized to send out abandonment retention e-mails, perhaps offering a free shipping promotion as enticement to return.
Testing proposal origins can stem from analytics hinting at areas for improvement, evaluation of a new feature or function, or internal battles on which practice works better. In this case there wasn’t a specific problem, our client was simply curious and wished to explore whether the shipping estimator on the cart page had any positive or negative effect on the site.
Screenshot of the cart item grid and shipping estimator:
Shoppers that did not like the costs would bounce whether the cost was shown at the cart page or during the checkout page. Therefore, it is up to the retailer whether they want more shoppers to bounce earlier in the funnel or later in the funnel. Of course, there are other factors that need to be considered, such as if shoppers trying to find the total price would be deterred by the extra step to fill out email address information.
Since the test stemmed from a “discover and see” approach, the prediction was more general. There should not be a significant effect on orders from removing the shipping estimator on the cart page since shoppers that did not like the shipping, taxes, or total price would bounce either way.
We ran 2 experiments, one during low season and one during high season, each with a 1.5 month time period with a huge data sample size. The test during the low season was a 50/50 traffic split. The test during the high season was a 90/10 traffic split to emphasize the results of hiding the estimator where only 10% of the traffic saw the estimator.
Show Estimator: $42.09 per visitor
Hide Estimator: $45.24 per visitor (+7.5% improvement)
Show Estimator: $79.57 per visitor
Hide Estimator: $77.53 per visitor (-2.6% loss)
The results were surprising, and we ran more tests just to be sure. The slower season trended towards removing the shipping estimator on the cart page. Yet the revenue statistics for the busy season were the opposite; showing the shipping estimator was the winner of the results.
The data led us to investigate how the shipping estimator’s presence affected the shopper’s behavior during different traffic periods. During slow periods the shopper profile is more heavily weighted with repeat shoppers that have a feel of what the shipping and taxes cost. Conversely, during a busier time the shopper profile is weighted more towards first-time shoppers, so showing the estimator may be preferential because they are not as familiar with the site. Without testing, the retailer would not have seen that the seasonality had a pronounced impact on if the shipping estimator should be present.
Some eCommerce sites already display different products and promotions to their various shopper profiles. On a different level, this line of experiments shows that shoppers have distinct habits and their overall behavior can change based on the time of the year. Similar experiments to this one could be run to find out if seasonal features, such as a gift finder or a quick order, may perform better or worse during slow and busy seasons.
Beyond seasonal feature sets, this experiment series opens up a whole other avenue of exploration for future experiments and possibly thousands of new opportunities to personalize the shopping experience based on customer segments. Retailers could test the purchase behaviors of different shopper profiles, how they move through the funnel and what features should show up to optimize conversion. At a micro level, time specific tests could be applied to further determine what should be shown depending if the shopping is done at work, at night, or during the weekend.
This is why cutting edge technology such as Optimizely is vital, so that retailers can quickly test and learn about their shoppers’ behaviors. Based on these results, futuristic eCommerce solutions will allow retailers to rapidly build these dynamic data driven features, an assortment of page layouts, and a variety of page flows so retailers can swiftly transform these ideas into revenue.