Comparing Google Shopping Tools: The Impact of Fluctuations

In this blog post, we share our Google Shopping knowledge using some examples to explain seasonality and fluctuations in Google Shopping. Based on that knowledge, we’ll show you what to look for when comparing Google Shopping tools.

 

Tl;dr:

• Expected seasonality can be problematic when comparing Google Shopping tools.
• Weekly or monthly fluctuation can be problematic, too.
• We show you the impact of fluctuation on your decision when evaluating different tools.

 

What should be…

In an ideal world for tool comparison, there would be no fluctuation. Unfortunately, there is always at least some fluctuation over time. When comparing monthly data without any changes made to your account, there won’t be equal sales every month. This is because of external factors like demand variation, competition, changes in your product range and seasonality. Let’s have a look at seasonality first.

 

Try our Shopping tool

 

Example 1: Expected Seasonality

 

Some goods are sold during one season only, while sales of other goods happen across seasons. In Europe and Northern America, skiing is a winter sport only. Of course, online shops also adjust to these circumstances and primarily offer skiing goods in fall and winter as demand is lower during other seasons. The opposite example happens to be bikinis: most bikinis are sold in spring and summer.

ski-bikini-graf02

Note: When evaluating a Google Shopping tool, be aware that at least parts of your product range could be seasonal.

 

Example 2: Fluctuation over months

 

However, in the real world, there are other fluctuations due to external factors — factors you cannot control. For instance, the demand for your goods is increasing (or decreasing) because of your competitors’ efforts and prices. A realistic Return On Advertising Spend (ROAS: the higher, the better) for one year could look like this:

ski-bikini-graf1

Note: Almost every account sees a certain fluctuation in ROAS, meaning that the monthly performance varies. If the ROAS does not turn out to be a valid KPI for comparison, you might want to take a look at the revenue or any other KPI of your choice.

 

What happens to your Shopping tool evaluation?

 

As you can see in the line chart above, there is some variation in ROAS even if you don’t change your behavior or test a new tool. The upcoming analysis is based on the following table, which fits the numbers of the line chart above and will compare 3 tools A, B and C.

 

tabs-1

 

Let me explain, how the upcoming tables are to be read. The short example below describes one test that evaluates tool A in January.

tabs-legend

 

You can see results from comparing the ROAS generated by tool A in January with returns generated in December without a tool.

Note: the ROAS of tool A was 10.1 in January. As 10.1 is smaller than 11.0, the ROAS decreased which would lead to a decision against tool A. This is indicated through the dark orange color. If tool A would beat the December ROAS, the cell would be highlighted green. In the case of a tie (equally high ROAS), the cell would be highlighted in yellow.

 

If your comparison includes data from the last 30 days, please consider delayed conversions which we explained in one of our previous blog posts.

 

What about equally strong tools?

 

Imagine you are testing a new tool (A). At first glance it looks like the tool delivers exactly the returns you and your team managed to achieve. What’s the problem? ROAS fluctuations!

The next table shows decisions for all months in a single table. You can see 12 different and independent test results for one year.

tabs-2

 

Let’s take a closer look: You can see the result if you tested in August. Doesn’t look good, right? You would have dismissed the tool because it performed worse than what you achieved in July. In the back of your head, you might already feel that this difference in ROAS might not derive from a switch to tool A.

After taking a look at the performance in September, on the contrary, you would decide, that tool A is better (ROAS of 9.8 vs. ROAS of 7.5). In October, you would see a tie, because the ROAS of the previous month equals the ROAS of October — both are 9.8.

All in all, tool A performs exactly the same as your previous tool and would win 5 comparisons, lose 6 comparisons and ties once.

 

What if the tested tool improves your KPIs?

 

Consider the following situation: The tested tool B always improves your ROAS by 0.5. The following table shows your possible decisions.

tabs-3

 

As you can see, even if tool B improved the ROAS, you won’t notice it every month just by comparing data from the previous month! In our example, there are 5 months where you would make a wrong decision against tool B (January, February, May, July and August).

 

Are you able to identify bad tools?

 

What about the opposite scenario? How easy is it to identify bad tools? Unfortunately, the same problem occurs. Take a look at tool C, which always decreases your ROAS by 0.5.

tabs-4

 

There would be 4 months, where you would choose tool C over your existing tool (April, September, November, October) as well as a month with a tie — which is also not correct (March).

 

Is your evaluation biased?

 

The more fluctuation over time, the easier it is to make wrong decisions. If you arrange all of the previously introduced tables below each other, you can see a pattern:

tabs-in-one

The performance of a tool doesn’t have the expected big impact on your decision when comparing e.g. ROAS performance to the previous month. In our example, in January, February, April, May, July, August, September and November the decision is always the same, independent of the performance of the tool! That is 9 out of 12 which tells us, that in our example only 25% of decisions depends on the performance of the tool, therefore it is biased. On average, there would be 5 out of 12 false decisions based on the example data above. Feel free to check your own data in the provided table!

 

Try it with your own data!

 

Now it’s your turn! Fill in the blank table with some of your monthly KPIs and see what happens. You can use a different KPI like KUR for our German friends or revenue, too. Just give it a try!

tabs-empty

Using your new knowledge, you can now look for the biggest difference between the rows “Your KPI” and “KPI previous month” in all months.  This is the difference, where a new tool would have beaten your status quo every single month. And even then it would be a close decision or a tie in some cases!

 

Lessons learned

 

Comparing Google Shopping tools is never easy, especially if you are using a sequential test like stated above. Natural fluctuations could – and in a lot of cases will – be stronger than the effect of a (new) tool. This doesn’t mean that the tool isn’t better than your existing tool or strategy, but you should consider a longer test period for a solid decision base. If the new tool can save you time, you should take that into account, too. When comparing recent data, don’t forget to estimate the delayed conversions!

Stay tuned for more exciting news from our “Whoop!” data science department!