Indeed, the new product has a surplus of 7103 units sold throughout the whole month, which corresponds to the average daily surplus of approximately 236.77 units.

But how can we say if this surplus is actually not random, meaning it is indeed the effect of the advertisement and not the artifact of the random fluctuation of daily customers’ behaviour?

In this second case, this observed surplus is only present in this particular sample of 30 daily observations (the first month after the introduction on the market) and is purely due to the random fluctuations caused by the "lucky chance" alone.

Our task would be to rule out this assumption of randomness and luck, and to actually prove the significant effect of the advertisement campaign.

For this purpose we will conduct a so called T-Test for paired Samples.

This test is one of the many available statistical significant tests that are used for leveraging the assumption of randomness versus the significant effect assumption.

If you ask, why specifically this test then?

Well, there are some hints in our data that point to the direction of this particular test.

First of all, one can observe that the patterns of the both series are somehow similar, especially if one considers the time period between the 15^th and the 20^th of June.

Furthermore, if we analyze the correlation between these two series, it is obvious that the corresponding correlation is positive and relatively high in magnitude.

This means that when the sales of the old product go up, the same happens to the sales of the new product and vice versa.

These two findings (the similar patterns and the strong positive correlation) pinpoint to the T-Test for paired samples as a test for statistical significance of the advertisement campaign.   

There are obviously the same factors that affect the both series in the same way simultaneously. Something like the particular day of week (weekend consumption behaviour may be similar for the both series), or mayhap even the weather on a particular day effects the both series in the same way.

That’s why in this case we assume that the two series are paired as they tend to go in the same direction as a response to the same affecting factors.

Thus, a strong positive correlation between the two series and a similar pattern in their course.

Below is the correlation graph:

The Code to produce this graph looks as follows:

----

dta <- ttest_data
dta$ID <- as.character(dta$ID)
dta$ID <- as.numeric(dta$ID)

control   <- dta$Control
treatment <- dta$Treatment

par(family = "mono")
plot(control, treatment, pch = 16, col = "steelblue2", las = 1, main = "Control vs. Treatment Correlation",
     font.main = 2, cex.main = 1.8, adj = 1)
grid()
title(sub = paste0("Corr: ", round(cor(control, treatment), 2)), font.sub = 2, cex.sub = 2.0)
 

----

But please remember that if your data doesn't show a strong positive correlation nor the similar structural pattern, then please consider using the T-Test for two independent samples instead!

So, basically what this test does, is checking whether this Mean Treatment Surplus (Mean Value at Stake) of app. 236.77 units sold per day is significantly different from zero, meaning the advertisement campaign does indeed have a positive effect after all.

The indication of this significant effect is the so called P-Value generated by the test.

Please remember as a rule of thumb that a P-Value below 0.05 only is considered to be a clear indication of the significant effect in the Treatment series.

In our case this value is app. 0.13, which postulates that this surplus in a Treatment series is actually absolutely random and should be attributed to the vagaries in the customers’ behaviour! It is definitely not due to the advertisement campaign, as was thought previously!

So, a product manager or a category manager should definitely rethink the product placement strategy, i.e. changing the advertisement spot on T.V. and other digital media.

(Or maybe just improve the quality of the new product per se… :-) )

In the picture below is the complete SAC Dashboard, where the actual data table itself is a SAC native table widget, and the rest of the graphs presented above are depicted inside the three separate R Widgets with the corresponding R Code listed previously and the same dataset attached to every one of them:

Final Remark

If you are willing to use the Code in your own dashboards, - it’s actually quite easy to adapt.

All you need to do is simply change the first line of every Code snippet in a corresponding R Widget:

"dta  <-   ttest_data"     into something like     "dta  <-   <your_data_name>".

The rest of the Code should remain as is.

And be sure that your data is structured in exactly the same way as the one we presented in this blogpost, namely the three columns with exactly the same column names (ID, Control, Treatment), where the ID column is a dimension attribute and the rest two are measures.

All things considered: Have fun while testing!  :-)