Statistical significance refers to the likelihood that a difference between two data points reflects reality. When using a larger sample size, you'll be more likely to see statistically significant data.
Statistical significance can be calculated using a statistical significance calculator (readily available through an online search).
When considering the statistical significance of data in our platform, keep in mind:
- The significance of any differences in percentages will depend on the sample sizes being used. For an audience of 1,000, a change in % between two waves is much more likely to be robust and significant than for an audience of 100.
- A difference between two data points that is statistically significant doesn't mean that the exact value shown is objectively true. Typically, statistics software will flag a value difference with 95% confidence, meaning it's 95% certain that the difference displayed will fall within the margin of error (which is typically, but not universally, set at +/- 3%.). For example: In a single choice question, 80% of respondents have selected option A and 20% option B. The statistical software used has flagged the interval of 60% between the two options as significant. This means that there is a 95% chance that the difference between Answer Option A and B is within the range of 57% and 63%.
- A data point not being flagged as statistically significant doesn't necessarily mean that it isn't reflective of reality. When breaking our data sets down into smaller audiences - thereby lowering the available sample size - some of the smaller percentage changes may not be statistically significant. This doesn't mean that the data can't be trusted or that there is nothing to be learned from it, but rather that there's a higher chance that the differences between data points sit outside of the margin of error. When working with data that isn't statistically significant, it's advisable to work with a degree of caution and also consider the wider context of the results.