Theoretical relevance is more important than statistical significance

Over the years, I have assisted many university students on their research projects, particularly in relation to data analysis were bivariate and multivariate statistical analyses are required. In the process, I have observed a tendency among some of the researchers of focusing primarily on finding statistically significant relationships between some variables in their studies as opposed to focusing on specific hypotheses testing.

Hypothesis testing, by definition, implies that either a statistically significant relationship will be found between some variables of interest; in which case the null hypothesis is rejected, or a non-statistically significant relationship will be indicated between those variables; in which case we fail to reject the null hypothesis.

So, instead of being excited just because some statistically significant relationships have been found or, conversely, being disappointed just because the analysis has not found any statistically significant relationships, the important thing is to check if those results have some theoretical grounding or support in existing literature.

Not every statistically significant or non-significant relationship between some variables in a study will make theoretical sense. For example, in a study done within a highly patriarchal community it would be rather odd to find that there was a significant association between gender and formal employment but in favour of females. The opposite would be expected instead. It would arguably also be odd and cast doubt on the study’s external validity if no association at all was found between employment and gender in such a case.

Although not as common, there are also instances where, due to the desire to have some statistically significant results to talk about in their research reports, some researchers will perform some rather redundant analyses under the guise of “secondary objectives”. An example of (completely made up) would be in a study where some height and weight data were collected for the purpose of computing the study participants’ body mass index (BMI), which in turn would be used to examine if there was an association between BMI and susceptibility to certain medical conditions X and Y. Now, imagine the subsequent analysis found that there was no association between BMI and medical condition X or BMI and medical condition Y. And then driven by the “thirst” to at least have some results with some p-value that is less than 0.05 somewhere in their paper, the researcher then runs some linear correlational analysis between height and weight. They could even also want some test performed to examine if the two medical conditions X and Y are associated, even if this would be somewhat redundant analysis in relation to the main study aim.

In summary, theoretical relevance or logic matters more than mere statistical significance. Put differently, statistical significance will not count for much if it lacks theoretical significance.