Q: What should one do with papers that largely consist of non-findings? Is it just a waste of time trying publish a study without significant effects? Or, should I simply aim for lower quality journals with such papers?
This is an important question that we must all confront as researchers. Not every study works out as we anticipated. No one wants to waste time on a project that has little chance of success. However, I think it is a myth that good journals only publish papers with statistically significant results. Yes, there is a bias in favor of publishing papers with statistically significant findings, but there are many great examples of papers published in top journals that are built around “non-findings.” Here are the lessons that I have learned from these papers:
Characteristics of successful papers with “non-findings”
1. Have a good theory. For starters, it is important to have a well-grounded theoretical explanation for why one would either expect, or not expect, to find a relationship between the variables in your analysis. If your paper doesn’t find a relationship between X and Y, and the reader doesn’t have a clear idea of why we would expect that X should (or should not) covary with Y, then your paper is doomed. Ainsworth-Darnell & Downey (AD&D)(ASR, 1998) relied upon Ogbu’s much cited “oppositional culture” framework to generate several hypotheses about black-white differences in attitudes, behaviors, and school performance. They didn’t find any support for the Ogbu’s hypotheses, but that’s what makes the paper so interesting (and one reason the paper is so often cited!).
2. Cover new ground. The best case scenario is that one has a theory that has not been tested, and there is a “research vacuum” to exploit. AD&D covered new ground by being the first paper to systematically evaluate Ogbu’s claims with a multivariate analysis of nationally representative data. Their “non-findings” filled an important void in the field. In separate studies, Hallinan & Kubitschek and I examined sector differences in learning among elementary school students, and found either no effects, or, positive public school effects. This went against the grain of prior research on the Catholic school advantage in high school (see point 3 below), but it also exploited the “research vacuum” in this area.
3. Re-visit old ground. Non-findings are especially interesting when they go against the grain of prior research. If prior research consistently finds that X matters for Y, and you find that X doesn’t matter for Y, that’s interesting! The big question is of course WHY you didn’t find that X matters for Y. If you used different data than other scholars, you need to show why your data are superior in quality to prior research. If your data are more recent, then you have a very good story to tell: X used to matter for Y, but it no longer does. Of course, you need a good explanation for why X no long matters for Y. Think of Wilson’s The Declining Significance of Race and The Truly Disadvantaged. Much debated works, but they are compelling and provocative narratives! Finally, along the lines of points 4 and 5 below, you have an interesting paper if you can show that, in contrast with prior research, your use of higher quality measures and/or more rigorous methods indicates that X doesn’t matter for Y.
4. Have impeccable measures. If your analysis has poor measures of your key concepts, it shouldn’t be surprising that you have insignificant findings in your analyses. In order for a “non-findings” paper to be compelling, the reader should be convinced that the measures are credible, with high validity and reliability. This is a common problem in “non-findings” papers that I reject as a reviewer. AD&D used multiple measures of “oppositional culture” in their study, and while one might prefer better measures, their measures were credible enough to make a significant contribution to the field.
5. Have a rigorous (and possibly novel) research design. Happily, the bar of what counts as “rigorous research” is constantly being raised. “Non-findings” are especially important when one can make a convincing argument that prior research findings are in fact “artifacts” of weak research design/methodology. For example, Guo and VanWey (ASR, 1999) used fixed effects models (rather than cross-sectional, between family models) to examine sibling effects on achievement, and found no effects of siblings. This study has been much debated, but it was compelling and innovative enough to merit publication in a top journal. Mouw (ASR, 2003) re-examined research on network ties and job searches using substantially more rigorous empirical models than prior research, and he found that job contacts did not affect job search outcomes. In each of these cases, reviewers were obviously convinced enough by the results (based on the rigor of the design and analysis) to deem the paper publishable.
To summarize, don’t immediately despair if your paper is filled with “non-findings.” Some “non-findings” papers should be abandoned, but typically, that is because they are not very good papers to begin with! Under the right circumstances, “non-findings” are actually “FINDINGS” that can be published in top journals, and may have an important impact in your subfield.