Paper Submissions

Questions relating to the submission of papers for publication to peer-reviewed journals.

Publishing “Non-findings”

Q: What should one do with papers that largely consist of non-findings? Is it just a waste of time trying publish a study without significant effects? Or, should I simply aim for lower quality journals with such papers?

Author: Bill Carbonaro, University of Notre Dame03ff41bd-56c6-4ea4-9fc1-5ef9ac0830d4

This is an important question that we must all confront as researchers. Not every study works out as we anticipated. No one wants to waste time on a project that has little chance of success. However, I think it is a myth that good journals only publish papers with statistically significant results.  Yes, there is a bias in favor of publishing papers with statistically significant findings, but there are many great examples of papers published in top journals that are built around “non-findings.” Here are the lessons that I have learned from these papers:

Characteristics of successful papers with “non-findings”

1. Have a good theory. For starters, it is important to have a well-grounded theoretical explanation for why one would either expect, or not expect, to find a relationship between the variables in your analysis. If your paper doesn’t find a relationship between X and Y, and the reader doesn’t have a clear idea of why we would expect that X should (or should not) covary with Y, then your paper is doomed. Ainsworth-Darnell & Downey (AD&D)(ASR, 1998) relied upon Ogbu’s much cited “oppositional culture” framework to generate several hypotheses about black-white differences in attitudes, behaviors, and school performance. They didn’t find any support for the Ogbu’s hypotheses, but that’s what makes the paper so interesting (and one reason the paper is so often cited!).

2. Cover new ground. The best case scenario is that one has a theory that has not been tested, and there is a “research vacuum” to exploit. AD&D covered new ground by being the first paper to systematically evaluate Ogbu’s claims with a multivariate analysis of nationally representative data. Their “non-findings” filled an important void in the field. In separate studies, Hallinan & Kubitschek  and I examined sector differences in learning among elementary school students, and found either no effects, or, positive public school effects. This went against the grain of prior research on the Catholic school advantage in high school (see point 3 below), but it also exploited the “research vacuum” in this area.

3. Re-visit old ground. Non-findings are especially interesting when they go against the grain of prior research. If prior research consistently finds that X matters for Y, and you find that X doesn’t matter for Y, that’s interesting!  The big question is of course WHY you didn’t find that X matters for Y. If you used different data than other scholars, you need to show why your data are superior in quality to prior research. If your data are more recent, then you have a very good story to tell: X used to matter for Y, but it no longer does. Of course, you need a good explanation for why X no long matters for Y. Think of Wilson’s The Declining Significance of Race and The Truly Disadvantaged. Much debated works, but they are compelling and provocative narratives! Finally, along the lines of points 4 and 5 below, you have an interesting paper if you can show that, in contrast with prior research, your use of higher quality measures and/or more rigorous methods indicates that X doesn’t matter for Y.

4. Have impeccable measures. If your analysis has poor measures of your key concepts, it shouldn’t be surprising that you have insignificant findings in your analyses. In order for a “non-findings” paper to be compelling, the reader should be convinced that the measures are credible, with high validity and reliability. This is a common problem in “non-findings” papers that I reject as a reviewer. AD&D used multiple measures of “oppositional culture” in their study, and while one might prefer better measures, their measures were credible enough to make a significant contribution to the field.

5. Have a rigorous (and possibly novel) research design. Happily, the bar of what counts as “rigorous research” is constantly being raised. “Non-findings” are especially important when one can make a convincing argument that prior research findings are in fact “artifacts” of weak research design/methodology.   For example, Guo and VanWey (ASR, 1999) used fixed effects models (rather than cross-sectional, between family models) to examine sibling effects on achievement, and found no effects of siblings. This study has been much debated, but it was compelling and innovative enough to merit publication in a top journal. Mouw (ASR, 2003) re-examined research on network ties and job searches using substantially more rigorous empirical models than prior research, and he found that job contacts did not affect  job search outcomes. In each of these cases, reviewers were obviously convinced enough by the results (based on the rigor of the design and analysis) to deem the paper publishable.

To summarize, don’t immediately despair if your paper is filled with “non-findings.”  Some “non-findings” papers should be abandoned, but typically, that is because they are not very good papers to begin with! Under the right circumstances, “non-findings” are actually “FINDINGS” that can be published in top journals, and may have an important impact in your subfield.

Advertisements

Placements in “Mid-Tier” Journals

Q: What would be considered “mid-tier” sociology of education journals (i.e., journals with a soc of ed audience other than Sociology of Education) but that are valued by sociology departments for hiring and/or tenure purposes?             

MullerAuthor: Chandra Muller, University of Texas

I’ll suggest a few guidelines to keep in mind when considering where to place a sociology of education article:

  1. Journals can change emphasis. Journals may shift in their emphasis or openness to articles in sociology of education, so it’s important to stay current about journals. For example, a change in editorship (e.g., sociologist of education, Pamela Quiroz, has just become co-editor of Social Problems) may signal a shift in the pool of reviewers, or the interests of the editors about what should be published. Oftentimes, new editors will write a statement about their goals or interests for the journal, providing an indicator about any shifts in emphasis.
  2. Developing your list of journals in sociology of education. Keep track of the journals that publish your cited references. Look at the more recent publications on CV’s of section members and colleagues who are working in sociology of education or on topics of interest to you. And, of course, reading the journals themselves is invaluable!
  3. Consider other sociology journals that may have an interest in your topic. Sociology of education has natural area overlap with many other sociological fields.  For instance, a paper on gender and STEM might be a good fit for Gender and Society; a paper on discipline and high school may good fit in Criminology; and, a paper on demographic trends in the relationship between race, family structure educational attainment might fit well in Demography. Each of these is the flagship journal of a professional association and potentially an excellent placement for an article.
  4. The definitions of “mid-tier” and “valued” are not uniform. Departments vary somewhat in what they consider an acceptable or valued journal.  Although indicators such as a journal’s impact scores are of questionable validity, they do provide some gauge of journal quality and people (like Deans) who are reviewing your CV may use them (along with acceptance rates) as indicators quality. Unfortunately, the hiring process can be unpredictable. Departments may have clearer expectations about journal quality required for tenure. Some hints about a department’s standards might be found on the CV’s of recently hired or tenured faculty.

Publishing in Education (or, Non-Sociology) Journals

Q: For sociologist of education, is it wise to publish in education journals?  Do sociology departments value these journals in the same way as they would sociology journals (i.e., for hiring and/or tenure)?

 Author: Chandra Muller, University of Texas Muller

If you think about it, almost every theorist you studied in your classical theory course conducted research on education (Weber, Durkheim, Marx, for instance) because of the centrality of education to sociology. However, sociology departments are typically composed of faculty members who have interests that may or may not be related to education.  This means a sociologist of education must establish the sociological legitimacy of his or her research to get hired and tenured in a sociology department. It is typically easier to do this with work published in sociology journals. Those who review your CV already understand differences in sociological journal quality and their impact. Further, the peer review process is likely to focus the content of your paper to be more sociological, even if it doesn’t start out that way. A criticism I’ve heard from sociology colleagues is that research in education journals may not be “theoretical” enough. Even if it is theoretical, it may be more difficult for your sociology audience to see the theoretical significance of articles in an education journal.

It is probably a bad idea to publish ONLY in education journals if you want to appeal to sociologists, broadly defined. That said, in many departments a mix of education and sociology journals may be acceptable. For the tenure process, it will be important to understand and fulfill your department’s expectations about journals. You must understand the “local culture” of your department, and thus you should ask your department chair (and your Dean) how they view publications in education journals. It may be necessary to “educate” administrators at your college/university about differences in quality among education journals. If you have an EEPA and AERJ on your record, you certainly want to be sure that your chair (and Dean) understand that scholars in your sub-field will see these as important strengths on your publication record.

If you have a mix of publications in sociology and education journals, you’ll want to make sure that the quality of sociology journals is reasonably high, as they may be used to gauge the overall quality of your research record. Ideally, you should try to publish in highly selective education journals as well, because it is easier to make the case that your “non-sociology” publications are high quality. Additionally, if you make theoretical connections between your sociology and education articles (maybe through citations or a research statement) then it may help people to better understand the significance and contributions of your education articles.

Responding to the Review(er)s

Douglas Downey

Q: How do you respond to reviewer comments when re-submitting a revised journal article (particularly if you disagree with a reviewer on something)?

Author: Doug Downey, The Ohio State University

Ten suggestions for handling reviewer comments:

1.  Write an angry, profanity-laced response to the editor about the lousy reviews you’ve just received.  Then tear this up.

2.  Distinguish between good and bad reviewer comments.  They can’t all be good—sometimes they contradict each other. Look to the editor’s letter for clues to help identify “good” reviewer comments that represent the central issue.

3. For reviewer comments that are simply off-base (e.g., they misread something), consider clarifying your text in a way that makes it very difficult for another reader to make the same mistake. Even though the reviewer was wrong, the fact that they misunderstood something is noteworthy.

4. Directly engage “good” reviewer comments.  This is where decent papers become great.  Articulate the issues they raise clearly and address their concerns head on in your manuscript.

5.  However, avoid “over revising.”  Papers need a central idea or selling point.  Make sure yours is clear by maintaining a central narrative.

6.  Appease reviewers by adjusting tone.  If I write that “the Reproductionist position regarding schools and inequality goes unchallenged,” I risk frustrating a reviewer who knows of some work that does challenge it.  I may be able to avoid this problem just by modifying the sentence slightly, “the Reproductionist position regarding schools and inequality goes virtually unchallenged.”   Or, I could consider using “largely” and be even safer.

7. For quantitative manuscripts, reviewers sometimes ask for changes in the models.  We all know that there are lots of decisions to be made when modeling (how to code variables, how to handle missing data, what covariates to include, how to weight the data, what dependent variables to predict, etc.) and that little adjustments here and there sometimes change the results.  Develop manuscripts around results that are robust to these different decisions.  If your argument depends heavily on a minor finding that only reaches statistical significance under a subset of modeling decisions, then you are setting yourself up for a difficult R&R.  And even if the paper does get published its likelihood of replication is low.

8.  If you can’t address a reviewers’ concern because of data limitations then admit it up front and make a case for why your manuscript still has value.

9.  A surprisingly large number of the papers I review have a basic problem—they lack a credible opposing view.  In other words, the research question lacks any controversy.  You can articulate the value of your manuscript most clearly to reviewers by highlighting the groups of scholars who view the issue at hand differently, and how your paper helps adjudicate among them.

10.  Consider contacting the editor for further advice.  Editors are sometimes willing to provide more insight via e-mail or phone conversations.

Q: Could you give any advice about how to write an effective R&R memo to an editor and reviewers? I want to address all the points made by the Editors and the Reviewers, but the memo is getting very long (and it is very time-consuming to write!) — is that a problem? What would you recommend?

03ff41bd-56c6-4ea4-9fc1-5ef9ac0830d4

Author: Bill Carbonaro, University of Notre Dame 

Doug does a great job of outlining the main features of an effective “R&R Memo” in his post. At a minimum, an effective memo demonstrates that the author has significantly improved his/her paper in response to the reviewers’ critiques and suggestions. After all — that is a major goal of the peer review process!

The question also specifically focuses on the length of the “R&R Memo,” which Doug did not comment on in his post. There is no “standard” length for an “R&R memo”- much depends on the substance of the reviews.  I have seen some reviews that are less than two single-spaced pages, and others that are nearly 10 single-spaced pages (with tables!). Some “Memos” are too long, and while others are too short. Here are a few guidelines to help keep your memo to “Goldilocks length” (“Just right!”):

  • Economize. It is unnecessary to describe EVERY single change that is made to the manuscript. It is preferable to simply say “I decided to adopt all of the minor changes suggested by Reviewer 2,” rather than describing every minor “bullet point” raised by that reviewer (e.g., “The author should change ‘STATA’ to ‘Stata.’).
  • But, Be Thorough. It is unwise to ignore specific comments and suggestions that reviewers make, particularly major ones. There is no better way to tick off a reviewer than to ignore his/her feedback! Good reviewers devote substantial time and thought to their reviews, and they want know that the author took their suggestions/critiques seriously. So – be sure that each reviewer sees that you have read, considered, and responded to the major points they made in their review of your work.
  • Prioritize.  Most of the text in  your “Memo” should be allocated to two main items: (1) justifications for suggested changes by the reviewers/editor that you decided AGAINST making, and (2) descriptions of supplementary analyses that you ultimately decided against including the paper. These are the make-or-break items that will likely determine whether the reviewers recommend to “reject” or “accept” your paper. So – be sure to devote adequate space to these crucial points in your letter.
  • Avoid redundancy. Always explain HOW and WHY major sections of your paper have changed, but avoid including the same revised sections of the paper in your letter. For example, it is sufficient to say: “As reviewer 1 requested, I incorporated ‘habitus’ into my theoretical framework to better motivate my research questions (see pp.4-5).”  The reviewers will (hopefully!) read both the paper and the letter, so it is unnecessary to make the same argument twice. 

Good luck!

Dealing with Rejection

At SEA, I had a good discussion with two junior scholars about the following question. We thought this would make a good blog entry.  Also — let us know how you handle this issue by responding to our first “Hidden Curriculum” poll! And — feel free (as always) to leave comments, especially if you disagree.

Q: When your paper is rejected, what do you do with the reviews? Do you revise the paper before sending it our again, and if so, how much revision is necessary?

Author: Bill Carbonaro, University of Notre Dame

03ff41bd-56c6-4ea4-9fc1-5ef9ac0830d4

There are two mistakes that authors typically make when their papers are rejected. The first is to ignore the reviews and resubmit the paper to a new journal without revisions. The second is to behave as if the paper received an R&R. Both approaches should be avoided, for different reasons.

Ignoring the reviews – or at least failing to make any revisions to the paper – is a perilous and shortsighted strategy. Authors should ask themselves: are you really so smart, and the reviewers so ignorant that none of their critiques and suggestions are worth addressing?  I appreciate the urgency in getting one’s work published (especially for untenured faculty and graduate students), given how long the review process takes. But, let’s face it: responding to feedback from reviewers usually makes our work better. By ignoring the reviews, authors are missing out on an opportunity to incorporate feedback that should improve one’s chances of getting his/her paper accepted upon submission to a new journal.

Also, it is not uncommon for a resubmitted paper to be sent to the same reviewers that originally evaluated the paper. It seems unlikely that such a reviewer would be generous or forgiving in his/her “re-review” if s/he finds that an author completely ignored his/her thoughtful suggestions in their initial review.  (Yes – I have been this reviewer, on more than one occasion.)

The second mistake — behaving as if the paper received an R&R – is more insidious. The author may have good intentions and genuinely want to improve his/her work. However, the problem with this is approach is that the author is being too deferential to the reviewers’ judgments.  Reviewers’ suggestions are not all equally good, and the author must do some serious sifting to determine which revisions should be incorporated in the paper. (See below.) Another problem with this approach is that it can take a great deal of time to fully respond to all of the reviewers’ comments. This is just wasted time that unnecessarily delays the ultimate publication of the paper.

Here is how I recommend proceeding after having one’s paper rejected:

First, take a deep breath, and read the reviews as soon as you get them.  Don’t delay. It’s disappointing to experience rejection but remember: the reviewers really want to help you make your paper better. Just get on with it!

Next, read the reviews and look for points of consistency and disagreement across them. If all three reviewers point out that the framing of your paper needs improvement, or that your models are mis-specified, then you should definitely address these concerns before resubmitting. Also – contemplate any differences of opinion about your paper. It is not uncommon to get conflicting evaluations and advice from reviewers. When this happens, you might consider getting a peer or your advisor/mentor to help you talk you through the problem. Finally, avoid the temptation to accentuate the positives in the reviews and dismiss the negatives. Remember – your goal is to improve your paper, so take all critiques and suggestions seriously.

After carefully reading the reviews, and giving them some serious thought, it’s time to make some decisions. Take three highlighters and assign a different color to each of three categories of comments: (1) a great idea; I will address it (green), (2) a judgment call; I may address it after some further thought (yellow), and (3) an unhelpful suggestion; I will ignore it (pink/red).

Based on the number of changes in categories one and two, set a firm deadline for resubmitting the paper. Prioritize this process and work fast.  I suggest a deadline of two to four weeks (depending on how much revision needs to be done).

Finally, remember: your goal is NOT to make the paper perfect before resubmitting. It never will be, so – let it go. It simply has to be good enough to get an R&R, and hopefully, your revisions in response to the reviewers’ feedback will help you get above the bar this time. Good luck! 

Appealing to the Editor?

Q: Is it proper (or advisable) to appeal an editor’s decision on a manuscript and, if so, under what circumstances? 

Author: David Bills, University of Iowa

 My first response to this question is that appeals to an editor to reverse a decision (and what “decision” means here is really “rejection,” since no one is going to appeal an acceptance) should be very, very rare.  But there are times when you should consider doing so.

Jilted authors should remember that top-drawer journals reject about 90 percent of the manuscripts that they receive.  This means that editors are going to reject some very good papers, papers that are theoretically tight, technically proficient, and nicely written.  But clearing these bars is not by itself enough to earn you a second look.  If you appeal on the grounds of “Hey, I didn’t make any mistakes,” you will get nowhere.  Editors are looking for papers that make significant and clear contributions to the literature.  They want to publish papers that will be cited a lot and that will move the discipline forward.

As an editor, I had to constantly ask myself “What 15-20 papers are mostly likely to have a lasting impact?”  These may not necessarily have been the 15-20 papers with the most sophisticated use of theory or the most impressive pyrotechnical wizardry. For an appeal of an editor’s decision to be successful, you need to convince the editor that your paper a) did not get a fair review AND b) that it stands to make a “top ten percent” contribution.

Every paper deserves a fair hearing.  But what counts as a fair hearing?  Reviewers make mistakes.  So do editors.  From an author’s perspective, these mistakes can be frustrating and infuriating.  Still, mistakes made by reviewers and editors, no matter how serious or neglectful or opinionated, only matter if they lead to a fundamental misrepresentation of your paper. Reviewers’ mistakes are consequential to the extent that they prevent the editor from seeing the potential contribution of your paper.  Similarly, an editor’s mistakes only matter if they prevent the publication of a paper that might have made a significant contribution.  If a reviewer misreads your tables or mangles your theory, you can point this out to the editor, but you can only expect this to carry the day if you can convince the editor that reading the tables properly or following your theoretical reasoning more adeptly would cast your paper in an entirely new light.  It’s not enough to play “Gotcha” with reviewers and editors. 

Much of the confusion over whether or not to appeal what you see as a wrong decision could be abated if reviewers would refrain from sending inappropriate signals to authors.  If you get a review that says your paper should be published, or revised and resubmitted, ignore this.  Listen to the editor, not to reviewers.  Don’t base your appeal on encouraging signs from the reviewers.

The takeaway from all this?  If reviewers or editors misread or wrongly criticize your paper, by all means let the editor know.  If reviewers or editors misread or wrongly criticize your paper AND you think you can persuade the editor that these failures are depriving the journal of a potential “top ten” paper, then and only then should you think about an appeal.

Recommending reviewers…?

Q: When I submit a paper to a journal, should I recommend possible reviewers to the editor?

Author: Bill Carbonaro, University of Notre Dame

I think the short answer is “yes, by all means!” The longer answer is a bit more complicated and I’m guessing that opinions vary.
From an author’s perspective, recommending reviewers to an editor makes good sense. Ideally, we all want knowledgeable, fair, and conscientious reviewers to read and respond to our work. In particular, we always want reviewers who are a good “fit” with the topic and methods used in our papers. Nothing is more frustrating than getting a review of one’s paper where the reviewer (a) isn’t an expert in the area that you are studying, and/or (b) doesn’t understand the data/methods that you are using. Perhaps I’m being idealistic here, but I think most of us actually want reviewers who can help us improve our work! After all, that is one important part of the review process, isn’t it? Good reviewers should evenhandedly judge the quality of work, while also providing useful and constructive suggestions for improvement. If you know scholars who the “fit the bill,” then by all means – tell the editor who these people are!
Having said this, it can be tempting to “stack the deck” with friendly reviewers (or just plain “friends!), who you know will like and support your work. (E.g., “my research finds strong support for theory X – please send my paper for review to the three main proponents of theory X.”) If you are rather blatant about this, it may make you look bad in the eyes of the editor, and – maybe s/he might be less likely to acquiesce to your request, and avoid your recommendations altogether.
I will also recommend against the following practice: asking a faculty member to “pre-review” your paper and then, after making revisions based on his/her feedback, recommending that person as a reviewer for your paper! I have had this happen on more than one occasion, and perhaps it was just a coincidence. But – personally, I can’t help but feel that I’ve “been had” when this happens. In fact, in these cases, I decline the request to serve as a reviewer. From my perspective, I’ve already affected the paper, and it’s time for someone else to weigh in and evaluate the work.
Now – I must confess that I have never asked an editor how they feel about this practice. I would think that, from an editor’s perspective, it certainly makes life easier, since it must be challenging to think of three or more reviewers who would be a good fit for every paper they receive. It has to be particularly useful for ASR and AJS, where editors are getting papers from many subfields with which they are quite unfamiliar. I’m guessing that most editors pick one or two people that they trust from the author’s suggested list, and then pick a few reviewers who are not on the list.
So – in short, it is advisable to recommend a few (two or three?) reviewers whose opinions and feedback you respect and look forward to receiving. It will probably increase the chances that you will get high quality reviews of your work – and, in an imperfect world, I think that about all that any of us can ask for.