Publishing “Non-findings”

Q: What should one do with papers that largely consist of non-findings? Is it just a waste of time trying publish a study without significant effects? Or, should I simply aim for lower quality journals with such papers?

Author: Bill Carbonaro, University of Notre Dame03ff41bd-56c6-4ea4-9fc1-5ef9ac0830d4

This is an important question that we must all confront as researchers. Not every study works out as we anticipated. No one wants to waste time on a project that has little chance of success. However, I think it is a myth that good journals only publish papers with statistically significant results.  Yes, there is a bias in favor of publishing papers with statistically significant findings, but there are many great examples of papers published in top journals that are built around “non-findings.” Here are the lessons that I have learned from these papers:

Characteristics of successful papers with “non-findings”

1. Have a good theory. For starters, it is important to have a well-grounded theoretical explanation for why one would either expect, or not expect, to find a relationship between the variables in your analysis. If your paper doesn’t find a relationship between X and Y, and the reader doesn’t have a clear idea of why we would expect that X should (or should not) covary with Y, then your paper is doomed. Ainsworth-Darnell & Downey (AD&D)(ASR, 1998) relied upon Ogbu’s much cited “oppositional culture” framework to generate several hypotheses about black-white differences in attitudes, behaviors, and school performance. They didn’t find any support for the Ogbu’s hypotheses, but that’s what makes the paper so interesting (and one reason the paper is so often cited!).

2. Cover new ground. The best case scenario is that one has a theory that has not been tested, and there is a “research vacuum” to exploit. AD&D covered new ground by being the first paper to systematically evaluate Ogbu’s claims with a multivariate analysis of nationally representative data. Their “non-findings” filled an important void in the field. In separate studies, Hallinan & Kubitschek  and I examined sector differences in learning among elementary school students, and found either no effects, or, positive public school effects. This went against the grain of prior research on the Catholic school advantage in high school (see point 3 below), but it also exploited the “research vacuum” in this area.

3. Re-visit old ground. Non-findings are especially interesting when they go against the grain of prior research. If prior research consistently finds that X matters for Y, and you find that X doesn’t matter for Y, that’s interesting!  The big question is of course WHY you didn’t find that X matters for Y. If you used different data than other scholars, you need to show why your data are superior in quality to prior research. If your data are more recent, then you have a very good story to tell: X used to matter for Y, but it no longer does. Of course, you need a good explanation for why X no long matters for Y. Think of Wilson’s The Declining Significance of Race and The Truly Disadvantaged. Much debated works, but they are compelling and provocative narratives! Finally, along the lines of points 4 and 5 below, you have an interesting paper if you can show that, in contrast with prior research, your use of higher quality measures and/or more rigorous methods indicates that X doesn’t matter for Y.

4. Have impeccable measures. If your analysis has poor measures of your key concepts, it shouldn’t be surprising that you have insignificant findings in your analyses. In order for a “non-findings” paper to be compelling, the reader should be convinced that the measures are credible, with high validity and reliability. This is a common problem in “non-findings” papers that I reject as a reviewer. AD&D used multiple measures of “oppositional culture” in their study, and while one might prefer better measures, their measures were credible enough to make a significant contribution to the field.

5. Have a rigorous (and possibly novel) research design. Happily, the bar of what counts as “rigorous research” is constantly being raised. “Non-findings” are especially important when one can make a convincing argument that prior research findings are in fact “artifacts” of weak research design/methodology.   For example, Guo and VanWey (ASR, 1999) used fixed effects models (rather than cross-sectional, between family models) to examine sibling effects on achievement, and found no effects of siblings. This study has been much debated, but it was compelling and innovative enough to merit publication in a top journal. Mouw (ASR, 2003) re-examined research on network ties and job searches using substantially more rigorous empirical models than prior research, and he found that job contacts did not affect  job search outcomes. In each of these cases, reviewers were obviously convinced enough by the results (based on the rigor of the design and analysis) to deem the paper publishable.

To summarize, don’t immediately despair if your paper is filled with “non-findings.”  Some “non-findings” papers should be abandoned, but typically, that is because they are not very good papers to begin with! Under the right circumstances, “non-findings” are actually “FINDINGS” that can be published in top journals, and may have an important impact in your subfield.

Advertisements

Placements in “Mid-Tier” Journals

Q: What would be considered “mid-tier” sociology of education journals (i.e., journals with a soc of ed audience other than Sociology of Education) but that are valued by sociology departments for hiring and/or tenure purposes?             

MullerAuthor: Chandra Muller, University of Texas

I’ll suggest a few guidelines to keep in mind when considering where to place a sociology of education article:

  1. Journals can change emphasis. Journals may shift in their emphasis or openness to articles in sociology of education, so it’s important to stay current about journals. For example, a change in editorship (e.g., sociologist of education, Pamela Quiroz, has just become co-editor of Social Problems) may signal a shift in the pool of reviewers, or the interests of the editors about what should be published. Oftentimes, new editors will write a statement about their goals or interests for the journal, providing an indicator about any shifts in emphasis.
  2. Developing your list of journals in sociology of education. Keep track of the journals that publish your cited references. Look at the more recent publications on CV’s of section members and colleagues who are working in sociology of education or on topics of interest to you. And, of course, reading the journals themselves is invaluable!
  3. Consider other sociology journals that may have an interest in your topic. Sociology of education has natural area overlap with many other sociological fields.  For instance, a paper on gender and STEM might be a good fit for Gender and Society; a paper on discipline and high school may good fit in Criminology; and, a paper on demographic trends in the relationship between race, family structure educational attainment might fit well in Demography. Each of these is the flagship journal of a professional association and potentially an excellent placement for an article.
  4. The definitions of “mid-tier” and “valued” are not uniform. Departments vary somewhat in what they consider an acceptable or valued journal.  Although indicators such as a journal’s impact scores are of questionable validity, they do provide some gauge of journal quality and people (like Deans) who are reviewing your CV may use them (along with acceptance rates) as indicators quality. Unfortunately, the hiring process can be unpredictable. Departments may have clearer expectations about journal quality required for tenure. Some hints about a department’s standards might be found on the CV’s of recently hired or tenured faculty.

Finding the Right Mentor – and Getting the Most from Working with Him/Her

ImageQ: What advice would you give to graduate students about identifying mentors and making the most of those experiences?

I am fortunate to have a terrific colleague who studies mentoring issues in graduate education, Jessica Collett. I asked Jessica to address a follow-up question in her post. – BC

Since you have been doing work on gender differences in mentoring received by women and men, do you think there is anything graduate students themselves can do to help bridge those gaps, or is that on the faculty?

Author: Jessica Collett, University of Notre Dame

Before I begin, I want to clarify that I see a difference between advisors and mentors. When I use this distinction, I imagine an advisor as someone who advises or directs. They work in the same area you do and help you identify interesting questions and relevant literatures and expose you to networks and norms in your specific area. A mentor, on the other hand, is a coach or a guide. They don’t need to work in the same area as you. They provide socialization, professionalization, and support through grad school and hopefully into your career. Many of us had advisor-mentors (people who served both roles in our development as scholars), but I try to remind students that these people do not have to be one in the same.

The first rule for identifying either a mentor or advisor is find someone who you respect. They should be someone who you think does good work, treats people well, and who you trust with your ideas. This respect and basic level of trust serve as the foundation for an honest relationship, a foundation that you can build on as time goes on and relates to the second rule: find someone who you can communicate with. Although this is ideally the case for either an advisor or a mentor, it is important that there is at least someone who you can be open with. Communication is at the heart of a successful mentoring or advising relationship. Many of the graduate students in my research who had decided not to pursue research-intensive, tenure-track positions had not yet shared their actual career goals with any faculty members. As a result, most of these students were woefully under-prepared for the positions that they did want. If no one knew that they wanted to focus on teaching, it was much less likely they would get teaching-assistantships or their own classes. Others struggled with communicating areas where they felt weak (e.g., grant-writing, statistics). This lack of open communication both stemmed from and contributed to concerns about inauthenticity and impostorism and negatively affected many students’ graduate careers.

Many of the students who I spoke with chose mentors based on superficial attributes, particularly gender and family status. Some did this because they were looking for people “like them.” They assumed that women would be better advisors for women, that people with kids would be better mentors for people with children, and so forth. Others did this drawing on particular stereotypes, assuming that women would be more supportive and nurturing, men would push them harder, older people would be better connected, young people hungrier to publish. Either way, the results seldom were as students hoped. Mentoring is an incredibly personal thing and personalities are much more important than characteristics in determining how well a faculty member and student will be able to work together. The first step is figuring out your own personality and what you want from the relationship. There is even a checklist in this guide to help you do that (and a companion guide you can share with faculty).  Having an open and honest relationship, with good communication and shared expectations, will enhance the mentoring you get. It is bringing all the qualities of interaction that those informal encounters bring to your own mentoring relationship, regardless of whether it occurs in an office or a soccer field.

My last piece of advice about student-faculty relationships is to be the student a faculty member wants to work with. This is not a buyer’s market, where you just select who you would most like to work with and you’re done. This is a dance and you need a partner. You want to have your pick of the faculty. Even before you set out to find an advisor or mentor, work hard in classes, be a good departmental citizen, take initiative as a graduate assistant, and so forth. It will make all of the above much easier. Good luck!

What’s Next – and Some Rapid-Fire Q&A

Hidden Curriculum LogoThanks to everyone for the recent surge of excellent questions! We are doing our best to get answers from knowledgeable, experienced academics who deserve our undying gratitude for their willingness to help make this blog a success. We have some big plans for the future.  Here is a preview of what we have planned for the next few months:

 

  • How to prepare for and be successful on the non-academic job market.
  • A series on how to maintain “work/life” balance in academic life. (We take the position that, yes, it IS possible!)
  • Advice on how to publish “non-U.S.” research in American sociology journals.
  • Tips on being a good discussant for a paper session (as a grad student) (who is asked to comment on a senior scholar’s work).
  • How to find a good dissertation topic.
  • Also — crowd-sourcing questions we have no good answers to!

For this week, we thought we would provide some quick answers to some recent queries posed by readers:

Q: I’ll be teaching my first class next semester and I can’t stop worrying about what my students should call me. I’m not technically a professor yet, but I’m also concerned that letting them call me by my first name will result in a less respect, which I’m already worried about as a young female grad student. And having them call me “Ms. (Last Name)” sounds strange as well. What is a reasonable way for undergraduate students to refer to graduate student professors?

The Hidden Curriculum agrees that it’s risky for grad students to allow undergraduates to be on a first name basis with you. If your students go around calling their other instructors “Professor Jones,” and they call you “Katie,” it seems likely your authority in the classroom will be undermined. We agree that “Ms. Smith” makes you sound like a substitute teacher. Or, maybe your mom. Might we suggest that you simply go with “Professor Smith?” It’s true – you aren’t really an “official” Professor but we suspect that if you quiz your students about the distinctions between adjunct, visiting, assistant, associate, and full professors, they will have NO idea what such distinctions mean. They mostly think that “Professors” are people who teach college classes. That’s you! So, Professor Smith – get to work on that syllabus!

Q: Why do almost all hiring committees keep candidates in the dark month after month after month? Haven’t we all been through the torture of not knowing what happened to our job application? Surely it can’t be that difficult to send quick updates to candidates to fill them in on when the committee is meeting and when various decisions have been/will be made.

Q: When is it appropriate for a job candidate to contact the search chair for an update on his/her application and/or the status of the search?

The Hidden Curriculum agrees that this lack of transparency is annoying, and even dispiriting. Some departments are much better at communicating with job applicants than others. As best we can tell, it doesn’t have much to do with the size, prestige, or institutional profile (R-1, R-2, T-1, etc.) of the department. Having said that, it is not uncommon for top departments to receive hundreds of applications for one position. Given that, it’s not surprising that many departments take a “don’t call us, we’ll call you” orientation. However – it is unconscionable for departments to fail to maintain regular contact with applicants who have interviewed for a given position. When that happens, people have very good reason to be irate and feel mistreated.

Regarding when it is appropriate to contact a search chair, the best time is (of course) after you have some interviews lined up. “Just checking in to let you know that my interview schedule is filling up, and I want to know if your department might be in the mix.”  Obviously, there is an ulterior motive here, and that’s perfectly fine. If you are not in such a fortunate position, and you want to “check in,” it is important to recognize that the pace of hiring in most departments is glacial. So, wait at least six weeks after submitting your materials.  However – be forewarned: there is an informal norm against “checking in” unless you have some news to report. Departments may perceive such inquires negatively.

Q: Rory McVeigh recently wrote on this blog that “hiring committees are looking for signs that the candidate will be actively involved in the intellectual life of the department,” but at a recent junior faculty mentoring session senior faculty told us “don’t volunteer all the time” and “find committees that require less work” (both of which I’d previously heard in various forms). Besides worrying about the troubling implications for the University if faculty actively avoid service, I’m confused about what junior faculty should actually do in this area. Are these statements actually conflicting? Does it only matter what one does to help his/her department and not the University? Or are there differing opinions?

Rory follows the above quote with:  “I am not talking here about getting bogged down in departmental service (we try to protect assistant professors from that).  Instead, I am thinking in terms of someone whose presence is felt in the department from day 1—rather than someone who hides in her/his office or works from home and only shows up on campus when teaching a class.”  We agree! Departments want to hire real live people, not name-plates on a door, or pictures on a web page.

Regarding the larger issue of service commitments, the advice about “not volunteering”  and finding “light” committee assignments is eminently sound. Good departments should protect their junior faculty from service obligations that prevent them from building a strong research and teaching record as they work towards tenure.  Typically, there is a post-tenure expectation that one’s service load will increase. Of course, it is important to be realistic: if you are joining a department with only three other faculty members, or one with almost no senior faculty, it will be hard to avoid significant service duties as a junior person. We strongly recommend job candidates ask about “service load” expectations in their conversations with junior and senior faculty when interviewing. Eyes wide open!

Writing Good Reviews

Q: “How do I write a good review for a journal? I don’t want to use the reviews I have received as an author because many are completely unstructured. ”Portrait: John Robert 'Rob' Warren

Author: Rob Warren, University of Minnesota

Current Editor of Sociology of Education

 

Everybody hates bad article reviews.  Editors, authors, even the other reviewers.  In this post I describe what I think makes a good article review, and I offer a few practical pointers for beginners.

Remember the manifest purpose.  There are lots of good reasons to agree to review an article—to find out about state of the art research; to contribute to the collective good;  to help shape the area, etc.—but don’t lose sight of the fact that what you are mainly doing is making a recommendation to the Editor.  Your review is, first and foremost, a letter to the Editor in which you make a recommendation about what decision your think she or he should make.

Whatever else you put in your review, be sure that you are very clear in justifying your recommendation.  And, make sure that the specific recommendation you make to the Editor (i.e., accept, reject, invite a revision and re-submission) matches the tone and content of the review.  Imagine a movie review that gives a new film 2 out of 5 stars—but does not explain what’s wrong with the film.  Or, imagine a movie review that gives a new film 5 out of 5 stars but then goes on to trash the plot, the script, the acting, and the special effects.  Neither movie review would be very helpful.  The Editor has to make a decision based on your recommendation.  Telling the Editor to “accept” the paper and then writing two pages about flaws in the research design puts the Editor in a tough spot.  So does recommending that the Editor “reject” the paper when your review contains little by way of critique.

Remember the “Golden Rule.” The manifest purpose of an article review is to provide a recommendation to the Editor about what to decide about a manuscript.  But we all know that the peer review process can be a major avenue for improving our research.  Reviewers have read things we haven’t read, they have theoretical perspectives that may be useful to us, they know about data or measures we don’t already know about, they are familiar with new methods we haven’t heard of, and they may interpret our results in surprising and constructive new ways.   As an article reviewer, you can make a huge difference in helping someone to improve their research—and thus the state of knowledge in the field.

As an author, you want reviewers to provide lots of thoughtful, constructive feedback based on a close and engaged reading of your manuscript.  Don’t be a free rider!  Be the reviewer you wish would read your next submission.  Take the time to be helpful.  Mention the new book or article that the author might not know about (and provide the citation).  Carefully explain why the author’s methodology is in need of improvement.  Engage in the theoretical discussion.  Challenge the author to think things through more carefully.  Etc.  Show the author the respect you would like to receive.

Timeliness is better than godliness.  A Pulitzer Prize-worthy review is useless if you send it in weeks (or months) after the journal’s deadline.  When you are invited to review an article, the invitation will usually say specifically how quickly the Editor hopes to receive your evaluation.  If you know you cannot stick to that time frame, then decline the invitation to review—and provide several possible alternate reviewers.  If you agree to review the manuscript, do it by the deadline.  Again, be the reviewer you wish would read your next manuscript.  Return reviews as quickly as you wish everyone else would.

Structure your review strategically.  A long, meandering, unstructured review is not very helpful.  Be clear, concise, and to the point.  Emphasize points by putting them first.  Here is how I wish every review were structured:

  1. Start by summarize the author’s research question, theoretical perspective, data, research design, and major findings.  Do it in just a few sentences, and don’t make any evaluative statements.  This tells the author you read the paper and understood what they were trying to say. (Length: 1 short paragraph)
  2. Provide an overview of your evaluation of the manuscript.  Many Editors will say not to include your specific recommendation, but others disagree.  Say, overall, what you think of the research and the manuscript.  (Length: 1 very short paragraph)
  3. Say what you like about the manuscript.  This is helpful to the Editor, but it also cushions the blow for the author since everything after this section will be more negative.  (Length: 1 short paragraph)
  4. Describe a few major weakness or limitations—and in declining order of importance.  If there is fatal flaw, lead with that.  Don’t list problems in the order they occurred to you.  List them in the order of how serious they are. (Length: Depends.  Maybe a page?)
  5. Describe a few less serious problems or issues.  These are things that are not “make or break” issues, but that you’d like to mention by way of being helpful and constructive.  (Length: Depends.  Maybe a long paragraph?)

If you add it all up, that comes to about two pages.  However, the Editor and author will know after less than a page whether (a) you read/understood the article; (b) what you basically think of it; and (c) what you see as its major strengths and weaknesses.

Don’t pretend to be expert on everything.  If you know a lot about the substantive area or empirical topic, but nothing about the methodology, that’s OK.  Likewise, if you know a lot about the methodology but nothing about the theoretical framework or empirical subject, that’s OK.  Comment on those areas about which you are expert.  Trust that the Editor will enlist other reviewers to comment on other aspects of the manuscript.

It’s not about you.   Evaluate the paper in front of you, not the one you wish was in front of you—or the one you would like to write.

Don’t overdo it.  Unless there is some exceptional circumstance, I do not think people should spend more than about 4 hours on an article review.  I recommend taking two hours to read the manuscript and take notes, and then another two hours to formulate and draft the review.  Or less. If you let it take longer, you will resent it—and thus be more likely to decline to review the next manuscript sent to you.  On average, you should expect to review three times as many manuscripts as you submit.  Make sure you review manuscripts well, but in a sustainable way.

They have people for that.  You do not need to comment on grammar, spelling, or smaller matters of style (unless the manuscript is so badly written that you feel compelled to make it a core issue).  Even in this low budget era, journals usually have copy editors.   You also do not need to tell authors how to format their references, tables, or figures.  Most journals contract with someone to do those things.  Spend your precious time and energy on the central task at hand: Making a recommendation to the Editor and providing constructive feedback that will help the author improve their research.

Publishing in Education (or, Non-Sociology) Journals

Q: For sociologist of education, is it wise to publish in education journals?  Do sociology departments value these journals in the same way as they would sociology journals (i.e., for hiring and/or tenure)?

 Author: Chandra Muller, University of Texas Muller

If you think about it, almost every theorist you studied in your classical theory course conducted research on education (Weber, Durkheim, Marx, for instance) because of the centrality of education to sociology. However, sociology departments are typically composed of faculty members who have interests that may or may not be related to education.  This means a sociologist of education must establish the sociological legitimacy of his or her research to get hired and tenured in a sociology department. It is typically easier to do this with work published in sociology journals. Those who review your CV already understand differences in sociological journal quality and their impact. Further, the peer review process is likely to focus the content of your paper to be more sociological, even if it doesn’t start out that way. A criticism I’ve heard from sociology colleagues is that research in education journals may not be “theoretical” enough. Even if it is theoretical, it may be more difficult for your sociology audience to see the theoretical significance of articles in an education journal.

It is probably a bad idea to publish ONLY in education journals if you want to appeal to sociologists, broadly defined. That said, in many departments a mix of education and sociology journals may be acceptable. For the tenure process, it will be important to understand and fulfill your department’s expectations about journals. You must understand the “local culture” of your department, and thus you should ask your department chair (and your Dean) how they view publications in education journals. It may be necessary to “educate” administrators at your college/university about differences in quality among education journals. If you have an EEPA and AERJ on your record, you certainly want to be sure that your chair (and Dean) understand that scholars in your sub-field will see these as important strengths on your publication record.

If you have a mix of publications in sociology and education journals, you’ll want to make sure that the quality of sociology journals is reasonably high, as they may be used to gauge the overall quality of your research record. Ideally, you should try to publish in highly selective education journals as well, because it is easier to make the case that your “non-sociology” publications are high quality. Additionally, if you make theoretical connections between your sociology and education articles (maybe through citations or a research statement) then it may help people to better understand the significance and contributions of your education articles.