Peer Review and Social Pyschology: Or Why Introductions are so Important!

Inspired by my colleagues Loren King and Anna Esselment, both of whom regularly make time in their busy schedules to read (I know! A crazy concept!), I’ve started to read a new book that Chris Cochrane recommended: Jonathan Haidt’s The Righteous Mind: Why Good People Are Divided By Politics and Religion.

I’m only in the first third of the book, but one of the main arguments so far is that when human make moral (and presumably other) judgements, we tend to use our intuitions first, and our reasoning second. That is to say, frequently we have gut feelings about all sorts of things and rather than reasoning out whether our feelings are correct, we instead search for logic, examples, or arguments to support those gut feelings. Haidt effectively illustrates this argument by drawing upon a broad set of published research and experiments he has done over the years.

At the end of chapter 2, he writes:

Continue reading

“I have tried to use intuitionism while writing this book. My goal is to change the way a diverse group of readers … think about morality, politics, religion, and each other …. I couldn’t just lay out the theory in chapter 1 and then ask readers to reserve judgement until I had presented all of the supporting evidence. Rather, I decided to weave together the history of moral psychology and my own personal story to create a sense of movement from rationalism to intuitionism. I threw in historical anecdotes, quotations from the ancients, and praise of a few visionaries. I set up metaphors (such as the rider and the elephant) that will recur throughout the book. I did these things in order to “tune up” your intuitions about moral psychology. If I have failed and you have a visceral dislike of intuitionism or of me, then no amount of evidence I could present will convince you that intuitionism is correct. But if you now feel an intuitive sense that intuitionism might be true, then let’s keep going.”

I found these first few chapters, and this paragraph in particular, to be extremely powerful and relevant to academic publishing (and other things!). If humans tend to behave in this manner, (e.g. we frequently rely on gut feelings to make moral judgements and we frequently try to find reasons to support those feelings), then the introduction of a journal article is CRUCIAL, both for peer review and afterwards. On the issue of peer review, I can’t tell you how many times I’ve received a referee report that was extremely negative, yet failed to: a) clearly show that they understood my argument; and b) demonstrate logically why my argument is wrong. I always blamed myself for not being clear enough, which is probably half true! But the real story is that sometimes my introductions were probably ineffective at connecting with people’s intuitions, and so these reviewers found reasons to reject it.

The lesson here, I think, is that introductions matter! You can’t ask or expect readers to withold judgement while you present the theory and evidence first. Instead, you have to find a way to tap immediately into their intuitions to make them open to considering the merits of your argument.

Professors, Elections Canada, and the Harper Government: Do Group Op-Eds Matter?

Early last week, the National Post ran a letter drafted by a small group of Canadian university professors (you can tell who the original letter writers by the order. Just look at where the alphabetized list starts and any names above it were the main writers), and signed (and edited) by a larger number of professors (including some of the most distinguished, senior, and smartest academic minds in Canada).

Several days before the letter ran in the Post, a draft of the letter hit my desk asking for my signature. Ultimately, I didn’t sign for a number of reasons. The main reason I didn’t sign was that I didn’t agree with all of the contents in the letter. There are parts of the “Fair Elections” bill that I agree with, other parts that I didn’t, and other parts that I simply wasn’t sure about. The timeline for signing the letter was tight and didn’t really give me much time to think these issues through.

Continue reading

Another reason why I didn’t sign was because, quite frankly, I’m not sure how effective these types of op eds are. If you have a chance, you should check out the comments left on the National Post page about the letter. To be succinct, they are nasty! There’s all sorts of anti-elitist rhetoric about professors being overpaid and narrow-minded (with some anonymous commentators regaling readers about their bad experiences at universities). Others claimed that professors are all Liberal-NDP supporters, or are at least ideologically aligned with those political parties. We are also experts with no real knowledge, apparently, who hide behind our PhDs and “peer review” to stifle dissent, etc.

To be fair, there were some defenders of professors (thank you!), remarking about the important expertise and knowledge that the profession have to offer. But the dominant discourse was negative.

In my view, it seems group letters bring out a certain amount of dissent towards the profession. Have a look at op-eds written by only one or two professors at a time; occasionally, you do see one or two anti-professor rants, but rarely very many. When we write these group letters, however, there seems to be many of these comments. So, do group-think letters actually help or hinder our ability to communicate with and affect public opinion and public policy?

In terms of public opinion, my sense is that these letters are not effective. Recent research suggests that most people have gut feelings about various issues or things, whether it be politics, religion or food, and then they search for justification for their gut reactions. (Frequently, I’ve been wondering whether the same phenomenon is at work in academic peer review?!) And so I think op-eds are built exactly for this type of behaviour. People don’t change their minds because of op-eds. They use them to feed their gut reactions.

How about public policy? Do these op-eds affect public policy? Well, I’m pretty sure Harper won’t be convinced by these types of letters but I do know that civil servants, including deputy ministers, scan op-ed pages for ideas. So maybe these letters will ultimately push civil servants to act in a way to thwart these reforms?

For me personally, I have yet to sign one of these group-think letters. Instead I’ve treated the op-eds that I write (and sign) as knowledge dissemination tools, trying to link current events either to my research, or the research of others.

But maybe I should have signed! I’ll guess we’ll see when the next letter gets circulated.

More on Knowledge Mobilization: What About Individual Citizens?

There’s been some recent discussion on this blog about the importance of knowledge mobilization. Dr. Erin Tolley, for instance, provided some excellent advice several days ago based on her own experiences in government and academia. But recently I’ve been wondering: what can us academics do to better share our research findings with regular citizens?

My usual strategy has been to write op eds in regional or national newspapers. I have no idea whether this is an effective strategy. I have a hunch that op eds rarely persuade but instead simply reinforce people’s existing opinions on the issue (one day I’d like to run an experimental study to test this proposition. I just need to convince my colleague, Jason Roy, to do it!) Sometimes, I receive emails from interested citizens or former politicians. In one op ed, published in the Toronto Star, I briefly mentioned the Kelowna Accord, dismissing it as a failure. The day after the op ed was published, former Prime Minister Paul Martin called me up in my office to tell me why my analysis of the Accord was wrong. That was quite the experience!

Continue reading

But on the issue of communicating research results to interested citizens, I wonder if there is more that I can do? At least once every six months, I receive an email from a random First Nation citizen asking for advice. Usually, the questions they send me focus on the rights of individual band members against the actions of the band council. One email I received, for instance, asked about potential legal avenues that were available to members for holding band council members accountable, because somehow they saw my paper in Canadian Public Administration on accountability and transparency regimes. Just last week, I received an email from a band member who was fielding questions from fellow band members about the rights of CP holders (e.g. certificates of possession) against a band council that wanted to expropriate their lands for economic development. Apparently, this individual tried to look up my work online but all of the articles I’ve written on this topic are gated (with the exception of one).

So, what to do?

Well, one easy and obvious solution is to purchase open access rights for these articles, which is something SSHRC is moving towards anyway. That way, anyone can download and read the articles.

But what else can we do? Taking a page from Dr. Tolley’s post, maybe I need to start writing one page summaries of my findings in plain language and post them on my website?

Another thing I want to try is to put together some short animated videos that explain my findings. This is what I hope to do with my SSHRC project on First Nation-municipal relations, if Jen and I can ever get this project finished!

Any other ideas? Suggestions welcome!

Knowledge Mobilization and the Academy: A Guest Post from Dr. Erin Tolley

Below is an excellent guest post on knowledge mobilization from Dr. Erin Tolley, Assistant Professor in the Department of Political Science at the University of Toronto, Mississauga. It provides some very practical and timely advice for those of us filling in the knowledge mobilization section of our SSHRC grants!

Enjoy!

A few weeks ago, Chris Alcantara wrote a great post about knowledge mobilization and the communication of research results to non-academic audiences. In his post and the comments that followed, Chris raised a number of questions about how best to facilitate knowledge transfer and asked, in particular, if we need a “rethink” of traditional modes of communicating research results.

Continue reading

I would come down firmly on the side of yes, particularly if your aim is to engage government decision-makers and policy analysts. This is true now more than ever, with austerity measures pinching bureaucrats’ time, and budgets for training and travel having been all but gutted. Whereas many government departments once maintained their own in-house libraries, these have largely been shuttered leaving policy analysts without any reference support and no access to the scholarly books and gated journals to which most academics direct their publishing efforts. With luck, an enterprising policy analyst might be able to direct some government resources toward the purchase of policy-relevant research publications, but this is rare. Even rarer would be the opportunity to attend a conference outside the bounds of the National Capital Region to hear academics present their research. That said, in the mid-level of the public service, there are increasing levels of education. Most new policy analysts now have at least a Master’s degree, and many have PhDs. Given this, there is an appetite for research, in addition to the skills and qualifications to understand and apply those insights.

This is not, then, a matter of “dumbing down” but rather of communicating the findings in ways that speak to the target audience. Long discussions about the theoretical framework or the extant literature will not interest most policy analysts. Too much methodological detail will bore or distract. No tables of regression coefficients! No p-values! Tell your audience which parts of your research are significant and really matter. If they would like more details, they will ask.

What will interest this audience most is the policy relevance and implications of your research. What do your findings tell policy-makers about their policy area? Where are the policy gaps? What are the most fruitful areas for action? (Hint: this should be something other than “More research is needed”).

To do this effectively, you need to know your audience. Read the legislation or policies that are most relevant to your field of study. Consult recent Standing Committee reports or other parliamentary publications. Take a look at the media releases from the government departments most centrally connected to your work. Read the department’s most recent Report on Plans and Priorities. Search the Government Employee Directory (GEDS) for the names of policy analysts who work in your field. Contact them. Talk to them. Ask them questions about what they do. Use this to inform your research.

When you communicate your results, remember the constraints that policy analysts face: limited dedicated reading time, tight deadlines, and a need for concise communication. Can you put your results into a “2-pager” that gives a brief synopsis of your work, your main findings and their policy relevance? CERIS has an excellent template. Include your email address. Send it to policy analysts working in your field. Post it on your personal webpage or in any other “Googleable” format. Write an op-ed about your work. Maintain a social media presence.

Contact one of your policy contacts and ask if they would be interested in having you present your work to their colleagues. Most departments have a “Brown Bag” lunch series, and they’re generally quite happy to host researchers with relevant new findings. Make the most of these opportunities when they arise. Don’t present a conference paper; policy analysts prefer PowerPoint or a handout. Provide it in advance. Make contacts once you’re there. Follow-up with them. And don’t ask—even jokingly—if they will give you money for your research. You can save that for the second date.

Erin Tolley is an assistant professor of Political Science at the University of Toronto. Prior to pursing doctoral studies, she worked for nearly a decade in the federal government.

Should We Change the Grant Adjudication Process? Part 2!

Previously, I blogged about the need to reconsider how we adjudicate research grant competitions.

Others agree:

Researchers propose alternative way to allocate science funding

HEIDELBERG, 8 January 2014 – Researchers in the United States have suggested an alternative way to allocate science funding. The method, which is described in EMBO reports, depends on a collective distribution of funding by the scientific community, requires only a fraction of the costs associated with the traditional peer review of grant proposals and, according to the authors, may yield comparable or even better results.
Continue reading

“Peer review of scientific proposals and grants has served science very well for decades. However, there is a strong sense in the scientific community that things could be improved,” said Johan Bollen, professor and lead author of the study from the School of Informatics and Computing at Indiana University. “Our most productive researchers invest an increasing amount of time, energy, and effort into writing and reviewing research proposals, most of which do not get funded. That time could be spent performing the proposed research in the first place.” He added: “Our proposal does not just save time and money but also encourages innovation.”

The new approach is possible due to recent advances in mathematics and computer technologies. The system involves giving all scientists an annual, unconditional fixed amount of funding to conduct their research. All funded scientists are, however, obliged to donate a fixed percentage of all of the funding that they previously received to other researchers. As a result, the funding circulates through the community, converging on researchers that are expected to make the best use of it. “Our alternative funding system is inspired by the mathematical models used to search the internet for relevant information,” said Bollen. “The decentralized funding model uses the wisdom of the entire scientific community to determine a fair distribution of funding.”

The authors believe that this system can lead to sophisticated behavior at a global level. It would certainly liberate researchers from the time-consuming process of submitting and reviewing project proposals, but could also reduce the uncertainty associated with funding cycles, give researchers much greater flexibility, and allow the community to fund risky but high-reward projects that existing funding systems may overlook.

“You could think of it as a Google-inspired crowd-funding system that encourages all researchers to make autonomous, individual funding decisions towards people, not projects or proposals,” said Bollen. “All you need is a centralized web site where researchers could log-in, enter the names of the scientists they chose to donate to, and specify how much they each should receive.”

The authors emphasize that the system would require oversight to prevent misuse, such as conflicts of interests and collusion. Funding agencies may need to confidentially monitor the flow of funding and may even play a role in directing it. For example they can provide incentives to donate to specific large-scale research challenges that are deemed priorities but which the scientific community can overlook.

“The savings of financial and human resources could be used to identify new targets of funding, to support the translation of scientific results into products and jobs, and to help communicate advances in science and technology,” added Bollen. “This funding system may even have the side-effect of changing publication practices for the better: researchers will want to clearly communicate their vision and research goals to as wide an audience as possible.”

Awards from the National Science Foundation, the Andrew W. Mellon Foundation and the National Institutes of Health supported the work.

From funding agencies to scientific agency: Collective allocation of science funding as an alternative to peer review

Johan Bollen, David Crandall, Damion Junk, Ying Ding, and Katy Börner

Read the paper:

doi: 10.1002/embr.201338068
On the one hand, some might argue that the system would be hijacked by logrolling and network effects. But if the system had strong accountability and transparency measures, which involved researchers disclosing expenditures, outcomes, and which researchers they financially supported, I think some of the possible negative effects would disappear.

It’s a neat idea and someone needs to try it. It could be SSHRC creating a special research fund that worked in this way: everyone who applied would receive money (equally divided among the applicants) and would have to donate a fixed portion of their money to others. Or maybe a university could try it with some internal funding.

Thoughts?

Hat tip to marginal revolution for this story.

Academia, Knowledge Mobilization, and Canadian Public Policy

Last week, I attended the State of the Federation Conference at Queen’s University. This year’s conference theme was on Aboriginal multilevel governance. The program was interesting, bringing together academics, activists, policymakers, and practitioners to talk about the role of Aboriginal peoples and the Canadian federation.

One of the keynote speakers was Michael Wernick, the current deputy minister of Aboriginal Affairs and Northern Development Canada. His talk was interesting in a number of respects, but one of the most interesting was a comment he made about the role of ideas in public policy making.

Among other things, he said that government bureaucracies were not in the business of knowledge creation and innovation. Instead, it was the responsibility of academics to produce new ideas for government to consider and implement.

Of course the big question is: how does government access the ideas produced by academics?

Continue reading

His answer was that one of the main sources was from newspapers and the op eds written by academics (and even social media)!

It was a surprising admission on a lot of fronts and the implications are significant.

First, academics clearly need to find creative ways to disseminate their research findings beyond the usual academic journals and books. Second, it reaffirms my current strategy of trying to summarize my academic findings through op eds. Third, in one sense, it shows that policymakers don’t have the time or interest to read and consume highly detailed, academic arguments. They value new ideas and research, but they want the bottom line; they want the policy implications, and are less interested, perhaps, in theory, empirics, and the like.

Finally, there is a serious need to rethink knowledge mobilization. The traditional route was conferences and invited talks, or good old fashioned reading! But perhaps there are better ways to facilitate knowledge transfer between policymakers and academics. Thoughts? Provide them in the comments.

The End of the Traditional Academic Career?

So says Alex Hope in the Guardian. Among other things, he writes:

You don’t necessary need to be an academic employed at an academic institution to contribute to the academy. Academics need to reflect these changes in their practice. We need to become more agile both in terms of our employment and our modes of communication. Academic tenure is the past – flexibility is the future.”

Steve Saideman disagrees:

“Universities depend on the full time profs to run programs, provide a variety of services, engage in research, and, oh, yeah, bring in research money.”

And:

“There is also something else–that universities are communities of scholars, not just buildings and administrators.  The pursuit of knowledge (yeah that is mighty high falutin) is a social endeavor, and universities, by bringing together students, professors, post-docs and other folks, facilitate the processes by which we can learn and argue and develop.  Yes, some scholars can work in isolation, but the social environment of universities is incredibly important for most work.”

On the one hand, I agree with Alex Hope. Flexibility is an important asset in a time when rapid change is needed.  And if there is one thing I’ve learned in my five years on the job, universities are slow and difficult to change!

Continue reading

So in one sense, adopting a model that relies more heavily on teaching flexibility, for instance, is a good thing. In many ways, the teaching competencies of university departments are fixed by who you hired in the past. Forever. A case in point is in my department where “legal studies” has become one of our most popular programs.  The problem is: none of our full time faculty members does research in this area and few of us are really qualified to teach an “intro to law” course, for instance. The easy answer is to hire a new tenure-stream professor in Canadian judicial politics but most universities are ratcheting back new hires and refusing, sometimes, to even replace retirees.

So I’m sympathetic towards calls for flexibility; I can understand why five- or ten-year renewable contracts would be attractive and why administrators are encouraging greater use of MOOCs and online courses (or at least blended learning courses).

I’m also not convinced by Saideman’s point that universities are important for knowledge production because they are sites of collaboration and interaction.  We work in an age of advanced communications that can facilitate meaningful and productive relationships irrespective of geography.  Gary Wilson at UNBC, for instance, and I have published two journal articles (one in CJPS and one in Regional and Federal Studies), won a SSHRC Insight Grant, and are presenting a paper at the annual state of the federation conference, yet we’ve never met in person. Ever (Does Gary even exist?!?).

If anything, I think the strongest reason for the status quo is the “research funding” rationale.  But of course, from a university administrator’s perspective, the five- to ten-year contract model still makes more sense then the current tenure model.  If I was a senior administrator, then I would give automatic renewal to those who constantly won research grants.  Those who didn’t: see yah!

Don’t get me wrong. I like the current system.  Tenure affords me the freedom to pursue whatever research I want without having to worry too much about the “politics” of my work.  It’s also nice to have an institutional home with the support that such a home provides (e.g. an office, library, support staff, teaching support services, and the like) but I think there’s merit in findings ways to increase academic flexibility both at the individual and institutional levels.

Should We Change the Grant Adjudication Process?

John Sides, commenting on recent criticisms that the U.S. governments (through the NSF) funds far too many “silly projects,” at the Monkey Cage writes:

“it’s very hard to determine the value of any research ahead of time.  It’s hard because any one research project is narrow.  It’s hard because you can’t anticipate how one project might inform later ones.  It’s hard because some funding goes to create public goods—like large datasets—that many others will use, and those myriad projects also cannot be anticipated.  It’s hard because some research won’t work, and we can’t know that ahead of time.”

I think John is completely right. Committees are faced with extreme uncertainty about the future value of the various proposed research projects and so the grant adjudication process is bit of a crapshoot, all else being equal. (It also explains why one of my SSHRC grants got funded after the third try, even though I made extremely minor revisions!)
Continue reading

Because it is difficult to figure out which proposed research will actually have value in the future, most committees and/or competition criteria put great emphasis on research record. And so, if you have a really strong record, you are likely to get funded.

But I’m not sure that’s the best model for adjudicating grant applications. If we take seriously the notion that predicting the future value of research is very hard, then there are at least two reasonable options for changing how we adjudicate grant applications:

a) fund all applications that come in (which is problematic since resources are limited; one solution would be to simply divide up the money among all of the applications); or better yet,

b) create a small pool of applications that meet a certain scholarly criteria and then randomly choose which applications to fund from that pool.  This process would be different from the current one where applications are ranked and funded beginning with #1 downwards.

My own view is perhaps we should move to model (b).  At least that way, some of the idiosyncrasies involved with personal preferences and networks are somewhat mitigated.

Or maybe not!

“Political scholars fiddle while Rome burns”

So reads the title of Lawrence Martin’s recent op ed in the Globe and Mail.

In essence, Martin laments the passing from public life of some of the giants of Canadian political science, such as John Meisel, George Perlin, Hugh Thorburn, and Richard Simeon (I would add Peter Russell!) and the lack of new scholars taking their place.  What should we make of his argument? Is he right?

Maybe.  As Tyler Cowan observes, it is only natural for any field, whether it be academia or music or literature, to see a decline in the production of superstars over time as the field produces its fundamental tenants and findings.

On the other hand, maybe Martin simply isn’t paying close enough attention to the discipline and to the media in general, as a number of commentators have observed (e.g. see Steve Saideman’s thoughts on these points).  Certainly people like Emmett MacFarlane, Royce Koop, Peter Loewen, Antonia Maioni, Tom Flanagan, all of whom are active in public life, come quickly to mind.
Continue reading

In all honesty, I think there’s truth in both sides of the argument.  Has their been a decline of Canadian politics scholars actively participating as public intellectuals in the vein of Ned Franks et al.? I think so. Is Canada producing replacements? Yes, but maybe in not the same numbers as in the past.

I think the fact that relatively few replacements are being produced, if that argument is indeed correct, is due to a slow but steady assault on the study of Canadian politics at Canadian universities (and especially at the larger universities).

Where is this assault coming from? One source is in hiring decisions.  I don’t have any concrete data, but it seems that as the older generation of Canadian political scientists retire, their departments aren’t interested in replacing them with Canadian politics scholars.

At the University of Toronto (downtown campus), for instance, Richard Simeon, Canada’s pre-eminant scholar of Canadian federalism, was replaced by an American-trained scholar who studies comparative political parties and voting behaviour in federal countries (I’m not sure if Canada is one of the countries she studies). Peter Russell, Canada’s top constitutional thinker, was replaced many years ago by a scholar who studies comparative constitutional law. In both cases, the replacements are without a doubt amazing and excellent scholars.  Peter’s replacement, for instance, is a Canada Research Chair, a Killam Research Fellowship Winner, and simply one of the smartest, hard-working, and most productive scholars in the world.  Period.  But the point here is that the big departments, like UofT, are replacing Peter Russell and Richard Simeon-like scholars with American-trained comparative politics scholars.

These trends aren’t limited to UofT.  McGill University, last year, had a position in Canadian politics.  Rumour had it that they had shortlisted three or four excellent candidates but declined to offer any of them the job.  Instead, a year later, the position was supposedly converted and filled with a comparative politics professor.

At UBC, a Canadian politics position several years ago was filled by a scholar who does comparative voting behaviour. Again, the new professor is a top-notch scholar in his field, but is not really a Canadianist (at least in terms of his recent research, but perhaps this will change over time).

These trends aren’t limited to hiring practices.  They are being felt in the classrooms and in the journals. New books argue that our discipline needs to embrace the “comparative turn“, rather than focusing solely on Canada. The number of graduate courses that focus solely on Canada seems to be on a steady decline (although at UofT, the satellite campuses in Mississauga and Scarborough have recently started to fill the void at the downtown campus with two excellent hires: Erin Tolley and Robert Schertzer).

In short, there are Canadian politics scholars actively engaged in public discourse, but perhaps not as many in the past.  Political science departments in this country need to do more to protect, prioritize, and publicize the study of Canadian politics.

More on Peer Review

Peer review seems to be the theme of the month on the blogosphere.  Here’s John Sides on the topic at the monkey cage, Stephen Walt at Foreign Policy, and Daniel Nexon at the Duck of Minerva.

The basic crux is that peer review is problematic, but it’s the best system we got.  My own random thoughts on peer review as a political scientist:

  • Frequently, reviewers “mail in” their reviews.  In one instance, a top journal in political science rejected my co-authored manuscript on the basis of three reviews: one review was a reject on the basis of two short criticisms that could easily have been addressed through a revise and resubmit (basically the reviewer criticized us for not including some literature); the second referee report was a revise and resubmit with very constructive suggestions; the third review provided no comments! Zero. Zilch. Nada. We wrote the editor and she confirmed that the reviewer did not provide any comments to the authors.  In another paper, one reviewer demanded major revisions on the basis that we didn’t include a government report that had come out one month after we submitted our paper, and that we failed to specify in a footnote whether we received financial support for our research. And again for another co-authored paper, we received one accept with minor revisions and one reject.  The reject was two sentences long and said we didn’t offer enough empirical support and that our theory was too simple.  End of review! No details on how and why our theory was too simple and no discussion of the kind of empirical support needed to fully support our argument. So to some extent, I agree with Stephen Walt about the usefulness (or uselessness!) of peer reviewer comments.  On the other hand, I’ve also received excellent and helpful comments that have dramatically improved my papers.  So it’s a mixed bag.
  • Editors matter.  The good editors recognize the value of different reviews and weigh them accordingly.  Most of the time, that happens.  But sometimes it doesn’t!
  • I’ve published a couple of papers in law journals and some law journals actually have a staff member whose job is to check all of your references.  I love this feature of the law journals and it’s made me think I need to get a research assistant to go through my papers and check my references!
  • Peer review tends to be really slow.  I must admit, I don’t understand why it can take so long.  Recently, review times have been speeding up, averaging two to three months but in the past, it could take anywhere from four to six months.  The longest I have ever taken to review a manuscript for a journal or university press was 10 days.  I hate the long delays that are so frequent in academic peer review and so I’ve always tried to turn things around as quickly as possible, and with as comprehensive of a review as possible.  If I knew I couldn’t get to a review for three or four months, I would turn the invitation down.
  • Peer reviewers tend to be reasonable when it comes to revise and resubmits, but not always.  Although I tend to provide long and comprehensive comments in my referee reports, I never expect authors to make all of the changes.  I understand that academia is about give and take as long as the authors consider the comments and have a response, I’m almost always willing to recommend acceptance of a revised and resubmitted manuscript.  As an author, only twice did I revise a manuscript and receive subsequent reviewer reports that basically trashed every revision and response I made.
  • I’ve heard many stories of reviewers rejecting a manuscript at one journal, and receiving it as a referee again at another journal.  I’ve never had this happen to me but I must admit, I don’t understand it.  If I was this reviewer, I would decline to review it a second time, or at least let the editor know that I reviewed at another journal and did they want me to review it again.
  • Double blind peer review is rarely double blind.  Authors post working papers online and don’t change the title or they list the papers and where they submitted them on their C.V.  I’m not sure why they do this.  Perhaps they hope reviewers will be more sympathetic if they know who the authors are? Other times, the authors of the papers are easily identifiable simply because the academic field is so small.  There have been several times where I knew the identity of the author right away, simply from writing style, references, and topic.  In those instances, I’ve always informed the editor that I think the author of the papers is x, and did they want me to continue to do the review.  Most editors say yes, with only a handful (maybe one or two in my experience) saying no.

In general, I like the peer review process if only because it’s the best process we have so far.  I’d like to see scholars be more up front with editors when they have reviewed papers for other journals or have a pretty good idea about who the authors are (how often does this happen? I have no idea)! It would be nice if reviewers took their job seriously and provided timely (e.g. quick!) and comprehensive reviews.  Frequently, peer reviewers bring their own biases (theoretical and methodological) to the peer review process.  The result is always rejection.  Sometimes we forget that research is a collective process, not an all out war between various theoretical and methodological camps.

 

 

What Happens When You Resubmit Published Articles for Peer Review?

Rejection!

According to a study, academic peer review is a crapshoot:

Douglas Peters and Stephen J. Ceci, “Peer-review practices of psychological journals: The fate of published articles, submitted again“. Behavioral and Brain Sciences / Volume 5 / Issue 02 / June 1982, pp 187-195.

Abstract:

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.

The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.

With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated.

 

Mentors and Giants of (Canadian) Political Science: An Interview with Peter Russell

Peter Russell is Professor Emeritus in the Department of Political Science at the University of Toronto.  He has written on a wide variety of subjects, including minority governments, parliamentary democracy, constitutional change and reform, and aboriginal and judicial politics. His classic book, Constitutional Odyssey: Can Canadians Become a Sovereign People? is a must read for students and scholars of Canadian politics as is his book,  Recognizing Aboriginal Title: The Mabo Case and Indigenous Resistance to English-Settler Colonialism, among others.  Not only has Peter had a strong influence on the academic world, but he has also been active in the real world of politics, advising governments and Royal Commissions on a wide range of issues and topics.

Much of my initial interactions with Peter were through his scholarship, which taught me the importance of taking into account history and agency for analyzing Canadian politics.  Later, he served as the final departmental reader on my dissertation, and has since then provided me with valuable advice about publishing, book writing, and how to make the most of my academic career.

The following is a transcription of a phone interview I did with Peter several weeks ago.  I’ve lightly edited and condensed it so any and all mistakes that may appear below are mine alone.

Enjoy!

I wish someone had told me at the beginning of my career:

how much I would enjoy it and how much fun I would have. As well, nobody told me that you could participate in and pontificate about politics. I sort of stumbled into participating.  Some of my colleagues think you shouldn’t do that.  They believe that you should stay in the ivory tower and not dirty your feet in the real world of politics but I’ve never agreed with that.

The individual I admire the most academically:

was C.B. Macpherson.  He was a political philosopher, par excellence, and he was a Marxist (small-m).  He was a wonderful and interesting philosopher of politics.  I actually didn’t agree with his fundamental argument in his most important book but that’s beside the point.  I really admired him because he spent the first 25 years of his life just working on his basic critique of what you might call liberalism. He didn’t publish his main book until well until his 40s.  In those days, you could get away with that.  He didn’t bother with a lot of little articles in refereed journals.  There wasn’t the same “publish or perish” expectation.  At the same time he was doing political theory, he was very active in the community around him, mostly in the civil liberties area.  For me, he was a wonderful example of a superb scholar and a real participant in the part of politics that really mattered to him. 
Continue reading

My best research project during my career:

was without a doubt, the Mabo case.  I stumbled into it and really learned so much by studying it.  I knew a little about Aboriginal peoples and politics but not nearly enough about western imperialism.  By doing that book, which took about 9 years, I learned so much about my own background as a person of British heritage and so much about the variety of experiences that Aboriginal peoples had; the comparative aspect really came out.  I also learned a lot about the limits of what can happen in courts because the Mabo decision really didn’t do much until it started to influence politics; and it still has a long way to go before fundamental improvements to the lives of Aborigines in Australia can occur.  So basically I learned about the limits of judicial power.

My worst research project during my career

was my work on trial courts, which I really worked hard on but which turned out to be mostly a dead end. I did a lot of writing about them and I ended up editing the only book we have in Canada on trial courts.  Part of my goal was to get political scientists interested in lower courts and to get their gaze off of the Supreme Court. I didn’t succeed. I also had a very strong view about reforming the structure of our trial courts and I had lots of support from the people who worked in those courts and experienced them.  I organized a conference, edited a book and did quite a bit of writing on the topic but it was the most inconsequential of any of my research. Nobody is interested in trial courts.  You mention trial courts to political scientists and, as they say, their “eyes glaze over.”

The most amazing or memorable experience when I was doing research

was when I was out at a place called Discovery Island.  It’s in the Torres Straits, just north of Cape York in Australia, and they call it Discovery Island because that’s where Captain Cook stopped and gazed southward at what we now call Cape York.  I was there with an Aboriginal friend who explained to me that this is where Captain Cook stopped and looked at everything across the strait before saying, “I claim all of this for my sovereign, King George III.”  Cook didn’t know much about the area.  He just sailed up the coast of what is now Queensland and claimed it. “Wow! That’s crazy!” I said.  “No, no, no, Peter. That’s not the word for it,” my Aboriginal friend responded. “That’s legal magic. It’s real magic.  Not only did Captain Cook claim ownership of lands he didn’t know, but then a whole bunch of lawyers over the centuries that followed, wearing wigs and looking like magicians with long black robes, would go into court and they took it seriously.  They believed it!  They put an army and navy and everything else behind it.  They made what was a really ridiculous statement come true!  That’s magic Peter! That’s legal magic.”  I use that phrase a lot.  People say “how did Britain get sovereignty over this?” and I say “Legal magic.” I thought it was such a perceptive moment in my life.  

The one story I always wanted to tell but never had a chance:

was when I was director of research for the Royal Commission on certain activities of the RCMP.  Those certain activities had been going on during the 1960s and the 1970s and they involved the security service, which was part of the RCMP at the time.  The security service had been collecting intelligence for the government about various things, including in those days, separatism, and they had done some terrible things that led to the creation of a Royal Commission.  There were public hearings and so on.  During the first six month’s of the Commission’s work, all we heard about was the many horrible things that the security service had done.  They had planted dynamite in the back of people’s cars to implicate them as terrorists.  They had burned a barn to implicate the PQ as terrorists.  They were trying to frame the PQ as a terrorist organization.  That was the main rotten thing they were doing. 

One day, it dawned on me that the commissioners should find out if the security service had done anything good to help Canada.  The commissioner said to me, “That’s a very good idea. Ask them if they can put together 20 good examples of the good stuff that they had done for Canada.”  The security service took several months to assemble a binder with 20 such case studies but of course they were all at this point, and will remain, top secret because they dealt with serious espionage and terrorist attempts that they had thwarted.  So we had two or three days of in-camera meetings with the security service on those success stories.

We at the commission could never tell those stories publicly because we would be giving away all sorts of secrets about their techniques and sources in other countries.  I’d love a chance to tell some of those stories one day because they convinced me that a country needs a security service to collect information and protect it against terrorism and espionage.  But of course I can’t tell any of those stories!  They are very revealing of what goes on in the world.  Canadians are very innocent and every once in while, we hear stories like someone selling secrets to the Russians but that’s fairly mild compared to the things I’m talking about.

A research project I wish I had done

is a study of native peoples in the United States.  I know the outline of the story, but I still don’t have a good handle on where things are with Aboriginal peoples in the United States.  I know they are not at the top of the political agenda, as they are here, and that’s a remarkable contrast.  I know there’s a lot more of them in the U.S., not as a percentage of the population, but in terms of raw numbers; there’s about three million and we just have over a million. And I know they’ve got some successes but they have also had a lot of problems.  I’ve never had a good handle on the situation in the U.S.  I have a very good understanding of the Maori, and what they’ve been through and where they are now. I also have a really in-depth understanding of Aboriginal people in Australia, including the Torres Strait islanders. But in terms of the U.S., I’ve always been interested in knowing more, and by knowing more I mean spending time with them because I feel comfortable talking about the Maori and the Aborigines in Australia and the Torres Strait islanders, and many of our own First Nations, and certainly the Inuit and the Métis, because I’ve spent time with them, hung out with them, and made friends with them.  But I’ve never had that experience in the United States.  It’s all second hand.  I don’t trust second hand.  I’d rather have first hand experience.

If I wasn’t doing this, I would be:

a lawyer.  I got talked out of being a lawyer by Bora Laskin.  I had started in 1958 at the University of Toronto teaching Canadian politics and I got very quickly to the lecture on the Judicial Committee of the Privy Council. I wanted to talk about the implications of the JCPC handing things over to the Supreme Court. To prepare a lecture on that, I went and knocked on the doors of some of my colleagues because I couldn’t find anything to read.  What difference does it make if the Supreme Court of Canada is supreme? “Oh Peter,” they said, “that’s law.  We don’t really study those kinds of things but there’s a nice fellow down on Philosopher’s Walk at the law school and his name is Bora Laskin and he’ll help you.  He doesn’t mind political scientists.”  I was a little nervous.  I didn’t know him and I didn’t know anyone at the law school.  It was my first year at UofT.  I went down and knocked on his door, and this guy with a great big smile welcomed me in.  Right away we were on a first-name basis.  It was clear he knew a lot.  He had been reading all of the Supreme Court cases since 1949 when the Supreme Court became supreme.  I lapped it all up and he says, “why don’t you sit in my course?” And I say “I’d love to that” and so I sat in the back row and took notes and so on.  Next year, I took another course, not with him, but with another Professor, on the U.S. Constitution, and another course the year after that on administrative law.  I had taken three courses and I loved them all. So I went down to see Bora in 1961 and I said, “Bora, I want you to make an honest lawyer of me.  I’m going to get a law degree.” And he looked at me, and for the first time, this really nice and friendly fellow, waved his finger and said to me, “Oh you mustn’t do that, Peter.  You mustn’t do that.” I said, “Really? You’re a lawyer, Bora, and you’re ok! I’ve enjoyed our conversations immensely.” “Oh,” he said, “yeah, but if you become a lawyer, you’ll lose any kind of perspective outside of law.  You’ll be on the inside where all of us are.  Stay on the outside.  Stay in political science. You’ll have a very distinctive, interesting, and valuable perspective from the outside.”  I really didn’t understand what he was talking about, quite honestly, but I took his advice because it would save me a lot of money.  The wife wasn’t too keen, we had two little kids, and so I didn’t do it. So he talked me out of it! I told that story at Bora’s memorial service.  People may say, “well Peter, you should have taken his advice.  You’ve been messing with the law ever since and have made a big mess of it!” But I’m glad I kept on the political science path.

The biggest challenge in Canadian politics in the next 10 years will be:

to create a better, more just, and mutually beneficial relationship with our First Nations, Inuit, and Métis.  I think we need to improve that relationship and make it a source of pride rather than a source of shame. We have done some good stuff and are doing some good stuff, but I think a lot more has to be done and we don’t quite know how to do it yet.

The biggest challenge in Canadian political science in the next 10 years will be:

I have an odd reaction to contemporary political scientists of your generation.  I find they are being moulded too much by the discipline.  They are being too constrained in what they can do by the discipline and I guess it’s partly the constraints of the tenure system which says to them, “get out some publications real fast. And what we really love are articles in refereed journals!” Political science does put some value on books, but I see so many young political scientists concentrating on these small pieces of the puzzle.  That’s all you do in a journal article, which is unlike C.B. Macpherson. In a journal article, you are not taking on something that’s really big and challenging.  I hope political science will break out of the tight discipline mould a bit.  I’d like to see that happen.

My advice for young researchers at the start of their career is:

enjoy yourself and if you are not enjoying yourself, get out fast.  Take some chances. I know that’s easier said then done, particularly if you have a family to feed and you are a young, starting political scientist, but be bold! Take some chances. Have some confidence in your own creative juices.  Don’t be so damn set on pleasing your peers! There’s too much deference to the leading writers in the field and too much caution about breaking away and taking new approaches.

Mentors and Giants of (Canadian) Political Science: An Interview with Tom Flanagan

This is the first of an occasional series I plan to launch today interviewing a number of Professors that have had an impact on my scholarly career, either directly or indirectly through their mentorship and/or work.  It’s an idea I borrowed from indecisionblog (I also borrowed most of their questions!), which is doing something similar on influential  economists in the United States. Enjoy!

Tom Flanagan is Professor of Political Science at the University of Calgary.  A political theorist by training, he has written on a wide range of topics, including Aboriginal politics, elections, electoral systems, the Reform and Conservative parties of Canada, the Supreme Court, rational choice and game theory, and Louis Riel, among many others.  In addition to his scholarly work, he has been active in public life: as an organizer for the Reform Party, the Conservative Party of Canada and the Wildrose Alliance, and as a public commentator for the CBC, the Globe and Mail, and other media outlets.

Tom was my thesis supervisor during my M.A. studies in political science at the University of Calgary and has had a powerful influence on my scholarly career.  Among other things, he showed me how to publish, how to be an efficient academic, and what it meant to be an intellectual, which means always remaining open to the possibility that one’s views and research are wrong.

Below is an interview I conducted with Tom via email in January 2013. 

I wish someone had told me at the beginning of my career

There’s a lot of advice that could have helped me, but I wouldn’t have listened!   I’m one of those people who only learn by making mistakes.

The individual I admire the most academically

The American economist Thomas Sowell.  He’s a great example of someone with solid accomplishments in his discipline who then broadened out to address public affairs in an illuminating way.

My best research project during my career

The Collected Writings of Louis Riel.  We had a great team, and we got the job done on time within budget.  It now provides a basis of information for scholars of all points of view.
Continue reading

My worst research project during my career

Over 40 years, ago I tried to write a book on the influence of Joachim of Fiore and his conception of the three ages of history.  It was a worthwhile topic, and other scholars subsequently published books about it, but it was too big a topic for me at my stage, and I was never able to get my manuscript published.  Most frustrating!

The most amazing or memorable experience when I was doing research

Finding reports in the William Pearce Papers at the University of Alberta archives that proved that the federal government did in fact deal with the Métis land grievances at Batoche before the 1885 Rebellion broke out.

The one story I always wanted to tell but never had a chance

When I was Stephen Harper’s chief of staff, a group of us were meeting one day in the leader’s office (Mackenzie King office).  It was at the time of the great debate over homosexual and heterosexual marriage.  Somehow the concept of scedasticity came up (Stephen has a degree in economics, so he understands these things); I honestly can’t remember why we started talking about such arcana.  Stephen said dead-pan, “Of course, our party prefers the traditional definition of scedasticity.”  He can be very funny in person, but it doesn’t come through very well in his public communications (they are written in what I call PMO blank verse).

A research project I wish I had done

For years I have dreamed of writing a book about the biological roots of human politics.  But I fear I have dawdled too long, and the field has exploded beyond my ability to keep up with all the new scientific discoveries.

If I wasn’t doing this, I would be

Fishing.  Also, I would have enjoyed being a litigator, and I think I would have been good at it.  I loved working with lawyers as an expert witness.  I love high-level polemics, maybe sometimes to the detriment of scholarly objectivity.

The biggest challenge in Canadian politics in the next 10 years will be

To keep the greenies and other lefties (yes, I’m talking to you!) from choking off the energy industry, which is about the only thing paying Canada’s bills.

The biggest challenge in Canadian political science in the next 10 years will be

I honestly don’t know.  Over the years, I’ve gotten away from our discipline, as I moved into a lot of other areas, including practical politics.  But I will say this:  There’s a big opportunity waiting for someone who understands modern multivariate analysis to take it public, if they can figure how to communicate it.  Most public polling is pretty simplistic, because the pollsters haven’t figured out how to communicate advanced analysis to politicians and their staff.

My advice for young researchers at the start of their career is

Keep it focused.   Research and write about manageable topics.  Milk your dissertation for all you can, but don’t stay bottled up with that topic forever.  Scholarship is full of great opportunities, but you have to be willing to take chances.