Writing a High-Quality Referee Report for Academic Research Papers

The scientific peer-review process is essential for the advancement of academic research. In this opinion article, we provide some thoughts and suggestions for writing a high-quality referee report. Our views are accumulated from many years of experience reviewing research papers submitted to various peer-reviewed academic journals. We intend for academic referees to use our opinion article as a checklist when reviewing a research paper. Our catalog may not be a comprehensive list of all of the possible points for referees to consider. Nevertheless, it can be a useful guide for referees to improve the quality and completeness of their review report. Finally, our viewpoints are based on experience serving as referees for academic research in the subject of business (e.g., accounting, information systems, etc.). Hence, some of our thoughts may not be applicable to research in all subject areas.       

To start the referee report, we suggest writing an introductory paragraph (or two) that summarizes the research paper being reviewed.  The paragraph would summarize the paper’s research questions or hypotheses, how the authors executed the study (e.g., data used, research methods, etc.), the findings of the study, and the authors’ arguments for how the study contributes to the literature.  Presenting this opening paragraph would demonstrate to the editor and the authors that you have a good understanding of the research paper. Then, you can proceed to provide comments and criticisms.

Referees structure the ordering (or ranking) of their comments differently. Some reviewers may order the comments by listing the most crucial concerns first and the least important concerns last. Some reviewers may order their comments based on the order of the sections presented in the research paper. In this case, the referee would list the comments about the Introduction section first, followed by comments about the Background & Hypothesis section, and so forth. There is no single best way, so we suggest that you follow the approach that suits you best.

Some referees go straight into their criticisms about the research paper. Other reviewers point out the strengths of the research paper first and then proceed to write their concerns about the study. We prefer to discuss the strengths of the research paper first before diving into the list of concerns. For example, you can first discuss the aspects of the study that you find interesting or commend the authors for collecting new and insightful data. Stated differently, try to bring up the positive attributes of the study before listing the negative attributes. This way, you can acknowledge the work and effort that the authors have already put into the study. 

You can provide significant value as a referee by offering constructive suggestions, guidance, and advice for how the authors can address your specific concerns and criticisms. Even if you ultimately recommend rejection for the paper, you can help the authors improve their work for submission to another academic journal. Hence, after writing a point of criticism, make an effort and take the time to suggest possible ways for the authors to address your concern. Here, you do not necessarily need to provide a detailed or step-by-step instruction; you can simply lead the authors in the right direction or provide general thoughts to consider.

Below, we provide a list of concepts and thoughts that referees can consider when writing a referee report. We divide the list into six main categories:

1. sufficiency of contribution,
2. motivation of the study,
3. scope of the study,
4. theoretical argument and hypotheses development,
5. execution of the study, and
6. writing issues.

1. Is the contribution of the research paper sufficient?
   1. Assessing contribution requires a subjective call and judgment. The sufficiency of contribution can depend on the journal for which the paper is being submitted. The threshold for contribution is higher for “top-tier” journals compared to lower-tiered journals.
   2. Does the paper contribute to the literature based on the prior studies that have been published? Here, you will need to have a thorough understanding of what has been done in the literature. Do a web search (e.g., Google Scholar, ProQuest, etc.) to investigate what studies have been published on the topic. It is also a good idea to search for working papers (i.e., unpublished works) related to the topic.
   3. Is the authorship team covering all of the bases regarding citing the relevant literature (i.e., is there a comprehensive coverage of the prior literature)?
   4. Based on what we can already learn from prior literature, does the research paper provide new and important insights?
   5. Does the paper appeal to a broad readership for the journal? Can you envision many people being interested in reading the paper? Would academics and practitioners care about the findings?
   6. Are the results interesting? Are results counter-intuitive? Are the results too obvious where we do not need to research to know the finding? Are the findings just mechanical in nature?
   7. Can the research study make a difference in the real-world? If yes, has the authorship team talked enough about it?
   8. Look for the contribution paragraphs in the Introduction section. Usually, they are located in the last paragraphs of the Introduction section. Evaluate these paragraphs in terms of effectiveness. Are the paragraphs convincing? Is there a convincing discussion about the paper’s contribution to academic research as well as contribution to business practice?

1. Is the authorship team adequately motivating the study?
   1. Look for this in the Introduction section (usually located in the beginning paragraphs of the paper). It is not enough for the authorship team to discuss what they are researching; the team needs to explain why the research is important and interesting to conduct.
   2. Is the research paper addressing a real-world problem? For example, does the research have the potential to change how certain business practices are being conducted? Is the research addressing an issue that is timely and relevant to the current environment?
   3. Does the authorship team explain why the study is important to disseminate? Beyond just mentioning that the study is the first one to tackle the question? Remember, it is not enough for the authorship team to simply say that they are the first to study a particular issue. Perhaps no prior study has tackled the research question because the topic is not important or interesting to tackle. The authorship team should convince the editor and referee that the issue is worth researching in the first place.  
   4. Does the authorship team make an interesting case about why we should read further beyond the first few pages?
   5. If the discussion on motivation is lacking, explain why you think this is the case. Offer some suggestions as to how the authorship team can improve their discussion about the study’s motivation.

1. Is the scope of the study sufficient?
   1. Assessing the scope or coverage of the research paper requires a subjective call and judgment
   2. Evaluating the scope of the paper can be dependent on the journal. Higher-tiered journals demand papers that are of a broader scope than lower-tiered journals.
   3. Is the authorship team doing enough in the research paper? Are the findings too limited in order to generate a broad appeal?
   4. Has the authorship team conducted enough analyses to answer the research question(s) comprehensively?
   5. Are there other hypotheses or research questions (relevant to the study) that have not been addressed in the study?
   6. Adding scope is one way is to enhance the contribution of the study. Consider suggesting ways for the authorship team to expand their study if the contribution of the work is a concern.
   7. At the same time, we need to consider that a research paper has limits. There are obviously things that can be beyond the scope of the study. However, go ahead and propose ideas that extend the scope of the current paper. It will be up to the authors to argue about whether the referee’s suggestion is beyond the scope of the paper. The editor will also make a judgment call as to whether the author ought to follow the suggestion to expand the scope.

1. Evaluating the theoretical arguments and hypotheses development
   1. Showing significant statistical associations (i.e., significant p-values) is not enough. There must be sound theoretical arguments underlying the hypotheses.
   2. Does the underlying theoretical mechanism for the findings make sense?
   3. Is the argument for the theoretical mechanism detailed enough? Or is it just scratching the surface?
   4. Do you accept all of the arguments that the authors put forth? Put on your skepticism hat! When appropriate, think about counterarguments that could challenge the author’s line of reasoning.
   5. Are the theoretical arguments complete? Has the authorship team thought about the issues from all angles? Are there alternative explanations and/or arguments that the authors are not discussing? Some authors may present arguments that only validate their findings (i.e., making one-sided arguments that help their cause). Be aware of this issue.
   6. It is not enough for the author to simply provide a laundry list of citations and references to the prior literature. Authors need to discuss how those prior studies support the paper’s arguments and help make the case.

1. Execution and analyses within the study
   1. Overall question: Is the study competently done in terms of analyses and execution?
   2. Does the authorship team measure the variables correctly? Are the variable measurements consistent with prior literature?
   3. Often times, there are multiple ways to measure a particular construct. Has the authorship team considered alternative ways to measure the constructs?
   4. Are the models complete? Are there any missing control variables that are important to include?
   5. Think about how you would model the research design and see if the approach that the authorship team has taken vastly differs from your approach. Are there alternative approaches that the authors should consider?
   6. Are there potential endogeneity concerns that can materially affect the documented results? Does the authorship team perform any methods to help reduce the concern of endogeneity?
   7. Often times, referees would suggest the authors perform additional analyses and robustness tests. It is certainly fair game to propose these additional tests to corroborate the main findings.
   8. Is the authorship team providing enough details/information in the Tables?

1. Writing Issues
   1. Does the paper flow well?
   2. Is the paper easy to read? Readability is crucial to attracting a broad readership to the article.
   3. It is easy for the authors to have repetitions, especially between the Introduction and the Hypotheses Development sections. Be on the lookout for this.
   4. Are there sentences that are too wordy or unnecessarily too long?
   5. Do you observe an abnormal amount of grammar errors?
   6. It is fair game to point to the authors certain passages in the paper that do not read well. Ask the authors to revise those sections.

Sometimes, it is easy to spot instances when multiple co-authors write different sections independently and then simply merge the pieces together. The problem is that writing styles can be different between co-authors and the overall flow of the paper suffers. Be aware of this potential issue.