Callaham et al used a scoring method to evaluate the quality of peer reviews. This can help stratify reviewers and can help identify areas for improvement.
Quality, strengths and weaknesses identified and commented on:
(1) study design and methods
(2) interpretation of data and conclusions
(3) overall written communication (independent of design, methods, results and interpretation)
Reviewer comments:
(4) provide useful suggestions to improve the manuscript
(5) are constructive and professional
(6) sufficient to allow the editor to make a proper decision about the manuscript
Evaluation |
Points |
unacceptable effort AND content |
1 |
unacceptable effort OR content |
2 |
acceptable |
3 |
commendable |
4 |
exceptional, hard to improve upon |
5 |
where:
• Normally no more than 10-20% of reviews should be classed as exceptional. A higher percentage may indicate overuse of the category.
total reviewer score =
= SUM(points for all 6 items)
Interpretation:
• minimal score: 6
• maximal score: 30
• The higher the score, the better the reviewer's performance on the review.
Limitations:
• A very high or low score in 1 or 2 items could mask performance issues.
• It should be acceptable if the person is consistent (performs comparably in all 6 measures).
ICD-10: ,