What is inter-rater reliability – and why does it matter in marking?

Would another marker agree?

Every student hopes their grade reflects the quality of their work – not who happened to mark it. Inter-rater reliability is the measure of whether that hope is met. If two markers independently assess the same exam and reach the same result, reliability is high. If their grades differ widely, the outcome depends partly on the marker rather than on the work.

The concept comes from research and testing, where reliability is a basic requirement for any measurement. In higher education, it sits at the heart of fair assessment – and behind familiar practices such as double marking, second marking and moderation.

How does it work?

Inter-rater reliability can be described and measured in several ways:

• Simple agreement. The share of submissions where two markers give the same grade, or a grade within an agreed margin.
• Statistical measures. Measures such as Cohen’s kappa or the intraclass correlation take into account how much agreement would be expected by chance.
• Patterns over time. Comparing markers across a cohort can show whether some are consistently stricter or more lenient than others.
• Agreement by criterion. Looking at individual criteria shows where markers interpret the standards differently.

Why does inter-rater reliability matter?

The main reason is fairness. A grade has real consequences for a student’s progression, further study and career. If two equally good answers can receive different grades because they were marked by different people, the grade loses its meaning.

It also matters for trust in qualifications and in the institution. Reliable marking is easier to explain and defend, and evidence of reliability is increasingly expected by quality assurance bodies, external examiners and students themselves. And measuring it helps institutions see where marking guides, training or calibration need to improve.

What inter-rater reliability cannot do

High reliability does not guarantee a good assessment. Two markers can agree perfectly and still both be wrong – for example if the criteria reward the wrong things. Reliability is about consistency; validity – whether the assessment measures what it is meant to – is a separate question.

Perfect agreement is also unrealistic, especially for essays and other open tasks. Research has repeatedly found substantial disagreement between experienced markers, and studies of UK assessors suggest that some variation is unavoidable because academic judgement is complex and partly tacit. The goal is not to remove judgement, but to make it as consistent and transparent as possible.

How does WISEflow approach it?

WISEflow cannot make markers agree. But it can help institutions build reliability into the marking process and see where it falls short:

• Shared criteria. Rubrics and marking guides give every marker the same framework to apply.
• Independent marking where it matters. Assessor types support blind double and second marking, so markers form their own view before seeing a colleague’s grade.
• Disagreements made visible. When co-assessors submit conflicting grades, assessors are alerted and reviewers see the full picture.
• Data to learn from. Grades and audit events can be exported through the WISEflow API, so institutions can analyse how often markers disagree and where.
• The marker decides. WISEflow supports academic judgement; it never replaces it.

What should institutions consider?

• Calibration: let markers discuss sample answers before marking begins.
• Clear criteria: describe quality concretely, and avoid vague performance descriptions.
• Measure: look at agreement data regularly – by marker, by module and by criterion.
• Be open: explain to students that marking involves judgement, and how the institution ensures it is fair.

Inter-rater reliability will never be perfect. Pursued well, as part of a considered approach to marking and moderation, it makes grades depend on the work – not on the marker.