The index of weighted percentage agreement is a summary score based on the frequency and relative weight at each level of agreement between 2 observers using ordinal categories.
Observations
Observations of First Observer
of Second Observer
definite
probable
possible
unlikely
definite
a
b
c
d
probable
e
f
g
h
possible
i
j
k
l
unlikely
m
n
o
p
total number of observations = N =
= SUM(numbers in all 16 cells)
Level of Agreement
Weight
perfect agreement
3
1 category disagreement
2
2 category disagreement
1
no agreement (maximum disagreement)
0
index of weighted percentage agreement =
= (((a + f + k + p) * 3) + ((b + g + l + e + j + o) * 2) + ((c + h + i + n) * 1) + ((d + m) * 0)) / (N * 3)
where:
• The cells are combined along the diagonals.
Interpretation:
• Perfect agreement would have an index of 100%.
• Complete disagreement would have an index of 0%.
To read more or access our algorithms and calculators, please log in or register.
