Cohen’s Kappa Calculator

Cohen's Kappa Calculator

Rater 1 Name (Rows)

Rater 2 Name (Columns)

Number of Rating Categories

Contingency Frequencies Matrix Enter non-negative integer observation counts for each agreement cell

All matrix cells must contain valid non-negative integers.

Significance Level (α) Default hypothesis threshold is 0.05 (95% Confidence Interval)

Significance level α must be strictly between 0 and 1 (e.g. 0.05).

Inter-Rater Reliability Diagnostic

What Is Cohen's Kappa?

Cohen's Kappa ($\kappa$) is a robust statistical metric used to measure inter-rater agreement between two independent raters or diagnostic classifiers categorizing items on a nominal or categorical scale.

Unlike simple percent agreement, Cohen's Kappa mathematically adjusts for the proportion of agreement expected purely by chance ($P_e$), providing a standardized coefficient between $-1.0$ (complete systematic disagreement) and $+1.0$ (perfect agreement).

When to Use It

Medical & Clinical Diagnosis: Evaluating consistency between two physicians diagnosing diseases or reading scans.
Content Analysis & Coding: Measuring inter-coder reliability in qualitative research and social sciences.
Machine Learning Evaluation: Assessing multiclass classification agreement beyond baseline majority class accuracy.
Psychometric Validation: Establishing instrument scoring repeatability between independent assessors.

The Mathematical Formulas

$$\kappa = \frac{P_o - P_e}{1 - P_e}$$ $$P_o = \frac{\sum_{i=1}^k n_{ii}}{N}, \quad P_e = \frac{\sum_{i=1}^k (R_i \cdot C_i)}{N^2}$$ $$SE(\kappa) = \sqrt{\frac{P_o(1 - P_o)}{N(1 - P_e)^2}}, \quad 95\% \text{ CI} = \kappa \pm z_{\alpha/2} \cdot SE(\kappa)$$

Where $n_{ii}$ are the diagonal agreement counts, $R_i$ and $C_i$ are row and column marginal totals, and $N$ is the total number of rated subjects.

Worked Example

Scenario: 2 radiologists rate 50 mammograms ($N=50$) as Normal / Abnormal: $[a=20, b=5, c=10, d=15]$ at $\alpha=0.05$.

Step 1: Compute Observed Agreement ($P_o$):
Diagonal agreement: $20 + 15 = 35 \implies P_o = \frac{35}{50} = 0.7000 \text{ (70.0\%)}$.

Step 2: Compute Chance Agreement ($P_e$):
Marginals: $R_1=25, R_2=25, C_1=30, C_2=20$.
$P_e = \frac{25 \times 30 + 25 \times 20}{50^2} = \frac{750 + 500}{2500} = \frac{1250}{2500} = 0.5000 \text{ (50.0\%)}$.

Step 3: Calculate Cohen's Kappa ($\kappa$):
$\kappa = \frac{0.7000 - 0.5000}{1 - 0.5000} = \frac{0.2000}{0.5000} = 0.4000$.

Step 4: Standard Error & Confidence Interval:
$SE(\kappa) = \sqrt{\frac{0.70(0.30)}{50(0.50)^2}} = \sqrt{\frac{0.21}{12.5}} = 0.1296$
$95\% \text{ CI} = 0.4000 \pm 1.96(0.1296) = [0.1459, 0.6541] \implies p = 0.0047$.

Inter-Rater Reliability Complete

Cohen's Kappa Results

Computed Cohen's Kappa Coefficient

κ = 0.4000

Fair Agreement Beyond Chance (p = 0.0047)

Observed Agree (P_o) 70.00%

Chance Agree (P_e) 50.00%

Standard Error 0.1296

z-Statistic 2.8284

p-value 0.0047

Total Rated (N) 50

Contingency Matrix & Marginal Proportions

Cohen’s Kappa Calculator