---
title: "Fleiss' kappa (k raters)"
method_id: SM.RLB.AGR.KRAT.FLEISS
family: Reliability
version: 1.1.0
date_modified: 2026-06-18
canonical: https://clarus.ofrencber.com/library/fleiss_kappa.md
source: Clarus method library
---

# Fleiss' kappa (k raters)

> A classical statistical method in the Clarus library (Reliability family).
> Clarus selects, assumption-checks and runs this method on your data with a
> deterministic rule engine and real numerical libraries (scipy / statsmodels);
> it never generates numbers, citations or results.

**Also known as:** Fleiss kappa; multi-rater kappa; generalized kappa.

## Hypotheses

- **Null (H0):** kappa = 0 (observed agreement equals chance-expected agreement)
- **Alternative (H1):** kappa != 0 (agreement differs from chance)

## When to use it

Use this method when your goal is to:
- assess inter rater agreement nominal multiple raters

## Data it expects

- **dependent:** 1 nominal/categorical rating
- **independent:** k raters (>=3 typical), fixed categories
- **pairing:** each subject rated by the same number of raters; raters need not be the same individuals across subjects

## Assumptions Clarus checks

- **fixed categories** (severity: design; on violation: warn_only)
- **equal raters per subject** (severity: error; on violation: warn_only)
- **rater independence** (severity: design; on violation: warn_only)

## Effect size reported

- **kappa** (Fleiss (1971); interpretive bands Landis & Koch (1977))

## Honest limitations

- Fleiss' kappa is sensitive to category prevalence: when one category dominates, kappa can be low even with high raw agreement (the kappa paradox) — always report observed agreement P_bar alongside kappa.
- Landis & Koch agreement labels are conventional heuristics, not universal cut-offs; different fields adopt different thresholds.
- Kappa estimates from few subjects or rare categories are unstable; interpret with a confidence interval (computed separately) where possible.
- Agreement is not accuracy: high agreement only means raters are consistent with each other, not that they are correct.
- The classic coefficient assumes the same number of ratings per subject; unequal raters require Gwet's AC1 or a generalized formulation.

## Primary sources

- Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382. https://doi.org/10.1037/h0031619
- Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174. https://doi.org/10.2307/2529310
- Gwet, K. L. (2014). Handbook of Inter-Rater Reliability (4th ed.). Advanced Analytics, LLC.
- Fleiss, J. L., Levin, B., & Paik, M. C. (2003). Statistical Methods for Rates and Proportions (3rd ed.). Wiley. https://doi.org/10.1002/0471445428

## How to cite this

To cite this Clarus method page (the page itself — for the method's own primary sources, see above):

**Plain text**

Clarus, "Fleiss' kappa (k raters)", version 1.1.0, Clarus method library, 2026. https://clarus.ofrencber.com/library/fleiss_kappa.md

**APA**

Clarus. (2026). Fleiss' kappa (k raters) (Version 1.1.0) [Statistical method, Clarus method library]. Retrieved from https://clarus.ofrencber.com/library/fleiss_kappa.md

**BibTeX**

```bibtex
@misc{clarus-fleiss-kappa,
  author       = {Clarus},
  title        = {Fleiss' kappa (k raters)},
  howpublished = {Clarus method library},
  version      = {1.1.0},
  year         = {2026},
  url          = {https://clarus.ofrencber.com/library/fleiss_kappa.md}
}
```

_No DOI is minted for Clarus method pages yet; this citation uses the canonical URL, version and date. Cite the primary sources above for the method's scientific provenance._

---

METHOD SM.RLB.AGR.KRAT.FLEISS · VERSION 1.1.0 · UPDATED 2026-06-18 · SOURCE Clarus method library