---
title: "Matcher training data"
summary: "We've produced a comprehensive dataset of record linkage judgements that can be used to develop entity matching technology. While we're using this data to build our own tools, we're also sharing the data as a community resource."
url: "https://www.opensanctions.org/docs/opensource/pairs/"
date_updated: "2026-08-20"
---

As soon as you start dealing with [entity data](/docs/entities/), one question becomes ubiquitous: are two given entities the same? Checking if a customer of your business is on a sanctions list, or if a politician has undeclared offshore wealth — both tasks demand a technical answer to this problem.

Integrating sanctions data from different sources raises the same challenge (*is the person sanctioned by country X the same as the one sanctioned by country Y?*). That's why, in order to [de-duplicate the OpenSanctions database](/articles/2021-11-11-deduplication/), we've made 760,000 (December 2025) manual judgements about entity pairs contained in our system.

The dataset of pairwise judgements is a valuable resource for building our own [matching API](/docs/api/) and [its matching algorithm](/matcher/). But as an open project, we're also sharing it for others who might want to develop their own technology.

## Grouped pairs

The grouped pairs dataset contains one row per human judgement (`positive`, `negative` or `unsure`) recorded in our deduplication system. It is produced by chronological replay: each row's `left` and `right` entities are the partially-merged clusters as our resolver knew them at the moment of that judgement — approximately the evidence the human decider actually saw, which is what makes the labels trustworthy. Judgements generated by matcher-driven tooling are excluded to avoid training a matcher on matcher output. Entity IDs in the rows are replay-time cluster IDs and are not stable across rows; every row instead carries `left_cluster` and `right_cluster` labels which support a leakage-safe train/test split (split by cluster partition, never by row). The full data semantics — identity rules, exclusions, field reference — are documented in [DATA.md](https://github.com/opensanctions/opensanctions/blob/main/contrib/matcher_training/DATA.md), a copy of which is published next to the data alongside a `summary.json` with scan counters and cluster statistics.

The dataset contains roughly one million rows, is regenerated monthly, and is byte-deterministic given the same judgements and source data.

### Downloading

Access requires a [data delivery token](/docs/bulk/delivery-service/). The latest dataset is always available at a stable URL:

```bash
curl -sfL -H "Authorization: Token $YOUR_TOKEN" \
  -o pairs.jsonl.gz \
  https://delivery.opensanctions.com/licensed/pairs/latest/pairs.jsonl.gz
```

Dated monthly snapshots are retained under `licensed/pairs/<YYYYMM>/` (e.g. `licensed/pairs/202609/pairs.jsonl.gz`), each with its `summary.json` and `DATA.md` alongside.

## Original pairs data

This earlier flat pairs file is no longer updated; it is superseded by the grouped pairs above.

* **[pairs.json](https://data.opensanctions.org/contrib/training/pairs-20251209.json.gz)** (2100MB+, line-based JSON, last updated 2025-12-09, [non-commercial use](/licensing/))

The data has the following format, with two [entities](/docs/entities/) on each line, and a judgement that states if the two records refer to the same logical item (`positive`) or if they are different logical items (`negative`):

```json
{
    "judgement": "negative|positive",
    "left": { ... entity data ... },
    "right": { ... entity data ... },
}
```

The data is intended for use with the [nomenklatura](https://github.com/opensanctions/nomenklatura) data integration framework (Python), but can easily be parsed outside of that context. Nomenklatura also contains the basic statistical model which is trained by the OpenSanctions project using the data.

The pairs data is derived from more granular matching judgements that are stored as a directed acyclical identifier graph. Using that raw data requires a more granular understanding of the way in which we de-duplicate entities and is not recommended.

If you use this data, please tell us about this! We're keen to hear how people use this data, and how the format or semantics could be improved.