Know-Your-Business (KYB) data

Alongside the main OpenSanctions dataset, we publish the Know-Your-Business (KYB) collection: full data extracts of several company registries and international reference datasets. We use these sources in an enrichment process that identifies companies and people who appear both in the KYB datasets and on watchlists, and copies their additional details and associated entities into the default dataset. The full extracts are a side product of that process, but you can also use them directly.

The datasets in the KYB collection are too large to include in the default collection, and usually include a majority of entities that have no risk indication. A company register, for example, is overwhelmingly made up of companies that are not sanctioned or subject to regulatory action.

How the KYB collection differs from the main dataset #

The KYB datasets are built for enrichment, not for direct screening, and that shapes how they behave:

• No deduplication. The datasets published in the KYB collection are not integrated — different references to the same person or company are left as separate entities.
• Updated less often. KYB sources usually refresh once a week or once a month, because their size makes intra-day updates an un-economical proposition.
• Overlap with the main dataset. The entities "copied in" by the enrichment process exist in both collections. Inside the default dataset, each KYB source is represented by a dataset with the ext_ prefix that carries the relevant entities — for example ext_gleif.

Because of that overlap, indexing both collections into the same instance of yente requires namespacing to avoid entity ID collisions — see importing the data into yente below.

What the KYB collection is not #

We use the term KYB to summarize the genre of these datasets, but the geographic coverage is not sufficient to build a global "know-your-business" service. If your use case requires a global database of company information, consider a data vendor such as OpenCorporates or Sayari.

We expand the KYB collection opportunistically, as we find data sources that cover regions and topics relevant to the core dataset. It is not our ambition to build a comprehensive global companies database — our friends at OpenCorporates also use open data to build such a product.

Using the full KYB datasets #

The KYB collection contains full copies of company registers and other international reference databases, crawled and exported to the FollowTheMoney data format. Before you rely on this data, read and understand the dataset description — it sets out the constraints on using it.

Plan for the size. The KYB collection holds roughly 50× more entities than the default OpenSanctions dataset, so it needs significant additional processing resources. Data updates in yente also take considerably longer, which can delay the availability of fresh sanctions and risk data in the same instance.

Importing the data into yente #

To load the KYB data into yente, add an additional catalog to your instance's manifest (see also: custom datasets):

catalogs:
  - url: "https://delivery.opensanctions.com/datasets/latest/default/catalog.json"
    auth_token: "$OPENSANCTIONS_DELIVERY_TOKEN"
    scope: default
    resource_name: entities.ftm.json

  - url: "https://delivery.opensanctions.com/datasets/latest/kyb/catalog.json"
    auth_token: "$OPENSANCTIONS_DELIVERY_TOKEN"
    scopes:
      - ru_egrul
      - icij_offshoreleaks
      - gleif
    resource_name: entities.ftm.json
    namespace: true
datasets: []

Note the namespace: true setting: it adds a dataset-specific suffix to the identifiers of every dataset loaded from the kyb catalog, so their entity IDs do not overlap with — and overwrite — the entities in the de-duplicated default dataset.