RSSChangelog

Please check that your configuration supports HTTP status 307 redirects for HTTP requests to data.opensanctions.org. Starting in August, the site may redirect users attempting to fetch a specific data export to an updated location.

The canonical location will now be at /artifacts/<dataset>/<version>/, where previously a mix of datasets/latest/<dataset> and /datasets/YYYYMMDD/<dataset> were used.

The goal of this change is provide immutable locations for each data export, helping to more clearly identify versions of the dataset.

Due to excessive bandwidth use, we are planning to limit access to historical data exports of our sanctions data (pre-2024). Bulk data customers and non-commercial users can request authenticated access by contacting our support team.

The following URL paths will become access-controlled:

• https://data.opensanctions.org/datasets/2021*
• https://data.opensanctions.org/datasets/2022*
• https://data.opensanctions.org/datasets/2023*
• https://data.opensanctions.org/graph/*

We kindly ask all users that retrieve bulk data extracts from data.opensanctions.org to ensure that the HTTP clients used to fetch bulk files are configured to support redirect response codes such as HTTP 302 (Found), 303 (See other), and 307 (Temporary Redirect).

Background: as our archive grows, we may need to virtualize some of our content delivery, or put in place redirects for deprecated locations within the data archive.

This change is relevant for users of the index.json and catalog.json metadata. API and yente users as well as those users who download static files based on location are not affected.

As the number of datasets in the default collections grows, the metadata size is becoming a factor: the main index file is now larger than two megabytes. In order to prevent scaling issues in the future, we're splitting up the dataset metadata into two files: index.json and statistics.json. index.json will contain fewer summary facts about each dataset. The extended statistics (e.g. the number of entities in the dataset linked to each country) will be in the more in-depth statistics.json.

The following fields will be removed from dataset metadata:

• schemata
• properties
• targets (and all nested items)
• things (and all nested items)

These fields continue to be published in a statistics.json file. A statistics.json file is published with each data export of a dataset, and referenced from the index.json via the statistics_url field.

Additionally, the following fields will be removed in favor of replacements that are already in index.json:

• sources - use datasets instead
• externals - use datasets instead