Names are social and political constructs that differ across cultures: they can indicate gender and marital status, be passed down through generations, or even be inspired by constellations and zodiac signs.
The nuances of names become even more intricate when we consider different writing systems, as we explored in this previous post. Liuhuaying Yang notes in this visual essay that there are 8,000+ General Standard Chinese Characters, and about 16.1% have multiple pronunciations, with some pronounced differently in names, meaning that converting logographic names into Latin characters often leads to information loss.
At OpenSanctions, our approach to name cleaning isn't about forcing messy, real-world names into a single canonical string, but rather about building a shared, transparent linguistic infrastructure that we can continue to build on and adapt. In this blog post, you’ll learn how our new framework standardizes validation checks for irregular or miscategorized names, optionally uses LLMs to suggest fixes, and ensures names ingested from lists around the world speak a single, FollowTheMoney language.
Why does a name need to be cleaned?
OpenSanctions contains over 400 different data sources. These sanctions, watchlists, and corporate and Politically Exposed Person (PEP) registries can be as varied as the countries that publish them. List formats range from PDFs to CSVs and semi-structured web pages, and each jurisdiction has its own conventions for cramming nicknames, abbreviations, or transliterations into a single “Name” field.
These technical inconsistencies, combined with the cultural nuances mentioned above, mean that every name entering our database needs to be checked for "irregularities".
Some common examples of “irregular” names include:
THE NATIONAL BANK PLC (FORMERLY AL RAFAH MICROFINANCE BANK)(two names, one apreviousName)Aleksandr(Oleksandr) KALYUSSKY(KALIUSKY)(a name and some alternative transliterations)John Smith; Jonny Smith(another form of multiple versions of a name in a single string)
To clean them, we need to split the string of names and categorize each name into its correct bucket, separating primary names, aliases, and former names, and removing any superfluous text that is not part of the name.
| Category | Meaning |
|---|---|
name | The canonical name |
alias | A real alternate name (explicit AKA, nickname) |
weakAlias | Too generic/short to trust for confident matching (single surnames, acronyms) |
previousName | An explicitly former name ("formerly...") |
abbreviation | A short acronym-style org name |
The cleaning pipeline
We previously relied on ad hoc, crawler-specific methods to do this, but soon realized that this isn’t a sustainable approach due to the almost infinite ways to split a name. For instance, if a publisher makes a small formatting tweak that we don’t catch, a name could slip through the net, triggering false positives or false negatives downstream. The cleaning process is a balancing act: cleaning too much or too little pushes errors in both directions.
We’ve replaced this approach with a new name-cleaning framework that aims to do four things:
- Detect: We’ve defined specific constraints that automate flagging irregularities, such as numbered lists, semicolons, parentheses, or multiple names within a single string
- Propose a fix: When a name is detected as "irregular", a fix is suggested automatically where possible. The cleaning and categorization might be heuristic or LLM-based, depending on the options selected for each data source.
- Review: Automated fixes are reviewed by an analyst, who can correct any errors before accepting
- Monitor: A dashboard allows us to monitor reviews, alerting analysts when a name-cleaning suggestion has been made
This enables us to:
Pull out weakAliases more aggressively. A
weakAliasis a relatively broad or generic alias that should not be used for matching in screening systems, but it may still be useful for identification purposes, particularly in confirming a possible match triggered by other identifier information. If a name appears as a full alias in one source but as a weakAlias in another, we treat it as a weakAlias everywhere — this prevents fragments like Abu Muhammad (part of a much longer name) from being published as a full name, which could generate false positives.Separate abbreviations into their own field. Abbreviations and acronyms are often a source of false positive matches because they contain limited distinguishing detail. Separating them out lets the matcher use precise matching instead of fuzzy matching, as is already the case with weak aliases (for instance, IKEA and IAEA are two very different things, and fuzzy matching would conflate the two). For now, abbreviation values are also copied to the older weakAlias property for backward compatibility, until that backstop is removed.
Our review system shows the name strings from the source on the left and the final extraction on the right. Here you can see two transliterations of Юрий identified — Yurii and Yuriy — and the last name Zherebtsov.
The LLM proposes; the analyst gets the final say
At this stage, the LLM’s proposal isn’t added directly to the live entity, but as a pending suggestion that sits alongside the original string. Once an analyst accepts the review, the cleaned version replaces the original on a future run. If nobody's reviewed it yet, the untouched string ships instead. The tooling still needs improvement before we feel it can publish data fixes “unsupervised”.
Rather than a handwritten prompt, OpenSanctions treats the name-splitting prompt as an artifact that’s evaluated quantitatively with each update. It's developed and optimized using the Python framework DSPy with the GEPA optimizer, which uses a set of examples with a “correct” or “gold standard” version of what a “clean” value looks like to evaluate the performance of the prompt and LLM.
Every time we update our library of examples, we repeat the process to improve the prompt and evaluate the performance on both new and existing cases. If its performance is acceptable and has not regressed, we deploy it to production, and new cases using LLM-cleaning benefit from the updated prompt.
- strings: ["Aleksandr(Alexander/Oleksandr) KARAMAN"]
entity_schema: LegalEntity
name: [Aleksandr KARAMAN]
alias:
- Alexander KARAMAN
- Oleksandr KARAMAN
- strings: ["ATTIJARIWAFA BANK (FORMERLY BANQUE COMMERCIALE DU MAROC)"]
entity_schema: Company
name: [ATTIJARIWAFA BANK]
previousName: [BANQUE COMMERCIALE DU MAROC]
The example shows two “gold standard” name-cleaning examples. The first example is part of numerous similar examples that result in a rule in the prompt like:
alias- Use for alternative spellings, transliterations, complete parenthetical person-name variants, and generated full-name combinations of such variants.
Right now, we’re not using LLM-based cleaning for sanctions datasets due to their high-risk nature, but the rest of the name-cleaning framework already allows us to standardize how we clean those names.
One of the things we’re discovering is that it’s difficult to ensure consistency within the team and as individuals on a range of name nuances; for instance, what should be categorized as an abbreviation as opposed to a weak alias. Refining those policies is an important next step.
At the same time, the cases where we are already using LLM-based cleaning are collecting data from human corrections, which feed back into prompt improvements. These baselines are allowing us to improve the quality of the LLM-based extraction and develop a safety net to confidently expand the use of these tools.
Our warnings and errors dashboard allows us to monitor reviews, alerting analysts when a suggestion has been made
An open methodology that continues to mature
As we roll this out to more of our crawlers, we’re finding new ways in which names are "irregular" — noise we want to detect automatically, to trigger the cleaning pipeline.
As with our data reviews in general, the database of reviewed extraction cases provides a basis for assessing the accuracy of various approaches to both automated and manual data extraction and review. We’re continuing development in this space with the goal of reliably publishing normalized screening data and minimizing the publication window, all while prioritizing data quality.
As ever, transparency remains key: we’re rolling out continual improvements in data provenance through our statements-based format, including recording the original string through original_value, and capturing the process involved in cleaning values through the origin field.
