Use AI-assisted data cleaning to propose duplicate candidates, then decide which records represent the same business object before merging anything. A normalized name is a useful search key. It is not an identity rule. Removing spaces, changing case or standardizing punctuation can bring inconsistent spellings together while also collapsing records that must remain separate.
For developers and operations teams, the practical safeguard is a reviewed mapping from source records to approved identities. Preserve the original rows, show the evidence for each proposed link and resolve conflicting fields separately. Do not let a request to “clean up duplicates” quietly authorize deletion, account consolidation or changes to related transactions.
This guide proposes a workflow and uses a fictional supplier-account example. Its small normalization example was checked locally with Python. No entity-matching model, AI assistant, production dataset or deduplication product was tested.
Decide what counts as the same thing
A business name can refer to several different objects: an organization, a trading brand, a billing account, a delivery site or a contact record. Those distinctions determine what a duplicate means.
Two records can describe the same organization and still be valid, separate accounts. Conversely, two differently spelled names can refer to one account entered twice. The matching question must name the object the application needs to preserve.
Before generating candidates, ask the data owner to state the identity rule and the evidence available to apply it. An authoritative account reference may settle some cases. Where identifiers are absent, reused or inconsistent across systems, the review will need other evidence and an unresolved state.
Do not infer identity from the presence of any convenient ID field. A source-row identifier distinguishes records in that source; it does not necessarily identify the underlying account. A reference from another system needs its namespace and meaning attached before it can be treated as an identity key.
Normalization helps retrieval without deciding identity
OpenRefine’s technical guidance on clustering distinguishes syntactic similarity from semantically informed reconciliation. Its clustering methods help find alternative-looking values, but similarity in character composition is insufficient to establish that the values denote the same thing.
That distinction also applies when a coding assistant writes a cleaning script. A transformation can be deterministic and still implement an unsuitable merge rule. A function that removes punctuation consistently does not prove that punctuation was irrelevant to the business distinction.
Retain both original and normalized values. Use the normalized form to organize a candidate set, while preserving the information needed to inspect why the strings differ. Avoid overwriting the source name before the review, especially if that destroys the clue that distinguishes two accounts or sites.
Keep candidate generation and final matching separately observable. If a relevant pair never enters the candidate set, even a good matcher cannot select it. If the set includes several similar but distinct records, a confident explanation is not enough to choose among them.
What entity-matching research adds
Wang and colleagues’ Match, Compare, or Select?, published at COLING 2025, investigates different ways to use language models for entity matching. The inspected arXiv revision is dated December 2024. It distinguishes independent pairwise judgments from comparisons and selection among candidates, and discusses consistency between record relationships.
The paper also examines limitations such as candidate-position effects and the difficulty of handling larger candidate sets. Its formulation makes assumptions about matching relationships that need checking before transfer to another application. Those experiments do not establish the reliability of today’s assistants on your supplier accounts.
Our implication is to inspect the proposed links together, not merely accept a series of isolated yes-or-no answers. If one record is linked to two accounts that the business says must remain distinct, the resulting group needs investigation. A fluent rationale for each individual link does not resolve the contradiction.
No benchmark gain or accuracy percentage from the study is used as a forecast for this workflow. The proposed review below remains an editorial method, not a replication of the paper’s system.
Three similar names, two supplier accounts
Imagine these fictional rows from an internal supplier export. For this example, the data owner has established that the account reference is authoritative for the object being cleaned. R01 and R02 are duplicate representations of account ACCT-A; R03 belongs to a separate account, ACCT-B.
| Source row | Supplied name | Approved account reference |
|---|---|---|
| R01 | Northstar Services | ACCT-A |
| R02 | North Star Services | ACCT-A |
| R03 | Northstar Services | ACCT-B |
A normalization that case-folds the name and retains only letters and digits produces northstarservices for all three rows. Grouping on that key produces one candidate group. Automatically retaining just one row per key would collapse two distinct accounts.
The correct output under the stated fictional rule is a mapping with three source links and two account identities. R01 and R02 point to ACCT-A; R03 points to ACCT-B. We checked those grouping and mapping counts locally. This is a constructed example, not an observed AI error or a measured matching result.
The example deliberately supplies decisive account evidence. Real data may not. If the account field were missing or disputed, the appropriate result could be “needs investigation,” not a guessed replacement account. The assistant should expose the missing evidence rather than invent a plausible identifier.
Ask for evidence and conflicts before a merge plan
Give the assistant a bounded task: propose candidate links using approved fields, preserve source IDs and identify supporting and conflicting information. Require a reason for each candidate and allow both “different records” and “insufficient evidence” outcomes.
A useful proposed review record contains the source-row IDs, proposed target identity, evidence used, contradictions, decision status and rule version. Keep model-supplied reasoning separate from source facts. “The names look like spelling variants” is an interpretation; the actual account references are supplied data whose reliability still depends on the source.
Robin Linacre’s 2022 Ministry of Justice practitioner explanation of Splink illustrates how different field agreements and disagreements contribute different amounts of evidence in probabilistic linkage. This is a historical account of a developed linkage approach, not a test of generative-AI cleaning or an endorsement of our fictional account rules.
The transferable caution is to consider what a field means in context. Do not assign an AI-generated confidence number the status of a calibrated match probability unless that interpretation has been established for the actual system and data.
Resolve field values separately from record links
Even an accepted match does not tell you which address, contact detail or status should survive. The records may contain values from different dates or systems. Choosing the most complete-looking row can discard a more authoritative value in another row.
Define field-level retention rules with the data owner. Identify the source of each retained value, preserve unresolved conflicts and avoid filling gaps from general model knowledge. The identity decision and the attribute-selection decision should be reviewable independently.
Retain source-to-target links so dependent records can be reconciled. If transactions refer to original account IDs, establish how those references behave before applying a consolidation. Prefer preparing a mapping and change preview over deleting rows as the first cleaning step.
A backup alone is not evidence that a merge can be reversed cleanly. Check what the application changes and what later actions could depend on the consolidated record. Scope a pilot to data and operations the responsible team can inspect and recover.
Test mistakes in both directions
Build an approved evaluation set that includes true duplicates with different spellings, distinct accounts with the same name, missing identifiers and conflicting account evidence. Include cases outside the proposed candidate groups so missed duplicates do not disappear from evaluation.
Check false merges and missed matches separately. They have different consequences, and a single accuracy figure can conceal which mistake the process makes. A review that only examines accepted links cannot establish how many valid matches were never proposed.
Keep the initial task read-only. Review candidate coverage, proposed groups, field conflicts and downstream references before authorizing changes. Record what was checked and what remains unresolved; a few constructed examples are not evidence that the entire dataset has been validated.
Our guide to AI-written SQL joins addresses a related downstream problem: how record relationships can distort totals. Here the earlier responsibility is to preserve the right identities before analysis or updates depend on them.
Start with one bounded export and one clearly defined account identity. Produce a candidate list and reviewed mapping first. A useful cleaning process should make the reason for keeping two similar names separate just as visible as the reason for linking two different spellings.