Duplicate Candidate Records: Why They Happen, What They Cost, and How to Prevent Them

Duplicate candidate records are not a sign of a careless team. They are what a working desk produces. In our audit of 3 million records at one staffing firm, roughly 10% were duplicates and another 10% held no usable company information. The cost is not storage. It is that the history of one person sits split across three records, and no single one of them tells you what happened.

How duplicate candidate records actually enter

Four routes, and none of them is anybody doing anything wrong.

Repeat applications

The same person applies to three of your roles over two years, through different channels, sometimes with a different email each time. Each application looks new because, to the system, it is.

Manual uploads

A recruiter pulls a CV from a job portal and uploads it without checking whether the person already exists. Checking takes forty seconds and there are two hundred CVs, so nobody checks.

Referrals and forwards

A CV arrives by email with a different name spelling, a personal address instead of a work one, or as an attachment somebody renamed.

Bulk imports

A migration or a purchased list lands without matching against what is already there. This is the one that creates duplicates in the tens of thousands rather than the tens.

The pattern is that all four are normal throughput. Which is why deduplication has to be continuous rather than an annual project that somebody eventually stops doing.

What identifier to match on, and where each one fails

Every dedup strategy is a bet on one identifier being stable. None of them is.

Our approach is to use the email address as the unique key, with the most recent CV retained on the record. That handles the common case cleanly and predictably, which matters more than it sounds: an unpredictable matching rule creates a different problem, where recruiters stop trusting that a record is complete.

The gap is worth naming plainly. A candidate who applied from a work address in 2023 and applies from a personal address in 2026 will create a second record. If your database is older than a few years, that pattern is a meaningful share of your duplicates, and it is the reason a periodic manual review still earns its place.

Why prevention beats cleanup

Every firm that has run a database clean-up has run it twice, because the first one did not hold.

A clean-up is a snapshot. The day after it finishes, applications, uploads and referrals resume at the same rate they always did, and the duplicate count starts climbing from zero again. Twelve months later somebody proposes a clean-up.

Matching at entry is the only intervention that compounds. It costs nobody any time, it applies to every record forever, and its effect is invisible, which is why it rarely gets prioritised over a project with a visible finish line.

The practical implication for anyone evaluating systems: ask what happens at the moment a record is created, not what the bulk merge tool looks like. The merge tool is a repair. The matching is the fix.

What a merge has to preserve

This is where the real damage happens, and it is permanent when it goes wrong.

A merge should retain, from every source record: all notes, all call recordings and transcripts, all email and message threads, all attachments and CV versions, all submission and placement history, every stage change with its timestamp, and every custom field value.

The reason is specific rather than abstract. Somewhere in your database is a note from 2023 explaining that a particular client rejected a particular candidate and why. Lose that note in a merge and you will submit them to that client again, and you will find out about it in front of the client.

The same logic applies to CV versions. Retaining the latest CV is the right default, because it is the current one. Keeping the earlier versions alongside it is better, because the 2022 CV sometimes contains a project the 2026 one dropped, and that project is occasionally the reason someone is a fit.

The other 10 percent: junk records

Separate problem, separate answer.

Junk is a record with no usable company or role information. Not enough to identify what the person does, let alone place them. In the audit, that was roughly 10% of the database.

These should be deleted rather than archived, and deliberately. A junk record is noise in every search result you will ever run, and it is a compliance exposure. Under India's DPDP Act and the GDPR, holding personal data with no lawful basis and no defensible retention position is a liability rather than an asset. Junk records fail both tests: you cannot justify keeping them and you cannot use them.

Data decay, and why stale is not the same as useless

The other 80% of that database were unique, real profiles that were simply out of date.

Three fields decay at three different rates. Job title and employer go first, because people move. Work email addresses die with the job. Phone numbers are the most durable, particularly in India where personal numbers travel between employers.

The important distinction: a backend engineer whose record says they were at a fintech in 2022 has almost certainly moved. They are still a backend engineer. What is stale is the record, not the person, and the fix is enrichment rather than deletion. We covered the economics of that in the talent pool audit.

What duplicates actually cost you

The cost is rarely storage. It shows up in four places, and three of them are invisible in any report.

Search results you cannot trust

A search returning the same person three times looks like three candidates. Recruiters learn to distrust counts, and a database recruiters distrust is one they stop searching, which returns you to paying a job portal for people you already own.

History that contradicts itself

One record says the candidate was rejected by a client. Another says they were never submitted. Both are true of their own fragment, and neither is true of the person.

Duplicate outreach

The same candidate receives the same campaign twice, sometimes from two recruiters on the same day. This is the one candidates notice and mention, and it costs more brand damage than anything else on this list.

Wrong benchmarks

Conversion rates calculated across a database with 10% duplicates are wrong by roughly that margin, which quietly corrupts every capacity and cost calculation built on top of them.

A dedup and hygiene routine that survives contact with a real desk

Most data hygiene plans fail because they require someone to do something that is never urgent. Three rules that do not.

Match at entry, not at cleanup. Every new record checks against the existing database before it is created. This is the only step that scales, because it costs nobody any time.

Refresh on use, not on a schedule. When a record surfaces in a search, check whether its contact data is older than your threshold and refresh it then. Effort concentrates on the records actually being worked.

Re-verify on every conversation. Current title, notice period and expectation should be captured automatically from the call rather than being a task for the recruiter, which we cover in automatic capture.

Cleaning up before a migration

The worst time to discover your duplicate rate is during a system move, and it is when most firms discover it.

Do not try to clean everything first. That is the most common reason migrations slip by months, and it is unnecessary. Segment instead.

1. Active and recent records move clean. These are the ones being worked, and they justify the effort.
2. Historical records with placement history move as they are and get refreshed on use. They have value and they are not urgent.
3. Junk is deleted before the move, not carried across. Migrating unusable data means paying to move it and then still having it.

Run the audit before the migration rather than after, because the numbers change what you migrate. A firm that discovers 20% of its database is duplicates or junk usually decides to move 80% of it, and the move gets faster as a result.

Frequently asked questions

Three signs your database has a duplicate problem

Before running a formal audit, three symptoms that usually mean the answer is yes.

Recruiters open a job portal by default. The clearest signal. Nobody bypasses a database they trust. If the first move on every new role is a fresh portal pull, the team has already concluded the internal search is not worth running.

Candidates mention being contacted twice. If this has come up more than once in a quarter, it is not an anomaly. It is a matching failure that happens to have become visible.

Two recruiters claim the same placement. Ownership disputes are often a duplicate problem wearing a commission costume. Two records, two owners, one person.

How to audit your own database this week

Take 500 records at random. Check three things: does the phone number work, is the job title still current, and does the person appear more than once.

The three percentages you get back are the real condition of your database. They are almost always worse than the record count suggests, and they are the only figures that will settle an internal argument about whether this is worth fixing.