How Should a SaaS App Handle Duplicate Records? A Framework

How Should a SaaS App Handle Duplicate Records? A Four Layer Framework

Handle them in four layers: prevent at the point of entry, detect on a schedule, present the evidence for a decision, and make resolution safe. Most products build only the fourth layer, which is why their users spend their afternoons merging records that should never have existed.

Duplicates are the quiet tax on every CRM, every applicant tracker, every inventory tool, and every product with a contacts table. They are also almost entirely a design problem rather than a data problem, because the moment to stop a duplicate is before it is saved.

Here is the framework we use, layer by layer, and where each one earns its cost.

Layer One: How Do You Prevent Duplicates at Entry?

Search before you create. The single highest value pattern is a create form whose first field searches existing records and shows matches as the user types. If the record already exists, they pick it. Nothing else you build will catch as many duplicates as this.

Nielsen Norman Group's framing of user error is the right lens. It distinguishes slips, which "occur when users intend to perform one action, but end up doing another (often similar) action", from mistakes, where "users have goals that are inappropriate for the current problem or task." A duplicate is usually a slip. Somebody meant to find a record and created one instead.

Their prevention advice maps directly onto this. "Include Helpful Constraints", "Offer Suggestions" such as autocomplete, "Choose Good Defaults", and "Use Forgiving Formatting". Forgiving formatting matters more than it sounds: if your email field rejects a trailing space, users retype and mistype, and a mistyped email is a new record.

Layer Two: What Should Automatic Detection Actually Do?

Run on a schedule against a defined set of fields, and surface candidates rather than acting. HubSpot documents that it "automatically compares record property values daily to surface potential duplicates", which is the right shape: continuous, background, and non destructive.

The field list is the design decision. HubSpot's duplicates manager compares contacts on "First Name, Last Name, Email address, IP country, Phone number, Zip Code, and Company Name", and companies on "Company Domain Name, Company Name, Country/Region, Phone Number, and Industry."

Notice that neither list is a single key. Matching on email alone misses the same person with a work and a personal address. Matching on name alone floods you with false positives. A useful detector scores across several fields, and the fields it uses should be visible to the user so they can understand why two records were paired.

Layer Three: How Do You Present a Possible Duplicate?

Side by side, with the differences highlighted and the evidence for the match stated. A user cannot make a good merge decision from two record names. They need to see which fields agree, which conflict, and what each record has that the other does not.

Show the asymmetries that make the decision. Which record is older, which has more activity attached, which one your integrations are writing to. A record with two hundred logged emails and a record with none are not equal candidates for survival, and the interface should say so rather than making the user work it out.

Give a clear way to say no. HubSpot's tool lets users reject pairs as well as merge them, and rejection has to be as prominent as merging. A queue that only offers "merge" trains people to merge things they should not, because the alternative is leaving the item there forever.

Layer Four: What Makes a Merge Safe?

Being reversible, or being extremely explicit that it is not. This is the hardest constraint in the whole area, and most products get it wrong by being quiet about it. HubSpot is at least direct: "merged records cannot be reverted."

If your merge is irreversible, the interface owes the user three things. A preview of the exact resulting record, not a description of it. A named survivor, so it is obvious which identifier and which URL will continue to exist. And a confirmation that states plainly that this cannot be undone, rather than a generic "are you sure".

The asymmetry between actions is instructive. HubSpot allows a rejected pair to be undone "within 14 days" while a merge is permanent. That is exactly right: the reversible action can be casual, the irreversible one cannot. We went into that principle in our piece on undo and destructive actions.

Who Should Be Allowed to Merge?

Fewer people than can edit records. A merge destroys one record permanently and changes the shape of another, which puts it in a different class from editing a phone number. It belongs with deletion in your permission model, not with editing.

Bulk merge deserves a further restriction. Individual merges are considered decisions. Bulk merges are policies applied by somebody who did not look at each pair, and the damage scales. HubSpot itself gates bulk management behind a higher subscription tier, which is a commercial decision that happens to encode a sensible caution.

Whatever you decide, record who merged what and when. A merge log is the only way to answer the question that always arrives eventually, which is where a particular piece of data went. We covered the general pattern in our piece on designing an audit log.

What Happens to the Data You Do Not Keep?

Decide explicitly, and show the decision. When two records disagree on a field, something has to win, and users need to know the rule. Newest value wins, survivor wins, or the user picks per field are all defensible. Silently choosing is not.

Non conflicting data should combine rather than compete. Activities, notes, attachments, and associations from both records generally belong on the survivor, and losing them is the outcome users fear most when they hesitate over a merge button.

Say what happens to the losing record's identifier. If an integration or a bookmark points at it, does that link break, or does it resolve to the survivor? Resolving is much kinder and it is a small amount of work compared to the support burden of broken links.

How Does This Change With Imports and Integrations?

It becomes the main source of duplicates rather than a side case. Manual creation produces duplicates one at a time. A CSV import or a two way sync produces them in thousands, and it does so overnight when nobody is watching.

Every import needs a matching step before it runs, with a preview of how many records will be created versus updated. That one number prevents more duplicates than any detection job, because the user can see that an import they expected to update 400 contacts is about to create 400 new ones.

The same applies to integrations. If two systems both create records, define which one owns identity and what key they match on. We wrote about the import experience specifically in our piece on import and export UX.

How Do You Know the System Is Working?

Track the duplicate rate over time, not the merge count. A rising merge count can mean your detection improved or your prevention got worse, and those require opposite responses. The ratio of duplicates to new records is the number that tells you which.

Watch the rejection rate too. If users are rejecting most of the pairs you surface, your matching is too loose and you are wasting their attention. If they never reject anything, your matching may be too tight, or they may have stopped reading.

Finally, measure how long a duplicate lives. A duplicate caught at entry costs seconds. A duplicate caught six months later has already sent two emails to the same person and split a deal history in half, and no merge fully repairs that.

Where Should a Small Team Start?

With layer one, always. Search before create is a few days of work and it removes the majority of the problem at the point where it is cheapest. Teams routinely skip it and go straight to building a merge queue, which is solving the symptom with the most expensive tool available.

Add detection second, presentation third, and treat bulk merge as something you may never need. A product with excellent prevention and a plain, careful single merge flow is in better shape than one with a sophisticated queue and an open front door.

If you are designing this part of a product and want a second opinion on where the leaks are, we are happy to look at it with you. Find us at phoenix.studio.