Duplicate finder is the process of identifying duplicates in your CRM. Duplicates can be found in all databases regardless if you use solutions such as dupe blockers or have very strict processes. However, not all duplicates are bad. There are many reasons why you would want to have duplicates in your system. Some of the reason includes: selling into multiple divisions of the same company, accounting needs to maintain separate records from your other teams, or selling multiple products to the same company. Therefore, your first step is to establish what makes a duplicate a true duplicate.

Step one is duplicate finder process is to establish what makes a true duplicate. For example, you may want only one record per each company regardless of how many locations are found in the system. Or a duplicate may mean that only if the company name and address is the same and division is the same then it is a duplicate. Each business will have different duplicate rules, which is the reason that a lot of the automated duplicate finder tools may not work at identifying true duplicates for your business.

Step two identifying duplicates, now that you have your rules you can identify duplicates based on your rules. If you CRM is under 10,000 rows you may even use excel. One trick is to add a field and use the concatenate function to combine the columns that you will want to match on. Then use conditional formatting to identify all the duplicates in your newly created field. You can also write a query in your SQL database to identify any instances where data is the same. Now this is very time consuming and will ignore similar data that is not identical. For example, if address is 123 St. and another record has the address 123 Street these will not match using excel or basic code. You will need to use fuzzy logic to find duplicates with the above addresses. Fuzzy logic matches similar data to each other, the problem with fuzzy logic is that it will catch data that is not the same. For example, ABC corporation and ABC partnership is not the same company but fuzzy logic will most likely flag them as the same.

Most likely you will identify duplicates using multiple fields, this is where a professional de-duping tool or de-duplication services can help you identify all duplicates in your system, usually at a fraction of the cost and time. Some of the fields that you can use to identify duplicates include:

  • Address
  • Name
  • Phone number
  • Email
  • Website
  • Duns #
  • Email Domain

The final step once you have identified all duplicates is to verify that they are in fact duplicates. You can verify it two ways, the first is manually check every record. The second is to calculate the confidence level of your duplicate match. To calculate the confidence level, you would see what percentage of your records match. For example, if you compare website, address, name and phone number and all 4 fields match each other you would get 100% confidence level, if only 1 of those fields match you would get 25% confidence level. You would then manually check all the duplicate groups with low confidence level to make sure you are comfortable with de-duping.

Now all that’s left is to update your CRM post duplicate finder, usually you have three choices once you have identified all duplicates. First is to mark the master record using business rules again, and deleting all other records. If you were to use this approach you want to make sure you copy all data to the new master record where the field is blank. You also want to make sure you copy over any associated notes, contacts, contracts and so on.

The second approach is to simply mark the non master duplicates as “dupe” in your record type. This approach ensures you do not miss any data that you can reference. You can also add the ID or name of the master record on the duplicate records so you can easily reference the active record.

The final approach, is to establish parent/child relationships. For example, if you have 20 duplicates that belong to the same company but they are in different locations you may want to keep the records but establish a master using the head-office address. You would add a new field called parent account where you will populate the name of your master account.

These are just some strategies to identify duplicates and de-dupe in your CRM, however, each business is different and will have different business needs. For professional data de-duplication of your data, contact StrategicDB at hello@strategicDB.com or +1-877-332-4923  for your free data audit.