Systematic reviews: Dedupe

Why use EndNote for deduping?

• EndNote can be set up to better spot duplicates next to each other (see instructions) 
• As at October 2024 we are not aware of a better tool available to SVHM staff

How can SVHM staff access EndNote?

• Onsite (SVHM computers) - contact IT if not already installed on your computer
• Offsite (personal computers) - email the library for software (staff only). Specify PC or Mac.

How to remove duplicates using EndNote

Removing duplicates in Deduplicator - https://sr-accelerator.com/#/deduplicator

Pros:

• Free online
• Good layout - records appear above each other, and fields such as DOI are easily compared
• Potential duplicates are broken into groups according to likelihood of correct matches
• No need to check the group classified as "Non duplicates" - so less records to compare

Cons:

•  "Likely Duplicates" appears to group records by title only, regardless of different authors, DOI or journal title
• This requires "splitting" groups to compare more similar records eg those with the same article and journal titles
• "Splitting" groups adds extra steps to the deduplication process, which takes time
•  "Splitting" large numbers of suggested duplicates would be very time consuming - and this would take longer than comparing records in EndNote.

Most useful for:

• small to medium sets of records being compared eg less than 2000 before deduplication.

Note: Deduplicator tested and reviewed by HW 21 Sept 2023

Removing duplicates in Covidence systematic review software (not currently recommended)

Pros:

• Covidence software available to SVHM staff on request
• Potential duplicates removed automatically from records imported for screening
• Good at picking up duplicates via DOI
• Can pick up additional duplicates missed using other systems

Cons:

• Testing in Oct 2024 showed the bulk of duplicate records were missed (which means they continued to title and abstract screening by 2 reviewers) and some records automatically removed were not actually duplicates and had to be reinstated
• Suggested duplicates still need to be checked manually
• Often does not differentiate between articles with the same title published in different journals (not duplicates)
• Difficult layout to compare - records appear side by side
• Can quickly become unwieldy if you have many potential duplicates to check

Most useful for: 

• a second check for additional duplicates after removing bulk of duplicates with another tool such as EndNote
• people without access to EndNote or an alternative method for deduping
• people using Covidence for screening anyway (don't use it just for deduplication)
• small number of records to import eg less than 500

Sample screenshot showing how you can compare duplicates automatically removed in Covidence (highlights added):

We do NOT currently recommend using Covidence for duplicate removal. Testing in Oct 2024 showed that it missed the bulk of duplicates and also incorrectly removed some records that were not duplicates. However Covidence is excellent software for other systematic review processes.

This was how we tested:

• On 10 Oct 2024 a librarian uploaded 740 records from Medline and Embase to Covidence - 292 from Medline and 448 from Embase. 
• Covidence automatically removed only 35 duplicates. 
• These databases have significant overlap which we expected to see reflected in the number of duplicates removed. We had expected about 200 duplicates.
• The librarian deduped the same set of 740 records manually in EndNote, simply sorting by title and checking titles, journal, authors, dates, page numbers to verify duplicate records.
• Deduping manually in EndNote identified 220 duplicates from the same set - so 185 extra duplicates.
• The remaining 520 records were uploaded into Covidence
• Covidence identified 4 further duplicates
• Manual checking showed that only 1 of the 4 further duplicates removed by Covidence was actually a duplicate.

As a result we do not currently recommend Covidence as a tool for accurately removing duplicate records, but will revisit this in the future.