Systematic reviews: Dedupe

Why use EndNote for deduping?

• EndNote can be set up to better spot duplicates next to each other (see instructions) 
• As at October 2024 we are not aware of a tool available to SVHM staff that reliably removes duplicates more efficiently. 

How can SVHM staff access EndNote?

• Onsite it should be installed on your hospital computer. If not, contact the IT department.
• Offsite, library staff can provide installation files for your home computer. You will need an SVHM login. Please email us for more information. 
• Library staff can also provide individual EndNote tutorials and extensive online resources.

How to remove duplicates using EndNote

Image by Helen Wilding

Removing duplicates in Deduplicator - https://sr-accelerator.com/#/deduplicator

Pros:

• Free online
• Good layout - records appear above each other, and fields such as DOI are easily compared
• Potential duplicates are broken into groups according to likelihood of correct matches
• No need to check the group classified as "Non duplicates" - so less records to compare

Cons:

•  "Likely Duplicates" appears to group records by title only, regardless of different authors, DOI or journal title
• This requires "splitting" groups to compare more similar records eg those with the same article and journal titles
• "Splitting" groups adds extra steps to the deduplication process, which takes time
•  "Splitting" large numbers of suggested duplicates would be very time consuming - and this would take longer than comparing records in EndNote.

Most useful for:

• small to medium sets of records being compared eg less than 2000 before deduplication.

Note: Deduplicator tested and reviewed by HW 21 Sept 2023

Removing duplicates in Covidence systematic review software (not recommended as at Oct 2024)

Pros:

• Covidence software available to SVHM staff on request
• Potential duplicates removed automatically from records imported for screening
• Good at picking up duplicates via DOI
• Can pick up additional duplicates missed using other systems

Cons:

• Testing in Oct 2024 showed the bulk of duplicate records were missed (which means they continued to title and abstract screening by 2 reviewers) and some records automatically removed were not actually duplicates and had to be reinstated
• Suggested duplicates still need to be checked manually
• Often does not differentiate between articles with the same title published in different journals (not duplicates)
• Difficult layout to compare - records appear side by side
• Can quickly become unwieldy if you have many potential duplicates to check

Most useful for: 

• a second check for additional duplicates after removing bulk of duplicates with another tool such as EndNote
• people without access to EndNote or an alternative method for deduping
• people using Covidence for screening anyway (don't use it just for deduplication)
• small number of records to import eg less than 500

Sample screenshot showing how you can compare duplicates automatically removed in Covidence (highlights added):

• On 10 Oct 2024 a librarian uploaded 740 records retrieved from Medline and Embase to Covidence - 292 from Medline and 448 from Embase. 
• Covidence automatically removed only 35 duplicates. 
• These databases have significant overlap which we expected to see reflected in the number of duplicates removed. We had expected about 200 duplicates.
• The librarian deduped the same set of 740 records manually in EndNote, simply sorting by title and checking titles, journal, authors, dates, page numbers to verify duplicate records.
• Deduping manually in EndNote identified 220 duplicates from the same set - so 185 extra duplicates.
• The remaining 520 records were uploaded into Covidence
• Covidence identified 4 further duplicates
• Manual checking showed that only 1 of the 4 further duplicates removed by Covidence was actually a duplicate.

As a result we do not currently recommend Covidence as a tool for accurately removing duplicate records, but will revisit this in the future.