Skip to Main Content

Systematic and Complex Reviews

Supplementary searching

Supplementary searching refers to search strategies used to address the known bias in database searching and to compensate for incomplete or missing indexing of journal articles. Supplementary searching comprises a number of different and complementary search strategies.

Although supplementary searching is mandatory according to both the Cochrane Handbook and the Joanna Briggs Institute, the number and selection of the strategies that you use will be determined by the scope and resources of your project. Most importantly the strategies used in the supplementary searches should be transparent, and documented well enough to be a reproducible as possible.

The CLUES mnemonic is a useful way to remember different strategies that can be used in supplementary searching. CLUES is a modification of the CLUSTER mnemonic developed by Booth et al. (2013). See the reference list for information on the original CLUSTER mnemonic.

CLUES method

There are two types of citation tracking: Backward and Forwards.

Backwards Citation Tracking refers to the process of scanning the reference list of key articles remaining in the final data analysis group of records. You conduct a backward citation scan to identify any new (previously unidentified) articles that meet your eligibility criteria. Include these articles in your data analysis. If the list is long, you can prioritize which articles you scan, perhaps the most cited or the most current. Keep a record of the sources searched and the resources located.

Forward Citation Tracking refers to looking for articles that have cited the articles included in your final data analysis group. Include any of these new (previously unidentified) articles that meet your eligibility criteria in your data analysis. If the list is long, you can prioritize which articles you scan, perhaps the most cited or the most current? Keep a record of the sources searched and the resources located.

Hand searching is a process of searching a specific journal rather than database, in an attempt to locate new (previously unidentified) records. The selected journals may be those that are appearing most commonly in the final data analysis group, a journal that is highly regarded within the industry or discipline or recommended by a subject specialist. Keep a record of the sources searched and the resources located. Limit this kind of search to 2-3 journals (Aromataris & Riitano, 2014).

Most articles will list a corresponding author. Email the authors of the articles included in your final data analysis group. Let them know the nature of your project and ask them if they can provide you with any leads to other articles, reports, studies or unpublished material that they think should be included in the analysis. If the list is long, you can prioritize which authors you contact, perhaps the most cited? Alternatively, you can brainstorm within your team to identify any key researchers not already identified and contact them in a similar way. Keep a record of the sources searched and the resources located.

A lot of research material is conducted by PhD students and is contained in theses and dissertations. Theses can be located in special databases such as:

Many universities will also have their own repositories available for searching e.g. ResearchDirectKeep a record of the sources searched and the resources located.

Grey material

Grey material is the most common form of supplementary searching that is reported in systematic reviews and scoping reviews. Grey material is any material that is published by organizations whose primary role is not publication, an example is a government report. There are many different types of grey material that will be relevant to projects or reviews e.g. clinical trials, organizational papers and reports, position statements, and parliamentary papers to name a few. Most of this material will be published on the internet. Here are some sources to locate this type of material.

Clinical trials

Locating unpublished clinical trials is essential for any kind of effectiveness review. There are a number of different directories you can search to locate this kind of material.

Organisational websites

  • Brainstorm a list of organizations relevant to the topic of the project or review. Locate their webpage. Use the search function of the site and keywords to locate resources. Alternatively look for relevant tabs on the web page to locate material for example: Research or Publications. This is sometimes called crawling the site.

Domain searching

  • Generally specific types of organizations are allocated specific types of URLs domain names. ICANN (2013) is the not-for-profit organization coordinating the allocation and assignment of these Internet’s unique identifiers. Some examples include:      an organizational or not for profit site (generally) site:.asn      an associations site      a government site      an educational site, in the UK

    Using this structure, it is possible to filter the grey material on the internet to locate more relevant information for a project or review. As with Google Scholar searches, multiple simpler and separate consecutive searches may be needed.

    Add site:domain to the web browser search string to restrict the search to those sites only. You can also extend the domain name to refine for country e.g. for Australian government, or for state government sites.

Grey material repositories

  • You may be able to locate grey material via Open access repositories. An open access repository is a searchable collection of resources available via online databases accessed through the internet. They may provide direct access for links and information about where this material may be stored. A University Repository is an example of a resource that may provide access to grey material for example thesis or reports written by research students and academic staff.

Other examples of grey material repositories include:

Ask an expert in the field or your stakeholders or supervisors for any recommendations of relevant articles or unpublished material. Keep a record of the sources searched and the resources located.

Google Scholar is not a database that is recommended for a systematic search, however it can be very useful when conducting a supplementary search. Use Google Scholar to locate theses, reports, or conference papers relevant to your project. Google Scholar searches will need to use multiple, separate simpler search strings in consecutive searches. Truncation and proximity searching work in different ways to databases and wildcards cannot be applied. Here is a list of other limitations with a Google Scholar search.

  • Google Scholar has a 256-character limit to its search string. Any terms beyond that will be deleted from the search.
  • Google Scholar does not use truncation. It uses automatic stemming. Automatic stemming identifies the ‘root’ of the keywords used, automatically searching for any additional letters. For example, the keyword 'learn' retrieves ‘learner’, ‘learning’, ‘learns’, ‘learnable’. But the keyword 'sociology' would not retrieve related keywords such as ‘sociologist’, ‘sociological’ as these synonyms do not contain the stem word 'sociologist '. Also stemming only works for complete words i.e. 'sociologist' rather than ‘sociolo’. This feature can be turned off by putting a '+' symbol in front of the keyword. For example: +learn will only return records that contain learn (6,870,000 results) rather than learn (6,910,000 results), this total would include records with 'learn', ‘learner’, ‘learning’, ‘learns’ and ‘learnable’.
  • Using the asterisk for truncation in Google Scholar returns whole word results rather than alternative endings e.g. road* will return roadway but not roadie.
  • Wildcard for variations in spelling do not work in Google Scholar.
  • There are some commands you can use to narrow the search e.g.: intitle:, allintitle:, allinurl:, intext:, allintext:, AROUND(X). You can find more info here advanced-search-operators/
  • You can also find some ways to limit a Google Scholar search in the Advanced Search option. Use with caution, these limits are extremely basic.

The amount of material that Google Scholar can locate is overwhelming and much of it will be irrelevant. Prior to searching, decide in your team on some limits. e.g., Decide how many different simple search strings you will use, set page number results to a specific number, 20, 50? Then decide how many pages you are prepared to review, 1, 2, 3? How many you decide will depend on the  resources in your team and the expectation or scope of the project. Keep a record of the sources searched and the resources located.

Download the work tool

The amount of supplementary searching results that you could consider for a review is vast. The amount and which methods you use will be determined by the scope of your project or research and the expectations of your discipline. Consider how much manpower, time and resources you have available within your project team. It is not expected that you will use all the strategies available. What is expected is that you are transparent about what strategies were used and document with enough detail to make your searches as reproducible as possible.


Aromataris, E., & Riitano, D. (2014). Constructing a search strategy and searching for evidence. A guide to the literature search for a systematic review. The American journal of nursing114(5), 49–56., A., & Carroll, C. (2015). Systematic searching for theory to inform systematic reviews: Is it feasible? Is it desirable? Health Information & Libraries Journal, 32(2), 220-235.

Booth, A., Harris, J., Croot, E., Springett, J., Campbell, F., & Wilkins, E. (2013). Towards a methodology for cluster searching to provide conceptual and contextual “richness” for systematic reviews of complex interventions: Case study (CLUSTER). BMC Medical Research Methodology13(1), 118.

ICANN. (2013). A beginner guide to participating in ICANN.

© Western Sydney University, unless otherwise attributed.
Library guide created by Western Sydney University Library staff is licenced under a Creative Commons Attribution 4.0 International (CC BY)