News

Guidance on terminology, application, and reporting of citation searching: the TARCiS statement

admin
admin

  1. Julian Hirt, research fellow and lecturer1 2 3,
  2. Thomas Nordhausen, research fellow4,
  3. Thomas Fuerst, medical information specialist5,
  4. Hannah Ewald, medical information specialist5,
  5. Christian Appenzeller-Herzog, medical information specialist5
  6. on behalf of the TARCiS study group

    1. 1Pragmatic Evidence Lab, Research Centre for Clinical Neuroimmunology and Neuroscience Basel, University Hospital Basel and University of Basel, Basel, Switzerland

    2. 2Department of Health, Eastern Switzerland University of Applied Sciences, St Gallen, Switzerland

    3. 3Department of Clinical Research, University Hospital Basel and University of Basel, Basel, Switzerland

    4. 4Institute of Health and Nursing Science, Medical Faculty, Martin Luther University Halle-Wittenberg, Halle (Saale), Germany

    5. 5University Medical Library, University of Basel, 4051 Basel, Switzerland
    1. Correspondence to: C Appenzeller-Herzog christian.appenzeller{at}unibas.ch

Evidence syntheses adhering to systematic literature searching techniques are a cornerstone of evidence based healthcare. Beyond term based searching in electronic databases, citation searching is a prevalent search technique to identify relevant sources of evidence. However, for decades, citation searching methodology and terminology has not been standardised. An evidence guided, four round Delphi consensus study was conducted with 27 international methodological experts in order to develop the Terminology, Application, and Reporting of Citation Searching (TARCiS) statement. TARCiS comprises 10 specific recommendations, each with a rationale and explanation on when and how to conduct and report citation searching in the context of systematic literature searches. The statement also presents four research priorities, and it is hoped that systematic review teams are encouraged to incorporate TARCiS into standardised workflows.

Synthesising scientific evidence by looking at the citation relationships of a scientific record (ie, citation searching) was the underlying objective when the Science Citation Index, the antecedent of Web of Science, was introduced in 1963.1 Although the availability of electronic citation indexes has increased, evidence syntheses in systematic reviews do not primarily rely on citation searching for literature retrieval but rather on search methods based on text and keywords.2 When used in systematic review workflows, citation searching traditionally constitutes a supplementary search technique that builds on an initial set of references from the primary database search (seed references).3

Citation searching is an umbrella term that entails various methods of citation based literature retrieval (fig 1). Checking references cited by seed references, also known as backward citation searching, is the most prevalent and a mandatory step when conducting Cochrane reviews.4 In forward citation searching, systematic reviewers can also assess the eligibility of articles that cite the seed references. Backward and forward citation searching are known as direct citation searching (fig 1). They can be supplemented by indirect retrieval methods—namely, by co-citing citation searching (retrieving articles that share cited references with a seed reference) and co-cited citation searching (retrieving articles that share citing references with a seed reference).

Fig 1

Overview of citation searching methods. Direct (dark blue boxes) and indirect (light blue boxes) citation relationships of references are shown, relative to a seed reference; arrows denote the direction of citation (ie, source A citing target B); horizontal axis denotes time (ie, the chronology in which references were published relative to the seed reference). Visual examples of cited references (accessible via backward citation searching), citing references (accessible via forward citation searching), co-citing references (accessible via co-citing citation searching), and co-cited references (accessible via co-cited citation searching) are shown. Note that the total number of the co-citing and co-cited references of a seed reference far exceeds the number shown in the light blue boxes

Citation searching can contribute substantially to evidence retrieval and can show similar or even superior effectiveness and efficiency compared with text and keyword based searches. An audit of the different search methods used in a systematic review of complex evidence, for instance, revealed that 44% of all included studies were identified by backward citation searching, and 7% by forward citation searching. In comparison, initial text and keyword searches accounted for only 25% of included studies.5 For the scoping review that collected methodological studies as a foundation for the present work, these figures were 28% and 12% for backward and forward citation searching, respectively, compared with 52% for extensive primary database searching.6

The conduct of systematic reviews is prominently guided by standard recommendations such as those in the Cochrane handbook,4 whereas their reporting is standardised by the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement.7 In contrast and despite its application by systematic reviewers for decades, standardised methodology and terminology for citation searching is not available. Of the three aspects on when to do citation searching, how to conduct citation searching, and how to report citation searching, limited guidance exists only for the third aspect in the PRISMA extension for reporting literature searches (PRISMA-S).8 Unsurprisingly, methodological studies show considerable heterogeneity in terms of citation searching terminology and recommended best practices.6 Even in a sample of Cochrane reviews, 13% did not use backward citation searching despite this being a mandatory step.9 The lack of standardisation not only impairs the transparency, reproducibility, and comparability of systematic reviews, but might also reduce article recall that could affect pooled effect estimates, guidance, and clinical decision making. On the other hand, uninformed use of citation searching in contexts where it is less useful might cause undue workloads.

We systematically collected evidence on the use, benefit, and reporting of citation searching6 and put it through a four round, online Delphi study. Together with the Terminology, Application, and Reporting of Citation Searching (TARCiS) study group, an international panel of methodological experts, we aimed to develop consensus for recommendations on when and how to conduct citation searching, and on how to report it, including a consensus set of citation searching terms. Furthermore, we framed research priorities for future methodological development of citation searching in the context of systematic literature searches.

Summary points

  • The TARCiS (Terminology, Application, and Reporting of Citation Searching) statement provides guidance in which contexts citation searching is likely to be beneficial for systematic reviewers

  • TARCiS comprises 10 specific recommendations on when and how to conduct citation searching and how to report it in the context of systematic literature searches, and also frames four research priorities

  • The statement will contribute to a unified terminology, systematic application, and transparent reporting of citation searching and support those who are conducting or assessing citation searching methods

Methods

To develop the TARCiS statement, a stepwise approach comprising a scoping review of the methodological literature (step 1; reported in detail in a separate publication6) and a Delphi study (step 2; reported in this publication) was chosen. The methods were prespecified in two study protocols.1011 The complete process is shown in figure 2.

Fig 2
Fig 2

Flow diagram of the development process of the TARCiS (Terminology, Application, and Reporting of Citation Searching) statement. Actions and outcomes of the development phases of the TARCiS statement are shown. Appendix 1 shows more detailed reporting of consensus scores

Step 1: Scoping review

We conducted a scoping review on the terminology that describes citation searching, the methods and tools used for citation searching, and its benefit. We considered methodological studies of any design that aimed to assess the role of citation searching, compared multiple citation searching methods, or compared technical uses of citation searching within health related topics. We searched five bibliographic databases, conducted backward and forward citation searches of eligible studies and pertinent reviews, and consulted librarians and information specialists for further eligible studies. The results were summarised by descriptive statistics and narratively. The detailed methods of the scoping review have been published elsewhere.610

Step 2: Delphi study

To develop consensus on recommendations and research priorities as tentatively derived from the results of step 1,6 we performed a multistage online Delphi study. Delphi refers to a structured process where collective knowledge from an expert panel is synthesised using a series of questionnaires, each one questionnaire adapted on the basis of the responses to a previous version.121314 We recruited an international panel of individuals experienced in conducting or reporting citation searching methods. For this, we invited authors of methodological studies, as identified in step 1,6 and methodological experts from international systematic review organisations or from our professional networks by email to participate in the Delphi study.

The Delphi study comprised four prespecified rounds.1011 The first round was pretested by four non-study related academic affiliates. Each round covered four to five thematic parts (appendix 2; table 1). Briefly, part A dealt with the terminology framework to describe citation searching methods in eight domains (for details, refer to table 4 in Hirt et al6). Part B contained pre-formulated recommendations on conduct and reporting of citation searching. Each recommendation was supported by a rationale and explanation text that were also subjected to collective consensus finding. Part C covered research priorities that were also derived from the scoping review.6 Part D contained a free text field to collect general comments from the panellists. Part E was designed to collect sociodemographic information and was limited to Delphi round 1.

Table 1

Data collection through four rounds of Delphi study to develop consensus on recommendations and research priorities of the TARCiS statement

Non-participating panellists were recorded as non-participators for a given round. Panellists who missed all rounds were recorded as non-responders. Recommendations and research priorities that had not yet reached the prespecified consensus of at least 75% were refined for the subsequent Delphi round. These refinements were based on the panellists’ comments. In rare cases, when additional valid suggestions from panellists for reformulation of rationale or explanation texts were submitted, recommendations that already reached the agreement threshold were also adapted and forwarded to the next Delphi round. For more methodological details on the Delphi study, see table 1 and the published protocols.1011

Deviations from the Delphi study protocol

For round 3 of the Delphi, we had originally planned to formulate one recommendation for each of the eight terminology domains (table 1, see also description to part A above). Depending on the votes, however, this approach might have led to the selection of inconsistent terms (eg, backward citation searching v forward citation tracking). Hence, we decided to use the terms that received the most votes in Delphi round 2 to formulate four term sets, which were consistent across all eight domains. Secondly, instead of using SosciSurvey15 as a survey tool,8 we switched to the Unipark/Enterprise Feedback Suite survey,16 which provided enhanced design and functional features. Thirdly, in addition to personalised emails (person based approach), we originally intended to recruit panellists using professional mailing lists and central requests to systematic review organisations (organisation based approach).8 However, because we had already recruited sufficient panellists using the person based approach (including individuals who were affiliated with various systematic review organisations), we waived the organisation based approach.

Results

Step 1: Scoping review

We identified 47 methodological studies that assessed the use, benefit, and reporting of citation searching. In 45 studies (96%), the use of citation searching showed an added value. Thirty two studies (68%) analysed the impact of citation searching in one or more previous systematic reviews. Application, terminology, and reporting of citation searching were heterogeneous. Details on the results of the scoping review can be found elsewhere.6

Step 2: Delphi study

Recruitment and characteristics of panellists

Of 35 experts identified and contacted, 30 declared an interest in participating and were invited to Delphi round 1. Three (10%) of the 30 panellists were non-responders. Table 2 summarises the personal and professional characteristics of the 27 participating panellists.

Table 2

Characteristics of 27 panellists* participating in the Delphi study to develop consensus on recommendations and research priorities of the TARCiS statement

TARCiS statement: final recommendations, rationale and explanations, and research priorities

Items for data collection through the four Delphi rounds in parts A-E are summarised in table 1. The Delphi study started with 41 terms describing different aspects of citation searching, eight draft recommendations with rationale texts on the conduct and reporting of citation searching, and one research priority (appendix 1). After Delphi round 4, the finalised TARCiS statement comprised 10 recommendations with rationale and explanation texts and four research priorities that reached consensus scores between 83% and 100%. Figure 2 and appendix 1 show details on content and consensus scores in rounds 1-4. An overview of all 14 TARCiS items omitting rationale and explanation texts is presented in box 1. A terminology and reporting item checklist based on TARCiS recommendations 1 and 10 is available in appendix 3 and on the TARCiS website.17

Box 1

TARCiS statement

Recommendations on terminology, conduct, and reporting of citation searching

  1. The following terminology should be used to describe search methods that exploit citation relationships:

    • “Citation searching” as an umbrella term.

    • “Backward citation searching” to describe the sub-method retrieving and screening cited references.

    • “Reference list checking” to describe the sub-method retrieving and screening cited references by manually reviewing reference lists.

    • “Forward citation searching” to describe the sub-method retrieving and screening citing references.

    • “Co-cited citation searching” to describe the sub-method retrieving and screening co-cited references.

    • “Co-citing citation searching” to describe the sub-method retrieving and screening co-citing references.

    • “Iterative citation searching” to describe one or more repetition(s) of a search method that exploits citation relationships.

    • “Seed references” to describe relevant articles that are known beforehand and used as a starting point for any citation search.

  2. For systematic search topics that are difficult to search for, backward and forward citation searching should be seriously considered as supplementary search techniques.

  3. For systematic search topics that are easier to search for and addressed by a highly sensitive search, backward and forward citation searching are not explicitly recommended as supplementary search techniques. Reference list checking of included records can be used to confirm the sensitivity of the search strategy.

  4. Backward and forward citation searching as supplementary search techniques should be based on all included records of the primary search (ie, all records that meet the inclusion criteria of the review after full text screening of the primary search results). Occasionally, it can be justified to deviate from this recommendation and either use further pertinent records as additional seed references or only a defined sample of the included records.

  5. Backward citation searching should ideally be conducted by screening the titles and abstracts of the seed references as provided by a citation index. Screening titles as provided when checking reference lists of the seed references can still be performed.

  6. Using the combined coverage of two citation indexes for citation searching to achieve more extensive coverage should be considered if access is available. This combination is especially meaningful if seed references cannot be found in one index and reference lists were not checked.

  7. Before screening, the results of supplementary backward and forward citation searching should be deduplicated.

  8. If citation searching finds additional eligible records, another iteration of citation searching should be considered using these records as new seed references.

  9. Standalone citation searching should not be used for literature searches that aim at completeness of recall.

  10. Reporting of citation searching should clearly state:

    • the seed references (along with a justification should the seed references differ from the set of included records from the results of the primary database search),

    • the directionality of searching (backward, forward, co-cited, co-citing),

    • the date(s) of searching (which might differ between rounds of iterative citation searching) (not applicable for reference list checking),

    • the number of citation searching iterations (and possibly the reason for stopping if the last iteration still retrieved additional eligible records),

    • all citation indexes searched (eg, Lens.org, Google Scholar, Scopus, citation indexes in Web of Science) and, if applicable, the tools that were used to access them (eg, Publish or Perish, citationchaser),

    • if applicable, information about the deduplication process (eg, manual/automated, the software or tool used),

    • the method of screening (ie, state whether the records were screened in the same way as the primary search results or, if not, describe the alternative method used), and

    • the number of citation searching results in the right column box of the PRISMA 2020 flow diagram for new or updated systematic reviews that included searches of databases, registers, and other sources.

Research priorities

  1. The effectiveness, applicability, and conduct of indirect citation searching methods as supplementary search methods in systematic reviewing require further research (including retrieval of additional unique references, their relevance for the review and prioritisation of results).

  2. Further research is needed to assess the value of citation searching. Potential research topics could be:

    • influence of citation searching on results and conclusions of systematic evidence syntheses,

    • topics or at least determinants of topics where citation searching likely/not likely has additional value, or

    • economic evaluation of citation searching to assess the cost and time of conducting citation searching in relation to its benefit.

  3. Further research is needed to assess the best way to perform citation searching. Potential research topics could be:

    • optimal selection of seed references,

    • optimal use of indexes and tools and their combination to conduct citation searching,

    • methods and tools for deduplication of citation searching results,

    • subjective influences on citation searching (eg, experience of researcher, prevention of mistakes), or

    • reproducibility of citation searching.

  4. Further research is needed to reproduce existing studies: Any recommendations in this Delphi that are based on only 1-2 studies require reproduction of these studies in the form of larger, prospectively planned studies that grade the evidence for each recommendation and propose additional research where the grade of evidence is weak.

  • The TARCiS checklist for terminology and reporting of citation searching is available for download.17

  • PRISMA=Preferred Reporting Items for Systematic reviews and Meta-Analyses; TARCiS=Terminology, Application, and Reporting of Citation Searching.

RETURN TO TEXT

Recommendation 1

The following terminology should be used to describe search methods that exploit citation relationships:

  • “Citation searching” as an umbrella term.

  • “Backward citation searching” to describe the sub-method retrieving and screening cited references.

  • “Reference list checking” to describe the sub-method retrieving and screening cited references by manually reviewing reference lists.

  • “Forward citation searching” to describe the sub-method retrieving and screening citing references.

  • “Co-cited citation searching” to describe the sub-method retrieving and screening co-cited references.

  • “Co-citing citation searching” to describe the sub-method retrieving and screening co-citing references.

  • “Iterative citation searching” to describe one or more repetition(s) of a search method that exploits citation relationships.

  • “Seed references” to describe relevant articles that are known beforehand and used as a starting point for any citation search.

Rationale and explanation supporting recommendation 1

As compiled in a recent scoping review,6 the reporting of citation searching methods is frequently unclear and far from being standardised. For example, “citation searching,” “snowballing,” or “co-citation searching” are sometimes used as methodological umbrella terms but also to denote a specific method such as backward or forward citation searching.6 For clarity, standardised vocabulary is needed. The set of terms brought forward in this recommendation is consistent in itself as well as with the terminology used in PRISMA-S and PRISMA 2020 guidelines818 and hence well suited for uniform reporting of citation searching.

Recommendation 2

For systematic search topics that are difficult to search for, backward and forward citation searching should be seriously considered as supplementary search techniques.

Rationale and explanation supporting recommendation 2:

Evidence indicates that the ability of citation searching as a supplementary search technique to find additional unique records in a systematic literature search varies between reviews.6 Searches for particular study designs (qualitative, mixed method, observational, prognostic, or diagnostic test studies) or health science topics such as non-pharmacological, non-clinical, public health, policy making, service delivery, or alternative medicine have been linked with effective supplementary citation searching.19202122 The underlying reasons include poor transferability to text based searching owing to poor conceptual clarity, inconsistent terminology, or vocabulary overlaps with unrelated topics.5 The ability of citation searching to find any publication type including unpublished or grey literature or literature that is not indexed in major databases (eg, concerning a developing country) might also be relevant.23 However, a clear categorisation of topics that are difficult to search for is currently not possible and it remains for the review authors themselves to judge whether their review topic is likely to fall into this category.

For people conducting the search who have difficulty assessing whether the topic is difficult or easier to search for, we recommend that they opt for citation searching or consult an experienced information specialist.24 If the search strategy does not exhaustively capture the topic, backward and forward citation searching might compensate for some of the potential loss of information.

Recommendation 3

For systematic search topics that are easier to search for and addressed by a highly sensitive search, backward and forward citation searching are not explicitly recommended as supplementary search techniques. Reference list checking of included records can be used to confirm the sensitivity of the search strategy.

Rationale and explanation supporting recommendation 3

Evidence indicates that the ability of citation searching as a supplementary search technique to find additional unique references in a systematic literature search varies between reviews.6 Searches for clearly defined clinical interventions as part of PICO (participant, intervention, comparison, outcome) questions have been linked with less effective supplementary citation searching, especially when the search strategies are sensitive and conducted in several databases. However, a clear categorisation of topics that are easier to search for is currently not possible, and it remains for the review authors themselves to judge whether their review topic is likely to fall into this category.

By checking reference lists within the full texts of seed references, review authors can test the sensitivity of their primary search strategy (ie, electronic database search).25 If no additional relevant, unique studies are found, the primary search might have been sensitive enough. If additional relevant, unique studies are found, these could indicate that the primary search was not sensitive enough.

For individuals conducting the search who have difficulty assessing whether the topic is difficult or easier to search for, we recommend that they opt for citation searching or consult an experienced information specialist.24 If, for whatever reason, the search strategy does not exhaustively capture the topic, backward and forward citation searching could compensate for some of the potential loss of information.

Recommendation 4

Backward and forward citation searching as supplementary search techniques should be based on all included records of the primary search (ie, all records that meet the inclusion criteria of the review after full text screening of the primary search results). Occasionally, it can be justified to deviate from this recommendation and either use further pertinent records as additional seed references or only a defined sample of the included records.

Rationale and explanation supporting recommendation 4

The more seed references used, the better the chance that citation searching finds additional relevant unique records. While using only a sample of the included records as seed references might be enough, there is currently no evidence that could help decide how many seeds are needed or how to decide which might perform better. Hence, we recommend using all the records that meet the inclusion criteria of the review after full text screening of the primary database search results.

However, review authors could deviate from this recommendation if they deal with a very small or large number of included records. A very small number of included records might not yield additional relevant records or only have limited value. In this case, review authors could use further records as seed references for citation searching (eg, systematic reviews on the topic that were flagged during the screening phase).26 A very large number of included records could lead to too many records to screen. In this case, review authors might use a selected sample of included records as seed references for citation searching. In the event of such deviation, authors should describe their rationale and sampling method (eg, random sample).

Recommendation 5

Backward citation searching should ideally be conducted by screening the titles and abstracts of the seed references as provided by a citation index. Screening titles as provided when checking reference lists of the seed references can still be performed.

Rationale and explanation supporting recommendation 5

Citation searching workflows encompass two consecutive steps: retrieval of records and screening of retrieved records for eligibility. When using an electronic citation index for citation searching, retrieval and screening are usually separated. While forward citation searching requires a citation index, backward citation searching can also be performed by manually checking the reference lists of the seed references. Reference list checking is sometimes part of an established workflow, for example, during the eligibility assessment of the full text record or during data extraction.25 Merging these two steps allows researchers to know the context in which a reference was used and that all references can be screened. However, reference list checking has three disadvantages:

  • The retrieval and screening phases are no longer separated, which makes reporting of the methods or results difficult and unclear

  • Citations from reference list checking cannot be deduplicated against each other or against the primary search results, which could add an unnecessarily high workload (see recommendation 7)

  • Eligibility assessments are restricted to the titles (instead of titles and abstracts) which could lead to relevant records being overlooked due to uninformative titles mentioned in vague contexts.

In recent years, online citation searching options via citation indexes or free to access citation searching tools have become more readily available leading to faster and easier procedures.27282930 More and even better tools to facilitate this workflow are expected in the future. Combining citation searching via citation indexes with automated deduplication (free online tools available)313233 makes this recommendation feasible. A caveat is that a search in one citation index will in most cases fail to retrieve all the cited references.3435 Thus, references to some documents (such as websites, registry entries, or grey literature) that are less likely to be indexed in databases might only be retrievable by checking reference lists or only in some citation indexes.3

Recommendation 6

Using the combined coverage of two citation indexes for citation searching to achieve more extensive coverage should be considered if access is available. This combination is especially meaningful if seed references cannot be found in one index and reference lists were not checked.

Rationale and explanation supporting recommendation 6

A single citation index or citation analysis tool might not cover all seed references and is likely to not find all the citing and cited literature. Citation indexes do not offer 100% coverage because some references are currently not indexed in one or several citation index(es)36 and because of data quality problems.37 Evidence indicates that when using more than one citation index for citation searching, the results of the different indexes can complement each other.383940 Thus, retrieval of backward and forward citation searching results from more than one citation index or citation analysis tool (eg, Lens.org via citationchaser, Scopus, citation indexes in Web of Science) followed by deduplication (see recommendation 7) can increase the sensitivity of citation searching. It is similar to the complementary effect of using multiple electronic databases for the primary database search, which is the preferred method in systematic search workflows.4 In recent years, online citation searching options have increased and many open access tools make rapid electronic citation searching universally accessible.27282930

Recommendation 7

Before screening, the results of supplementary backward and forward citation searching should be deduplicated.

Rationale and explanation supporting recommendation 7

The concept of citation searching as a supplementary search method relies on the notion that reference lists and cited-by lists of eligible references are topically related to these references.6 This topical relation implies a considerable degree of overlap within these lists leading to several duplicates. Furthermore, the overlap likely also extends to the results of the primary database search that was performed on the same topic. Based on these considerations and on the fact that the results of the primary database search have already been screened for eligibility, the screening load of citation searching results can be substantially cut by removing those references that have already been screened for eligibility (deduplication against the primary database search) and those references that appear as duplicates during citation searching.34 Depending on the method of deduplication, this procedure can be done in one go.

While deduplication can be conducted manually, standard bibliographic management software and specialised tools currently provide automated deduplication solutions, allowing for easier and faster processing.344142 If citation searching leads to only very few results, omission of the deduplication step can be considered to save time and administrative effort.

Recommendation 8

If citation searching finds additional eligible records, another iteration of citation searching should be considered using these records as new seed references.

Rationale and explanation supporting recommendation 8

Citation searching methods can be conducted over one or more iterations, a process that we refer to as iterative citation searching.43 The first iteration is based on the original seed references (see recommendation 4). If eligibility screening of the results of this first iteration leads to the inclusion of further eligible records, these records serve as new seed references for the second iteration, and so forth. Evidence indicates that conducting iterative citation searching can contribute to the identification of more eligible records.6434445

Iterations beyond the first round of citation searching require additional time and effort and could interrupt the ongoing review process, so the decision in favour of or against further iterations should be guided by an informal cost-benefit assessment. Relevant factors to be assessed include the review topic (difficult or easier to search for), sensitivity of the primary search, aim for completeness of the literature search, and the estimated potential benefit of the iteration(s) (eg, based on the number or percentage of included records found with the previous citation searching iteration).

Review authors should report the number of iterations and possibly the reason for stopping if the last iteration still retrieved additional eligible records. Furthermore, stating “citation searching was done on all included records” can lead to confusion. Most authors might mean all records were included after full text screening of the primary search results. But strictly speaking, “all included records” also includes those records retrieved via citation searching. The second interpretation implies that iterative citation searching is required until the last iteration leads to no further identification of eligible records.

As outlined in the rationale of recommendation 7, results of citation searching iterations can be deduplicated against all previously retrieved records to reduce the screening load.

Recommendation 9

Standalone citation searching should not be used for literature searches that aim at completeness of recall.

Rationale and explanation supporting recommendation 9

We refer to standalone citation searching when any form of citation searching is used as the primary search method without extensive prior database searching.6 This is contrary to citation searching as a supplementary search method to a primary database search. Seed references for standalone citation searching could, for example, be records from researchers’ personal collections or retrieved from less sensitive literature searches. Standalone citation searching can be based on a broad set of seed references. It can comprise backward and forward citation searching as well as indirect methods that collect co-citing and co-cited references.

When study authors have replicated published systematic reviews with standalone citation searching, they have mostly missed literature that was included in the systematic review.27464748 Since search methods for systematic reviews and scoping reviews should aim at completeness of recall, standalone citation searching is not a suitable method for these types of literature review.

Recommendation 10

Reporting of citation searching should clearly state:

  • the seed references (along with a justification should the seed references differ from the set of included records from the results of the primary database search),

  • the directionality of searching (backward, forward, co-cited, co-citing),

  • the date(s) of searching (which might differ between rounds of iterative citation searching) (not applicable for reference list checking),

  • the number of citation searching iterations (and possibly the reason for stopping if the last iteration still retrieved additional eligible records),

  • all citation indexes searched (eg, Lens.org, Google Scholar, Scopus, citation indexes in Web of Science) and, if applicable, the tools that were used to access them (eg, Publish or Perish, citationchaser),

  • if applicable, information about the deduplication process (eg, manual/automated, the software or tool used),

  • the method of screening (ie, state whether the records were screened in the same way as the primary search results or, if not, describe the alternative method used), and

  • the number of citation searching results in the right column box of the PRISMA 2020 flow diagram for new or updated systematic reviews that included searches of databases, registers, and other sources.

Rationale and explanation supporting recommendation 10

Relevant guidance for researchers conducting citation searching in systematic literature searching can be found in item 5 of PRISMA-S.8 Accordingly, required reporting items are the directionality of citation searching (examination of cited or citing references), methods and resources used for citation searching (bibliographies in full text articles or citation indexes), and the seed references that citation searching was performed on.8 Additional information for the reporting of citation searching can be found in PRISMA-S items 1 (database name), 13 (dates of searches), and 16 (deduplication).8 Although PRISMA-S can be seen as the minimum reporting standard for citation searching as a supplementary search technique, other important elements that emerged from our scoping review6 need to be reported to achieve full transparency or reproducibility. These elements are listed in recommendation 10 as a supplement to PRISMA-S to comprehensively guide the reporting of supplementary citation searching in systematic literature searching.

Concerning reporting of citation searching results in the PRISMA 2020 flow diagram,49 two variants are possible: reporting only those records that are additional to the primary search results after deduplication, or reporting all retrieved records followed by insertion of an additional box where the number of deduplicated records is reported.

Researchers should be aware that the detail of the citation searching methods do not have to be reported in the main methods of a study. Detailed search information can be provided in an appendix or an online public data repository.

Examples of good reporting

Example 1

“As supplementary search methods, we performed . . . direct forward and backward CT [citation searching] of included studies and pertinent review articles that were flagged during the screening of search results (on February 10, 2021). For forward CT, we used Scopus, Web of Science [core collection as provided by the University of Basel; Editions = SCI-EXPANDED, SSCI, A&HCI, CPCI-S, CPCI-SSH, BKCI-S, BKCI-SSH, ESCI, CCR-EXPANDED, IC], and Google Scholar. For backward CT, we used Scopus and, if seed references were not indexed in Scopus, we manually extracted the seed references’ reference list. We iteratively repeated forward and backward CT on newly identified eligible references until no further eligible references or pertinent reviews could be identified (three iterations; the last iteration on May 5, 2021).”6

Example 3

“Reference lists of any included studies and retrieved relevant SRs [systematic reviews] published in the last five years were checked for any eligible studies that might have been missed by the database searches.”51

Research priority 1

The effectiveness, applicability, and conduct of indirect citation searching methods as supplementary search methods in systematic reviewing require further research (including retrieval of additional unique references, their relevance for the review and prioritisation of results).

Rationale and explanation supporting research priority 1

Indirect citation searching involves the collection and screening for eligibility of records that share references in their bibliography or citations with one of the seed references (ie, co-citing or co-cited references).10 Indirect citation searching typically retrieves a large volume of records to be screened.4648 Therefore, prioritisation algorithms for the screening of records and cut-off thresholds that might discriminate between potentially relevant and non-relevant records have been proposed with the aim to reduce the workload of eligibility screening.2747 The methodological studies that have pioneered indirect citation searching methods for health related topics have so far exclusively focused on standalone citation searching.6 It is currently unclear whether the added workload and resources for searching and screening warrant indirect citation searching methods as supplementary search techniques in systematic reviews of any type (qualitative or quantitative studies, difficult or easier topics to search for).

Research priority 2

Further research is needed to assess the value of citation searching. Potential research topics could be:

  • influence of citation searching on results and conclusions of systematic evidence syntheses,

  • topics or at least determinants of topics where citation searching likely/not likely has additional value, or

  • economic evaluation of citation searching to assess the cost and time of conducting citation searching in relation to its benefit.

Research priority 3

Further research is needed to assess the best way to perform citation searching. Potential research topics could be:

  • optimal selection of seed references,

  • optimal use of indexes and tools and their combination to conduct citation searching,

  • methods and tools for deduplication of citation searching results,

  • subjective influences on citation searching (eg, experience of researcher, prevention of mistakes), or

  • reproducibility of citation searching.

Research priority 4

Further research is needed to reproduce existing studies: Any recommendations in this Delphi that are based on only 1-2 studies require reproduction of these studies in the form of larger, prospectively planned studies that grade the evidence for each recommendation and propose additional research where the grade of evidence is weak.

Discussion

TARCiS recommendations and research priorities

In keeping with our study aims, the TARCiS recommendations cover three aspects of citation searching in the context of systematic literature searches. They offer guidance regarding when to conduct, how to conduct, and how to report citation searching. The strength of each recommendation reflects the panellists’ assessment of the strength of evidence to support them.

In systematic evidence syntheses, citation searching techniques can be used to fill gaps in the results of primary database searches, but their application is not universally indicated. TARCiS recommendations 2 and 3 provide critical assistance in cost-benefit considerations (ie, whether a systematic search is likely to benefit from the use of citation searching). Systematic searchers of defined pharmaceutical interventions, for instance, might take from this guidance that they can skip citation searching because their primary database search might already allow for high sensitivity at reasonable specificity and expedite other supplementary search techniques, such as clinical trial registry searching.52 Accordingly, TARCiS does not recommend the use of citation searching in easier-to-search-for topics, but—as formulated in research priority 2—more research is needed to more reliably discriminate between topics that are easier to search for and those that are difficult to search for.

TARCiS recommendations 4-8 comprise guidance for technical aspects of citation searching. This guidance includes the selection of seed references, use of electronic citation indexes, deduplication, and iterative citation searching. While composing these recommendations, the TARCiS study group has considered that individual workflows must be framed in line with institutional licenses for subscription only databases and software. For illustration, one such workflow that is based on the licenses as provided by the University of Basel was deposited as an online video.53

Concerning guidance for reporting of citation searching, we developed a consensus terminology set for citation searching methods (TARCiS recommendation 1) as well as a recommendation for preferred reporting items for citation searching (TARCiS recommendation 10), along with a downloadable checklist.17 TARCiS recommendation 10 increases the reporting standards provided by PRISMA-S8 by dealing with the reporting of citation searching iterations, software tools that facilitate citation searching via a citation index, and the method of eligibility screening. Furthermore, TARCiS recommendation 10 standardises the reporting of citation searching results in the PRISMA 2020 flow diagram. We suggest that systematic reviewers, methodologists, journal reviewers, and editors use the TARCiS statement terminology and reporting checklist17 (appendix 3) as an additional checklist until future work by the PRISMA-S study group produces an updated reporting guideline that renders the TARCiS checklist obsolete.

Dissemination

TARCiS is intended to be used by researchers, systematic reviewers, information specialists, librarians, editors, peer reviewers, and others who are conducting citation searching or assessing citation searching methods. To enhance dissemination among these stakeholders, we aim to provide additional open access publications in scientific and non-scientific journals relevant in the field of information retrieval and evidence syntheses.

We have launched a TARCiS website (https://tarcis.unibas.ch/) and plan to disseminate the TARCiS terminology and reporting checklist17 on various platforms, including EQUATOR. We aim to make the TARCiS statement available via the Library of Guidance for Health Scientists (LIGHTS), a living database for methods guidance54; the Systematic Review Toolbox, an online catalogue of tools for evidence syntheses55; and ResearchGate, a social scientific network to share and discuss publications.

We will also share the TARCiS terminology and reporting checklist17 with editors of journals relevant in the field of information retrieval and evidence syntheses to request for inclusion in their instructions for authors and raise awareness of this topic. We hope that this effort will guide authors and peer reviewers to use TARCiS and assist their conduct, reporting, and evaluation of citation searching. We will also share the TARCiS statement with primary teaching stakeholders in evidence syntheses and systematic literature searching (eg, York Health Economics Consortium, RefHunter, Cochrane, Joanna Briggs Institute, and the Campbell Collaboration) and suggest its inclusion in future editions of their handbooks. We will present and discuss the TARCiS statement on international conferences and share our publications and presentations via relevant mailing lists and newsletters, X (formerly Twitter), and LinkedIn.

Limitations

A limitation of the TARCiS statement is that, despite the expectation and intent to recruit panellists from all parts of the world, their locations were limited to Australia, Europe, and North America. In addition, only a few panellists were recruited from countries where English was not the dominant language. Furthermore, both the evidence collected in our scoping review and the participating panellists are primarily involved with health related research. These considerations might reduce the generalisability of our recommendations and research priorities in other countries, languages, and research areas.

Conclusions

TARCiS comprises 10 specific recommendations on when and how to conduct citation searching and how to report it in the context of systematic literature searches. Furthermore, TARCiS frames four research priorities. It will contribute to a unified terminology, systematic application, and transparent reporting of citation searching and support researchers, systematic reviewers, information specialists, librarians, editors, peer reviewers, and others who are conducting or assessing citation searching methods. In addition, TARCiS might inform future methodological research on the topic. We encourage systematic review teams to incorporate TARCiS into their standardised workflows.

Ethics statements

Ethical approval

This study is based on published information and uses surveys of topical experts and therefore did not fall under the regulations of the Swiss Human Research Act, and we did not need to apply for ethical approval according to Swiss law. Data protection and privacy issues for the survey are outlined in the main text.

Data availability statement

The survey sheets and questionnaires used for this study are included in the supplementary content. Data generated and analysed during this study (except for sociodemographic information) are available on the Open Science Framework (https://osf.io/y7kh3).

Acknowledgments

We would like to dedicate this work to Cecile Janssens, who died soon after agreeing to join our Delphi panel. We thank Jill Hayden (Dalhousie University) and Claire Duddy (UK) for participating in our Delphi panel; and Christian Buhtz (Martin Luther University Halle-Wittenberg), Jasmin Eppel-Meichlinger (Karl Landsteiner University of Health Sciences), Tania Rivero (University of Berne), and Monika Wechsler (University of Basel) for participating in the pretest of the Delphi survey.

Footnotes

  • TARCiS study group: Alison Avenell (University of Aberdeen, UK), Alison Bethel (University of Exeter, UK), Andrew Booth (University of Sheffield, UK; and University of Limerick, Ireland), Christopher Carroll (University of Sheffield, UK), Justin Clark (Bond University, Australia), Julie Glanville (Glanville.info, UK ), Su Golder (University of York, UK), Elke Hausner (Institute for Quality and Efficiency in Health Care, Germany), Tanya Horsley (Royal College of Physicians and Surgeons of Canada, Canada), David Kaunelis (Canadian Agency for Drugs and Technologies in Health, Canada), Shona Kirtley (University of Oxford, UK), Irma Klerings (Donau University, Austria), Jonathan Koffel (USA), Paul Levay (National Institute for Health and Care Excellence, UK), Kathrine McCain (Drexel University, USA), Maria-Inti Metzendorf (Heinrich-Heine University Duesseldorf, Germany), David Moher (University of Ottawa, Canada), Linda Murphy (University of California at Irvine, USA), Melissa Rethlefsen (University of New Mexico, USA), Amy Riegelman (University of Minnesota, USA), Morwenna Rogers (University of Exeter, UK), Margaret J Sampson (Children’s Hospital of Eastern Ontario, Canada), Jodi Schneider (University of Illinois at Urbana-Champaign, USA), Terena Solomons (Curtin University, Australia), Alison Weightman (Cardiff University, UK)

  • Contributors: All authors made substantial contributions to conception and design, or acquisition of data, or analysis and interpretation of data; drafted the article or revised it critically for important intellectual content; and approved the final version to be published. JH, TN, TF, HE, and CA-H had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. JH, TN, TF, HE, and CA-H contributed to the study concept and methodology; acquisition, analysis, interpretation, validation, and visualisation of data; and critical revision of the manuscript for important intellectual content. JH, HE, and CA-H conducted the statistical analysis. JH and CA-H drafted the manuscript; provided administrative, technical, and material support; and supervised the study. The TARCiS study group authors are the Delphi panellists who were involved in Delphi rounds 1-4; they received the final manuscript draft for critical revision, important intellectual input, and approval for publication. CA-H is the guarantor for the study. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

  • Funding: The authors did not receive a specific grant for this study.

  • Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/disclosure-of-interest/ and declare: no specific support for the submitted work. CA-H received payments to his institution for a citation searching workshop by the University of Applied Sciences Northwestern Switzerland. JH received consulting fees from Medical University Brandenburg and payments for lecturing from the University of Applied Sciences Northwestern Switzerland, Catholic University of Applied Sciences, and Netzwerk Fachbibliotheken Gesundheit. From the TARCiS study group: JS received support from Alfred P Sloan Foundation; was funded by the US National Institutes of Health, National Science Foundation, US Office of Research Integrity, United States Institute of Museum and Library Services, and University of Illinois Urbana-Champaign; received book royalties from Morgan and Claypool; received consulting fees or honorariums from the European Commission, Jump ARCHES, NSF, and the Medical Library Association; received travel support by UIUC; contributes to the CREC (Communication of Retractions, Removals, and Expressions of Concern) Working Group; has non-financial associations with Crossref, COPE (Committee on Publication Ethics), the International Association of Scientific, Technical and Medical Publishers, the National Information Standards Organisation, and the Center for Scientific Integrity (parent organisation of Retraction Watch); and declares the National Information Standards Organisation as a subawardee on her Alfred P Sloan Foundation grant G-2022-19409. JG received payments for lecturing by York Health Economics Consortium. MJS received consulting fees at the Canadian Agency for Drugs and Technologies in Health and National Academy of Medicine (formerly Institute of Medicine) and for lecturing and support for attending a meeting at Institute for Quality and Efficiency in Health Care; and has a leadership role as secretary of the Ottawa Valley Health Library Association. AW received payments to her institution for a citation analysis workshop run via York Health Economics Consortium. SK declares non-financial interests as a member of the UK EQUATOR Centre and a coauthor of the PRISMA-S reporting guideline and was funded by Cancer Research UK (grant C49297/A27294); the current work was unrelated to this funding. PL is an employee of the National Institute for Health and Care Excellence. MR received payments by the Medical Library Association and declares non-financial interests as a member of the PhD programme affiliated with BMJ Publishing Group. ABo is a co-convenor of the Cochrane Qualitative and Implementation Methods Group and has authored methodological guidance on literature searching. All the other authors have no competing interests to disclose.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

Source link

TAGGED:

Categories

error: Content is protected !!