AI Tools for Evidence Synthesis

How do we define Artificial Intelligence (AI)?

It is the theory and development of a computer system capable of performing tasks that typically require human intelligence, such as visual perception, speech recognition, decision-making, and language translation. 

Examples of AI: 

• Automated synthesis of information
• Natural language processing
• Speech and text analysis 
• Facial recognition

Evidence Synthesis

AI has become an increasingly valuable asset in the realm of systematic and scoping reviews, as well as other evidence synthesis. These advanced tools can be leveraged at various stages of the process:

- Search strategy development
- Identification of relevant articles and resources
- Data screening and extraction
- Synthesis of information
- Creation of plain language summaries

While AI tools offer significant advantages, experts emphasize the importance of:

1. Understanding potential biases and limitations
2. Using new AI tools alongside established, validated methods
3. Considering ethical implications, copyright issues, and intellectual property concerns

The National Institute for Health and Care Excellence (NICE) position statement regarding the use of AI in evidence generation and reporting

This statement aims to:

- Outline expectations for AI usage in evidence processes
- Provide guidance on regulations, best practices, and standards
- Support committee members and assessment groups in evaluating AI applications

A collaborative effort led by international organizations, including:

- International Collaboration for Automation in Systematic Reviews
- Cochrane and Campbell
- JBI (formerly Joanna Briggs Institute)

This initiative is developing guidance and recommendations for the responsible use of AI in evidence synthesis, currently in draft form for consultation and revision.

As AI continues to evolve, its role in systematic reviews and evidence synthesis is likely to expand, necessitating ongoing evaluation and adaptation of best practices.

APU’s guidance on generative AI for teaching, assessment, and feedback

• Syllabus Guidelines
• See the academic integrity policy in the current Academic Catalog. 
• Guidelines for Citing AI-Generated Text in Dissertations and Theses at APU

Purpose

• Using AI as a mediating step in between sections of the systematic review process

• Creates efficient operations and reduces the amount of time spent on more time-heavy portions

• Using AI as an aid to make faster decisions

• Increasing transparency and clarity in review questions

Strategies

• Determine the strengths and weaknesses of different sections of the systematic review process

• Identify the areas that take the most amount of time

• Assess the risk in automation 

• Talk to the research and library team about where automated processes would benefit the process

Human Review Primary (in between first and second step):

• AI can synthesize information to form a protocol

• Checking to make sure elements of DEI are included in the protocol and all components are present

Human Review Secondary (in between the second and third step):

• Autogenerated search strings

• Automated literature selections; Conducting the quality check after return results 

Human Review Tertiary (in between the third and fourth step):

• Automated selection of studies; review selection criteria and process

• Automated data extraction; review the type of data and what is included and excluded

• Automated synthesis of data; review for any biases and exclusions

Machine Bias

• Overestimations of research data input

• Inaccurate or unfair predictions

• Information exclusion

• Over specification 

• Discrimination against specific groups

Research Bias

• Lack of representation for marginalized groups in medical research 

• Grey literature may not always be considered

Equity Considerations

• May not consider equitable practices

• With the presence of machine discrimination, equity may go out the window

• Equity can be highlighted from the human lens

To strengthen the processes that utilize AI, it is essential to provide feedback and speak up about any inconsistencies or biases observed in the intermediate reviews. Also, always remember to assess the role of AI in your project and document its use in the methods section.

For more information on writing a Systematic or Scoping Review, go to the Library Research Guide. 

The following tools are available via subscription or limited free use; some are AI-based, while others do not utilize AI.

• Systematic Review Toolbox

  The Systematic Review Toolbox is an online catalog of tools that support various tasks within the systematic review and wider evidence synthesis process.

• Rayyan

  Free web tool designed to speed up the process of screening and selecting studies

• Abstrackr

  Aids in citation screening. Please note you will need to create a free account before accessing the tool.

• DistillerSR

  An online application designed to automate all stages of the systematic literature reviews. "Another powerful tool for screening, DistillerSR uses AI to assist in the initial screening of titles and abstracts. It can prioritize studies based on relevance, apply predefined criteria, and flag potential duplicates. DistillerSR also supports collaborative work, allowing teams to share screening tasks and discuss decisions in real-time" (Verma, 2024). Additionally, it is a valuable tool for the full-text screening phase. Priced packages are available (please note we cannot offer support on using this system).

• Elicit 
  Use in keyword suggestions and data extraction. This AI tool specializes in enhancing research questions by suggesting relevant keywords related to your PICO elements. By providing initial terms, Elicit can generate a list of related keywords that might be used in literature searches, ensuring comprehensive coverage. Full data extraction, including adding additional fields that may not be standard. It can be prompted to extract aspects of interest from the results, such as population, outcomes measured, main findings, etc. Information can be downloaded via RIS or CSV with a subscription. This YouTube video has a great example of data extraction from an uploaded PDF in comparison with SciSpace.

• SciSpace
  Full data extraction, including adding additional fields that may not be standard. It can be prompted to extract aspects of interest from the results, such as population, outcomes measured, main findings, etc. Information can be downloaded via RIS or CSV with a subscription. This YouTube video has a great example of data extraction from an uploaded PDF in comparison with Elicit.

• ExaCT (not currently available)

  Information Extraction system. Please note you will need to request a free account. The system is trained to find key information from scientific clinical trial publications, namely the descriptions of the trial's interventions, population, outcome measures, funding sources, and other critical characteristics. Please note you will need to request a free account. Research article on this product. 

• RobotReviewer

  RobotReviewer is a machine learning system that aims to support evidence synthesis. The demonstration website allows users to upload RCT articles and see automatically determined information concerning the trial conduct (the 'PICO', study design, and whether there is a risk of bias).

• EPPI-Reviewer 
  An application for all types of literature review, including systematic reviews, meta-analyses, 'narrative' reviews, and meta-ethnographies. It is suitable for small or large-scale reviews (with some of our existing reviews containing over a million items). Connects with Zotero to help you locate full text. Visualizer, mapping, and multiple other awesome tools in the product to create reviews. Supports the development of systematic reviews from study screening through data collection, analysis, and synthesis. It supports authors and editors in writing all types of reviews, particularly in complex areas including meta-analysis, framework synthesis, and thematic synthesis. Used with Cochrane reviews and Campbell Reviews, and is free for authors who write for those publishers. 

• RevMan (ReviewManager)

  RevMan is Cochrane's bespoke software for writing Cochrane reviews. RevMan has been designed to integrate with other systematic review software, and new features and updates are added regularly. Cochrane review authors can log in to RevMan to view the dashboard (all reviews) and edit reviews online. Watch a 6-minute YouTube tutorial for authors using RevMan. RevMan is now available for non-Cochrane reviews. Click here to find out more.

• Covidence
  Covidence allows you to collaborate online with your co-authors on tasks such as uploading and sharing search results, screening and selecting studies, data collection, and risk of bias assessment.

• ASReview

  A free and open source tool that uses active learning as you screen to sort the remaining items from most to least likely to be included in your review. Like Rayyan, it doesn't do the screening for you. It sorts items to screen based on your previous inclusion/exclusion decisions.

• Litsearchr

  The web interface for the R package litsearchr helps identify search terms and generate reproducible search strategies. The web version and the R package are both completely free of charge.

Recording of 1 hour webinar exploring Artificial Intelligence (AI) and its potential impact on the process of systematic reviews (August 15th, 2023). Note PICO Portal is a systematic review platform that leverages artificial intelligence to accelerate research and innovation.

Moderator Dr Greg Martin. Presenters: Eitan Agai - PICO Portal Founder & AI Expert; Riaz Qureshi - U. of Colorado Anschutz Medical Campus; Kevin Kallmes - Chief Executive Officer, Cofounder; Jeff Johnson - Chef Design Officer.

An update on machine learning AI in systematic reviews

June 2023 webinar including a panel discussion exploring the use of machine learning AI in Covidence (screening & data extraction tool).

Web Clinic: Artificial intelligence (AI) technologies in Cochrane

The session was delivered in May 2024 and you will find the videos from the webinar, together with the accompanying slides to download [PDF]. Recordings from other Methods Support Unit web clinics are available here.

Part 1: How Cochrane currently uses machine learning: implementing innovative technology
Part 2: What generative AI is, the opportunities it brings and the challenges regarding its safe use
Part 3: Cochrane's focus on the responsible use of AI in systematic reviews
Part 4: Questions and answers

The CLEAR path: A framework for enhancing information literacy through prompt engineering.

This article introduces the CLEAR Framework for Prompt Engineering, designed to optimize interactions with AI language models like ChatGPT. The framework encompasses five core principles—Concise, Logical, Explicit, Adaptive, and Reflective—that facilitate more effective AI-generated content evaluation and creation.

Lo, L. S. (2023). The CLEAR path: A framework for enhancing information literacy through prompt engineering. The Journal of Academic Librarianship, 49(4), 102720.

A selection of recently published articles exploring AI tools in evidence synthesis:

• Khalil, H., Ameen, D., & Zarnegar, A. (2022). Tools to support the automation of systematic reviews: A scoping review. Journal of Clinical Epidemiology, 144, 22–42. https://doi.org/10.1016/j.jclinepi.2021.12.005

• Kiritchenko, S., de Bruijn, B., Carini, S., Martin, J., & Sim, I. (2010). ExaCT: automatic extraction of clinical trial characteristics from journal publications. BMC medical informatics and decision making, 10, 56. https://doi.org/10.1186/1472-6947-10-56

• Verma, P. (2024, September 1). How to write systematic reviews step by step with the help of ai tools. LinkedIn. https://www.linkedin.com/pulse/how-write-systematic-reviews-step-help-ai-tools-dr-preeti-rrs4c

• Hill, G. (2024, October 15). Libguides: Systematic and literature reviews: using ai’s in reviews. Royal Malborne Hospital Library. https://libguides.mh.org.au/systematic_and_literature_reviews/_AI

• Flanagin, A., Kendall-Taylor, J., & Bibbins-Domingo, K. (2023). Guidance for authors, peer reviewers, and editors on use of AI, language models, and chatbots. JAMA, 330(8), 702. https://doi.org/10.1001/jama.2023.12500

• Flanagin, A., Pirracchio, R., Khera, R., Berkwits, M., Hswen, Y., & Bibbins-Domingo, K. (2024). Reporting use of AI in research and scholarly publication—Jama network guidance. JAMA, 331(13), 1096. https://doi.org/10.1001/jama.2024.3471

• Affengruber, L., van der Maten, M.M., Spiero, I. et al. An exploration of available methods and tools to improve the efficiency of systematic review production: a scoping review. BMC Med Res Methodol 24, 210 (2024). https://doi.org/10.1186/s12874-024-02320-4 https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-024-02320-4 
  Concludes: "For title and abstract screening and literature searching, various evaluated methods and tools are available that aim at improving the efficiency of SR production. However, only a few studies have addressed the influence of these methods and tools in real-world workflows. Few studies exist that evaluate methods or tools supporting the remaining tasks. Additionally, while validity outcomes are frequently reported, there is a lack of evaluation regarding other outcomes".

• Alshami, A.; Elsayed, M.; Ali, E.; Eltoukhy, A.E.E.; Zayed, T. Harnessing the Power of ChatGPT for Automating Systematic Review Process: Methodology, Case Study, Limitations, and Future Directions. Systems 2023, 11, 351. https://doi.org/10.3390/systems11070351
  Explores the use of ChatGPT in (1) Preparation of Boolean research terms and article collection, (2) Abstract screening and articles categorization, (3) Full-text filtering and information extraction, and (4) Content analysis to identify trends, challenges, gaps, and proposed solutions.

• Blaizot, A, Veettil, SK, Saidoung, P, et al. Using artificial intelligence methods for systematic review in health sciences: A systematic review. Res Syn Meth. 2022; 13(3): 353-362. doi:10.1002/jrsm.1553
  The review below outlines automated tools and platforms that employ artificial intelligence (AI) approaches, and evaluates the reported benefits and challenges associated with their use. They report the usage of Rayyan Robot Reviewer EPPI-reviewer; K-means; SWIFT-review; SWIFT-Active Screener; Abstrackr; Wordstat; Qualitative Data Analysis (QDA);  Miner and NLP and assess the quality of the reviews which used these.

• Janka H, Metzendorf M-I. High precision but variable recall – comparing the performance of five deduplication tools. JEAHIL [Internet]. 17Mar.2024 [cited 28Mar.2024];20(1):12-7. Available from: http://ojs.eahil.eu/ojs/index.php/JEAHIL/article/view/607 

• Kebede, MM, Le Cornet, C, Fortner, RT. In-depth evaluation of machine learning methods for semi-automating article screening in a systematic review of mechanistic literature. Res Syn Meth. 2023; 14(2): 156-172. doi:10.1002/jrsm.1589
  "We aimed to evaluate the performance of supervised machine learning algorithms in predicting articles relevant for full-text review in a systematic review." "Implementing machine learning approaches in title/abstract screening should be investigated further toward refining these tools and automating their implementation".

• Khalil H, Ameen D, Zarnegar A. Tools to support the automation of systematic reviews: a scoping review. J Clin Epidemiol 2022; 144: 22-42 https://www.jclinepi.com/article/S0895-4356(21)00402-9/fulltext 
  "The current scoping review identified that LitSuggest, Rayyan, Abstractr, BIBOT, R software, RobotAnalyst, DistillerSR, ExaCT, and NetMetaXL have the potential to be used for the automation of systematic reviews. However, they are not without limitations. The review also identified other studies that employed algorithms that have not yet been developed into user-friendly tools. Some of these algorithms showed high validity and reliability, but their use is conditional on user knowledge of computer science and algorithms."

• Khraisha Q, Put S, Kappenberg J, Warraitch A, Hadfield K. Can large language models replace humans in systematic reviews? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages. Res Syn Meth. 2024; 1-11. doi:10.1002/jrsm.1715
  "Although our findings indicate that, currently, substantial caution should be exercised if LLMs are being used to conduct systematic reviews, they also offer preliminary evidence that, for certain review tasks delivered under specific conditions, LLMs can rival human performance."

• Mahuli, S., Rai, A., Mahuli, A. et al. Application ChatGPT in conducting systematic reviews and meta-analyses. Br Dent J 235, 90–92 (2023). https://doi.org/10.1038/s41415-023-6132-y
  Explores using ChatGPT for conducting Risk of Bias analysis and data extraction from a randomised controlled trial.

• Ovelman, C., Kugley, S., Gartlehner, G., & Viswanathan, M. (2024). The use of a large language model to create plain language summaries of evidence reviews in healthcare: A feasibility study. Cochrane Evidence Synthesis and Methods, 2(2), e12041. https://onlinelibrary.wiley.com/doi/abs/10.1002/cesm.12041 

• Qureshi, R., Shaughnessy, D., Gill, K.A.R. et al. Are ChatGPT and large language models “the answer” to bringing us closer to systematic review automation?. Syst Rev 12, 72 (2023). https://doi.org/10.1186/s13643-023-02243-z
  "Our experience from exploring the responses of ChatGPT suggests that while ChatGPT and LLMs show some promise for aiding in SR-related tasks, the technology is in its infancy and needs much development for such applications. Furthermore, we advise that great caution should be taken by non-content experts in using these tools due to much of the output appearing, at a high level, to be valid, while much is erroneous and in need of active vetting."

• van Dijk SHB, Brusse-Keizer MGJ, Bucsán CC, et al.Artificial intelligence in systematic reviews: promising when appropriately used. BMJ Open 2023;13:e072254. doi: 10.1136/bmjopen-2023-072254 
  Suggests how to conduct a transparent and reliable systematic review using the AI tool ‘ASReview’ in the title and abstract screening.

• An open source machine learning framework for efficient and transparent systematic reviews

• Using artificial intelligence methods for systematic review in health sciences: A systematic review

• The Case for Artificial Intelligence in Systematic Reviews

• Systematic Review Writing by Artificial Intelligence: Can Artificial Intelligence Replace Humans?

• Rayyan AI Systematic Review System (a company that conducts systematic reviews with AI processing)

• Artificial intelligence and automation of systematic reviews in women's health

• Can Systematic Reviews Be Automated? 

• Artificial Intelligence in Systematic Reviews: Evaluation of Emerging Electronic Tools for Researchers in Knowledge Synthesis

• Abstrackr: Software for Semi-Automatic Citation Screening

• OpenMetaAnalyst: Powerful Open-Source Software for Meta-Analysis

• Text mining for searching and screening the literature: https://libraryguides.mcgill.ca/text-mining/home

• Clark J, McFarlane C, Cleo G, Ishikawa Ramos C, Marshall S. The Impact of Systematic Review Automation Tools on Methodological Quality and Time Taken to Complete Systematic Review Tasks: Case Study. JMIR Med Educ. 2021 May 31;7(2):e24418. doi: 10.2196/24418. PMID: 34057072; PMCID: PMC8204237.

• Cierco Jimenez R, Lee T, Rosillo N, Cordova R, Cree IA, Gonzalez A, Indave Ruiz BI. Machine learning computational tools to assist the performance of systematic reviews: A mapping review. BMC Med Res Methodol. 2022 Dec 16;22(1):322. doi: 10.1186/s12874-022-01805-4. PMID: 36522637; PMCID: PMC9756658. https://pubmed.ncbi.nlm.nih.gov/36522637/

Various AI tools are invaluable throughout the systematic review or evidence synthesis process. While the consensus acknowledges the significant utility of AI tools across different review stages, it's imperative to grasp their inherent biases and weaknesses. Moreover, ethical considerations such as copyright and intellectual property must be at the forefront.

• Application ChatGPT in conducting systematic reviews and meta-analyses

• Are ChatGPT and large language models “the answer” to bringing us closer to systematic review automation?

• Artificial intelligence in systematic reviews: promising when appropriately used

• Harnessing the power of ChatGPT for automating systematic review process: methodology, case study, limitations, and future directions

• In-depth evaluation of machine learning methods for semi-automating article screening in a systematic review of mechanistic

• Tools to support the automation of systematic reviews: a scoping review

• The use of a large language model to create plain language summaries of evidence reviews in healthcare: A feasibility study

• Using artificial intelligence methods for systematic review in health sciences: A systematic review

Publishers may have different policies regarding the use of generative AI and how to cite it. Check your publisher's information for the authors' webpage, or contact their editorial staff, for details.

If using a chatbot or other generative AI-created content, here are ways to acknowledge that usage:

• Cite it in the text and references of your work
• Describe your use as part of your methodology
• Include an appendix with screenshots or transcripts of prompts and AI-generated responses

Examples of different citation styles: