Proof-of-Concept Study: Utilizing AI for Streamlined Thematic Coding and Analysis in QES

This proof-of-concept study will evaluate the performance of AI (artificial intelligence) compared to standard manual input as a methodological shortcut in the QES process, focusing on thematic coding and analysis stages. Specifically, it will examine variations in resource demand, accuracy, depth, and concordance between AI-driven and manual approaches. Using an adaptive protocol to accommodate AI's rapid evolution, the study will employ Claude 2 for analysis. Thematic coding prompts will be generated from a random sample of non-open access QESs and verified by a second researcher. A convenience sample of 30 studies from three unpublished QESs across different health science topics (PROSPERO CRD42024531522; PROSPERO CRD42023430908; PMCID: PMC890566) will be analyzed. Data collection will involve an Excel table to record and compare outputs from manual and AI-generated processes. AI thematic coding will be conducted by uploading each study into Claude 2 with instructions to generate codes and quotations, which will then be compared to the original QES for accuracy, depth, and completeness. A blinded researcher will evaluate thematic coding results using a 5-point semantic similarity scale inspired by Wang et al. (2018). The analysis stage of the QES process will involve both manual and AI-driven methods, with AI codes uploaded for evaluation. Concordance between manual and AI analyses will also be assessed by a blinded researcher based on a pre-specified taxonomy. Findings from thematic coding and analysis will be narratively synthesized, providing an overall impression of AI's usefulness and reliability in the QES process.