This data comes from a study to explore expert consensus on the importance of criteria for assessing certainty of evidence in systematic reviews on the effects of complex interventions. We conducted an online modified-Delphi process from March to May 2017 with 116 international experts in complex intervention research. Participants reviewed, rated, and discussed the importance of 50 criteria derived from a review of systems for assessing certainty of evidence on intervention effects. We analysed quantitative rating data using the RAND/UCLA Appropriateness Method and qualitative data using thematic analysis to identify areas of agreement on the importance of criteria. There was no significant disagreement among participants on the importance of any criterion. Sixteen criteria (32%) were determined to be critically important and had an interquartile range above the “critically important” threshold. Indirectness of the Evidence Base was the only domain for which all criteria were rated as critically important, while Initial Certainty Rating: Study Design was the only domain containing a criterion determined to be of limited importance. The remaining domains had a mix of criteria rated as either critically important or important but not critical. Participants’ comments produced several themes on intervention complexity and certainty criteria that elucidate the rationale for their ratings. Findings provide insight into experts views on how to incorporate a complexity perspective when assessing certainty of evidence in systematic reviews estimating the effects of interventions in public health. Findings from this study will inform ongoing initiatives for developing GRADE guidance specifically for complex interventions.It is now standard practice for policy makers to draw on systematic reviews as the superior source of evidence to inform decision-making regarding effective interventions. One of the crucial steps in systematic review conduct and recommendation formulation involves a synthesis and rating of the quality of research evidence to help determine its validity and relevance for practice. However, social interventions, commonly applied within the disciplines of International Development, Psychology, Education, Criminology, Public Health, Social Work and Welfare are often complex. This means they might involve a number of interacting components, multiple outcomes, and diverse delivery formats. In addition, they might be more amenable to contextual factors and intervention implementation issues. For the research synthesis endeavours to be effective in guiding policy and practice, they must utilise adequate methodology that reflects the unique features of these interventions. Without this, research conclusions may be incorrect rendering policy recommendations erroneous and interventions inefficient. The most prominent system to guide evidence-informed decision-making has been developed by the Grading of Recommendations Assessment, Development and Evaluation Working Group (GRADE) in clinical medicine. The GRADE approach provides a systematic and transparent process for rating the "best-available evidence" to inform recommendations for practice. Having been endorsed by more than 80 organisations worldwide, including the Cochrane Collaboration and the World Health Organisation, the applicability of the GRADE approach outside of clinical practice has been questioned, in part because of its neglect of important considerations of complex social interventions. Results from our previous investigation on the application of the GRADE approach in complex interventions indicate that GRADE may have limitations for these interventions. By way of illustration, GRADE initially starts with "low" quality rating for all types of observational studies; meanwhile, many social and health policy interventions are not possible to evaluate other than through observational and quasi-experimental study designs. This may lead to misinterpretations of evidence when transferred into practice, and therefore discourage important decisions. Furthermore, GRADE does not integrate important considerations in evaluation of these interventions, such as the implementation of an intervention, and integration of different types of the evidence. This suggests the need for a new methodological framework to summarise and rate the quality of evidence to inform decision-making in social disciplines. The proposed project will develop and disseminate a framework for rating the quality of evidence and the strength of recommendations in complex social interventions. This new framework will be an official extension to the GRADE approach. For the purposes of guidance, a preliminary Steering Committee of leading experts and evaluation specialists across social disciplines has been established. With their coordination, this project will first organise an international online panel involving multidisciplinary expertise to generate a list of evidence quality definitions and criteria that will reflect the particular aspects of social interventions. Prior to the panel, a systematic review of the existing quality assessment tools and evidence grading systems will be undertaken with a specific focus on these interventions to elucidate the main areas of target. Following the online expert panel, a consensus meeting will be hosted with a select group of participants to generate the final list of definitions and criteria for the new framework. The generated GRADE extension will be published in high-impact social science journals, disseminated online, as well as in academic and professional conferences and meetings.

The method for this study was an online modified-Delphi process that aimed to identify areas of agreement and disagreement among on the domains and criteria for rating the certainty of evidence in systematic reviews of complex interventions. The online-modified Delphi expert panel then ran from 22 March to 16 May 2017 using ExpertLens – a mixed-method online system, which combines two rounds of structured questionnaires with a feedback, and one round of asynchronous and anonymous discussion. For more information, see the file 'GRADE_Delphi_Study_Information'.