10.1576/toag.7.2.120.

27071

www.rcog.org.uk/togonline

EDUCATION The Obstetrician & Gynaecologist 2005;7:120125

The Delphi technique

Shakila Thangaratinam, Charles WE Redman
The Delphi technique is a way of obtaining a collective view from individuals about issues where there is no or little definite evidence and where opinion is important. The process can engender group ownership and enable cohesion among individuals with diverse views. It is an iterative questionnaire exercise with controlled feedback to a group of panellists who are anonymous. The design avoids the often counterproductive group dynamics that can occur where individuals are swayed or intimidated by others but allows panellists to reappraise their views in the light of the responses of the group as a whole. Introduction
The Delphi technique was named after the Ancient Greek oracle, who could predict the future. It involves the collection and aggregation of expert opinion and was initially used by the military to estimate the probable effects of massive atomic bombing. It now has applications in economic and financial settings, civic planning and health care: very much a case of turning swords into ploughshares. Its original purpose was to obtain the most reliable consensus of opinion of a group of experts by a series of intensive questionnaires interspersed with controlled opinion feedback,1 particularly in areas where hard data was unavailable or too costly to obtain.2 The cardinal features of the Delphi method are the use of a number of questionnaire rounds, feedback of responses, the opportunity for participants to modify their responses and anonymity of responses.3 the researcher feeds back results from the previous rounds there tends to be convergence to a consensus of opinion (Box 1).4

Keywords
consensus, Delphi technique, opinion, survey

When should the Delphi technique be used?

The technique is about harnessing and organising judgement, particularly in problems that are complex and require intuitive interpretation of evidence or informed guesswork. A Delphi can be used as an alternative to conventional meetings, avoiding problems arising from powerful personalities, group pressures and the effects of status. This is manifest more in the hierarchical structure of the health profession, where more junior practitioners may be reluctant to challenge the opinion of their seniors.5 The structure and sense of direction of the technique provides focus and avoids entropic and often counterproductive discussions and digressions that bedevil face-to-face group discussions.

Author details

Conduct of a Delphi survey

A valid Delphi process would consist of at least a three-iteration questionnaire survey, although decision about the number of rounds is largely pragmatic. The purpose of the initial iteration is to identify broad issues related to the various components of the issue at hand.A questionnaire consisting of open-ended questions is circulated to a panel of experts and opinion leaders. The responses to the open-ended questions are analysed qualitatively by sorting, categorising and searching for common themes. These responses are edited and then used to construct the second questionnaire. The second and subsequent rounds are more specific, with the questionnaire seeking the rating or ranking of various items in terms of their significance and analysed quantitatively. As

Methodological considerations What is an expert?

Some would argue that an expert is any individual with relevant knowledge and experience of a particular topic.6 This depends upon the setting and objectives of the Delphi in question.

Shakila Thangaratinam MRCOG, Clinical Research Fellow in Evidence Based Medicine, Academic Unit of Obstetrics and Gynaecology, Keele University School of Medicine, UK.

Size of panel
There are no hard and fast rules. Linstone7 suggests that a suitable minimum panel size is seven but panel sizes have ranged from 4 to 3000. It seems, therefore, that the decision about panel size is empirical and pragmatic, taking into consideration factors such as time and expense8. Representation is assessed by the qualities of the expert panel rather than its numbers.9

Charles WE Redman MD FRCOG FRCS(Ed), Consultant Gynaecologist, University Hospital of North Staffordshire, Stoke on Trent, ST4 6QG, UK. Email: cweraf@btopenworld.com (corresponding author)

120

2005 Royal College of Obstetricians and Gynaecologists

Selection of the panel

The Delphi technique straddles the divide between qualitative and quantitative methodologies; this is especially evident in the issue of how to select the panel.There is a danger of bias, where experts are selected on the basis of acquaintance with researchers, and this may be a difficult factor to address in intensely specialised areas.10 Some express concern about bias resulting from poor selection methodologies, while others dismiss these concerns. The

implication is that random sampling may not always be appropriate, particularly in areas of clinical intervention where it is more appropriate to select specialists in that area. It is sensible to describe the criteria that were used.

EDUCATION The Obstetrician & Gynaecologist 2005;7:120125

Anonymity of panellists
Anonymity of individual responses is one of the key features said to characterise the Delphi technique.2 It means that panellists will not know

Box 1 Example of generating a list of expected core competencies for a specialist registrar (year 13) in obstetrics and gynaecology
The various components of obstetrics and gynaecology training include: antenatal care basic surgical skills management of labour gynaecological problems contraception

First iteration Please provide as many suggestions as you can for each one of the open-ended questions addressing the various components of obstetric and gynaecological training below: Antenatal care What competencies pertaining to antenatal care are expected from a year 13 specialist registrar in obstetrics and gynaecology? Second iteration For each section below please rate the importance of each one of the issues relating to the various components of obstetric and gynaecological training. Circle your response according to the following 5-point scale: 1: 2: 3: 4: 5: not at all important slightly important moderately important very important absolutely essential

Antenatal care What competencies pertaining to antenatal care are expected from a year 13 specialist registrar in obstetrics and gynaecology? Conduct booking visit Perform cervical cerclage 5 5 4 4 3 3 2 2 1 1

Third iteration For each section below please take into consideration the mean importance of each one of the items as rated by the panel and your individual score for the items. If you want to reconsider your rating, mark your new score in the space provided. Antenatal care What competencies pertaining to antenatal care are expected from a year 13 specialist registrar in obstetrics and gynaecology? Antenatal care 1.1 Conduct booking visit 1.2 Perform cervical cerclage Panel rating Mean SD 4.3 0.54 2.5 0.83 Your previous rating 3 4 Your new rating

Analysis for the third iteration data will be conducted by counting frequencies of designations of the various issues. Competencies that are rated as 4 or more by at least 90% of the respondents will be considered as essential.

2004 Royal College of Obstetricians and Gynaecologists

121

EDUCATION The Obstetrician & Gynaecologist 2005;7:120125

who made what response, although more extensive constructions of anonymity have been suggested, even to the point that the responses are anonymous to the researchers.11,12 The advantage of anonymity is that it is a leveller of opinion, removing the effects of status, personalities and group pressures that can arise in meetings.1214

Questionnaire design and scoring methods

Panellists views have been recorded using a variety of scoring methods. Linear numerical scales, such as described by Likert,15 are often used. Concern has been expressed about the validity of some of the scoring methods used and the way that data have been aggregated. Delphi reports often give little information on the actual scoring and aggregation methods used and greater clarity in reports would be desirable.

are presented or perhaps in the way they are analysed. Simple statistical summaries will conceal important variations in distribution, such as bimodal patterns that could demonstrate a lack of consensus.9 Mullen18 points out that the question of whether or not consensus should be sought depends on the aim: if the aim is to find the correct answer, as in a positivist sense, outliers have to be considered as they might be right; when the aim is to obtain normative views, seeking consensus is appropriate.

Scientific merit
One of the arguments against the Delphi technique is that these studies mostly overlook reliability measurements and scientific validation of the findings.19 However, Delphi is of significant use in resolving situations where no definite evidence is available, by relying on the knowledge and experience of experts.Therefore, it might not be appropriate to use the same validation criteria as for hard science.20 Alternate means have been suggested to express the validity and credibility of Delphi findings.9 The findings of one study can be tested or confirmed in another study with a different sample as a means of validation.21 Inclusion of a clear decision trait that explains the appropriateness of the method selected to address a problem, choice of expert panel, data collection procedures, identification of justifiable consensus levels and means of dissemination and implementation are features that determine the credibility of the method.20

Feedback
Feedback with an opportunity to revise earlier responses is an important Delphic feature but it is also a potential weakness.The researcher should be impartial and provide feedback in a reliable and valid way, highlighting the degree of dissent and divergence among participants views.10 Most feedback is numerical or statistical with some form of aggregated group response. Sometimes these data can be supplemented by non quantitative information such as justifications or other comments. Statistical feedback often uses medians, usually accompanied by minima and maxima, quartiles and/or the interquartile range. Participants should also be informed about the position of their scores in relation to the overall picture, giving them an opportunity to revise previous scores in view of the group response.9

Examples of health service applications

Since 1969, there have been over 1400 publications demonstrating use of this technique in a healthcare setting, most of which have been published in the last 10 years. Its applications have included forecasting disease patterns and health funding requirements, addressing clinical problems and education.

Number of rounds
The Delphi method requires a minimum of two rounds (three if round one is open-ended). Beyond that, the number of rounds is disputed. Walker and Selfe16 make the sensible point that repeated rounds may lead to fatigue by respondents and increased attrition. Most studies use only two or three rounds.

Forecasting
As a forecasting tool, researchers have used Delphi methodology to predict developments in a variety of healthcare areas, including child and maternal health. For example, Longhurst22 used a Delphi to predict how improvements in nutrition, family income and prenatal care would impact on birth weight and subsequent intellectual development. In this study, experts were asked to predict what factors might influence outcome and the extent to which this might occur. As a starting point they used real data but had then to use their knowledge or

Consensus
The Delphi was originally developed as a means for gaining consensus, although this is not always the case. It has, nonetheless, been argued that the Delphi is designed to force consensus.17 This might occur by the way the questionnaire data 122

2004 Royal College of Obstetricians and Gynaecologists

expertise to judge, by theoretical extrapolation, what might happen. This provided information that enabled cost-benefit analyses of government programmes to improve the nutrition for pregnant mothers and young children.

Consensus in clinical problem-solving

Delphi can be used as a way of obtaining consensus about tackling clinical problems and, in this context, it is useful as a clinical management tool. It has been particularly useful in the development of performance indicators, a notoriously controversial area.23 This type of approach could be used to proactively determine performance indicators in a variety of clinical settings, including obstetric and gynaecological wards.Two concurrent Delphi studies conducted by Kennedy in 2003 describe the best midwifery practices possible.A consensus was reached in the study by panels consisting of midwives and recipients of midwifery care on the essential qualities and traits of an exemplary midwife.24 Managing clinical activity and solving operational problems can be compromised by a lack of consensus about key issues, which tend to be multifactorial and complex. Identifying and obtaining agreement on these underlying factors can be frustrating. This is an area which the Delphi technique can usefully influence. For example, in a study looking at length of stay in hospital beds, not only were a number of important factors identified but the exercise itself was said to foster a better mutual understanding between clinicians and managers.25

The consensus-forging nature of the Delphi technique can be used to good effect in enabling evidence-based medical guidelines to become adopted by clinical groups. In essence, this approach involves the development of evidence based guidelines which are then modified and finally ratified by a Delphi process. Good examples of this are seen in paediatrics, in the development of guidelines for childhood seizures and diarrhoea. This approach has also worked well in the multinational context of European colposcopy, where a set of evidencebased colposcopic clinical guidelines on treatment have been produced.27 Discussion and exploration of differences were an important part of this process and this was served well by the iterative methodology inherent in the Delphi process. Guidelines developed in this way need a process of validation with some possible ensuing modification. One approach, termed the RAND appropriateness method, is based on a structured review of the scientific literature and the collective judgement of an expert panel using Delphi methodology.This panel considers whether certain clinical activities are appropriate or necessary. Such analyses often identify high rates of inappropriate activities or ones of uncertain value.

EDUCATION The Obstetrician & Gynaecologist 2005;7:120125

Education
In clinical education, the Delphi technique has been used in a variety of ways including forecasting, planning and curriculum development.

Planning and forecasting

A notable planning example is a large Delphi study that was undertaken by the Graduate Medical Education National Advisory Committee (GMENAC) in 1976, in order to estimate the number of doctors needed in the foreseeable future. Despite the lack of hard and objective data, it was recognised that this approach represented the most detailed scientific study at that time.29 Unfortunately, it did not take into consideration the role of other healthcare professions so the suggested number of doctors needed was inflated.The point is that opinion can be useful, especially when robust or adequate data are not available, but there can be no real guarantees that it is entirely right.

Clinical guidelines
Delphi can influence the formulation and use of clinical guidelines in one of three ways:

to develop the guidelines to enable agreement on using the guidelines to evaluate how well clinical activity conforms to the guidelines.

Delphi is used as a means of actually producing the guidelines, particularly when the available evidence in insufficient or conflicting. A modified Delphi approach was used by the Swiss Society of Gynaecology and Obstetrics to set up nationally accepted guidelines on the indications for hysterectomy. An expert panel of 17 Swiss gynaecologists rated the indications twice for appropriateness and once for necessity. After a second panel, a consensus was obtained on both appropriateness and necessity.26

Curriculum development
The Delphi technique is regarded as a reasonable strategy for achieving consensus over curricular needs. A clear consensus about what constitutes good practice is essential to establish competences for curriculum development. This often requires detailed consideration of a variety of differing views and opinion. The Delphi technique has 123

2004 Royal College of Obstetricians and Gynaecologists

EDUCATION The Obstetrician & Gynaecologist 2005;7:120125

usefully provided a means for constructively addressing these issues.The Delphi has been used to identify competencies needed for a variety of healthcare professionals including pharmacists,21,30 nurses21 and colposcopists.31 The European Federation for Colposcopy undertook the Delphi method in order to identify a list of core competencies for colposcopy.31 A group of European colposcopy experts were each asked to review an initial list of competencies and add any competency that they felt might have been omitted. These responses were collated and the revised questionnaire circulated to see if there were any further additions. In subsequent rounds, further lists were circulated and the panellists were asked to rate each competency using a five-point Likert scale (1 = definitely unnecessary for diagnostic colposcopy; 2 = not important for diagnostic colposcopy; 3 = advisable for diagnostic colposcopy; 4 = important for diagnostic colposcopy;5 = essential for diagnostic colposcopy). Participants were asked to review and reconsider their ratings compared with those of the panel as a whole and amend them if desired. These steps were repeated until a consensus was reached, which occurred by the fourth round. Those competencies that were rated as 4 or more by at least 90% of the respondents were regarded as necessary for the core curriculum (Figure 1).

Delphi is a significant tool in curriculum development for undergraduate and postgraduate students and determining generic and specific standards for the trainees. A three generational Delphi method was used by the Womens Health Initiative Task Force, University of Texas Medical Branch, to identify and prioritise the key concepts in issues relating to womens health in order to develop a curriculum for medical students and resident physicians.32

Conclusions
As in any area of clinical practice, obstetrics and gynaecology generates an array of problems that are complex and not amenable to simple didactic analysis. In this context, the Delphi technique is a particularly useful tool for developing group consensus and can avoid many of the counterproductive pitfalls that can bedevil face-to-face meetings. Conflicts of Interest: None. Author contributions: CWER conceived idea, conducted literature search and composed initial draft; senior author. ST conducted literature search and provided examples.

Figure 1. Development of European colposcopy curriculum using the Delphi process

124
2004 Royal College of Obstetricians and Gynaecologists

References
1. Dalkey NC, Helmer O. An experimental application of the Delphi method to the use of experts. Manage Sci 1963;9:45867. 2. Helmer O, Linstone HA,Turroff M, editors. Delphi Method:Techniques and Applications. Mass: AddisonWesley; 1975. 3. McKenna H.The Delphi technique: a worthwhile approach to nursing? J Adv Nurs 1994;19:12215. 4. Jairath N,Weinstein J.The Delphi methodology: a useful administrative approach. Can J Nurs Adm 1994;7:2942. 5. WIlliams PL,Webb C.The Delphi technique: a methodological discussion. J Adv Nurs 1994;19:1806. 6. Cantrill JA, Sibbald B, Buetow S.The Delphi and nominal group techniques in health services research. International Journal of Pharmacy Practice 1996;4:6774. 7. Linstone HA. The Delphi technique. Handbook of Futures Research.Westport, CT: Greenwood; 1978. p. 271300. 8. Hassan F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs 2000;32:100815. 9. Powell C. Myths and realities of the Delphi technique. J Adv Nurs 2003;41:37682. 10. Murphy MK, Black N, Lamping DL, McKee CM, Sanders CFB, Askham J. Consensus development methods and their use in clinical guideline development. Health Technol Assess 1998;(2). 11. Robinson JBL. Delphi methodology for economicimpact assessment. Journal of Transportation Engineering 2004;117:33549. 12. Sumsion T.The Delphi technique: an adpative research tool. British Journal of Occupational Therapy 191;61:1536. Charlton JR, Patrick DL, Matthews G,West PA. Spending priorities in Kent: a Delphi study. J Epidemiol Community Health 1981;35:28892. Rudy SF. A review of Delphi surveys conducted to establish research priorities by specialty nursing organizations from 1985 to 1995. ORL Head Neck Nurs 1996;14:1624. Likert R. A technique for measurement of attitudes. Archives of Psychology 1932;140:4453. Walker AM, Selfe J.The Delphi technique: a useful tool for the allied health researcher. British Journal of Therapy and Rehabilitation 1996;3:67780. Parston G. The Future of Public Services 2007. London: Office for Public Management; 1995. Mullen PM. Delphi: myths and reality. J Health Organ Manag 2003;17:3752. Sackman H. Delphi Critique. Boston: Lexington Books; 1975. Fink A, Kosecoff J, Chassin M, Brook R. Consensus Methods: Characteristics and Guidelines for Use. Santa Monica, CA: RAND; 1991. Duffield C.The Delphi technique: a comparison of results obtained using two expert panels. Int J Nurs Stud 1993;30:22737. Longhurst R. An Economic Evaluation of Human Resources Program with Respect to Pregnancy Outcome and Intellectual Development. Ithaca: Cornell University; 1971. Beattie E, Mackway-Jones K. A Delphi study to identify performance indicators for emergency medicine. Emerg Med J 2004;21:4750. Kennedy HP. Enhancing Delphi research: methods and results. J Adv Nurs 2004;45:50411. Xiao J, Douglas D, Lee AH,Vemuri SR. A Delphi evaluation of the factors influencing length of stay in Australian hospitals. Int J Health Plann Manage 1997;12: 20718. Schilling J,Wyss P, Faisst K, Gutzwiller F, Haller U. Swiss consensus guidelines for hysterectomy. Int J Gynecol Obstet 1999;64:297305. Behrens K, Shafi M, Dollery E, Jordan JA, Redman CWE. European standards in treatment of CIN. Proceedings of the BSCCP Annual Scientific Meeting, Cardiff, April 2004. Lazaro P, Fitch K. From universalism to selectivity: is appropriateness the answer? Health Policy 1996;36:26172. Morgan BC. Projecting physician requirements for child health care-1990. Pediatrics 1982;69: 1506. Dunn WR, Hamilton DD, Lilja J. A competencebased study of the receptarie in the Swedish Pharmacy. J Soc Adm Pharm 1984;2:15261. Redman CWE, Dollery E, Jordan JA. Development of the European Colposcopy Core Curriculum: use of the Delphi technique. J Obstet Gynaecol 2004;24:7804. Philips B Jr, Andeson G, Ridl K. Establishing a womens health curriculum using the Delphi method. Educ Health 2003;16:15562.

13.

24. 25.

14.

26.

15. 16.

27.

17. 18. 19. 20.

28.

29.

30.

21.

31.

22.

32.

23.

2004 Royal College of Obstetricians and Gynaecologists

125