AN ANTHOLOGY OF HIGHER LEARNING
Volume Two
Research
Methodology
Edited
by
Dr. Cheryl A. Lentz
The Refractive Thinker™: An Anthology of Higher Learning Volume II: Research Methodology
Smashwords Edition, License Notes
This ebook is licensed for your personal enjoyment only. This ebook may not be re-sold or given away to other people. If you would like to share this book with another person, please purchase an additional copy for each person you share it with. If you're reading this book and did not purchase it, or it was not purchased for your use only, then you should return to Smashwords.com and purchase your own copy. Thank you for respecting the hard work of this author.
The
Lentz Leadership Institute LLC
9065 Big Plantation Avenue
Las
Vegas, NV 89143-5440
USA
Orders@lentzleadership.com
Website:
www.lentzleadership.com
All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording or by any information storage and retrieval system without written permission from the editor, except for the inclusion of brief quotations in a review.
The Lentz Leadership Institute books are available at special discounts for bulk purchases for the purposes of sales promotion, seminar attendance, or educational purposes. Special volumes can be created for specific purposes and to organizational specifications. Please contact us for further details.
*Individual authors own the copyright to their individual materials.
The Lentz Leadership Institute LLC has each author's permission to reprint.
The statue on the cover, called Soaring Figure by Stella Shawzin, is located on the plaza at Fordham University–Lincoln Center, NY, NY. http://www.library.fordham.edu
Copyright © 2009 by The Lentz Leadership Institute LLC
Library of Congress Control Number: 2009923656
Volume ISBNs
Soft Cover 978-0-9823036-4-1
E-book/PDF 978-0-9823036-5-8
Refractive Thinker logo by Joey Root, Cover design by Jacqueline Teng and Gary A. Rosenberg; Final production by Gary A. Rosenberg.
Contents
Foreword
Preface
Acknowledgements
PART I. THE DELPHI PRIMER
CHAPTER 1
The Delphi Primer: Doing
Real-World or Academic Research Using a Mixed-Method Approach
Dr.
Elmer Hall References,
CHAPTER 2
The Delphi Epiphany: Research
into Emerging Market Success Factors
Dr. Edward Knab
References,
CHAPTER 3
The Modified Ask-the-Experts
Delphi Method: The Conundrum of Human Resource Experts on Management
Participation
Dr. Cheryl A. Lentz References,
PART II. MIXED METHODS
CHAPTER 4
Stages of Internet Adoption in
Preventive Health
Dr. Diane M. Cortner References,
PART III. QUALITATIVE
CHAPTER 5
The Phenomenological Research
Method as a Valid Human Sciences Research Tool Into the Investigation
of Human Behavior
Dr. Barbara Turner References,
CHAPTER 6
A Qualitative, Ethnographic
Review of Teachers' and Administrators' Perceptions of a Prescribed
Writing Program
Dr. Neysa T. Sensenig References,
CHAPTER 7
Exploring Consumer Perceptions of
Global Branding and Iconization
Dr. Gail Ferreira References,
PART IV. QUANTITATIVE
CHAPTER 8
Use of Spearman's Rank
Correlation to Determine Pilot Survey Validity for Doctoral
Dissertation on Workplace Bullying
Dr. Judy Fisher-Blando
References,
PART V. RESEARCH AS ART
CHAPTER 9
The Elements of the Gaze:
Building Internal Coherence in Research Design
Dr. Linda Wing
References,
Conclusion
Index
Order Forms
The important thing is not to stop questioning. Curiosity has its own reason for existing. One cannot help but be in awe when he contemplates the mysteries of eternity, of life, of the marvelous structure of reality. It is enough if one tries merely to comprehend a little of this mystery every day. Never lose a holy curiosity.
—ALBERT EINSTEIN, GERMAN-BORN U.S. PHYSICIST (1879–1955)
FOREWORD
The Box
We all know about “The Box”—in business, we'd better stay in it. In academia, we are challenged and encouraged to jump out of it. In the business world, The Box signifies safe boundaries, represented by standards, best practices, rules, and regulations. In academia, The Box signifies limitations, represented by the lack of critical thinking and the ability to challenge the status quo. You're either in it or out of it—a paradigm that has and continues to separate the worlds of business and academia.
In the past century, the world has experienced advances in technology, increased communication, expanded globalization, the creation of e-information, and multi-cultural diversity. The operational boundaries within today's business environment inhibit the necessary creativity for companies to evolve within the changing world. In turn, emerging research methodologies require application to real world scenarios to validate the creative thought fostered through academia. To satisfy the needs of both academia and business and bridge the gap that separate both worlds, we need to transform the paradigm, combining the relative safety provided by the business world's ‘In the Box’ mentality with the creativity and possibilities presented through the academic world's ‘Out of the Box’ thinking. The key is to merge the static business world mentality with the transformational methods from academia, thereby reshaping The Box.
The Refractive Thinker: Volume II Research Methodology provides insight into the studies conducted by doctoral scholars. The authors share experiences using traditional research methodologies and how they applied ‘Out of the Box’ creative thought to enhance the traditional academic methods and also reform the ‘In the Box’ approach in a business environment. Authors portray how they created a link between the academic and business worlds, leveraging their ability to perceive the potential of an environment by bending traditional concepts, notions, and rules, reshaping them into new ideas, and in turn applying them to foster evolutionary change—the very essence of refractive thinking.
The members of G. Bailey & Associates (GBA) apply refractive thinking to the concept of leadership, whereby bridging the gap between academic methodologies and the assessment of specific attributes, and the business world's notion and application of the behavior. GBA members conducted research studies that examined leadership from the perspectives of organizational growth, strategic planning, and corporate performance for a particular segment of the small business community. Through the application of new concepts derived from the assessment of our research findings, GBA has altered traditional assessment methods for leadership. In doing so, we developed a holistic approach that identifies and measures specific elements that comprise leadership as a whole, rather than assessing the methods, attributes, and behavior as mutually exclusive rudiments. The result is a measured analysis of leadership alignment in terms of organizational growth, strategic planning, and performance. Our company has given us the vehicle to merge our academic ‘Out of the Box’ creative thinking with business best practices taken from ‘Within the Box’—where the result is a new, transformed box that bridges academia and business, incorporating the best of both worlds.
By reading this book, you will have taken your first step on the refractive thinking path to reforming ‘The Box’ paradigm. May you find success on your personal journey of reconstructing traditional concepts and evolving the basic foundations that continue to shape our world.
Dr.
Jody L. Sandwisch
GBA Sr. Partner
www.gba-nexus.com
Stuart
C. Bailey
GBA Sr. Partner
www.gba-nexus.com
Preface
I think therefore I am.
—SOCRATES
I critically think in order to be.
I refractively think in order to change the world.
Welcome to The Refractive Thinker™:
Volume II:
Research Methodology.
Thank you for joining us for Volume II to celebrate the accomplishments of these doctoral scholars affiliated with the University of Phoenix School of Advanced Studies. The purpose of this anthology is to offer a glimpse into the scholarly works of these authors, specifically within the world of research methodology. The purpose is to provide a resource that is beyond the boundaries of a textbook with regard to the doctoral dissertation. Part I for example is a Delphi Primer—three different author's interpretations and application of the Delphi Method— all unique, all within the vein of The Refractive Thinker™ where each challenged the conventional wisdom and expanded the traditional boundaries. These authors dared to not just think outside of the box. Instead the box continues to evolve into exploring nearly entirely new ideas for the construction of ‘the box’ i.e. the applicable methodology. Along the same lines, Part II includes elements of the Mixed Methodology, Part III: Qualitative Techniques, and Part IV includes Quantitative Techniques. The last section, Part V Research as Art, builds upon this refractive thinking by offering an entirely new research methodology for consideration.
In addition to exploring various research methodologies, the purpose of The Refractive Thinker™ is also to serve the tenets of leadership. Leadership is not simply a concept outside of the self, but comes from within, defining our very essence; where the search to define leadership becomes our personal journey, not yet a finite destination.
The Refractive Thinker™ is an intimate expression of who we are—the ability to think beyond the traditional boundaries of thinking and critical thinking. Instead of mere reflection and evaluation, one challenges the very boundaries of the constructs themselves. As in Volume I, the authors within these pages are on a mission to change the world, never satisfied or quite content with what is or asking why, instead these authors intentionally strive to push and test the limits to ask why not. Join us on this next adventure of The Refractive Thinker™ where Volume II continues the discussion specifically themed for this volume to explore the wonders of research methodology. This offers yet another bite of the apple from the tree of knowledge upon an ever expanding canvas from which these authors choose to cast their paint, envisioning new horizons in which to move forward and explore in the future.
New to this volume is the edition of a Peer Review Board, comprised of both content experts as well as those well versed in the style of APA format. While the purpose of this board was to offer additional layers of editing to enhance the brilliance of the ideas put forth by these authors, the final version for each chapter was at the discretion of each, to preserve the integrity and authenticity of their individual work.
I invite you to join with me as we venture forward to showcase these authors of Volume II. The goal is to offer a chance to bring to publication more ideas for which the audience may be interested in the expertise and guidance that they offer.
Please contact me for further information regarding these authors and the works contained within these pages. Perhaps you or your organization may be looking for their expertise to incorporate as part of your annual corporate meetings as a key note or guest speaker(s), perhaps to offer individual or group seminars or coaching, or require their expertise as consultants.
We look forward to your interest in discussing future opportunities. Let this continue our journey begun with volume I to which The Refractive Thinker™ will serve as our guide to this and future volumes. Come join us in our quest to be refractive thinkers and add your wisdom to the collective. We look forward to your stories.
Acknowledgments
The foundation of leadership embraces the art of asking questions—to validate and affirm what we do and why. Leaders often challenge this status quo, to offer alternatives and new directions, to dare to try something that has not yet been done as proved true in this case with Volume II. This publication required the continued leap of faith and belief in this new model by those willing to continue forward on this voyage. As a result, please let me express my gratitude for the help of the many that made this project possible.
First, let me offer a special thank you to Trish Hladek, my husband, Bill Lentz, and Dr. Tom Woodruff and his wife, Diane, for their unwavering support and belief that traversing unchartered waters is worthy of the journey. My gratitude extends to our Peer Review Board to include Dr. Barbara Turner, Carol Amato, Dr. Laura Grandgenett, and Dr. Tom Woodruff; and our Board of Directors to include: Dr. Elmer Hall, Dr. Edward Knab, Dr. Judy Blando, Dr. Lisa Kangas (and myself); as well as our production specialist, Gary Rosenberg; Refractive Thinker logo designer, Joey Root; and companion website and cover designer, Jacqueline Teng. The cover was produced in conjunction with the combined talents of these last three, our production team.
Let me also extend my sincere thanks to all participating authors who continue to believe in this project as we move forward with our eyes to the future. We appreciate their commitment to leadership, and to the concept of what it means to be a refractive thinker.
Dr.
Cheryl A. Lentz
Managing Editor
Las Vegas, NV •
October 2009
Part I
The Delphi Primer
CHAPTER 1
The Delphi Primer: Doing Real-World
or Academic Research Using a Mixed-Method Approach
Dr. Elmer Hall
Business researchers struggle with ways of doing affordable research that will provide meaningful information without alerting the competition or revealing trade secrets. Sometimes they might use focus groups of their typical target customer, but often the research is in areas where not enough is known to start with focused questions. Sometimes it is about an uncertain future where working with a few experts is essential in order to bring experience to bear on the forecasting or scenario planning process. The Delphi technique for doing mixed-method research is uniquely adept at such research.
In academic settings, for theses and dissertations, learners struggle with the types of research they would like to do in a specific area. If there is no prior research in a specific area, the learner will have to start with open-ended, qualitative research questions to identify the relevant factors. If there is good background information, the learner will likely do quantitative research in order to measure the importance of the factors or correlational relationships involved. An ambitious learner may not be content with simply a qualitative study that identifies the issues, or a learner may not be sufficiently comfortable with the prior research to attempt to quantify it. Additionally, there are many situations where the sign-off process by research review boards or the target audience might make more conventional research impractical or even impossible. In such cases of academic research, the Delphi technique is uniquely versatile.
The Delphi Method for doing mixed-method research is a valuable way to do research in business, government and in academic settings. Since there are many ways to utilize this powerful technique it is helpful to see several variations. This discussion is organized into the following sections:
Overview and history of the Delphi technique
The Delphi process
When to use the Delphi
Examples of Delphi research
Tips for effective research
Conclusions about using Delphi for research
Much of the available writing and research on Delphi research is available from the RAND web site at www.RAND.org/pubs/ where publications, old and new, can be downloaded or purchased. In 1975, Linstone and Turoff compiled articles into a comprehensive book, The Delphi Method: Techniques and Applications (made available electronically in 2002), which is still one of the most comprehensive sources of information on the subject with an extensive bibliography and abstracts. The Delphi technique, or simply Delphi, will be used here to describe all studies using Delphi-types of research in order to avoid confusion with the conventional Delphi Method.
OVERVIEW AND HISTORY OF THE DELPHI TECHNIQUE
The Delphi Method was developed by researchers at the RAND Corporation in the 1950s. It was an outgrowth of Douglas Aircraft's RAND project to study inter-continental warfare for the Air Force. As the Cold War set in, U.S. military strategists wanted to empirically study various strategies for nuclear armament; this required trying to view the issue of military buildup from the point of view of the Soviet strategic planners. Obviously, asking the Soviets how they would react to different buildup scenarios was out of the question. Project Delphi was the name given to the project to utilize experts to help formulate the best U.S. strategies (Dalkey & Helmer, 1963). It was not until 1963 that a RAND publication discussing long-range forecasting that the Delphi technique was fully introduced outside of the defense community (Gordon & Helmer, 1964, RAND publication P-2982).
The Delphi project was to develop the methodology and also to establish substantive results.
The authors found themselves in “a near-vacuum as far as tested techniques of long-range forecasting are concerned.” The study covered six topics: scientific breakthroughs; population control; automation; space progress; war prevention; weapon systems. Individual respondents were asked to suggest future possible developments, and then the group was to estimate the year by which there would be a 50 percent chance of the development occurring. Many of the techniques utilized in that Delphi are still common to the pure forecasting Delphi [studies] being done today. (Linstone & Murray, 1975, p. 10)
The Delphi technique is now used in vastly different situations including more complex research. A more conventional approach that aims for consensus is shown in Figure 1. It will often be used related to long-term scenarios of the future where a deep knowledge of the emerging technologies, the politics, or market forces are essential for coming to informed decisions (Straus, Parker, Bruce, & Dembosky, 2009). At its base, the Delphi is a team processes; it has the strengths and weaknesses of team dynamics. The iterative process starts out with providing the entire team with available information. Round 1 will ask all participants general and open-ended questions and attempt to gather information from the diverse expertise in information silos. The researcher will then organize the factors and information and present them back to the team for review. The second round will involve grouping, organizing, prioritizing, and discussing what is meant or implied by each factor. The researcher again organizes the results and presents them to the team to see if consensus on the factors and the importance of them can be reached. Additional iterations are done as needed.
Figure 1. The Conventional Delphi Process.
The research method for Delphi starts with qualitative research and moves toward quantifying the factors identified in the first round as illustrated in Figure 2. Creswell (2003; 2005), Jonassen, Tessmer, and Hannum (1998), Gay, Mills, and Airasian (2006); Leedy and Ormrod (2005), Linstone and Turoff (1975), Hasson, Keeney, and McKenna (2000), University of Phoenix (2003) and others talk about methods for research in general and when to use Delphi. Helmer and Dalkey (1999) discuss modifications to the Delphi. A joint effort by RAND and the University of California is illustrated in The RAND/UCLA appropriateness method user's manual. (Fitch, Bernstein, Aguilar, Burnand, LaCalle, Lazaro, Loo, McDonnell, Vader, & Kahan, 2001, RAND publication MR-1269) which provides guidelines for conducting research to identify the consensus from medical practitioners on treatment protocol that would be most appropriate for a specific diagnoses.
Figure 2. Process for Delphi Research.
In Delphi research, the participants and the ultimate target population are not only the most likely reason why the process is used, but they also will often determine how the groups will interact (Rowe & Wright, 2001; RAND, 1971, R-612-ARPA; Straus, Parker, Bruce, & Dembosky, 2009). There may only be a few qualified experts around the world, for example, so it may not be possible or cost-effective to meet in person. Often the participants are anonymous.
THE DELPHI PROCESS
The Delphi is the ultimate mixed-method study tool. It can be used for real-life research used by business, government, and even academia. When used for grounded research, Delphi will not only identify the issues with qualitative questions but also attempt to quantify the results into more actionable information for business strategies or policy recommendations. Knab (2008) demonstrates how grounded research can utilize the Delphi method to dynamically construct new theories and attempt to test them in subsequent rounds.
Figure 1 shows what would be more of a conventional approach that aims for consensus and would generally be expected to have the following characteristics:
Experts. Experts, or very informed people, would be the participants who would typically be uniquely qualified to know about the problem being researched because of their immediate knowledge, scientific, technical, economic, or other experience.
Proxies for the target population. Often participants are asked to respond about their expectations of the target population, a group that they are well informed about. Examples of people who could serve as proxies for the real target audience are: military generals about enemy generals, rehab nurses related to their patients, teachers regarding students, and wardens regarding inmates.
Inaccessible target population. In many cases the target audience is inaccessible or asking the true target audience cannot be expected to give reliable results. Examples are retired generals regarding active service generals (who may be preoccupied with a real war!), and elementary teachers related to typical responses they would expect from their students.
Figure 2 shows the process for Delphi research. This process is common to all Delphi research. Critical to the successful use of the Delphi technique is the effective use of groups or experts.
Underlying the effective use of Delphi techniques is the effective use of groups to come up with better decisions than an individual would. Surowiecki (2004) refers to this as the wisdom of crowds and collective wisdom. There are many ways to make the best use of team analysis and decision making such as the effective use of Delphi processes (Straus et al., 2009).
WHEN TO USE THE DELPHI
With all of the variations of use of the Delphi, it is important to understand when it can best be used. The things to look for when considering the Delphi are when:
All issues or factors have not been established by prior (qualitative) research
Doing the qualitative research alone is not sufficient for the purposes of the study
Forward thinking and forecasting is often required
Brainstorming and participant feedback is desired to fully understand: 1) the factors, and 2) the implications of the various possible outcomes or scenarios
Only a relatively few people are available to participate or it is preferable to maintain the anonymity of the participants
The target audience is inaccessible or cost prohibitive to reach
The problem being researched is not a simple question of amounts ($, units) or importance (ranking such as 1 to 10); it requires that the group share information and jointly analyze the issues in order to fully understand the facts and implications
The act of conducting the research on the true target population might be detrimental, destructive, or impossible
For scenario planning, where various scenarios of the future can be defined and then analyzed
Complex or ambiguous issues
For grounded research in order to produce real-life solutions or strategies
Some characteristics that would suggest not using Delphi are:
The problem being researched can be identified or assessed with one instrument (or one round of research)
Producing simply a qualitative study in an area where little or no research is available is sufficient (i.e., it is not necessary for the purposes of the study to also quantify the findings)
When prior research has firmly established the factors or issues so it would now be simply a matter of quantifying the findings from prior research
The target audience is accessible by sample or survey
There are no adverse consequences with direct contact with the target population(s)
When other methods or studies (such as focus groups for product marketing) would work better or provide better results.
Care should be exercised by the researcher to avoid unintentionally influencing the outcome of the research. There is much written about the misuse or abuse of the process by a consultant who acts as an impartial facilitator in opinion assessments for regional development planning but then actively manipulates the process in order to shape the outcomes of the study. It is unethical for a researcher to have conflicts of interest or to have pre-determined results expect by the client. Actually, this type of analysis would unlikely be Delphi; this group activity would be a focus group or a situational analysis such as a Strengths-Weaknesses-Opportunities-Threats (SWOT) analysis. (See Hall's 2007 study for an example of situational analysis related to economic development.)
EXAMPLES OF DELPHI RESEARCH
Prior to 1975, there were 670 Delphi studies but 400 were during the last 5 years of that period (Lindstone & Turoff, 1975; Appendix). These studies were mostly published in forecasting and futuristic-type magazines.
Often, the Delphi technique makes research possible that would otherwise not be possible. It is hard to estimate how frequently Delphi research is done. It is reasonable to expect a huge number that are not published. Delphi research could be very effectively utilized in all aspects of strategic planning, including product pipeline and scenario planning (Hall & Hinkelman, 2007). A McKinsey study in 2009 by Dye, Sibony and Viguerie found that 81% of companies had changed their planning process this year with more than 50% of respondents saying that scenario planning was either added or will now take on a more prominent role. “In a highly uncertain environment, the advantages of scenario planning are clear: since no one base case can be regarded as probable, it's necessary to develop plans on the assumption that several different futures are possible and to focus attention on the underlying drivers of uncertainty” (Dye, Sibony, & Viguerie, 2009, p. 1). Hall (2009) discusses the process of planning during times of extreme uncertainty. Planning as it pertains to sustainability should also be a hot area of Delphi research.
Here are examples of Delphi research that take different approaches and are in different industries:
Ed Knab (2008, 2009) asking global marketing managers about critical success factors of introducing consumer products to emerging markets. This study took a typical Delphi approach for business and academic research. A surprise from this study was not exactly the top factors for success, it was an epiphany. The study showed that entering emerging markets is a process that starts with engaging local human resources, having those locals help decide what products to introduce to the market and, lastly, have them help implement the product entry. The success factors itemized in the results are important for all global business managers, but the identification of the criticality of the process is invaluable. One interesting adjustment in the process was to continue to recruit additional participants to help with the rankings in the second round (with controls to assure that there were no statistical differences from the original group). This helped to overcome attrition and the smaller-than-desired original group of participants. Note, though, that there are risks to changing the dynamics of the group, especially with heavy group interaction and convergence. This study used lived experience from the global marketing professionals in order to avoid a two-tier sign-off process that would have rendered the research functionally impossible.
Cheryl Lentz (2007) asking human resource (HR) experts about their expectations of managers' involvement in strategic planning. This study took research by Floyd and Wooldridge (1990, 1996, 1997) showing that the involvement of mid-level management in strategy formation improves companies' performance. The plan was to replicate aspects of the prior study and extend it to mid- and lower-level management. This study produced a conundrum: using HR experts did not result in a replication—using the Delphi approach—of the Floyd and Wooldridge studies. Even though the correlations found 20 years previously were inconsistent and weak (with 0.44 as the highest R-Squared correlation), Lentz had hoped to use a similar relationships using HR experts as a validation of this use of Delphi before expanding it to lower-level managers. Lentz discussed many possible reasons why there was not at least the ability to replicate the prior study's findings. To fully understand this conundrum will require doing parallel studies of organizations using traditional surveys of employees combined with a simultaneous Delphi using HR experts. Ideally, this should be done longitudinally for multiple industries. This will not be a trivial undertaking. It is unfortunate that Lentz did not find support for using HR experts in management and employee research; as it would have provided future researchers significant savings of time and money by using experts for HR research. Now such research has questionable validity and—at least until the conundrum is understood—must be used cautiously. Note that it was possible to do this modified Delphi with only one round because of the detailed research that had gone before. Additionally, the HR experts were asked about each level of management, i.e., they were asked to represent several target populations that they knew well.
Shivie Bhagan (2008) asked two groups, Chief Executive Officers (CEOs) and Chief Information Officers (CIOs), about their priorities and their involvement in IT strategic planning. This was a very clever study to identify the different perspectives of CEOs related to information technology (IT) strategic planning compared to that of the CIOs. CEOs view life differently from the CIOs. They seem to be from different planets, something of a Mars vs. Venus syndrome. Bhagan did two separate Delphi groups, CEOs and CIOs, and then compared the results. He had pretty good success with each of the groups; however, they did have different priorities in mind related to the most important factors in IT strategic planning. This study shows the value of such a Delphi, mixed-method study. For example, each of the prioritized lists is a valuable result of this study for management. Looking at the differences in the perspectives of CEOs and CIOs is a Mars-Venus type of situation where simply understanding that they are from different planets is a critical first start in improving the working relationship. Maybe the analogy should be Mercury vs. Pluto, where the CIOs are up close to the fire in the electronic hot-seat and the CEOs have a distant view of the organization in general including IT. Given the criticality of IT to competitive advantage, CIOs should probably allow more strategic involvement and responsibility from the CIO. Maybe even a Delphi-type planning process with IT folks and other strategic planners would help bring all parties of the corporate planning process into the same orbit.
Delphi type research is extensively used in education for both assessing efficacy of teaching and teaching methods as well as helping to determine policy. A timely study by Patricia Champion (2007) studied No Child Left Behind (NCLB) initiative in education and gifted student programs. Champion engaged experts on gifted programs in education to assess the projected ramifications that these programs will have on future gifted programs.
Much research has been done related to the use of Delphi in medical research. The RAND/UCLA Appropriateness Model is probably the best defined and most consistently used Delphi research with well documented materials and guidelines (Fitch et al., 2001, RAND publication MR-1269). Based on specific diagnosis, the appropriateness model attempts to identify the best possible treatment protocol. In the typical usage specialists are provided the available journal literature and other statistics to review prior to the first round. The first round of forms asks them how much time they spent reviewing the information, how familiar they were with it, and how relevant the information was. They will share information and experience before attempting to come up with the most appropriate treatment protocols for this diagnosis. These Delphi materials can be used for best practices outside of the medical area. An example related to policy decisions is Campbell, Braspenning, Hutchinson, and Marshall (2003) who also do a short overview of using Delphi related to quality indicators for primary care.
Policy planning, psychology and social science research may use variations of Delphi. Nominal Group Technique (NGT) is common for policy development (Van de Ven & Delbecq, 1972, 1974). NGT is especially appropriate for conducting Delphi studies in one setting (similar to the RAND/UCLA method above). Two types of factor analysis methods used in psychology and social sciences are R-Method and Q-Method techniques. R-Method uses normal factor analysis to find correlations between variables across a sample of subjects, so it is rather straight-forward (Webler, Danielson, & Tuler, 2009). The Q-Method uses inverted factor analysis and gathers information about respondents' personal viewpoint, beliefs, etc., correlates them, and then looks for clusters of attributes/ beliefs (Brown, 1993; van Exel & Graaf, 2005). In Q-Method, subjective feelings can be quantified and analyzed. For more information on Q-Method research and tools visit www.QMethod.org, www.siri-us.org and www.QMethodology.net.
TIPS FOR EFFECTIVE RESEARCH
At the foundation of Delphi process is the efficient and effective use of teams. Several RAND and other researchers discuss this (see Linstone & Turoff, 1975). Straus, Parker, Bruce, and Dembosky (2009, RAND publication WR-580-USG) summarizes current intelligence research issues that have been identified related to group dynamics. These issues are:
Productivity losses during brainstorming. Fewer ideas come up from brainstorming than if the individuals drafted their list of ideas first by themselves. Productivity losses include waiting your turn in group settings.
The common knowledge effect. Even if the group has diverse members with a broad background, frequently these pockets of relevant knowledge are not shared with the group. The result is a bias toward alternatives that fit with the views that are common (or more accepted) knowledge to the group.
Group polarization. It is possible for groups to become polarized and settle on group decisions that are more extreme than the position of any of the individuals in the group. Persuasive arguments and social comparison are two possible explanations for this phenomenon.
Confirmation bias. As information is presented, individuals and groups may give more emphasis to the facts and ideas that meet their preexisting biases. Thus, even though all evidence and all sides to an issue may have been presented, this bias of the information may lead groups to be over-confident in their early or prior conviction on the issues.
Overconfidence. Individuals may over state their expertise on an issue. It is, after all, difficult to know what one does not know. This study noted:
Overconfidence appears to worsen in... situations that do not have demonstrably right or wrong answers... in larger groups and in homogenous groups. Overconfidence may be due, in part, to the convergence of both the common knowledge effect and confirmation bias, given that these processes falsely reinforce commonly and strongly-held perspectives in groups. (p. xii)
Pressure toward uniformity and groupthink. Social influencing processes can result in individuals going along with the group or dominant view. Pressures “toward uniformity inhibit members from sharing dissenting information, challenging others, and considering alternative solutions or courses of action—thereby eliminating or reducing the benefits of cognitive diversity in groups” (p. xiii).
All of these potential problems are mutually reinforcing; and, likewise, steps to avoid them are also reinforcing. Modifying the group structure, improving the facilitation process, and training are ways to minimize these possible negative effects. This working paper by Staus et al. with its annotated bibliography is a must-read for anyone doing Delphi research (or doing work with planning groups).
Academic research might take the form of experimental research where the environment is controlled to see the efficiency of groups and analyze group dynamics. Straus et al. (2009), Rowe and Wright (1999; 2001), as well as Linstone and Turoff (1975) summarize these findings well. Most research, however, does not have the control of a laboratory-type setting. In the real world it is often not easy to get access (responses) from the true target population (like CEOs). There is a story— almost certainly true—of a doctoral researcher sending a survey to all the Fortune 1000 CEOs and getting zero positive responses. Choosing a different audience, like strategic planning consultants or past presidents, might be ways to still conduct the research. Some of the things that (academic) researchers might consider are:
Aim for fewer rounds. The second round may be sufficient to gain a good and representative understanding of the issues or factors. Unfortunately, in each additional round, participants fall out (called mortality), causing validity and reliability issues. A single round study might even be possible as Lentz (2007) did.
Aim for one setting. If it is possible to get the participants together (physically or virtually), it is often possible, and usually desirable, to finish all rounds of the study in one setting. This is probably a study that should be conducted using the Nominal Group Technique (NGT) as defined by Van de Ven and Delbecq (1972; 1974). Bamford and Warder (2001) provide a great example and case study of NGT for formulating health care policy.
Institutional Review Board (IRB). To protect privacy rights and to assure research integrity universities and other institutions have an Institutional Review Board (IRB) that provides research guidelines and preapproves all research (e.g., University of Phoenix, 2003). The laws and ethics related to doing research are significant (APA, 2001, appendix C); Collaborative Institutional Training Initiative (CITI, n.d.) provides various types of information and training on the laws and ethics of doing human subjects research.
Privacy and confidentiality releases. Privacy is the Achilles' heel of modern research. Privacy laws provide protection for various groups (including prisoners, minors, students and medical patients), but it severely restricts the ability of researchers. Additionally, most organizations have privacy policies that define complex approval research procedures— rejection procedures, really. Doing Delphi research may not obtain confidential individual or organizational information which might make it unnecessary to obtain individual informed consent releases or a Use of Premises release form for gathering organizational information (Cooper & Schindler, 2006).
Lived Experience. There are more sign-offs in order to be review-board approved when asking specific people to talk about their specific position within a specific company. Asking the participants about ‘a company they know very well’ or about their ‘lived experience’ might avoid the need to get two levels of sign-offs to do research, one for the company and one for the participant.
The participants. In Delphi research, the quality of the participants determines the possible quality of the research results (Bamford & Wander, 2001). Getting no participation, say by contacting Fortune 100 CEOs, means that the study has to be redefined and likely reapproved by research boards. Delphi allows, and even encourages diversity from the participants. Branching out to include smaller company CEOs, past CEOs, and management consultants may yield only a few from each group, but a sufficient number overall to execute the study. Also, the final report must convey the types of participants and their expertise so that the potential users of study can assess how strongly they can rely on the findings.
Consensus. The consensus process—getting teams to think through complex problems and come up with the best solutions—is critical to effective teamwork and to the Delphi process. In most cases, however, it is not necessary—or even desirable--to come up with the one and only best solution. So long as there is no confusion about the facts and the issues, forcing a consensus when there is none is counter-productive (see Fink, Kosecoff, Chassin, & Brook, 1984).
Results. Because the Delphi results will be used for many business and policy decisions a clear concise table will likely be the only part of the entire study that gets wide circulation. Rank ordering a list of 10 items would be much more valuable if it also indicated the importance (maybe a on a scale of 1 to 10) and the degree of consensus (maybe standard deviation).
Limitations. There are many limitations of doing a Delphi study; limitations and caveats in using the results should be clearly conveyed. It is important to include the comfort level of the respondents as well as the facilitator(s) in the results. Knowing where there is not consensus is as important as know where there is.
Vigilant efforts must be made by the facilitator to design the study and to execute it in a way that will produce the best possible results in terms of accurate findings and usable results. A careful review of the RAND/UCLA guidelines as well as discussions on group effectiveness are important to optimize the process as much as possible given the constraints (Hasson, Keeney, & McKenna, 2000; Rowe & Wright, 1999).
CONCLUSIONS ABOUT USING THE DELPHI FOR RESEARCH
The Delphi technique for doing mixed-method research is a very powerful tool that can produce fast and affordable research results, even in complex circumstances with uncertain information. Essentially, it is two studies for the price of one: qualitative followed by quantitative. It starts with a process of qualifying the relevant factors or options and then moves into a quantification process in order to determine the priority and importance. The experts involved can work together to construct scenarios or forecasts of the future, and then assess them.
Typically, the participants are both the reason for using Delphi in the first place and also the determinants of how the study is executed. They are often either specialized in the area of interest (and therefore experts) or uniquely informed about the true target population to be studied (like a teacher regarding the student population). The team process of the group of participants allows for a shared learning where expertise and facts can be shared and analyzed before attempting to prioritize or propose solutions. An effective process, with an experienced moderator, is critical to produce the best group performance and the best results.
Frequently, Delphi is done in circumstance where there would be no study without it. A single instrument many not be acceptable for academic purposes for example, or the sign-off process may be insurmountable with the target population directly. Actionable, grounded research may be required for business decisions or government policy. Also, costs or accessibility may make research on the target population impossible. Academic research using Delphi should increase in popularity since it provides flexibility and allows the learner to obtain more usable results than a single-instrument study.
After the recession of 2008–2009 there will be major efforts to do more scenario planning and to affordably assess strategic options. It has become clear that planners at all levels of government and business had become complacent, expecting a higher level of stability in the future. The Delphi technique is uniquely suited for many types of academic, government policy and business research. Research using Delphi processes will likely be utilized much more extensively in the future. Using it well and wisely by making the best of available knowledge and expertise will result in better decisions toward products, health, education, security, sustainability, and more...
References
American Psychological Association [APA]. (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: The American Psychological Association.
Bamford, M., & Warder, J. (2001, July). Occupational health nurses' perceptions of their education and training needs to meet the new public health agenda using the nominal group technique. International Journal of Lifelong Education, 20(4), 314–325. Retrieved July 3, 2009, doi:10.1080 /02601370110059500.
Bhagan, S. (2008). A chief executive officer and chief information officer consensus decision-making model for information technology investments. (Doctoral dissertation). Retrieved March 8, 2009, from Dissertations & Theses @ University of Phoenix database. (Publication No. AAT 3333901).
Campbell, S., Braspenning, J., Hutchinson, A., & Marshall, M. (2003, April 12). Research methods used in developing and applying quality indicators in primary care. BMJ: British Medical Journal, 326(7393), 816–819. Retrieved July 3, 2009, from Academic Search Complete database.
Champion, P. A. (2007). Equity vs. excellence: A Delphi study examining the future ramifications of NCLB on gifted education. (Doctoral dissertation). Retrieved July 1, 2009, from Dissertations & Theses @ University of Phoenix database. (UMI number 3277191).
Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (2nd ed.). Thousand Oaks: CA: Sage Publications, Inc.
Creswell, J. W. (2005). Educational research: Planning, conducting, and evaluating quantitative and qualitative research. (2nd ed.). Upper Saddle River, NJ: Merrill-Prentice Hall.
Cooper, D. R., & Schindler, P. S. (2006). Business research methods (9th ed.). New York: McGraw-Hill/Irwin.
Collaborative Institutional Training Initiative [CITI]. (n. d.). Collaborative institutional training initiative. Retrieved July 16, 2009, from: https://www.citiprogram.org
Dalkey, N., & Helmer, O. (1963, April). An experimental application of the Delphi method to the use of experts. Management Science, 9(3), 458–467. Retrieved July 3, 2009, from Business Source Complete database.
Dye, R., Sibony, O., & Viguerie, S. P. (2009, April). Strategic planning: Three tips for 2009. McKinsey Quarterly. Retrieved May 8, 2009, from: www.mckinseyquarterly.com
Fink, A., Kosecoff, J., Chassin, M., & Brook, R. (1984, September). Consensus methods: Characteristics and guidelines for use. American Journal of Public Health, 74(9), 979–983. Retrieved July 3, 2009, from Academic Search Complete database.
Fitch K., Bernstein S.J., Aguilar M.D., Burnand, B., LaCalle, J.R., Lazaro, P., Loo, M., McDonnell, J. & Vader, J.P., Kahan, J.P. (2001). The RAND/UCLA appropriateness method user's manual. Santa Monica, CA: RAND Corporation. Document MR-1269. Retrieved July 3, 2009, from: www.rand.org/publications/
Floyd, S., & Wooldridge, B. (1996). The strategic middle manager: How to create and sustain competitive advantage. San Francisco: Jossey-Bass Publishers.
Floyd, S., & Wooldridge, B. (1997, May). Middle management's strategic influence and organizational performance. Journal of Management Studies, 34(3), 465. Retrieved March 3, 2005, from ProQuest database.
Gay, L. R., Mills, G. E., & Airasian, P. (2006). Educational research: Competencies for analysis and applications (8th ed.). Upper Saddle River, NJ: Pearson Education Inc.
Gordon, T. J., & Helmer, O. (1964). Report on a long-range forecasting study. Santa Monica, CA: RAND Corporation. Document MR-2982. Retrieved July 3, 2009, from: www.rand.org/publications/
Hall, E. (2009). Strategic planning in times of extreme uncertainty. In C. A. Lentz (Ed.), The refractive thinker: Vol. 1. An anthology of higher learning (1st ed., pp. 41–58). Las Vegas, NV: The Lentz Leadership Institute.
Hall, E. B. (2007). Strategic economic development & marketing plan for Highlands County. Morrisville, NC: LuLu Press.
Hall, E. B., & Hinkelman, R. M. (2007). Perpetual Innovation™: A guide to strategic planning, patent commercialization and enduring competitive advantage. Morrisville, NC: LuLu Press.
Hasson, F., Keeney, S., & McKenna, H. (2000, October). Research guidelines for the Delphi survey technique. Journal of Advanced Nursing, 32(4), 1008–1015. Retrieved July 3, 2009, from Academic Search Complete database.
Helmer, O., & Dalkey, N. (1999, Spring). The modified Delphi technique: A rotational modification. Journal of Vocational and Technical Education. (15), 2.
Jonassen, D. H., Tessmer, M., Hannum, W.H. (1998). Task analysis methods for instructional design. Mahwah, NJ: Lawrence Erlbaum.
Knab, E. F. (2008). Going global: Success factors for penetrating emerging markets. (Doctoral dissertation). Retrieved March 8, 2009, from Dissertations & Theses @ University of Phoenix database. (Publication No. AAT 3326207).
Knab, E. F. (2009). Globalizing your business: A practical approach to international expansion. In C. A. Lentz (Ed.), The refractive thinker: Vol. 1. An anthology of higher learning (1st ed., pp. 73–89). Las Vegas, NV: The Lentz Leadership Institute.
Leedy, P. D., & Ormrod, J. E. (2005). Practical research: Planning and design (8th ed.). Upper Saddle River, NJ: Merrill Prentice Hall.
Lentz, C. A. (2007). Strategic decision making in organizational performance: A quantitative study of employee inclusiveness. (Doctoral Dissertation). Retrieved March 8, 2009, from Dissertations & Theses @ University of Phoenix database. (Publication No. AAT 3277192).
Linstone, H. A., & Turoff, M. (1975). The Delphi method: Techniques and applications. Reproduction electronically in 2002. Retrieved on June 30, 2009, from: www.is.njit.edu/pubs/delphibook/
Rowe, G., & Wright, G. (1999, October): The Delphi technique as a forecasting tool: Issues and analysis. International Journal of Forecasting, 15(4).
Rowe, G., & Wright, G. (2001): Expert opinions in forecasting. Role of the Delphi technique. In: Armstrong (Ed.): Principles of forecasting: A handbook of researchers and practitioners, Boston: Kluwer Academic Publishers.
Straus, S. G., Parker, A. M., Bruce, J. B., & Dembosky, J. W. (2009, April). The group matters: A review of the effects of group interaction on processes and outcomes in analytic team. Santa Monica, CA: RAND, National Security Research Division (Document No: WR-580-USG). Retrieved on June 30, 2009, from: www.Rand.org
Surowiecki, J. (2004). The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations. New York: Doubleday.
University of Phoenix. (2003). Doctor of business administration research handbook (2nd ed.). Phoenix, AZ: University of Phoenix Press.
Van de Ven, A. H., & Delbecq, A. L. (1972). The nominal group as a research instrument for exploring health studies. Journal of the American Public Health Association, 337–342.
Van de Ven, A. H., & Delbecq, A. L. (1974). The effectiveness of nominal, Delphi and interacting group decision making processes. Academy of Management Journal, 17(4), 605–621.
van Excel, J., & de Graff, G. (2005). Q methodology: a sneak preview. Retrieved on August 12, 2009, from: www.QMethodology.net
Webler, T., Danielson, S., & Tuler, S. (2009, February). Using Q method to reveal social perspectives in environmental research. SERI Report # 09-001. Greenfield MA: Social and Environmental Research Institute. Retrieved August 12, 2009, from: www.seri-us.org
Wooldridge, B., & Floyd, S. (1990). The strategy process, middle management involvement, and organizational performance. Strategic Management Journal, 11(3), 231–241.
About the Author
Dr. Hall holds several nationally accredited degrees: a Bachelor of Arts (BA) and Master of Business Administration (MBA) from the University of South Florida; and a Doctorate of International Business Administration (DIBA) from Nova Southeastern University. Throughout his schooling, he was a management and research assistant involved in business/trade research and systems development. He has taught at the undergraduate and graduate levels (MBA and MIS) at several Florida universities and currently is a Professor of Business at Warner University where he teaches masters and undergraduate business classes and is the Interim Sustainability Officer. He also is a Facilitator and Dissertation Mentor for the University of Phoenix. His “real” education, however, is from his personal entrepreneurial ventures and those of his clients.