Espacios. Vol. 36 (Nº 15) Año 2015. Pág. 3
Jhon Wilder ZARTHA Sossa 1; Héctor Samuel VILLADA Castillo 2; Raúl HERNÁNDEZ Zarta 3; Amanda Lucía FERNANDEZ Pérez 4; Bibiana ARANGO Alzate 5; Gina Lía OROZCO Mendoza 6; Rafael BERMUDEZ Ortega 7; Diego Fabián JOAQUI Daza 8; Alcy René CERÓN Mosquera 9; John Fredy MORENO Sarta 10
Recibido: 31/03/15 • Aprobado: 12/05/2015
ABSTRACT: Delphi methodology was chosen to perform a foresight study up to 2032 in biodegradable packaging and environmentally friendly products. To develop the study, a questionnaire with 196 variables classified into 5 areas was elaborated: Technological innovations, commodities for biodegradable polymers production, commodities for biodegradable packaging production, additives and technologies, the questionnaire was sent to 92 national and international people with knowledge on the topic. The expert panel was made up of 30 people who gave response to the questionnaire for three rounds. According to statistical analysis raised, in the first round, 51 priority variables and 140 variables under discussion were obtained, the 5 variables remaining were classified as non-priority. In the third round, a total of 78 prioritized variables were obtained. The results obtained in the area of technological innovations, natural coatings, active packaging and smart packaging, were considered priorities during the three rounds. In the area of commodities for the production of biodegradable polymers, the potato, cassava and corn starch were considered a priority, in addition to Agroindustrial wastes and in the technology area, reactive extrusion, blow molding and extrusion injection molding were kept for three rounds as priority variables. |
RESUMEN: La Metodología Delphi fue elegida para llevar a cabo un estudio de previsión hasta 2032 en envases biodegradables y productos respetuosos con el medio ambiente. Para desarrollar el estudio, se elaboró un cuestionario con 196 variables clasificadas en 5 áreaz: Las innovaciones tecnológicas, materias primas para la producción de polímeros biodegradables, materias primas para la producción de envases biodegradables, aditivos y tecnologías. El cuestionario se administró a 92 personas nacionales e internacionales con conocimiento sobre el tema. El panel de expertos estaba formado por 30 personas que dieron respuesta al cuestionario en tres rondas. De acuerdo con el análisis estadístico planteado, en la primera ronda, se obtuvieron 51 variables de prioridad y 140 variables bajo análisis, las 5 variables restantes fueron clasificados como no prioritarias. En la tercera ronda, se obtuvieron un total de 78 variables de priorizadas. Los resultados obtenidos en el ámbito de las innovaciones tecnológicas, revestimientos naturales, envases activos y envases inteligentes, se consideraron prioridades durante las tres rondas. En el ámbito de las materias primas para la producción de polímeros biodegradables, la papa, la yuca y el almidón de maíz se considera una prioridad, además de los residuos agroindustriales y en el área de tecnología, extrusión reactiva, moldeo por soplado y moldeo por inyección de extrusión se mantuvieron durante tres rondas como las variables prioritarias. |
Since 1950 the Rand Corporation developed the Delphi methodology (Nielsen and Thangadurai, 2007; Landeta, 2006; Dalkey and Helmer, 1963), in order to investigate the impact of technology for the US Air Force during the war (Gil-gomez and Pascual, 2012; Fletcher and Marchildon, 2014; Rowe and Wright, 1999; Linstone and Turoff, 1975), it has been used by different disciplines for multiple purposes because its main objective has been resort to the expertise to predict how a phenomenon in the future would behave (García and Suarez, 2012). According to Kim, Jang and Lee, (2013), this technique is considered one of the most used, as it contains information consensuated by a group of people considered experts in relation to the objective; the contrast of the arguments issued by the experts allow reducing uncertainty in decision making.
In this case, the Delphi methodology was used in a foresight study on the biodegradable packaging development in Colombia, in order to find the current trends developed in other world places and the possible alternatives commodities for the production of the same, given that from latest times, it has been a global challenge to find alternative solutions to environmental pollution caused by improper usage of synthetic plastic used for such purpose; however, the development of these materials in Colombia is still growing, although Oxo degradable plastics have been used as alternative because its degradation occurs in less time than current plastic but the degradation process is not complete and its production sources are not renewable.
Therefore, a foresight study through the Delphi method in this topic, provides information about to development of biodegradable packaging and environmentally friendly products, not only in areas such as technological innovation and commodities, but technology and type of additives to be used, due to its biodegradable nature, the elaborated material must have international biodegradable standards.
This article is part of a group of results that were obtained in the prerogative project titled "Foresight study for the year 2032 on friendly biodegradable packaging in the Colombian environment"; financed by national department of planning, general prerogative system in Colombia, the government of Cauca, University of Cauca, and the research group of CYTBIA.
For one procedure to be defined as "Delphi", for general aspects must be taken into account: anonymity, insistence, controlled feedback, and a statistical analysis on the answers generated by the group. (Zartha et al. 2014)
Many applications of the Delphi method exist at an international level, in diverse sectors. The following chart shows the applications made by 15 authors who have made scientific publications.
Table 1. Applications of the Delphi method in diverse sectors
Author |
Country and organization |
Purpose |
Number of experts |
Coduras and Señarís (2009) |
Astrazéneca Foundation |
Study that tries to anticipate the future of the pharmaceutical benefits in Spain |
67 y 65 respectivamente |
Wakefield and Watson (2013) |
Brigham Young University (Estados Unidos), Bournemouth University (Reino Unido) |
Used on public relations and research on communications regarding the 2.0 web era. The purpose is to introduce, analyze, and explain the Delphi and its evolution; and its applications on public relations. |
Duke (2009): 10 to 15, Mckinnon et al. (2001): 7, Boynton and Watson (2006) and Watson (2008): 30 Fink and Fuchs, (2012): 32 |
Kauko and Palmroos (2014) |
Bank of Finland |
It presents how the panel members are invited to present foresight variables on the financial market in a controlled experiment, and presents 2 post survey methods as an adjustment to the Delphi method. |
10 experts were used, Rowe and Wright were cited (1999): Ten people are enough, and there is no evidence that increasing the size from 7-10 people would improve precision. |
Fletcher and Marchildon (2014) |
Canada |
A modified draft Participatory Action Research (PAR) in health leadership in Canada Delphi method was used. |
First round with 39 experts |
Cancelo, Neyro, Baquero (2014) |
Spain |
Recognize the degree of agreement/disagreement between a group of Spanish gynecologist about the role .of probiotics as a help treatment of vaginitis. |
123 experts |
Cabero (2013) |
Spain |
Modified Delphi method with the objective of forming blocks of knowledge that should have future action in the formation process of the university's TIC teaching. |
First round with 68 experts, second round with 65 experts. |
Liimatainen et al. (2014) |
Finland |
This study is about CO2 emissions in terrestrial cargo transport in Finland |
24 and 20 respectively |
Debin, Souty, Turbelin, et al. (2013). |
France |
The objective of this study is to establish an expert based determination from the beginning and end of influenza epidemics in France |
57 answered the three rounds |
Creange and Careyron (2013) |
France |
This study shows that a group of criterion and diagnose strategies are not enough to enrich a consensus for the diagnostic of a typical CIDP "Chronic inflammatory demyelinating polyneuropathy" in the clinical practice. |
32 |
Okoli and Pawlowski (2004) |
Canada and the United States |
The article provides a guide regarding the adequate expert selection process for the Delphi study, and gives detailed principles for decision making during the process, to assure the validity of a Delphi study, It mentions an example on e-commerce in sub-Saharan Africa. |
Between 10 and 18 for each panel (4 panels in total) |
Landeta (2006) |
Spain |
Evaluate the methodology and revise its actual validity, especially in areas concerning the social sciences. |
Cites Cataluña study with 14 experts |
Graham, Regehr and Wright (2003) |
Canada |
Achieve consensus between a panel of experts to establish the best clinical criteria for the diagnosis of carpal tunnel syndrome |
14 y 12 respectively |
Gordon and Pease (2006) |
United States |
Describe the Delphi process in real time, and illustrate its usage in an application regarding decision making; conducted from the millennium project of the American council of the United Nations University. |
10 a 15 respectively |
Moore (2011) |
United States |
States in consumer packaged goods. Uses a modified Qualitative Delphi and descriptive statistics for leadership profiles of the NDP stages in businesses in the United States in consumed bottled goods. |
11 to 12 experts evaluating each stage of the development process of new processes. 37 panel members, in groups of 5 to 6. |
Zeedick (2012) |
United States |
This deals with the instructional design theory and its applications in line education programs. A Delphi was used in three rounds with descriptive analysis and interquartile ranges. According with the results, they did not obtain the adequate results about the necessity of applying this theory in online programs, |
Cites: Helmer (1963), Delbecq et al. (1975) and Limestone and Turoff (1975): 9 experts are enough. . |
Source: (Zartha et al. 2014)
Between the most important characteristics of the Delphi method, Castelló y Callejo (2000) mention that a Delphi is a consultation to a large number of chosen experts from different sectors or specific topics for an exercise, based on a survey; prepared by a panel or commission of experts. On a similar side, (Rowe and Wright, 1999) the Delphi has four key characteristics: anonymity, (Becker and Bakal, 1970; Helmer and Rescher, 1959) iteration, controlled feedback and added statistics from a group of answers, (Gordon and Helmer, 1964). Besides, (Martino, 1993; Konow and Pérez, 1990) add another characteristic related with monitor equipment. They produce a structured survey from different points of view, opinions, and judgements form the panel members, which can be raised in a quantitative manner in latter rounds.
The Delphi method is an anonymous and structured consultation to a group of individuals about topics where the opinion is important and a consensus is achieved. Helmer y Rescher (1959). The expert gets feedback through questionnaires made for the corresponding study, because they can give their opinion about the subjects considered and it allows the experts reconsider their points of view according to a set of responses (Thangaratinam and Redman, 2005).
Regarding topic selection, technologies and priority variables, the criterion remained constant. This consists on the calculation of the mode, modal frequency and consensus percentage. Concerning this data, the consensus average percentage was determined for each pooler (Zartha et al. 2014). Based on this information, the following criterion was established:
Topic, technology, priority variable: Consensus over the average pooler and a mode of 4 or 5. The scale is 0 to 5 where the prioritization order was the following:
0= No priority
1= Low Priority
2= Low
3= Medium
4= High
5= Very High
N= Does not Know/ Did not Answer
For the study, the following poolers were taken into account:
Technological innovations, raw materials for the production of biodegradable polymers, Raw materials for the acquisition of biodegradable packaging, additives, and technologies.
Delphi Method:
Stage 0: definition of the transversal axis or poolers form the company's perspective related to their areas of interest
Stage 1: Socialization of the methodology and communication with the monitoring team
Stage 2: Construction of the expert profile, definition of the problem/necessity, and the revision of the list of experts
Stage 3: Construction of the technological thematic tree diagram
Stage 4: Survey for the first round
Stage 5: Delivery of the first round to the experts
Stage 6: Statistical analysis and report of the first round
Stage 7: Analysis of the technologies/themes of priority/no priority, discussion, construction and the delivery of the second Delphi round
Stage 8: Construction of the second round Delphi
Stage 9: Delivery of the second round
Stage 10: Analysis of the second round
Stage 11: Construction of the second round Delphi
Stage 12: Delivery of the third round
Stage 13: Construction of the final report and Socialization with the monitor team
This first stage was started with the election of a accompanying team chosen from the research CYTBIA team, who have made developments in the biodegradable packaging subject. The team was commissioned to provide the base information for the Delphi study construction.
Information search of innovation trends, commodities and technology used in biodegradables packaging production was performed for building a theme tree with 196 variables, after that they were classified in 5 areas according to functionality and thematic application: 1. Technological innovations, 2. commodities for biodegradable polymers production, 3. commodities for biodegradable packaging production, 4. additives and 5. technologies; based on this, the questionnaire for the first round Delphi was built in excel format and it was sent to various experts in the subject by email.
For this first round 97 people from national and international universities, government, private sector and other organizations were invited, of this people, 30 experts formed the panel responding to the structured consultation. By this first round, the experts must have to qualify as non priority and priority topics in a scale of 0 to 5 respectively.
Subsequenly the statistical analysis was accomplished, was obtained four parameters like Moda, Modal Frecuency, Consensus percentage and the consensus average percentage for each area; with this results was determinated that one subject would be a priority (TP) if the califfication exceded the category consensus percentage and it had a modal value equal or higher than 4.
Likewise, was determined that a subject would be considered as No Priority (NP) if it had a modal value less or equal to 2 and a consensus average percentage higher than corresponding category.
The subjects who did not accomplished the above conditions were considered as subjects under discussion (TD).
For the second round, a new questionnaire in the categories with items considered "priority" and "discussion" was done. In this stage, the goal was to identify from experts some subjects under discussion in order to classify them like priority subjects instead of some other discussion subjects. In other words, the priority subjects group may not increase, for this reason, it is necessary that for each topic leaving the group, another discussion subject could take its place justifying any changes, given that this will be the base argument for the final round.
In the third round, a new questionnaire with subjects considered as "priority" and "discussion" and the appropiate justifications in the previous round was sent to provide the better arguments to the experts when they choose a subject as priority again
After that, the statistical treatment was completed again, taking into account the statistical parameters used in the first round, according to the consensus done by the experts, a final classification was obtained. With these results a final report was made with most important subjects in the biodegradable packaging area and the need of CYTBIA team to make new researches in this subject.
According to Balaraman, Venkatakrishnan, (1980); Zartha, et al, (2011) for analysis of consensus for each of the topics in the third round, there were considered the following:
According to the qualification assigned by the participants of the survey in each of the topics in the first round, the modal value qualification was calculated. If the value was 4 or 5, It would be constant for the next rounds as a priority.
The qualification for the first round were taken as the pattern for the third round.
In the third round, participants assigned X to the topics considered priority, after this, the X was replaced with the value of the qualification assigned in the first round. The lower modal qualification value (Er3) in the frist round was determined for each subject, since they were not considered as a priority initially.
Similarly, there were identificated the participants that considered some topics as a priority in the first round but did not check them with an X in the third round, therefore these items were removed of consensus, this result was called (Sr3).
In the third round, the modal frecuency was determined through the next equation:
Where: Fm1 is the modal frecuency subject in the first round with the total number of people who participated in the three rounds.
The consensus percentage in the thrid round was determined through equation:
The structured questionnaire was sent to 97 people invited to participate in the study by email and it was answered by 30 people who formed the expert panel. The panel included the participation of national and international researchers; The international researches were Mexico 7%, Argentina 3%, France 3%, Spain 4%, Italy 3% and Colombia 80%, as well as government people and national private sector people. Figure 1 shows the expert panel participation percentage for each nationality.
Figure 1. Participants nationality
According to statistical criteria taken into account for the classification of the 196 variables, subjects who had modal value equal or higher than 4 and higher than average consensus percentage were considered as priority; subjects who obtained less or equal to 2 modal value and a percentage higher than the average consensus percentage were classified non-priority and finally, subjects who did not accomplished the specifications were considered under discussion. Table 2 shows the average consensus percentage of each area
Table 2. Average consensus percentage for each area
Area |
Consensus percentage average |
|
35% |
|
29% |
|
28% |
|
31% |
|
34% |
According to this, figure 2 shows the segmental variables classification in the first round, the distribution was: 26% priority variables, non-priority 3% and 71% variables under discussion.
Figure 2. Segmental variable classification in the first round.
Below figure 3 shows the expert panel training, it can be seen that in the panel of experts, the highest percentage was of doctors with 40%, followed by bachelors by 36%, then professionals with master's degree 17% and 7% of professionals with degree of specialization.
Figure 3. Expert panel training
The survey was sent to 30 experts panel to make corresponding changes according to their judgement; after obtaining the answers, items count under discussion and changes were made. The unchanged subjects were considered priorities for the third round.
Subsequently, the final questionnaire was made, it was sent to panel again, table 3 shows subjects classified as priority for the third round according to the consensus obtained by the experts
Table 3. Priority topics
Area |
Priority topics |
Technological Innovations |
Natural coatting |
Food container |
|
Active packaging |
|
Shrink films |
|
Foamed |
|
Intelligent packaging |
|
Bioactive films |
|
Commodities for biodegradable polymer production |
Corn |
Mandiocca |
|
Potato |
|
Wood fiber |
|
Agroindustrial wastes |
|
Collagen |
|
Fatty acids |
|
Monoglycerids |
|
Commodities for biodegradable packaging production |
Banana fiber |
Chaff reed fiber |
|
Microbiane cellulose |
|
Polyhidroxyalkonates (PHAs), |
|
Polyhidroxybutyrate valerate (PHBV), |
|
Polilactic acid (PLA) |
|
Microcrystalline cellulose |
|
Seaweed wet biomass |
|
Chitin |
|
Aditives |
glycerol |
antimicrobial agents |
|
powdered pigments |
|
coupling agents |
|
Technologies |
Twin screw extrusion |
Reactive extrusion |
|
Co-extrusion |
|
Blow film extrusion |
|
Injection molding |
|
Spray |
|
Co-laminathing |
|
Bio processes |
|
Nanotechnologies |
Table 4, shows classified subjects as priority for each round in the study.
Conventions:
. | Priority topics in the first and second round |
. | Priority topics in the third round |
Table 4. Topics classified in each rounds
Topics |
First round |
Justifications second round |
Third round |
1. Technological innovations |
|||
Monomer and Polymer development through biorefinery process |
X |
X |
|
Thermoformed trayds |
X |
X |
|
Bags for seedbeds |
|
X |
X |
Bags for Ecological stations |
|
X |
X |
Plastic bottles |
|
X |
X |
Natural coatting |
X |
X |
X |
Multilayer packaging |
X |
X |
|
Wrapping |
X |
X |
|
Food container |
X |
X |
X |
Active packaging |
X |
X |
X |
Shrink films |
. |
||
Foamed |
. |
||
Intelligent packaging |
X |
X |
X |
Flexible packaging |
X |
X |
|
Bioactive films |
X |
X |
X |
Nanocomposites |
X |
X |
|
2. Commodities for biodegradable polymer production |
|||
2.1 Vegetal origin commodities |
|||
Starches |
|||
Corn |
X |
X |
X |
Mandioca |
X |
X |
X |
Potato |
X |
X |
X |
Sweet potato |
|
X |
|
Arracacia |
|
X |
|
Banana |
|
X |
|
Cidra potato |
|
X |
|
Cellulose |
|||
Wood fiber |
X |
X |
X |
Agroindustrial wastes |
X |
X |
X |
Vegetal lipids |
|||
Fatty acids |
X |
||
Others |
|||
Orange Peel |
|
X |
X |
Soya been peel |
|
X |
X |
Rice Peel |
|
X |
X |
Pineapple peel |
X |
X |
|
2. 2 Animal origin commodities |
|||
Collagen |
. |
||
Fatty acids |
. |
||
Monoglycerids |
X |
X |
X |
3. Commodities for biodegradable packaging production |
|||
3.1 Vegetal origin commodities |
|||
Metilcellulose MC |
X |
X |
|
Hidroximetil cellulose HMC |
X |
||
Carboximetil cellulose CMC |
X |
X |
|
Banana fiber |
X |
X |
X |
Chaff reed fiber |
X |
X |
X |
Acetoglycerids |
X |
|
|
Pectins |
|
X |
X |
Zeina |
|
X |
|
Corn gluten |
|
X |
X |
Resins |
|
X |
X |
High amylose |
|
X |
X |
3.2 Microbian origin |
|||
Microbian cellulose |
. |
||
Polyhidroxyalkonates (PHAs) |
X |
X |
X |
Polyhidroxybutyrate valerate (PHBV) |
X |
X |
X |
Polyhidroxybutyrate terephthalate (PHAT) |
X |
X |
|
Polilactic acid (PLA) |
X |
X |
X |
Microcrystalline cellulose |
X |
X |
X |
Seaweed wet biomass |
X |
X |
X |
3.3 Animal origin |
|||
Marine and pisciculture |
|||
Chitin |
X |
X |
X |
Alginate |
X |
||
Carrageenan |
X |
X |
|
Others |
|||
Keratin |
X |
X |
|
Biodegradable synthetic origin |
|||
Polyvinyl Alcohol (PVA) |
X |
X |
|
Polycaprolactone (PCL) |
|
X |
X |
4. ADITIVES |
|||
4.1 Plastiziers |
|||
Glycerol |
X |
|
X |
Saccharose |
|
X |
|
Fructose |
|
X |
X |
4.2 Other |
|||
Masterbach |
X |
X |
|
Antimicrobial agents |
X |
X |
X |
Powdered pigments |
X |
X |
X |
Liquids colorants |
|
X |
X |
Thermosetting adhesive hot melt |
|
X |
X |
Estearic acid |
|
X |
X |
Citric acid |
|
X |
X |
Anphiphilics substances |
|
X |
X |
Coupling agents |
X |
X |
X |
5. TECHNOLOGY |
|||
5.1 Extrusion |
|||
Single screw extrusion |
X |
||
Twin screw extrusion |
X |
|
X |
Reactive extrusion |
. |
||
Co-extrusion |
X |
X |
X |
Film blown extrusion |
. |
||
Coating extrusion |
X |
X |
|
5.2 Molding |
|||
Thermoforming |
X |
X |
|
Injection molding |
. |
||
Injection blown molding |
X |
X |
|
Reactive injection molding |
X |
X |
|
5.3 Other |
|||
Force Spinning |
|
X |
X |
Electro spray |
|
X |
X |
Spray |
X |
X |
X |
Compressed process |
X |
||
Laminathing |
X |
X |
|
Co-laminathing |
X |
X |
X |
Sealing systems |
X |
X |
|
Bioprocesess |
X |
X |
X |
Nanotechnologies |
X |
X |
X |
5.4 Print system |
|||
3D printing |
|
X |
X |
Inkjet |
|
X |
X |
Printing inks |
|
X |
X |
TOTAL |
51 |
|
78 |
According to results obtained up to 2032, there is priority in techonological innovation development in subjects such as foam and active packaging production; it´s also an opportunity for searching new commodities in biodegradable polymers from potato, corn, agroindustrial wastes, microbian cellulose and faty acids production. In biodegradable packaging there were priorized subjects as polilactic acid, microbian cellulose and polyhroxyalkanoates and others. The use of technologies as twin screw extrusion, reactive extrusion and injection molding show great opportunity
These results allow to visualize opportunities that can be implemented in research and business development by Cytbia team.
From the methodological point of view, the foresight Delphi method used for this project development allows to priorize technological innovation, new commodities and technology in biodegradable packaging production in Colombia up to 2032
Through tools and methodologies for foresight studies, the uncertainty can be reduced by means of Delphi method in biodegradable packaging purchase and technology research in the decision making as well as new research lines and innovation entrepreneurship projects identified
37 priority subjects in common and 78 priority subjects in the third round were stablished with the biodegradable packaging foresigth study and with these results the innovation system actors can focus their strategies and projects.
Priority subjects up to 2032 in technology innovation area were: Natural coating, food container, active packaging, shrink films, foamed, intelligent packaging, bioactive films.
The most important commodities for biodegradable polymers production up to 2032 according to experts could be: corn, mandioca, potato, wood fiber, agroindustrial wastes, collagen, fatty acid, monoglycerids.
Banana fiber, chaff reed fiber, microbiane cellulose, polyhidroxyalkonates (PHAs), polyhidroxybutyrate valerate (PHBV), polilactic acid (PLA), microcrystalline cellulose, seaweed wet biomass, chitin, are the most important topics in the next years according to the experts.
The prominent additives are: glycerol, antimicrobial agents, powdered pigments, and coupling agents
Higher consensus technology in the three rounds: Twin screw extrusion, reactive extrusion, co-extrusion, blow film extrusion, injection molding, spray, co-laminathing, bio processes and nanotechnologies will be priority up to 2032.
Balaraman, S., & Venkatakrishnan, K. S. (1980). Identifying engineering education goals and priorities for the future: An experiment with the Delphi technique. Higher Education, 9(1), 53–67. http://doi.org/10.1007/BF00149135
Becker, G., & Bakal, D. A. (1970). Subject anonymity and motivational distortion in self-report data. Journal of Clinical Psychology, 26(2), 207–209.
Cabero Almenara, J. (2013). Formación del profesorado universitario en TIC. Aplicación del método Delphi para la selección de los contenidos formativos. Educación XX1, 17(1), 21. http://doi.org/10.5944/educxx1.17.1.10707
Cancelo Hidalgo, M. J., Neyro Bilbao, J. L., & Baquero Úbeda, J. L. (2013). Probiotics as adjuvant therapy for vaginitis. Agreement based on the Delphi method. Progresos de Obstetricia Y Ginecologia. http://doi.org/10.1016/j.pog.2013.07.002
Créange, A., & Careyron, A. (2013). The diagnosis of chronic inflammatory demyelinating polyneuropathy: A Delphi-method approach. Journal of Neurology, 260(12), 3015–3022. http://doi.org/10.1007/s00415-013-7100-2
Dalkey, N., & Helmer, O. (1963). An Experimental Application of the Delphi Method to the Use of Experts. Management Science, 9(3), 458–467. http://doi.org/10.1287/mnsc.9.3.458
Debin, M., Souty, C., Turbelin, C., Blanchon, T., Boëlle, P.-Y., Hanslik, T., Falchi, A. (2013). Determination of French influenza outbreaks periods between 1985 and 2011 through a web-based Delphi method. BMC Medical Informatics and Decision Making, 13, 138. http://doi.org/10.1186/1472-6947-13-138
Fletcher, A. J., & Marchildon, G. P. (2014). Using the Delphi Method for Qualitative, Participatory Action Research in Health Leadership. International Journal of Qualitative Methods. Retrieved from http://ejournals.library.ualberta.ca/index.php/IJQM/article/view/19025
De Liaño, B. G. G., & Pascual-Ezama, D. (2012). La metodología Delphi como técnica de estudio de la validez de contenido. Anales de Psicologia, 28(3), 1011–1020. http://doi.org/10.6018/analesps.28.3.156211
Gordon, T., & Helmer, O. (1964). Report on a long-range forecasting study. The RAND Corporation, 1–71. http://doi.org/10.1126/science.1088667
Gordon, T., & Pease, A. (2006). RT Delphi: An efficient, "round-less" almost real time Delphi method. Technological Forecasting and Social Change, 73(4), 321–333. http://doi.org/10.1016/j.techfore.2005.09.005
Graham, B., Regehr, G., & Wright, J. G. (2003). Delphi as a method to establish consensus for diagnostic criteria. Journal of Clinical Epidemiology, 56(12), 1150–1156. http://doi.org/10.1016/S0895-4356(03)00211-7
Helmer, O., & Rescher, N. (1959). On the Epistemology of the Inexact Sciences. Management Science. http://doi.org/10.1287/mnsc.6.1.25
Kauko, K., & Palmroos, P. (2014). The Delphi method in forecasting financial markets-An experimental study. International Journal of Forecasting, 30(2), 313–327. http://doi.org/10.1016/j.ijforecast.2013.09.007
Landeta, J. (2006). Current validity of the Delphi method in social sciences. Technological Forecasting and Social Change, 73(5), 467–482. http://doi.org/10.1016/j.techfore.2005.09.002
Liimatainen, H., Kallionpää, E., Pöllänen, M., Stenholm, P., Tapio, P., & McKinnon, A. (2014). Decarbonizing road freight in the future - Detailed scenarios of the carbon emissions of Finnish road freight transport in 2030 using a Delphi method approach. Technological Forecasting and Social Change, 81(1), 177–191. http://doi.org/10.1016/j.techfore.2013.03.001
Linstone, Harold A. Turoff, M. (1975). Delphi Method: Techniques and Applications. 1975. Retrieved from https://www.ncjrs.gov/App/Publications/abstract.aspx?ID=256068
Martino, J. P. (1993). Technological Forecasting for Decision Making. Interfaces (Vol. 25).
Kim, M., Jang, Y. C., & Lee, S. (2013). Application of Delphi-AHP methods to select the priorities of WEEE for recycling in a waste management decision-making tool. Journal of Environmental Management, 128, 941–948. http://doi.org/10.1016/j.jenvman.2013.06.049
Nielsen, C., & Thangadurai, M. (2007). Janus and the Delphi Oracle: Entering the new world of international business research. Journal of International Management, 13(2), 147–163. http://doi.org/10.1016/j.intman.2006.07.003
Okoli, C., & Pawlowski, S. D. (2004). The Delphi method as a research tool: an example, design considerations and applications. Information & Management. http://doi.org/10.1016/j.im.2003.11.002
Wakefield, R., & Watson, T. (2014). A reappraisal of Delphi 2.0 for public relations research. Public Relations Review, 40(3), 577–584. http://doi.org/10.1016/j.pubrev.2013.12.004
Zartha, J., Herrera, J. Prospectiva tecnológica. Aplicación del Método Delphi en Facultades y Programas de Ingeniería. Caso Universidad Pontificia Bolivariana. ISBN: 978-958-696-902-4. Primera edición. 2011.
Zartha, J., Montes, J., Toro, I., Villada, H. Método Delphi - Propuesta para el cálculo del número de expertos en un estudio Delphi sobre empaques biodegradables al 2032. Espacios. 2014. Vol. 35 (Nº 13), Pág. 10.
Zeedick, D. The Modified Delphi Method to Analyze the Application of Instructional Design Theory to Online Graduate Education. 2012
1. Universidad Pontificia Bolivariana. Escuela de Ingenierías – Facultad de Ingeniería Agroindustrial. Email: Jhon.zartha@upb.edu.co
2. Universidad del Cauca. Email: hsamuelv@yahoo.com
3. Universidad Pontificia Bolivariana. Email: rhernandezz@uqvirtual.edu.co
4. Universidad del Cauca. Email: alfernandez06@gmail.com
5. Universidad Pontificia Bolivariana. Email: bibiana.arango@upb.edu.co
6. Universidad Pontificia Bolivariana. Email: gina.orozco@upb.edu.co
7. Centro Regional de Productividad e Innovación del Cauca. CREPIC. Email: jrafaelbermudez@gmail.com
8. Universidad del Cauca. Email: diego_lds85@hotmail.com
9. Universidad del Cauca. Email: alcyrenecm@yahoo.com.mx
10. Instituto Tecnológico Metropolitano. Email: johnmorenosarta@gmail.com