ISSN 0798 1015

Vol. 39 (Number 46) Year 2018. Page 8

Characteristics of the e-learning implementation in higher education. A literature review

Características de la implementación del e-learning en educación superior. Una revisión de literatura

Diana Marcela CARDONA-ROMÁN 1; Jenny Marcela SÁNCHEZ-TORRES 2

Received: 26/05/2018 • Approved: 10/07/2018

Contents

1. Introduction

2. Conceptual framework

3. Methodology

4. Characterization and analysis of articles on the implementation of e-learning

5. Discussion and conclusions

Bibliographic references

1. Introduction

The inclusion of the Information and Communication Technologies (ICT) in higher education is part of the introduction of innovation in an educational organization (Schneckenberg, 2009). The ICT allow offering permanent training at vast scale (O’Neill, Singh, & O’Donoghue, 2004). The most innovative Higher Education Institutions (HEIs) could take advantage of several opportunities that arise from technological advances to contribute to the satisfaction of the needs of various consumers (González-Videgaray, 2007; O’Neill et al., 2004). However, some HEIs are pressed by the market to introduce new commercial products for education instead of products related to their quality requirements (González-Videgaray, 2007). The pressure to incorporate these commercial products, in some cases, constitutes a valid form of advertisement that could increase the enrollment, and generate a better institutional status. In other cases, the adoption of a commercial product is not enough (González-Videgaray, 2007).

E-Learning is a type of educational innovation and its implementation can be seen as a process that involves changes and modifies the way people work. Consequently, the implementation of e-learning involves challenges, difficulties and barriers for employees to accept the organizational change (Gedik, Kiraz, & Ozden, 2013; Naveed, Muhammed, Sanober, Qureshi, & Shah, 2017; Schneckenberg, 2009). In some cases, these technological innovations contribute to the improvement of academic performance, although only innovations by themselves do not guarantee any development in this aspect (Fetaji & Fetaji, 2009; Shahtalebi, Shatalebi, & Shatalebi, 2011; Yengin, Karahoca, Karahoca, & Uzunboylu, 2010). The e-learning implementation generates changes in the employees’ behavior, which means higher implementation costs. According to Tucker & Gentry (2009), the implementation costs are estimated in around 70%, compared to the acquisition costs of the necessary technological infrastructure for operating e-learning.

Hence, the HEIs that want to implement an online education solution need to create a vision of the desired end-product in order to start with the strategic planning (Kucina Softic & Bekic, 2008). Additionally, they must define the structure of the courses, their contents, and whether these will be entirely or partially online supported (Gedik et al., 2013). They also must assume the organizational changes required and, finally, the evaluation of the e-learning implementation process (Schneckenberg, 2009).

Nevertheless, the implementation of e-learning in HEIs lacks a standardized or systematic way of planning, comparing and evaluating the e-learning projects, as well as, their results and their effectiveness (Fetaji & Fetaji, 2009; Shahtalebi et al., 2011; Yengin et al., 2010). As suggested by Fetaji & Fetaji (2009), hundreds of e-learning projects have been carried out with significant differences in their results: some of them are success stories, others are complete failures.

Taking into account that the success of an e-learning implementation project depends on many characteristics, it is convenient to explore those required in HEIs. Cardona-Román & Sánchez-Torres (2016) reported 105 institutional characteristics grouped in ten categories for the implementation of e-learning when the HEIs include an educational innovation. Zhang & Duan (2017) also proposed seven dimensions at the institutional level to ensure the implementation of e-learning.

In any case, the characteristics of the organizational axis are necessary, as well as those of the technological and pedagogical axes. This paper aims to present the extracted characteristics from selected experiences reported in scientific databases, by classifying them into three axes: organizational, technological and pedagogical. To achieve this objective, it is necessary to answer the following questions: What characteristics are taken into account for the implementation of e-learning in different experiences around the world? How are these characteristics related to the organizational, technological and pedagogical axes?

This work is structured as follows: Section 2 describes the conceptual framework of e-learning, its implementation and the importance of its evaluation. Section 3 presents the description of the method used for structuring this paper and includes the exploration in scientific databases, the identification and the selection of the experiences to be analyzed. Section 4 contains the characterization of the experiences, the extraction of their characteristics after the content analysis and the answer found for the guiding question that allowed us to achieve the scope of this work. Likewise, the conclusions are condensed in Section 5 and, finally, special thanks are given to the agency that financed the work, and the bibliographic references are listed.

Before establishing the main characteristics involved in the implementation of e-learning, it is necessary to define some concepts related to the research. This section contains the documentary analysis and conceptual exploration of the following concepts: e-learning, implementation of e-learning, and evaluation of both e-learning and its implementation.

2.1. e-Learning

In this paper, we used the definition proposed by Cardona-Román & Sánchez-Torres (2011), who included the main attributes of e-learning, which is understood as:

an educational or teaching/learning process with a physical separation between the tutor and the student. The students acquire competencies and skills that they strengthen through the use of ICT, the use of internet and the help of multi-directional communication – synchronous and asynchronous tools –. The student is the center of the independent formation, thus, the student has continuous assimilation of knowledge, skills, and competencies with the support of collaborative learning. The contents are instantly updated, structured according to each one or the organization. The tutor supports the student. The formation has flexible access to both time and space, which allows adequate training and teaching.

García-Peñalvo & Seoane-Pardo (2015) did an updated review of the e-learning concept, its evolution, and the different generations proposed by guiding authors in this field of the information society. Besides the features of innovation, it includes the evaluation of learning, such as learning analytics, and it specifies that the subject of technology platforms is somehow overcome. This updated review opens a way for the technological ecosystem of e-learning and its services, which work as support of the actors involved in the teaching/learning process.

2.2. Implementation of e-learning

The implementation is "the execution in the organization of an adopted program, a process, the use of a product or an accepted idea" (Rivera, 1995). The implementation of e-learning is a dynamic process that requires a series of organizational activities and changes, and educational innovation, which is fundamental for an adequate execution and use of informatics systems (Pérez, 2006; Chen, Wu, & Ma, 2010; Kerimbayev, 2016). According to Guri-Rosenblit & Gros (2011), the experience and theory reach the same conclusion that implies a transformation of the institutional culture in the arrangement of the academic, pedagogical, curricular and administrative work

2.3. Evaluation of e-learning

Evaluating e-learning shows important initiatives and experiences around the world, aiming to establish standards that allow the identification of specific criteria and indexes (Ala-Mutka et al., 2010; Anyelis et al., 2010; Rubio, 2003). The evaluation of e-learning is focused on technology, i.e. it focuses on the use of a particular technological resource, or on the reaction or learning of the students and teachers (Al-Rahmi et al., 2018; Cardona-Román, 2012; González-Videgaray, 2007; Medina, 2013; Retno Sayekti, 2018).

2.4. Evaluation of the implementation of e-learning

The evaluation of the implementation process seeks to assess the way the plan is executed systematically. This assessment takes into account both, an administrative point of view, due to the available resources, and an assessment viewpoint, since the services that the project intends to provide are set in motion (Rossi & Freeman, 1993). This type of evaluation is interesting for the managers of the project themselves, to optimize its management. But it is also relevant for those who decide the continuity of the project and its financing  (Sánchez-Torres, 2006).

The documents on the evaluation of the e-learning implementation are centered around the analysis of an online course production (Carchiolo, Longheu, Malgeri, & Mangioni, 2007; Clegg & Bradley, 2006; Goh, Leong, Kasmin, Hii, & Tan, 2017; Hubackova, 2015; Jung, 2011). Some of them address aspects related to the planning of a project of implementation of e-learning (Hanna, 1998; Mahmud & Gope, 2009). Others are associated with the definition of policies (de Freitas & Oliver, 2005)  or to the difficulties in access and quality identified from the student's perspective (Akaslan, Law, & Taskin, 2012; Vagarinho, 2012).

In summary and according to some authors (Fetaji & Fetaji, 2009; Shahtalebi, Shatalebi, & Shatalebi, 2011; Yengin, Karahoca, Karahoca, & Uzunboylu, 2010), the implementation of e-learning in HEIs lack a standardized or systematic form of planning, comparing, and, of course, evaluating e-learning projects, their results and their effectiveness.

This research corresponds to the exploratory type based on the stages described below and shown in Fig. 1. This work was also structured with documentary and content analysis approaches (Kitchenham et al., 2009; Krippendorff, 2004).

Fig. 1
Stages of the method employed

Source: Prepared by the authors

3.1. Exploration in scientific databases

This stage defined a search protocol based on Kitchenham et al. (2009). The goal of the review was to characterize the experiences selected from the literature and to extract the main characteristics required to implement a successful e-learning solution. To achieve the goal we formulated three key questions (KQ). The KQs describe the research question specified in the introduction.

The key questions addressed by this study are:

KQ1. How are distributed the selected articles according to their type (journal, conference) and on what year were they published?

KQ2. What type of study does the experience address or propose (models, strategies, approaches, methodologies, frameworks, challenges or barriers, reviews)?

KQ3. Which perspective approach (institution, professor, student, conceptual) did the experience has? 

3.1.1 Selection of databases and definition of search strategy

To answer the KQs, we selected academic and scientific databases. This selection considered databases with complete articles on areas of knowledge such as education sciences, engineering, and administrative sciences. These are the main areas in which the present study was defined. The scientific databases selected were: EBSCO-ERIE, JSTOR, IEEE Xplore, Emerald, Springer, Science Direct and Taylor & Francis.

The search strategy with the key terms of the study, and a specific period between the years 2000 and 2015 was defined as:

((measur* OR evaluat* OR asses*) AND (implement* OR implant*) AND ("e-learning" OR elearning OR "virtual learning" OR "virtual education" OR "online education" OR "web-based education" OR "web-based learning" OR "virtual program") AND ("higher education" OR universit* OR HEI)) within 2000-2015.

3.1.2. Identification and selection of articles in the e-learning field

To identify the potential papers, we considered their availability (full text), titles and keywords. Then, the abstracts were reviewed for the final selection of the experiences and the selection criteria were assessed in each experience.

The selection criteria for the experiences were identified as:

1. Articles peer-reviewed obtained from databases structured from the search equation.
2. Articles published between the years 2000 and 2015.
3. Full text.
4. Articles aligned to the guiding question and the key questions.
5. Articles that included the terms "implementation of e-learning", "higher education", "university", "project", "practice", "evaluation", "quality", "factors", "characteristics", "criteria", "model", "strategy", "approach", "methodology", "process", "aspects", "indicators", "challenges of implementation", "organization" or "management of change", mainly written in English.
6. Exclusion of the articles with a title focused on an evaluation of technology, Learning Management Systems (LMS), e-learning architecture, or technological infrastructure.

The search in the selected scientific databases yielded a total of 5044 documents, which were subsequently filtered for documents available for download. A total of 4231 articles was identified, meaning 84%. The databases that recovered the most registries were: Springer (63%) and IEEE Xplore (25%). The remaining 12% corresponded to articles from the remainder of the consulted databases.

Applying the exclusion criteria, 720 articles were identified. The title and abstract were read to verify the alignment to the KQs and guiding question and 83 potential articles were select to be analyzed in depth. After a double check of both the inclusion and exclusion criteria, we found five repeated experiences, which were deleted. Then, in total, 78 potential articles of e-learning experiences were selected for an in-depth reading.

3.2. Characterization and analysis of articles on the implementation of e-learning

This stage was based on the methodology proposed by Krippendorff (2004). The initial characterization of the selected experiences was carried out with a classification into the type of article (journal or conference) and year of publishing. The content was analyzed and the details of the selected articles were revised with the purpose of answering the KQs. For this analysis, we identified the perspective from which the experiences were addressed (institution, professor, student, conceptual). Likewise, the experiences were classified according to the type of study they proposed (model, methodology, approach, strategy, framework, challenges and review). Finally, we obtained a set of characteristics, which were grouped in the organizational, pedagogical and technological axes.

The results show the general characterization of the 78 selected experiences. In addition, the content analysis allowed answering the KQs presented previously, thus, extracting the characteristics that define the implementation of e-learning.

4.1. General characterization of the articles

Fig. 2 contains the distribution of the selected experiences according to the year and type of publication, answering the KQ1. The year with the most publications was 2009 with 11 articles, followed by 2010 and 2012, both with ten articles. It was found that from the selected experiences, 51 correspond to journals articles (65%) and 27, to conferences articles (35%).

Fig. 2
Distribution of experiences selected according to year and type of document

Source: The authors

The type of study proposed by each experience was determined during the deep-analysis, to establish the groups and their comparisons. The types of study proposed were: model, challenges/barriers, strategy, approach, methodology, framework, and reviews. Their distribution is shown in Fig. 3 and answers the KQ2. The experiences classified as models were the most reported, reaching 19 articles (24%). The strategies amounted to 16 articles (20%). The challenges or barriers obtained 14 articles (18%). Besides, the approaches achieved 12 articles (15%), while the rest of the types of study obtained less than ten articles.

Fig. 3 Experiences classified according to the type of study they address or propose

The perspectives addressed by the experiences were classified as institution, professor, student, or conceptual. The experiences could have more than one perspective, thus, single-perspective and multi-perspective classifications were made, as can be seen in Fig. 4. On the one hand, in the single-perspective classification, 32 articles belong to the student perspective (42%) and 21 articles to the perspective of the educational institution. Two articles considered the professor perspective and 12 articles were from the conceptual perspective. On the other hand, for the multi-perspective classification, the most common was the student-professor perspective with six articles. The student-institution and the institution-professor perspectives had one article each. Three articles were classified into three perspectives addressing (student-professor-institution). These findings provide an answer to the KQ3.

Fig. 4
Perspectives addressed in the selected experiences

Source: The authors

4.2. Identification of characteristics

This work identified the characteristics of the implementation of e-learning, which were grouped in three axes: organizational, pedagogical and technological.

Organizational: It corresponds to everything related to the educational institution, its structure and functioning. This axis includes the planning, establishing of policies and guidelines related to online education, financial aspects, quality, organizational culture, organizational changes, institutional vision, and goals. Besides, it includes the flow of communication, leadership, means of implementation, project management, perceived benefits from e-learning, establishing of teamwork and roles, monitoring, control, evaluation, among others.

Pedagogical: It contains the characteristics related to educational and training aspects such as the construction of contents, courses, and curriculum. This axis also includes evaluation of learning, formation, change of roles, educational resources, student learning, teaching, training, attitude of the actors, motivation, among others.

Technological: It implies the technical solutions, infrastructure, connectivity, accessibility, equipment (both hardware and software), LMS, and administrative support for the technology. This axis includes quality testing for technology, adoption, use, and management of content. Besides, it considers their capacity for integrating the technical resources necessary to manage, organize, coordinate, design and impart teaching programs, among others.

The content analysis identified 158 characteristics in the 78 selected experiences. Fig. 5 shows the findings by type of study and axis.

Fig. 5
Characteristics grouped by type of study and axis

Source: The authors

We found out 27 characteristics of the experiences classified as reviews: 13 characteristics correspond to the organizational axis, eight (8) to the pedagogical axis, and six (6) to the technological axis. These experiences included the implementation plans, project feasibility, planning, effectiveness and evaluation, selection or development of a curriculum, contents, and technology to support the instruction process.

The experiences classified as strategies had 26 characteristics that contain both the processes that must be kept in mind when implementing e-learning and the way it has to be implemented. This group of the experiences answers how to carry out the solution, implementation, and operation of e-learning. The characteristics were assigned, thus: 15 for the organizational axis, four (4) for the pedagogical axis, and seven (7) for the technological axis.

The distinctive attributes of the experiences classified as frameworks were related to the impact of e-learning on teamwork, culture, perceived benefits, barriers to be faced, the role of the actors within the online learning environment, among others. The characteristics identified were 24, which were grouped by axis as follows: 12 for the organizational axis, six (6) for the pedagogical axis, and six (6) for the technological axis.

The characteristics of the experiences identified as methodologies were 22. These include the plans for implementation, such as the evaluation of the technological infrastructure and of the whole e-learningproject. However, it is necessary to note that these experiences lack details when it comes to the specifics of each phase, stage or step they propose. In the organizational axis were grouped 10 characteristics, 11 in the technological axis, and one (1) in the pedagogical axis.

The experiences classified as approaches group the practices related to what should be taken into account when implementing e-learning, as well as, how to start the solution. We identified 21 characteristics, which were assigned as follows: eight (8) for the organizational axis, six (6) for the pedagogical axis, and seven (7) for the technological axis.

In experiences classified as barriers and challenges, we found out 20 characteristics. Nine (9) characteristics were assigned to the organizational axis, six (6) characteristics to the pedagogical axis, and the last five (5) to the technological axis. The common characteristics were: technological and educational implementation, attitude of professors and students, organizational structure, leadership, policies, standards, procedures for e-learning in the organization, and financial resources.

We identify 18 characteristics of the experiences classified as models: seven (7) characteristics were aligned with the organizational axis, five (5) characteristics with the pedagogical axis, and six (6) characteristics with the technological axis. These characteristics address the processes to be executed in the start-up of the e-learning.

The summary of the common characteristics of each type of study and grouped by axis are shown in Table 1 . The findings indicate that 22 relevant characteristics are required for the implementation of e-learning. The characteristics associated with the pedagogical axis appear in most of the experiences analyzed. One of the characteristics is related to the planning and construction of educational materials, and another is the evaluation of the quality of these materials. The fact that these characteristics have a more significant frequency implies that those are the most researched aspects when it comes to e-learning. The same thing happens with the characteristics of the technological axis. The two common characteristics in this axis were identified as, first, the selection, development, and use of an e-learningtechnology, and second, the evaluation of the quality of e-learningtools.

Table 1
Summary of the characteristics organized by axis and type of study.

Source: Prepared by the authors. Note: The symbol (●) indicates the classification
according to the type of study where the characteristic was found.

In summary, it is understood that the implementation of e-learning appears in most of the educational organizations, and it can be categorized in experiences of technological implementations (Akaslan et al., 2012; Gedik et al., 2013; González-Videgaray, 2007; Sharpe, Benfield, & Francis, 2006). The e-learning implementation models or plans must be clear and specific (Akaslan et al., 2012; Sharpe et al., 2006) or considered as a value chain (Barsky & Catanach, 2011; Gómez, Reyes, & Romero, 2007). The implementation of e-learning requires success factors that must be taken into account in the execution of the e-learning project (Borotis, Zaharias, & Poulymenakou, 2008; Gedik et al., 2013; González-Videgaray, 2007; Mallinson & Vos, 2009; Shahtalebi et al., 2011). It also requires leaders to identify and confront the challenges that arise from the e-learning implementation process (Barron & Schneckenberg, 2012; Comisión Europea, 2006; González-Videgaray, 2007; Kucina Softic & Bekic, 2008; O’Neill et al., 2004; Owusu-Ansah, Neill, & Haralson, 2011; Schneckenberg, 2009; Yengin et al., 2010). It is necessary to highlight that the organizational characteristics are essential in the implementation of online programs. These organizational characteristics vary according to the organizational structure, planning, objectives, goals, leadership, communication, culture, beliefs and values of the actors of the educational institution.

This work highlights the review of several experiences that address the implementation of e-learning. Their characteristics were grouped into three principal axes explained above (organizational, pedagogical and technological), and classified according to types of study (models, challenges, strategies, approaches, methodologies, frameworks and reviews). The review of these experiences made it possible to recognize that the implementation of e-learning depends on these three axes. The extraction of the characteristics according to the type of study facilitated the content analysis, allowing that the information be manageable and comparable among similar articles.

The inclusion of publications in a period of sixteen years (2000 to 2015) extracted from recognized academic databases allowed to achieve the purpose outlined for this work, that is, summarizing the characteristics for the implementation of e-learning around the world. The sample of the experiences selected implied a significant representation of countries (40 countries). The countries with the most considerable amount of experiences referred were: USA (12%), Australia (9%), Turkey (6%) and Europe (6%). The previous provided an answer to the research question, and the KQs formulated.

The findings show very few academic papers on the implementation of e-learning in Latin American countries. Nevertheless, there are results published by VirtualEduca & ACESAD (2013), Lupion-Torres & Rama (2010), and Silvio et al. (2004) that show advances in the field of online education in Latin America and the Caribbean.

The content analysis of the experiences allowed to elucidate that the predominant type of research was qualitative, with an appearance in 51% of the experiences. Besides, the preferred method for data collecting was the survey (reported in 25 articles). However, the most notable weakness of these experiences was the lack of details to perform the measurement or evaluation.

The implementation of e-learning continues to have an instrumentalist vision focused on technologies, their acquisition and adoption. Most of the articles were related to the selection of a technology platform and its evaluation. The technological evaluation took into account the pertinence, usability, accessibility, acquisition, and adaptability (Al-Rahmi et al., 2018; Tingoy & Gulluoglu, 2012; Vagarinho, 2012). Furthermore, it is essential that the technological infrastructure lose importance and become a component that creates added value within the general strategy of an educational organization (Calderón & Marín, 2011; VirtualEduca & ACESAD, 2013). Other critical features identified for implementing an e-learning solution were the construction of contents, and the creation of courses and their evaluation (Berechet & Istrimschi, 2014; Carchiolo et al., 2007; Stefaniak, 2015; Van, 2012).

It is necessary to understand that the development of the course includes a protocol for instructional design, production of e-learning materials, learning resources, learning objects, and distribution of the materials. Besides, this protocol must be adjusted to the pedagogical model of the institution. In the same way, the educational organizations must perform evaluations on the pertinence of the courses they offer, as well as an assessment of the student-professor and student-student relations, of the learning and of the support services provided to guarantee quality. 

Organizational, financial, political and support characteristics become important institutional capacities that add value when an educational innovation is launched. Likewise, the training and awareness of all the actors on e-learning are permanent actions that allow the correct execution of new projects. The features extraction allows overcoming the challenges and barriers that may arise while paying attention to the critical success factors. Besides, it allows analyzing the distinctive characteristics of the context, due to their influence on the success or failure of an e-learning initiative.

The implementation of e-learning in the HEIs lacks a standardized or systematic way for being carried out. Thus, activities such as planning, comparing and evaluating e-learning projects, their results, and their effectiveness, are so diverse from one institution to another, even if they are found in the same microenvironment.

This literature review showed that standardized metrics are lacking to evaluate the implementation of an entirely online program and not only the virtualization or use of a course. Although it is understood that virtual courses are part of the general process for the implementation of online programs.

The evaluation of the implementation of e-learning in HEIs is a still-developing subject, taking into account that there are few documents focused on measuring their progress and monitoring when a work plan is executed.

One of the limitations of this research is related to the types of documents reviewed and the search protocol followed. As mentioned previously, the search was focused on documents published on scientific databases, without considering institutional documents or documents from quality evaluation agencies. Consequently, this limitation may be one of the topics of future research, which could enrich the characteristics required to implement a successful online program.

5.1. Acknowledgement

This work is a product of the doctoral thesis "Model of Measurement of the Implementation of Virtual Programs of Postgraduate in Colombian HEIs. An Approach from the Organizational Management". Authors thank the Office of Research and Extension of the Faculty of Engineering of the Universidad Nacional de Colombia for the style review. The first author acknowledges the Administrative Department of Science and Technology of Colombia (COLCIENCIAS for its acronym in Spanish) for the financing through the doctoral fellowship in the national call for doctoral studies No. 567/2012.

Akaslan, D., Law, E. L., & Taskin, S. (2012). Analysis of issues for implementing e-learning: The student perspective. In ISSN:2165-9559, E-ISBN:978-1-4673-1455-8, ISBNPrint:978-1-4673-1457-2, & INSPECAccessionNumber:12746734 (Eds.), Proceedings of the 2012 IEEE Global Engineering Education Conference (EDUCON) (pp. 1–9). Marrakech: IEEE. http://doi.org/10.1109/EDUCON.2012.6201122

Al-Rahmi, W., Alias, N., Othman, M. S., Alzahrani, A. I., Alfarraj, O., Saged, A. A., & Rahman, N. S. A. (2018). Use of E-Learning by University Students in Malaysian Higher Educational Institutions: A Case in Universiti Teknologi Malaysia. IEEE Access, 6. http://doi.org/10.1109/ACCESS.2018.2802325

Ala-Mutka, K., Redecker, C., Punie, Y., Ferrari, A., Cachia, R., & Centeno, C. (2010). The Future of Learning : European Teachers ’ Visions. JRC59775 - 2010. IPTS Institute for prospective Technological Studies and JRC European Commission. Seville.

Anyelis, A., Blahová, V., DöĞer, M. F., Doratis, L., Kantcheva, S., Kangasniemi, J., … Skjerve, T. (2010). Learning, Innovation and ICT: Lessons learned by the ICT cluster Education & Training 2010 programme. (V. den B. Lieve, C. Malin, & G. Barbara, Eds.)European Commision. Lisboa: Lieve, Van den Brande Malin, Carlberg Barbara, Good. Retrieved from http://www.kslll.net

Barron, A., & Schneckenberg, D. (2012). A theoretical framework for exploring the influence of national culture on Web 2 . 0 adoption in corporate contexts. The Electronic Journal Information Systems Evaluation, 15(2), 176–186. http://doi.org/ISSN 1566-6379

Barsky, N. P., & Catanach, A. H. (2011). Transforming the Business Education Value Chain with E-Learning Technologies. Evidence from a Principles Course at a Private US University. In P. Van den Bossche, W. H. Gijselaers, & R. G. Milter (Eds.), Building Learning Experiences in a Changing World (pp. 149–164). Dordrecht: Springer Netherlands. http://doi.org/10.1007/978-94-007-0802-0

Berechet, L. D., & Istrimschi, P. A. (2014). Education Reboot: Reinventing the University. Procedia - Social and Behavioral Sciences, 142, 755–761. http://doi.org/10.1016/j.sbspro.2014.07.611

Borotis, S., Zaharias, P., & Poulymenakou, A. (2008). Critical Success Factors for E-Learning Adoption. In T. Kidd (Ed.), Handbook of Research on Instructional Systems and Technology (pp. 496–511). New York: Idea Group Publishing.

Calderón, A., & Marín, J. C. (2011). Modelo de gestión integral de TIC en procesos de producción de educación virtual. (Tesis de maestría). Universidad ICESI.

Carchiolo, V., Longheu, A., Malgeri, M., & Mangioni, G. (2007). A model for a web-based learning system. Information Systems Frontiers, 9, 267–282. http://doi.org/10.1007/s10796-006-9014-3

Cardona-Román, D. M. (2012). Propuesta de diseño de metodología para evaluar el impacto del e-learning en el proceso de aprendizaje de estudiantes de educación a distancia. Universidad Nacional de Colombia.

Cardona-Román, D. M., & Sánchez-Torres, J. M. (2011). La educación a distancia y el e-learning en la sociedad de la información: una revisión conceptual. Revista UIS Ingenierías, 10(1), 29–52. Retrieved from http://revistas.uis.edu.co/index.php/revistauisingenierias/article/view/39-52

Cardona-Román, D. M., & Sánchez-Torres, J. M. (2016). A Literature Review of the Institutional Characteristics Required for Implementing E-Learning in Higher Education. In 8th International Conference on Education and New Learning Technologies, EDULEARN16 (pp. 2219–2231). Barcelona, Spain: IATED. http://doi.org/10.21125/edulearn.2016.1444

Chen, J. W., Wu, D. B., & Ma, H. L. (2010). A Strategic Alignment of e-learning Implementation Process in a University Setting. In Proceedings - 2010 International Conference on Web Information Systems and Mining, WISM 2010 (Vol. 1, pp. 109–112). http://doi.org/10.1109/WISM.2010.121

Clegg, S., & Bradley, S. (2006). Models of personal development planning: practice and processes. British Educational Research Journal, 32(1), 57–76. http://doi.org/10.1080/01411920500402003

Comisión Europea. (2006, March 30). ¿Cómo puede el aprendizaje combinado ayudar a la integración de las TIC en la educación continua?”, boletín de prensa. 30 marzo 2006. Elearningeuropa.info/es/, p. 1. Europa. Retrieved from http://elearningeuropa.info/es/taxonomy/term/995

de Freitas, S., & Oliver, M. (2005). Does E-learning Policy Drive Change in Higher Education?: A case study relating models of organisational change to e-learning implementation. Journal of Higher Education Policy and Management, 27(1), 81–96. http://doi.org/10.1080/13600800500046255

Fetaji, B., & Fetaji, M. (2009). e-Learning Indicators: a Multi-Dimensional Model for Planning and Evaluating e-Learning Software Solutions. Electronic Journal of E-Learning, 7(2), 1–28.

García-Peñalvo, F. J., & Seoane-Pardo, A. M. (2015). Una revisión actualizada del concepto de eLearning. Décimo Aniversario. Journal Education in The Knowledge Society (JEKS), 16(1), 119–144. http://doi.org/10.14201/eks2015161119144

Gedik, N., Kiraz, E., & Ozden, M. Y. (2013). Design of a blended learning environment: Considerations and implementation issues. Australasian Journal of Educational Technology, 29(1), 1–19.

Gómez, L. M., Reyes, A. R., & Romero, W. (2007). Aplicación de un modelo de vigilancia tecnológica al e-learning. In F. Malaver & M. Vargas (Eds.), Vigilancia tecnológica y competitividad sectorial: lecciones y resultados de cinco estudios (pp. 220–247). Bogotá D.C., Colombia: OCYT.

González-Videgaray, M. (2007). Evaluación de la reacción de alumnos y docente en un modelo mixto de aprendizaje para educación superior. Revista ELectrónica de Investigación Y EValuación Educativa - RELIEVE, 13(1), 83–103.

Guri-Rosenblit, S., & Gros, B. (2011). E-Learning: Confusing Terminology , Research Gaps and Inherent Challenges. The Journal of Distance Education / Revue de I’Éducation À Distance, 25(1), 1–12.

Hanna, D. E. (1998). Higher Education in an Era of Digital Competition: Emerging Organizational Models. Journal of Asynchronous Learning Networks- JALN, 2(1), 66–95.

Hubackova, S. (2015). Evolution and Evaluation of e-learning. Procedia - Social and Behavioral Sciences, 171, 231–235. http://doi.org/10.1016/j.sbspro.2015.01.114

Jung, I. (2011). The dimensions of e-learning quality: from the learner’s perspective. Educational Technology Research and Development, 59(4), 445–464. http://doi.org/10.1007/sl 1423-010-917

Kerimbayev, N. (2016). Virtual learning: Possibilities and realization. Education and Information Technologies, 21(6), 1521–1533. http://doi.org/10.1007/s10639-015-9397-1

Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009). Systematic literature reviews in software engineering - A systematic literature review. Information and Software Technology, 51(1), 7–15. http://doi.org/10.1016/j.infsof.2008.09.009

Krippendorff, K. (2004). Content Analysis: An introduction to its methodology (Second). California: Sage Publications.

Kucina Softic, S., & Bekic, Z. (2008). Organizational aspects of supporting e-learning at university level. In ITI 2008 - 30th International Conference on Information Technology Interfaces (pp. 153–158). Dubrovnik: IEEE. http://doi.org/10.1109/ITI.2008.4588399

Lupion-Torres, P., & Rama, C. (2010). La educación superior a distancia en América Latina y el Caribe. Realidades y tendencias. (PUCPR, VirtualEduca, & UNISUL, Eds.)La Educación Superior a Distancia en América Latina y el Caribe. Realidades y Tendencias. Brasil: Editora Unisul. Retrieved from http://virtualeduca.org/documentos/observatorio/oevalc_2010_(tendencias).pdf

Mahmud, K., & Gope, K. (2009). Challenges of Implementing E-learning for Higher Education in Least Developed Countries: A Case Study on Bangladesh. In 2009 International Conference on Information and Multimedia Technology (pp. 155–159). Jeju Island: IEEE. http://doi.org/10.1109/ICIMT.2009.27

Mallinson, B., & Vos, L. (2009). A Theoretical Investigation of the Synergy Between Enterprise E-Learning and Knowledge Management. In C. Barry, K. Conboy, M. Lang, G. Wojtkowski, & W. Wojtkowski (Eds.), Information Systems Development. Challenges in Practice, Theory and Education (Springer U, pp. 1–602). Boston, MA: Springer US. http://doi.org/10.1007/978-0-387-68772-8

Medina, L. (2013). La evaluación en el aula: reflexiones sobre sus propósitos, validez y confiabilidad. Revista Electrónica de Investigación Educativa - REDIE, 15(2), 34–50.

Naveed, Q. N., Muhammed, A., Sanober, S., Qureshi, M. R. N., & Shah, A. (2017). Barriers effecting successful implementation of E-learning in Saudi Arabian Universities. International Journal of Emerging Technologies in Learning, 12(6), 94–107. http://doi.org/10.3991/ijet.v12i06.7003

O’Neill, K., Singh, G., & O’Donoghue, J. (2004). Implementing eLearning Programmes for Higher Education: A Review of the Literature. Journal of Information Technology Education, 3(Structural Issues for Traditional Universities), 313–323.

Owusu-Ansah, A., Neill, P., & Haralson, M. K. (2011). Distance Education Technology: Higher Education Barriers During the First Decade of the Twenty-First Century. Online Journal of Distance Learning Administration, XIV(II), 1–13.

Pérez, C. A. (2006). Implementación de un sistema de educación virtual en una institución de educación superior. Fundación Univesitaria Católica del Norte.

Retno Sayekti. (2018). The Implementation of E-learning System at UIN Sumatera Utara in Response to Technology Challenge in Education. In Retno Sayekti. (2018). The Implementation of E-learning System at UIN Sumatera Utara in Response to Technology Challenge in Education. In IOP Conf. Series: Journal of Physics: (Vol. 970).IOP Conf. Series: Journal of Physics: (Vol. 970).

Rivera, J. (1995). La implementación: Un fenómeno organizativo multidimensional (Serie de Economía de la Empresa 04 No. 4). Documento de Trabajo (Vol. 95–22). Getafe, España. Retrieved from http://e-archivo.uc3m.es/bitstream/handle/10016/6419/db952204.pdf?sequence=1

Rossi, P., & Freeman, H. (1993). Evaluación. Un enfoque sistémico para programas sociales. (Trillas, Ed.). México: Trillas.

Rubio, M. J. (2003). Focus and Models of Evaluation of the e-learning. Revista ELectrónica de Investigación Y EValuación Educativa - RELIEVE, 9(2), 101–120. Retrieved from http://www.uv.es/RELIEVE/v9n2/RELIEVEv9n2_1.htm

Sánchez-Torres, J. M. (2006). Propuesta metodológica para evaluar las políticas públicas de promoción del e-government como campo de aplicación de la Sociedad de la Información. El caso colombiano. (Tesis doctoral). Universidad Autónoma de Madrid.

Schneckenberg, D. (2009). Understanding the real barriers to technology-enhanced innovation in higher education. Educational Research, 51(4), 411–424. http://doi.org/10.1080/00131880903354741

Shahtalebi, S., Shatalebi, B., & Shatalebi, F. (2011). A Strategic Model of Virtual University. Procedia - Social and Behavioral Sciences, 28(1), 909–913. http://doi.org/10.1016/j.sbspro.2011.11.167

Sharpe, R., Benfield, G., & Francis, R. (2006). Implementing a university e-learning strategy: levers for change within academic schools. Research in Learning Technology, 14(2), 135–151. http://doi.org/10.1080/09687760600668503

Silvio, J., Rama, C., & Lago, M. T. (2004). La educación superior virtual en América Latina y el Caribe. UNESCO IESALC. México: UNESCO IESALC, Instituto Internacional para la Educación Superior en américa Latina y el Caribe.

Stefaniak, J. E. (2015). The implementation of service-learning in graduate instructional design coursework. Journal of Computing in Higher Education, 27(1), 2–9. http://doi.org/10.1007/s12528-015-9092-7

Tingoy, O., & Gulluoglu, S. S. (2012). Effects of culture on distance learning in higher education in the light of digital divide and multicultural: Curriculum and design strategies. In 2012 IV International Conference “Problems of Cybernetics and Informatics” (PCI) (pp. 1–3). Baku: IEEE. http://doi.org/10.1109/ICPCI.2012.6486275

Tucker, J. P., & Gentry, G. R. (2009). Developing an e-learning strategy in higher education. Foresight, 11(2), 43–49. http://doi.org/10.1108/14636680910950147

Vagarinho, J. P. (2012). Quality in e-learning processes State of art. In P. ISBN: & 978-1-4673-4743-3 (Eds.), 2012 International Symposium on Computers in Education (SIIE) (pp. 1–6). Andorra la Vella: IEEE. http://doi.org/13247378

Van, N. (2012). The Solution for Building a Strategic Plan for e-learning at Universities in Vietnam. In International Conference on Education and e-Learning Innovations (Vol. 7, p. 5). IEEE.

VirtualEduca, & ACESAD. (2013). la educación superior a distancia y virtual en Colombia: nuevas realidades. (N. Arboleda Toro & C. Rama Vitale, Eds.). Bogotá D.C., Colombia: ACESAD / VIRTUAL EDUCA.

Yengin, İ., Karahoca, D., Karahoca, A., & Uzunboylu, H. (2010). Re-thinking virtual universities. Procedia - Social and Behavioral Sciences, 2(2), 5769–5774. http://doi.org/10.1016/j.sbspro.2010.03.941

Zhang, W., & Duan, C. (2017). Construction and Implementation of Institutional E-learning Development Framework in Continuing Higher Education. Journal of Educational Technology Development & Exchange, 10(1), 11–25.

1. Master of Computing and Systems Engineering. Computing Systems and Industrial Engineering Department, Universidad Nacional de Colombia. Ph.D. Candidate in Engineer – Industrial and Organizations. dmcardonar@unal.edu.co

2. Full-time Professor. Ph.D. in Economics, Innovation Management, and Technological Policies. Master of Management of Science and Technology Analysis and Management. Master of Systems Engineering. Computing Systems and Industrial Engineering Department. Universidad Nacional de Colombia. jmsanchezt@unal.edu.co