dynaDYNADyna rev.fac.nac.minas0012-7353Universidad Nacional de Colombia10.15446/dyna.v84n202.59554ArtículosAssessing the efficiency of science, technology and innovation using Data Envelopment Analysis (DEA): The case of ColombiaEvaluación de la eficiencia de ciencia, tecnología e innovación mediante Data Envelopment Analysis (DEA): Caso ColombiaAristizábal-TorresDanielaCastro-PeñaMónica YulenibEcheverri-GutiérrezPaola AndreacValencia-SalazarJesús Davidc Centro de Investigaciones, Universidad Libre Seccional Pereira, Pereira, Colombia. daristizabal@unilibrepereira.edu.coUniversidad LibreCentro de InvestigacionesUniversidad LibreSeccional PereiraPereiraColombiadaristizabal@unilibrepereira.edu.co Universidad Católica de Pereira, Pereira, Colombia. monica.castro@ucp.edu.coUniversidad Católica de PereiraUniversidad Católica de PereiraPereiraColombiamonica.castro@ucp.edu.co Facultad de Ingeniería, Universidad Libre Seccional Pereira, Pereira, Colombia. paecheverri@unilibrepereira.edu.co, jdvalencia@unilibrepereira.edu.co Universidad LibreFacultad de IngenieríaUniversidad LibreSeccional PereiraPereiraColombiapaecheverri@unilibrepereira.edu.cojdvalencia@unilibrepereira.edu.coJul-Sep201784202215220110820162203201730072017This is an open-access article distributed under the terms of the Creative Commons Attribution LicenseAbstract
This paper provides insights into how to approach the assessment of scientific production efficiency using Data Envelopment Analysis (DEA). The methods employed in this paper take into account both the CCR and Assurance Region (AR) models, which are based on science, technology and innovation (STI) indicators. The sample focuses on the departments in Colombia as well as the Capital District (C.D.). The research concludes that both models calculate the efficiency of the units of analysis. Furthermore, according to the country’s STI targets, AR is susceptible to hierarchies generated within the variables.
Resumen
Se plantea una aproximación de medición de la eficiencia de la producción científica mediante el Data Envelopment Analysis (DEA): modelos CCR y Assurance Region (AR) con base en los indicadores de ciencia, tecnología e innovación (CTI) en los departamentos y en el Distrito Capital (D.C.) de Colombia. Se concluyó que ambos modelos calculan la eficiencia de las unidades de análisis, siendo AR susceptible a jerarquizaciones generadas en las variables de acuerdo a los objetivos país en temas de CTI.
Keywords:assessment of scientific productionefficiencyscience, technology and innovation indicatorsData Envelopment Analysis (DEA).Palabras clave:medición de producción científicaeficienciaindicadores de ciencia, tecnología e innovaciónAnálisis Envolvente de Datos (DEA).1. Introduction
Science and innovation indicators demonstrate the potential of the technical scientific information that is generated by a country [1].
Scientific journals are the fundamental channel of formal communication within the scientific community [2]. The number of patents applied for or granted and the number of papers published in journals that have a high impact factor are indicators that are used to measure the quality of research [3].
On the global scale, the Organization for Economic Co-operation and Development OECD has published hand-books to assess activities relating to science, technology and innovation, such as the Oslo Manual (1997), the Canberra Manual (1995), the Manual of Patents (1994), the Technology Balance of Payments (TBP) (1990) and the Manual of Frascati (1963) [4].
The Administrative Department of Science, Technology and Innovation (Colciencias) in Colombia frequently monitors research groups. Colciencias has implemented a pattern to classify research groups using the categories A1, A, B, C, D and recognized (Rec), which follows the research group indicator. The coefficient is assigned according to the categorization of four scientific elements: development of innovative knowledge, technological development, social appropriation of the knowledge and training of human resources. Furthermore, the pattern classifies researchers into three categories (Senior, Associated and Junior) according to their academic degree and their academic production [5].
The report entitled ‘State of Science in Colombia’ was written in 2015 and detailed the production of research in the different administrative divisions (departments) according to the four product categories. The indicators take into account investments in science, the number of scholarships granted, research centers and research groups, recognized researchers and registered products [5].
Some scholars have encouraged researchers to measure scientific production, which includes: measuring university research [6], the influence of the size of the institution, and the number of researchers producing scientific research in the centers [7]; scientific production in biological and health sciences in twenty Brazilian universities [8]; measuring scientific production in the field of the psychology [9]; and the empirical study of scientific measurement [10].
The above areas of research evaluate the number of products developed by the units of analysis. Other scholars make measurements in terms of efficiency; however, this involves two serious challenges: First, the research needs to develop standard criteria to compare the data collected from reality. Second, some cases analyzed are designed with complex organizational structures with multiple inputs and outputs for endowments and products, which do not assess efficiency with suitable precision. Therefore, the researcher should allocate weighted values according to trade-offs and factor endowments [11]. The mathematical application Data Envelopment Analysis (DEA) is such an alternative solution.
Some research that has used the DEA model has measured efficiency in the following areas: research projects in two clinical centers [12], national innovation systems [13], innovation in each German region [14], international comparisons of efficiency among the national innovation systems [15], analysis of scientific and technological activity [16], technology and knowledge transfer in Spanish public research centers [17], research groups in Andalucía [1], innovation regional systems in 31 Chinese provinces [18], classification of research group in Colombia [19], R&D activities according to the number of patents submitted in the European Union [20] and research activity at Valladolid University [21].
Based on data from Colciencias, this paper details the advantages and features of DEA application in order to measure scientific activity in Colombia.
2. Theoretical approach2.1. Science, Technology and Innovation.
Science, Technology and Innovation (STI) are core elements that should be used to develop sustainable knowledge in societies. STI has the potential, nationally, to be a significant engine for social and economic development. National and regional policies promote both investments and human resources training. These activities strengthen the abilities to link STI to sustainable growth [22].
Colciencias annually up-dates the assessment framework for scientific production in Colombia. The adjustments made are based on global standards that classify the research abilities and technological development of recognized research groups and researchers in Colombia. Despite this, research activity in Colombia remains far removed from developed countries. Investments into activities relating to science, technology and innovation (ASTI) represent only 0.5% of GDP in Colombia; investments into Research and Development (R&D) represents 0.2% [5].
2.2. Data Envelopment Analysis (DEA)
DEA is a technique based on linear programming the purpose of which is to help maximizing the efficiency of each decision making unit (DMU). Every DMU could be a functional unit, a process or a group that demands inputs and supply outputs [23]. There are two ways of measuring DMU efficiency [24]:
The input-oriented method: the quantity of resources is minimized in order to get the best level of production.
The output-oriented method: the quantity of manufactured products is maximized by a fixed level of inputs.
DEA offers a method to find a set of proportions based on the assumption that all observed data is homogeneous. The aim is to maximize the function of relative efficiency of the DMU; however, this is subject to the radius of the DMU not being bigger than the unit. The border of efficiency can be variable with the BCC model or constant with the CCR model. Specifically, this research uses the CCR output-oriented model, which is represented by the following formulation (1):
CCR-O is taken as a baseline for the application of the Assurance Region (AR) model, which establishes the limits at the top and the bottom for the weighted values. This allocation depends on whether the restrictions only affect the weights input and / or output [25].
3. Research methods
The application of the model was carried out in four phases: The first was the categorization of the information that is consolidated in the ‘State of Science in Colombia’ document using input and output variables; this information was subsequently selected depending on its relationship and direct influence on the states’ research efficiency. In the second phase, the constant scale generated by the CCR model was selected, which is oriented to the outputs as the products are susceptible to improvement. In the third stage, the CCR-O is applied and is taken as the basis for the Assurance Region model in order to adjust the efficiency measurement by means of weight restrictions assigned to products. This is more in line with the Colciencias’ evaluation of each one of the types of products. The final phase refers to the interpretation of the efficiency levels determined by the two models applied.
3.1. Selection of variables
The research calculated the global results of 32 departments in Colombia. The Capital District (Bogota) was also observed due to its significant level of research production: the result was that a total of 33 DMU were obtained. The “State of Science in Colombia” exhibits information that shows the dynamics and general lines of Colombia’s science and technology (S&T), which are the variables used to apply the DEA model to each of the items in the report. They are classified as being either input and output variables and have a set of values for each unit of analysis. This operation (shown in Fig. 1) assumes that both monetary investments and human capital of every DMU are the inputs used develop research products.
Selected variables
Source: The authors.
Fig. 1 shows 15 input variables, which include categories and subcategories, 33 DMU and four output variables. The number of variables was reduced because of the operationalization of the DEA model. Otherwise, the number of efficient DMUs would have been high, which would have allowed them a higher number of possible combinations and been a grade closer to the efficiency border. Nevertheless, the reduction in variables does not generate substantial loss of information for the application.
First, three variables related to ‘categorization of researchers’ were united into one variable. The same process was then applied for the six ‘categorization of research groups’ variables, and it was taken into account that these variables have a hierarchical order that is represented in particular categories relating to effort and research relevancy. The following paragraph explains the criteria used to unify the variables.
The weighting scale with the values proposed in Colciencias’ last scheme of measurement was applied to assess research and research groups. This scale is coherent with the relevancy and the level of contributions to the science described within the classification (c.p Table 1). Therefore, the value of these variables is a weighted sum. The two variables presented a co-relation of 0.994. Overall, ‘research groups’ was the representative variable, taking into account that researchers’ production contributes to the classification of the research groups.
Weighting value for category of research groups
Source: The authors.
Moreover, Colciencias also observes the investment in ASTI and R&D as independent categories in order to differentiate them. However, this research grouped them into one variable as they are complementary [26].
Scholarships awarded for Master’s and PhD programs were not included in the project as input variables because the products of this investment will become reflected in the selected output variables in the medium-term after the students have finished their education. The following are used as the input variables:
Research centers and technological development that are recognized and strengthened by Colciencias, according to information from the period 2009-2013.
Research groups that were recognized and classified by their particular field of knowledge and for their scientific and technological production (results based on call for applications 693 of 2014).
Approved STI projects financed by Colciencias, the values of which were calculated at constant 2013 prices and correspond to the period 2010 - 2013.
Investment in science, technology and innovation (ASTI) and research and development (R&D) activities as a percentage of the GDP. This investment relates to the year 2013.
Four types of products were taken as output variables: new knowledge (NK), technological development (TD), social appropriation of the knowledge (SAK) and human training (HT). This information allows researchers to identify the target of each DMU’s effort.
3.2. Application of the model
This research used two DEA models as a reference. The first is the Assurance Region (AR), which maintains a hierarchy within the output variables. In this research, this refers to the different types of products developed by the research in Colombia; the second is the CCR-O model. Unlike the previous model, CCR-O does not provide any type of restriction when allocating weighted values for both input variables and output variables.
The AR model uses the CCR-O model as a baseline, which results in a constant scale. Specifically, in the Colombian case, every type of product counts on a different significance level, which depends on the research groups in Colciencias’ scheme of assessment. The calculated weighted values can be found in Table 2.
Weighted values for the type of research products
Source: The authors.
According to the weighted values (Table 2), each variable has a restriction when qualifying the efficiency of the DMUs, which is related to the proportion of weight assigned by DEA. Both the best and the worst stages were calculated for each of the possible relations between the products in order to establish the limits of the restrictions. Those limits can be found at the top and the bottom of data. Table 3 includes the limits of restrictions between product relations.
Limits of the restrictions between product relations
Source: The authors.
The Assurance Region model allows us to identify the weighting values assigned to each of the output variables for the DMUs. Considering their restrictions, they maintain the levels of relevance that are assigned to the types of research products by Colciencias, as shown in Fig. 2.
Level of weighted values allocated to the output variables
Source: The authors.
The CCR-O model does not take into account any kind of restriction. CCR-O freely allocates the most convenient weighted values to each DMU, according to the levels of the input variables and output variables.
4. Main findings
The application of the CCR-O model and AR model identified the DMU, which appears to be both efficient and inefficient depending on the levels that are described for the input variables and output variables. Furthermore, the research indicates the group of reference for the inefficient units. In addition, the results allow us to find the variables which need to be improved in order to be efficient. Finally, the research looks at the weighting assigned to the variables.
Four DMU were eliminated from 33 departments when the models were applied (Guainía, Putumayo, Vaupés and Vichada) as they offered values equal to zero in all their output variables. They were, therefore, disqualified from the analysis. Both models were output-oriented and were solved by the Software DEA Solver because the aim of this research is to determine the efficiency of the departments in Colombia and the Capital District in terms of their scientific production. The frequency of efficiencies for the DMUs were obtained and can be seen in Fig. 3.
Efficiency of the <italic>AR</italic> model and CCR-O model
Source: The authors.
Fig. 3 offers the DMUs according to the different ranges of efficiency levels for each model. The CCR-O model identified a higher quantity of technically efficient DMUs than the AR model. This difference is explained due to the AR model being more rigorous with restrictions that limiting the weighting values allocation. Neither the CCR- O nor the AR models described were Pareto-efficient DMUs because at least one of their variables was assigned with a zero weighting value.
The CCR-O model developed a border of constant efficiency that is shaped by the DMUs, which appear to be efficient. This takes into account that the aim of the projections in the model is to lead the inefficient units to the border. Based on this, efficient DMUs are converted into references that do not appear in the same situation. Fig. 4 shows how many times a DMU was referenced in each of the models, and a ranking is generated within the efficient DMUs since a higher frequency as a reference, implies a better performance of that unit compared to other efficient units [1].
Group of reference in the models
Source: The authors.
Overall, Boyacá stood out as a global leader as it was efficient in both models as it had the biggest indicator such as modality of inefficient units. Amazonas and Atlántico came second and third in the number of DMUs. Cesar, Córdoba, Capital District, Caldas and Magdalena, which has a ‘zero’ frequency in the AR model, stopped being efficient after they were included in the restrictions of hierarchical structuring of the level of importance that was awarded to the research products.
DEA shapes the groups of reference in order to propose improvements to the inefficient DMUs, according to the levels presented by them. The goal is to achieve efficiency through identifying the variables and quantities which must be modified.
The main improvement is proposed for output variables because the CCR model and AR model are output-oriented. Despite this, some DMUs need the input variables to be further modified in order to become efficient. In this case, the models coincided with the type of variables that need to be changed in either a major or less manner so that the analysis units become efficient. The following variables should be adjusted with a descending relevance: products of technological development, social appropriation, human training and products of new knowledge.
Despite the models being output-oriented, both proposed modifications for some of the input variables. Investment in ASTI and research centers developed major amounts of change. This indicates that some departments are not developing enough products that justify the resources used as there are DMUs which achieved major production with less resources for those variables.
5. Discussion and conclusions
There is no evidence in Latin-American scientific reports regarding assessing the efficiency of the scientific activity on a regional level. 59% of research used CCR and 40% of research used BCC from all the projects that used DEA. The remaining 1% of projects used Super Efficiency, Crossed Efficiency and a proper guideline. None of the cases used the Assurance Region model. Therefore, there were no restrictions considered with incidence for the share of the weighting values assigned to the variables by DEA.
More than 95 % of the research analyzed used the human and financial resources as input variables. Moreover, 70 % of the research used patents and publications (papers and books) as output variables. The share of efficiency for the units of analysis ranged between 20 % and 66 %. For this exercise with CCR, the proportion of efficient DMU was 38 %, and when the AR was adjusted, the proportion was lowered to 21 %. This share makes more rigorous estimation possible because it includes restrictions that refer to levels of importance regarding the output variables. In the both cases, there were no Pareto-efficient DMUs as, at least in one of the variables, a weighting value of zero was allocated. Therefore, they were technically efficient.
Broadly speaking, the both the CCR model and the AR model allow researchers to undertake a measurement of STI approach. The AR model is suitable for hierarchical structures developed in the variables by the particular entity that manages the STI in each of the countries.
Researchers are concerned about efficiently measuring scientific up-dates. The main patterns used to be able to evaluate scientific activity are the high human resource skills and the relation they have with proper, public and private funds. These patterns allow research centers to generate products such as new knowledge, technological development, social appropriation and human training.
Furthermore, there is an opportunity to carry out a comparative study in other countries using the AR model, which allows researchers to determine hierarchical structures within the output variables, according to the CTI targets in the country. A multi-centric model can also be applied to compare the DMU case study.
Aknowledgements
We thank the Universidad Libre Seccional Pereira for their support during the research process.
References[1] Pino, J., Solís, F., Delgado, M. y Barea, R., Evaluación de la eficiencia de grupos de investigación mediante análisis envolvente de datos (DEA). El profesional de la Información, 19(2), pp.160-167, 2010. DOI: 10.3145/epi.2010.mar.06PinoJ.SolísF.DelgadoM.BareaR.Evaluación de la eficiencia de grupos de investigación mediante análisis envolvente de datos (DEA)El profesional de la Información192160167201010.3145/epi.2010.mar.06[2] Togia, A. and Tsigilis, N., Impact factor and education journals: A critical examination and analysis. International Journal of Educational Research, 45(6), pp. 362-379, 2006. DOI: 10.1016/j.ijer.2007.02.001TogiaA.TsigilisN.Impact factor and education journals: A critical examination and analysis.International Journal of Educational Research456362379200610.1016/j.ijer.2007.02.001[3] Joshi, M., Bibliometric indicators for evaluating the quality of scientific publications. The Journal of Contemporary Dental Practice [online]. 15(2), 2014. [date of reference October 12th of 2015]. Available at: Available at: http://www.jaypeejournals.com/eJournals/ShowText.aspx?ID=6058andType=FREEandTYP=TOPandIN=_eJournals/images/JPLOGO.gifandIID=457andisPDJoshiM.Bibliometric indicators for evaluating the quality of scientific publications.The Journal of Contemporary Dental Practice1522014October 12th of 2015Available at: http://www.jaypeejournals.com/eJournals/ShowText.aspx?ID=6058andType=FREEandTYP=TOPandIN=_eJournals/images/JPLOGO.gifandIID=457andisPD[4] Godin, B. and Ratel, S., Jalons pour une histoire de la mesure de la science. Conférence Internationalisme statistique, pratiques étatiques et traditions nationales (UQAM), Montreal. [online]. 1999. Available at: http://www.cirst.uqam.ca/Portals/0/docs/note_rech/99-11.pdfGodinB.RatelS.Jalons pour une histoire de la mesure de la scienceConférence Internationalisme statistique, pratiques étatiques et traditions nationalesMontreal1999http://www.cirst.uqam.ca/Portals/0/docs/note_rech/99-11.pdf[5] Departamento Administrativo de Ciencia, Tecnología e Innovación - Colciencias. El estado de la ciencia en Colombia, Colombia, 2015, 212 P.Departamento Administrativo de Ciencia, Tecnología e Innovación - ColcienciasEl estado de la ciencia en ColombiaColombia2015212[6] Cherchye, L. and Vanden, P., On research effciency: A microanalysis of Dutch university research in economics and business management, Research Policy, 37(4), pp. 495-516, 2005. DOI: 10.1016/j.respol.2005.03.005CherchyeL.VandenP.On research effciency: A microanalysis of Dutch university research in economics and business managementResearch Policy374495516200510.1016/j.respol.2005.03.005[7] Bonaccorsi, A. and Daraio, C., Exploring size and agglomeration effects on public research productivity, Scientometrics, 63(1), pp. 87-120, 2005. DOI: 10.1007/s11192-005-0205-3BonaccorsiA.DaraioC.Exploring size and agglomeration effects on public research productivityScientometrics63187120200510.1007/s11192-005-0205-3[8] Zorzetto, R., Razzouk, D., Dubugras, M., Gerolin, J., Schor, N., Guimarães, J. and Mari, J., The scientific production in health and biological sciences of the top 20 Brazilian universities. Brazilian Journal of Medical and Biological Research, 39(12), pp. 1513-1520, 2006. DOI: 10.1590/S0100-879X2006005000040ZorzettoR.RazzoukD.DubugrasM.GerolinJ.SchorN.GuimarãesJ.MariJ.The scientific production in health and biological sciences of the top 20 Brazilian universities.Brazilian Journal of Medical and Biological Research391215131520200610.1590/S0100-879X2006005000040[9] García, A., Guerrero, V. and Moya, F., World scientific production in psychology. Universitas Psychologica [Online]. 11(3), 2012. [date of reference November 21th of 2015]. Available at: Available at: http://www.scielo.org.co/scielo.php?script=sci_arttextandpid=S1657-92672012000300002GarcíaA.GuerreroV.MoyaF.World scientific production in psychologyUniversitas Psychologica1132012November 21th of 2015Available at: http://www.scielo.org.co/scielo.php?script=sci_arttextandpid=S1657-92672012000300002[10] Crespi, G. and Geuna, A., An empirical study of scientific production: A cross country analysis, 1981-2002, Research Policy , 37, pp. 565-579, 2008. DOI: 10.1016/j.respol.2007.12.007 CrespiG.GeunaA.An empirical study of scientific production: A cross country analysis, 1981-2002Research Policy37565579200810.1016/j.respol.2007.12.007[11] Hernangómez-Barahon, J., Pérez, V.M. y Martín-Cruz, N., Implicaciones de la organización interna sobre la eficiencia. La aplicación de la teoría de la agencia y la metodología DEA a las ONGD españolas. Cuadernos de Economía y Dirección de la Empresa [online]. (40), pp. 17-46, 2009. [date of reference November 12th of 2015]. Available at: Available at: http://www.redalyc.org/articulo.oa?id=80711788002Hernangómez-BarahonJ.PérezV.M.Martín-CruzN.Implicaciones de la organización interna sobre la eficiencia. La aplicación de la teoría de la agencia y la metodología DEA a las ONGD españolasCuadernos de Economía y Dirección de la Empresa4017462009November 12th of 2015Available at: http://www.redalyc.org/articulo.oa?id=80711788002[12] Dilts, D., Zell, A. and Orwol, E., A novel approach to measuring efficiency of scientific research projects: Data envelopment analysis. Clinical and Translational Science, 8(5), pp. 495-501, 2015. DOI: 10.1111/cts.12303DiltsD.ZellA.OrwolE.A novel approach to measuring efficiency of scientific research projects: Data envelopment analysis.Clinical and Translational Science85495501201510.1111/cts.12303[13] Kotsemir, M., Measuring national innovation systems efficiency - A review of DEA approach, Higher School of Economics Research. WP BRP, 16(1), pp 1-24, 2013. DOI: 10.2139/ssrn.2304735KotsemirM.Measuring national innovation systems efficiency - A review of DEA approachHigher School of Economics Research161124201310.2139/ssrn.2304735[14] Broekel, T., Rogge, N. and Brenner, T., The innovation efficiency of German regions - A shared-input DEA approach. Working Papers on Innovation and Space. [Online]. 8(13), 2013. [date of reference October 25th of 2015]. Available at: Available at: ftp://137.248.191.199/RePEc/pum/wpaper/wp0813.pdfBroekelT.RoggeN.BrennerT.The innovation efficiency of German regions - A shared-input DEA approachWorking Papers on Innovation and Space8132013October 25th of 2015Available at: ftp://137.248.191.199/RePEc/pum/wpaper/wp0813.pdf[15] Zhang, J., International comparison of national innovation system efficiency. Tech Monitor. [Online]. 2013. [date of reference October 25th of 2015]. Available at: Available at: http://www.techmonitor.net/tm/images/d/da/13apr_jun_sf2.pdfZhangJ.International comparison of national innovation system efficiencyTech Monitor2013October 25th of 2015Available at: http://www.techmonitor.net/tm/images/d/da/13apr_jun_sf2.pdf[16] Gao, W. and Yuan, L., Analysis of science and technology activity based on DEA model, Third International Conference ICICA, pp. 604-609, 2012. DOI: 10.1007/978-3-642-34038-3_83GaoW.YuanL.Analysis of science and technology activity based on DEA modelThirdInternational Conference ICICA604609201210.1007/978-3-642-34038-3_83[17] Guede, M., La eficiencia de los centros públicos de investigación en el proceso de transferencia de conocimiento y tecnología. PhD dissertation. Universidad Rey Juan Carlos de Madrid, Madrid, España. 2011.GuedeM.La eficiencia de los centros públicos de investigación en el proceso de transferencia de conocimiento y tecnología.PhDUniversidad Rey Juan Carlos de MadridMadrid, EspañaMadrid, España2011[18] Tong, L. and Liping, C., Research on the evaluation of innovation efficiency for China's regional innovation system by utilizing DEA. 2009 International Conference on Information Management, Innovation Management and Industrial Engineering, pp. 192-195. 2009. DOI: 10.1109/ICIII.2009.53TongL.LipingC.Research on the evaluation of innovation efficiency for China's regional innovation system by utilizing DEA2009International Conference on Information Management, Innovation Management and Industrial Engineering192195200910.1109/ICIII.2009.53[19] Restrepo, M. y Villegas, J., Clasificación de grupos de investigación colombianos aplicando análisis envolvente de datos. Revista de la Facultad de Ingeniería de la Universidad de Antioquia, [Online]. 42, 2007. [date of reference September 29th of 2015]. Available at: Available at: http://www.redalyc.org/articulo.oa?id=43004209RestrepoM.VillegasJ.Clasificación de grupos de investigación colombianos aplicando análisis envolvente de datos.Revista de la Facultad de Ingeniería de la Universidad de Antioquia422007September 29th of 2015Available at: http://www.redalyc.org/articulo.oa?id=43004209[20] Quindós, M., Vicente, M. y Rubiera, F., La eficiencia de las actividades de I+D desde el punto de vista de las patentes registradas en los países de la Unión Europea. Estudios de Economía Aplicada [Online]. 23(3), 2005. [date of reference January 21th of 2016]. Available at: Available at: http://www.redalyc.org/articulo.oa?id=30123305QuindósM.VicenteM.RubieraF.La eficiencia de las actividades de I+D desde el punto de vista de las patentes registradas en los países de la Unión Europea.Estudios de Economía Aplicada2332005January 21th of 2016Available at: http://www.redalyc.org/articulo.oa?id=30123305[21] Castrodeza, C. y Peña, M., Evaluación de la actividad investigadora universitaria: Una aplicación a la Universidad de Valladolid. Estudios de Economía Aplicada . [online]. 20(1), 2002. [date of reference January 13th of 2016]. Available at: Available at: http://www.redalyc.org/articulo.oa?id=30120111CastrodezaC.PeñaM.Evaluación de la actividad investigadora universitaria: Una aplicación a la Universidad de ValladolidEstudios de Economía Aplicada2012002January 13th of 2016Available at: http://www.redalyc.org/articulo.oa?id=30120111[22] UNESCO. Programa de política científica y fortalecimiento de capacidades de la Unesco, Montevideo, Uruguay, 2016. UNESCOPrograma de política científica y fortalecimiento de capacidades de la UnescoMontevideo2016[23] Soto, J. and Arenas, W., Análisis envolvente de datos: De la teoría a la práctica, Pereira, Postergraph, 2010.SotoJ.ArenasW.Análisis envolvente de datos: De la teoría a la prácticaPereiraPostergraph2010[24] Angulo, L., Soares, J. and Figueiredo, S., A multiobjective approach for non-discretionary variables in data envelopment analysis, DYNA 83(195), pp. 9-15, 2015. DOI: 10.15446/dyna.v83n195.45396AnguloL.SoaresJ.FigueiredoS.A multiobjective approach for non-discretionary variables in data envelopment analysisDYNA83195915201510.15446/dyna.v83n195.45396[25] Coll, V. and Blasco, O., Evaluación de la eficiencia mediante el análisis envolvente de datos, Universidad de Valencia, España, 2006.CollV.BlascoO.Evaluación de la eficiencia mediante el análisis envolvente de datosUniversidad de ValenciaEspaña2006[26] Observatorio Colombiano de Ciencia y Tecnología., Percepciones de las ciencias y las tecnologías en Colombia: Resultados de la III encuesta nacional de percepción pública de la ciencia y la tecnología, Colombia, Bogotá, 2014.Observatorio Colombiano de Ciencia y TecnologíaPercepciones de las ciencias y las tecnologías en Colombia: Resultados de la III encuesta nacional de percepción pública de la ciencia y la tecnologíaBogotá2014
How to cite: Aristizábal-Torres, D., Castro-Peña, M.Y., Echeverri-Gutiérrez, P.A. and Valencia-Salazar, J. D., Assessing the efficiency of science, technology and innovation using Data Envelopment Analysis (DEA): The case of Colombia. DYNA, 84(202), pp. 215-220, September, 2017.