Espacios. Vol. 36 (Nº 09) Año 2015. Pág. E-1

Development of a sustainable behavior measurement scale of undergraduate students

Desenvolvimento de uma escala de medida de comportamento sustentável dos estudantes universitários

Débora SPENASSATO 1; Andréa C. TRIERWEILLER 2; Antonio Cezar BORNIA 3; Beatriz Marcondes de AZEVEDO 4; Rolf Hermann ERDMANN 5; Lucila M. S. CAMPOS 6

Recibido: 29/01/2015 • Aprobado: 16/02/2015

Contenido

1. Introduction

2 Sustainable Behavior

3 Planned Behavior, Norm-Activation Theories and Triple Bottom Line

4 Item Response Theory

5 Methodological procedures

6 Results and discussion

7 Conclusion

References

Appendix A. Items used to measure the sustainable behavior of students

1. Introduction

There is increasing interest among researchers in establishing measures for sustainability issues and sustainable behavior. Among these efforts we can highlight a study undertaken to analyze the attitudes of undergraduate students about the environment using the revised New Ecological Paradigm (NEP) scale and to measure behavior with the Environmentally Responsible Behavior Index (Thapa, 1999). Daneshmandi and MacLauchlan (2006) affirm that the evaluation of environmental education interventions is at an early stage, especially with the use of quantitative approaches. They examined the psychometric properties of the Children's Environmental Attitudes and Knowledge Scale, and found the instrument to be reliable and valid.

Studies conducted throughout the world have indicated that there are differences between the environmental attitudes and environmentally responsible behaviors of undergraduate students (Müderrisoğlu, Altanlar, 2011). According to Juárez-Nájera et al. (2010, p. 688): "Prediction of sustainable behavior is not simple. It appears to involve a number of variables, none of which is likely to operate without interacting with others". Some studies played a vital role in understanding this subject, among them is the study conducted by Hines et al. (1987) which developed a meta-analysis on responsible environmental behavior, and twenty years after, Bamberg and Möser (2007) presented a new meta-analysis of psycho-social determinants of pro-environmental behavior.

Assessing behaviors that cannot be measured directly is complex, unlike measuring the height of an individual, which can be measured directly with a tape-measure. For this reason, "IRT is predicated on the existence of latent variables: constructs that cannot be directly measured, yet are inferred to exist based on peripheral associations among measurable qualities" (Thomas, 2011, p. 292). Therefore, it allows measuring behavior using a set of items included in a questionnaire that is designed to estimate a latent trait (Sustainable behavior). Müderrisoğlu and Altanlar (2011), in an initial qualitative approach, used a set of items to determine environmental attitudes through The New Ecological Paradigm Scale (NEP) that consisted of 15 items grouped in 5 main ecological attitudes, and they measured Environmentally Responsible Behaviors (ERB) using 24 items. Then, based on a quantitative approach, they explained the environmental attitudes and environmentally responsible behaviors of students using factor analysis with the Varimax Rotation method.

The most commonly applied methodology for test evaluation is Classical Test Theory (CTT), which uses the total score to estimate the latent trait in the individual. Item Response Theory (IRT) uses probabilistic models to make inferences, making it possible to develop a standardized measurement scale and to identify the latent trait level of each individual. It also allows comparing items, individuals and different populations, which is an advantage over classical test analysis. Thus, IRT has become a widely used technique in many fields of study.

The purpose of this paper is to use Item Response Theory to develop a sustainable behavior measurement scale of undergraduate students. Sustainable behavior will be measured in terms of the relationship between human behaviors and actions focused on sustainability. This sample was selected because undergraduate students perform an important role in our society "[...] because they will be the future custodians, planners, policy makers, and educators of the environment and its issue" (Thapa, 1999, p. 426). This article is based on the Theory of Planned Behavior, Norm-Activation Theory and Triple Bottom Line (TBL) facets.

This paper is organized in six additional sections. The next section presents a background on the study of sustainable behavior. Section 3 presents the research theories that guided the development of the set of items. Section 4 presents the Item Response Theory used for analyzing these items and the development of the scale. In Section 5 we present the methodological procedures adopted in the study, followed by the results and discussion in Section 6 and finally, the conclusions in Section 7.

• 2. Sustainable Behavior

Environmental problems caused by human behavior are longstanding and have worsened with technological advances: new production methods that use machines and other procedures that are not environmentally friendly have been destroying the natural world. Thus, concern for the fate of the planet has focused on the relationships that people have with the environment (Polli, Kuhnen, 2011).

Sustainable behavior comprises the set of effective, anticipated and deliberate actions aimed at accepting responsibility for the conservation and preservation of physical and cultural resources. This responsibility includes the integrity of animal and plant species, social and individual welfare and the safety of present and future generations (Corral-Verdugo, Pinheiro, 2004).

Cordano et al. (2011) conducted a survey among business students in the United States and Chile to compare three theories of pro-environmental behavior; one of these theories was Schwartz's Norm-Activation Theory, which was also used in this article. The researchers concluded that each theory explained a significant amount of variance in behavioral intentions, but no theory was clearly dominant in the samples investigated.

Universities are major employers and resource users, have a crucial role in promoting sustainable behavior, and have the opportunity to contribute to the sustainability of the societies in which they exist (Merkel, Litten, 2007). As such, institutions of higher education must be concerned with designing and implementing curriculum, and training undergraduate students about the principles of sustainability (Ryu, Brody, 2006). An example of the theme of sustainability in higher education is the study conducted by Pappas et al. (2013, p. 01) that states: "Rapidly declining environmental, economic, and social conditions world-wide require a response from the engineering community, especially in higher education, since many of the solutions to these problems are related to the discipline, particularly engineering design for sustainability". Lozano et al. (2013, p. 10) propose that universities should assume the role to become sustainability leaders, ensuring that the needs of present and future generations are better understood. Thus, professionals who are well versed in sustainable development can educate students in the transition to "sustainable societal patterns".

3. Planned Behavior, Norm-Activation Theories and Triple Bottom Line

The Theory of Planned Behavior focuses on decision-making based on rational choices and self-interest, while Norm-Activation Theory focuses on values and moral norms. The Theory of Planned Behavior assumes that behavior is influenced by subjective norms and perceived behavioral control, this guides an individual's conduct, which is defined as his or her sustainable attitude or willingness to behave in a sustainable way (or not) (Ajzen, 2002).

Psychology explains that the concept of attitude originated in an attempt to predict social behavior, and defines attitude as a disposition to respond favorably or not to a psychological object (Morris, Maisto, 2004). Therefore, attitudes are expected to predict and explain human behavior. Positive attitudes would explain trends towards sustainable behavior, and negative attitudes would explain trends away from sustainable behavior. Norm-Activation Theory attempts to explain the mechanisms that lead a person to act in an altruistic way. Altruistic behavior is influenced by the activation of personal norms (moral obligations), and this activation depends on the values of each individual (Schwartz, 1977). There is an increasing demand for studies that predict environmental attitudes based on values, which are defined as standards that guide the lives of individuals (Tamayo, 2012).

Norm-Activation Theory and its derivations have been able to explain with greater certainty such behavior by more effectively contemplating the moral domain. Juárez-Nájera et al. (2010) conclude that sustainable behavior is a reflection of the values and norms internalized by individuals. Thus, we decided to explore Planned Behavior by creating items related to the locus of control concept that is part of this theory.

Dimopoulos et al. (2008) conducted a study that was crucial for helping environmental educators to generate positive attitudes towards the National Marine Park of Zakynthos among local early ages. They designed a conservation education module with activities, to affect knowledge and attitudes of students from elementary school based on pre-test and post-test control and experimental group to measure the effects on knowledge, understand and concern, locus of control, and verbal commitment. The results showed a significant effect on knowledge, but not on attitudes. However, post-test correlations in the experimental group showed that as the knowledge level increased both (a) the locus of control and (b) understanding and concern for the sea turtle issue, became more defensible.

Almeida and Sobral (2009) affirm that since the second half of the twentieth century there has been significant interest in values as psychological variables, culminating in the development of theoretical analyses and empirical research designed to understand human values beyond the narrow field of philosophical reflection and provide solutions for operationalizing the concept and studying it empirically. In terms of scientific trends, Almeida and Sobral refer to the pioneering work of Rokeach in the 1970s, which has been considered a reference in the study of human values over the past decades in many countries. In addition, the theory of universal values was created by Schwartz in 1992. Schwartz proposed a synthesis of previous contributions in a model that understands that values are general objectives aimed at satisfying basic human needs. He recognized ten motivational types that he uses to explain the implicit reasons for human behavior and choices made throughout life (Schwartz, 1994).

In this article, universal values are considered to be the life goals of individuals and are related to social behaviors. Schwartz (1992) believes that human values are beliefs that produce positive and negative feelings when activated. Universal values are motivational constructs that refer to desirable goals that guide choices and the evaluation of actions, people and situations, and serve as criteria for making judgments. To operationalize this theory, the SVS scale (Schwartz Value Survey) was created, which is composed of a set of items grouped into 10 categories. Schwartz (2005) presents the categories as follows:

1. Hedonism is derived from organic needs and pleasure associated with satisfaction;
2. Stimulation is derived from organic needs for variety, stimulation and excitement;
3. Self-Direction is derived from organic needs for control and dominance and interactional requirements for autonomy and independence;
4. Power is the achievement or maintenance of a dominant position of prestige within a social system;
5. Achievement is a demonstration of competence according to socially approved cultural patterns within concrete interactions;
6. Tradition is related to the customs valued by the group; they express group solidarity and contribute to the group's survival;
7. Conformity is derived from the requirement that the personal inclinations that might damage the operations and interactions of the group be inhibited;
8. Security is derived from the basic requirements of individuals and groups; the two subtypes of security values, some serve primarily individual interests, others wider group interests;
9. Universalism is derived from the survival needs of groups and individuals, in contrast to an in-group focus on of benevolence values, tolerance and protection for the welfare of  all individuals and for nature;
10. Benevolence is derived from the basic requirements for the functioning of the group and the organic need for affiliation; it emphasizes voluntary concern for the welfare of others.

Finally, the Triple Bottom line (TBL) implies that a firm's responsibilities are much broader than simply those related to economics are. This theory considers social and environmental measures of performance in addition to traditionally used economic measures. Environmental performance generally refers to the amount of resources a company uses in its operations and the by-products that its activities create. Social performance generally refers to the impact that a firm, including its suppliers, has on the communities in which it works. Social and environmental performance is normally unique to each organization or at least to each industry. Measuring performance in terms of social and environmental aspects is thus a difficult task (Hubbard, 2009, p. 180).

4. Item Response Theory

Item Response Theory uses statistical models to make inferences about latent traits based on the responses of individuals to test or questionnaire items that are designed specifically for this purpose (Andrade et al., 2000).

The mathematical models employed in IRT specify that an examinee's probability of answering a given item correctly or positively depends on the examinee's ability and the parameters of the item. This relationship is represented by Item Characteristic Curve (ICC) (Hambleton et al., 1991). IRT also provides Item Information Curve (IIC) and the Test Information Function (TIF). The IIC indicates the quality of each item, i.e., how much each item contributes to the construction of the measurement scale. The TIF is the sum of IICs (Baker, 2001). The ICC represents the relationship between the probability of an individual presenting sustainable behavior and the parameters.

The statistics obtained using this methodology are independent of the group of individuals who were used to calculate them if the following assumptions are met: (1) all parameters of the items must be in the same scale (usually a normal scale, with a mean of zero and variance of one), (2) there must be only one latent trait responsible for carrying out all the items of test (unidimensionality) and (3) there must be local independence, which determines that, considering this single latent trait, responses to items are independent of one another (Baker, 2001; Andrade et al., 2000; Hambleton et al., 1991). Because of the invariance of parameters, these items may be applied again to another group.

The one-dimensional logistic model with two parameters in IRT was proposed by Birnbaum (1968), as follows:

Where:

P(Uij=1|θj) represents the probability of a positive response from student j to item i, given its latent trait;

θj represents the latent trait level, which in this study is the student's Sustainable Behavior;

bi represents the difficulty of item i, which is measured on the same scale of the latent trait; the higher the value, the greater the level of the latent trait required for the student to respond positively to a given item;

and ai is the discrimination parameter of item i. It is proportional to the slope of the ICC at point bi; the higher the value, the greater the power to discriminate individuals.

• 5. Methodological procedures

Initially, the research took a qualitative approach, having relied on literature review to prepare the set of items. A specialist in psychology and an environmental specialist assessed these items. The second phase of research was quantitative, involving statistical analysis to assess the quality of the items and the test and to measure sustainable behavior.

We developed the questionnaire by conducting exploratory research to understand the factors found in earlier studies to influence the demonstration of sustainable behavior using two approaches in psychological science: (1) the Theory of Planned Behavior, from which were derived the items regarding the locus of control and (2) Norm-Activation Theory, which was used for the items relating to Universal Values. We found studies by Tamayo (2012), Juárez-Nájera et al. (2010), and Thapa (1999), among others, by filtering the search using the descriptor "sustainable behavior". In addition to these theories derived from psychology, (3) we used the TBL (Elkington, 1997) for creating items related to environmental, social and economic aspects of sustainable behavior.

Juárez-Nájera et al. (2010) elaborated the model to investigate the behavior for sustainability of individuals in higher education institutions with the use of key variables; and one of them was universal values, which is also used in this study.

The theory of the locus of control assumes that the locus is a dispositional state of the individual that changes according to the situation. According to Tamayo (2012), it is a dynamic concept that explains and describes the more-or-less stable beliefs based on which an individual identifies the source of control of events and their own behavior.

The questionnaire contains three sections:

1. The first section of the locus of internal control includes 14 items.
2. The second section of universal values contains 20 items referring to the 10 categories of values: hedonism, stimulation, self-direction, power, achievement, tradition, conformity, security, universalism and benevolence.
3. The third section contains 9 items related to the TBL that involve issues of sustainability, which inherently consider the environmental, social and economic spheres (3 items for each sphere).

The sample consisted of 120 students from courses in Technology Analysis and Systems Development and Environmental Management and Tourism Management in two Brazilian Federal Institutes of higher education. To measure the sustainable behavior of the students, a questionnaire with the 43 initial items was employed in the period from the 7th to the 11th of May 2012. The students completed the questionnaire online, through individual logins.

There were three response categories for each item ("no", "sometimes" and "yes"). However, during analysis, the response categories were dichotomized due to the low response rate in some categories; therefore, "sometimes" was considered to be "no", so "no" answers and "sometimes" answers = 0 and "yes" answers = 1.

Of the 43 initial items in the questionnaire, 6 were excluded from the analysis because they had problems in the parameters estimation, i.e., low power of discrimination (a < 0.6) and a high standard error. Thus, we did not consider them in the construction of the scale, and measurement of the latent trait. The items eliminated can be classified as follows: 1 item about stimulation, 1 item about conformity, 1 item about power and 3 items about economic TBL. Thus, the final questionnaire contained 37 items.

For data analysis, a model of dichotomous responses with two parameters of IRT (Birnbaum, 1968) and the method of the Marginal Maximum Likelihood were used within the Bilog software (Mislevy, Bock, 1990) to estimate the model parameters and to check the test quality. This method was composed of two phases: (1) estimation of the items parameters considering a certain distribution of the ability and (2) assumption that the items parameters are known and the abilities are estimated using the Maximum Likelihood.

The test's dimensionality was determined using the R software (R Core Team, 2013) with the aid of packages "mirt" (Chalmers, 2011) for the factor analysis with full information (Bock et al., 1988) and with the package "psych" (Revelle, 2012) for the analysis of the main components with a tetrachoric correlation matrix (Mislevy, 1986); both techniques are appropriate for dichotomous responses.

To make inferences about the sustainable behavior of students, the estimated values were analyzed to diagnose whether there was any value that could have been poorly estimated, we identified the items and anchor levels to construct the scale. There are three conditions that must be met for an item to be an anchor on a certain scale level: (1) it must have been answered positively by at least 65% of individuals with this latent trait level, (2) answered positively by a maximum of 50% of individuals with the immediately preceding latent trait level, and (3 the difference between conditions 1 and 2 must be at least 30%" (Andrade et al., 2000).

6. Results and discussion

We present the results for three topics: (1) test dimensionality; (2) estimation of item parameters; (3) construction of the scale and latent trait estimation. The metric used to estimate the parameters of the items and the latent trait was the scale N(0,1), i. e., normal distribution with a mean of zero and variance of one.

Table 1 shows the percentage of positive answers for each item. We can see that item 30 (I seek success) had a higher percentage of positive responses (93.3%).

Table 1. Percent of positive response to the each item

Item 34 had the lowest percentage of positive responses: I buy environmentally friendly products even if they are more expensive than conventional ones (8.3%). Cronbach's alpha, which refers to the internal consistency of an instrument, was 0.84 for the survey.

• 6.1. Dimensionality of the test 

To assess the test's dimensionality, we applied a PCA with a tetrachoric correlation matrix, providing an overview of the relative contribution of the factors to the total variance. The Figure 1 shows the eigenvalues, and the first component was predominant relative to the others.

Figure 1. Principal Component Analysis: eigenvalues versus number of components

The results of the full information factor analysis indicated that the first factor was responsible for explaining 29% of the total variance, making acceptable the supposition of a dominant factor, and confirming the results of the PCA.

• 6.2 Estimation of item parameters

Table 2 shows the estimated values for the discrimination and difficulty parameters of the items, obtained from the outputs of the Bilog software. Appendix A shows the 37 items used to measure the sustainable behavior of students.

Table 2. Estimated values of the parameters (a) and (b) of the 37 items

All items had a discrimination power greater than 0.63. The items with the highest values were, 7 (I feel a sense of personal obligation to take action to prevent environmental problems) and 10 (the quality of the friends I have depends on how nice I am), meaning that they were better at discriminating individuals with different levels of sustainable behavior. The items with the lowest discrimination power were items 3 (When I make plans, I am sure that I will accomplish them) and 18 (I do things my way). In terms of difficulty, item 30 showed low b values (I seek success), meaning that the probability of students with a low level of the latent trait responding positively to these items is high. Items 12 (I am concerned about making plans, because I do not like to rely on luck) and 34 (I buy environmentally friendly products even if they are more expensive than conventional ones) had the highest b values, meaning that the probability of students responding positively to these items is high only for a high level of the latent trait.

Figure 2 shows the ICC and IIC of item 7 (I feel a sense of personal obligation to take action to prevent environmental problems) which has a high discrimination (2.049), characterized by the slope of the curve, demonstrating that it is an easy item (b = -0.393). The IIC shows that the item provides good information for the measurement of the latent trait.

Figure 2. Item Characteristic Curve and Item Information Curve of the item 7

Based on TIF (Figure 3), the questionnaire provided greater information about individuals who had a sustainable behavior level ranging from -3 to 1.5. Therefore, we should add items with the highest degree of difficulty to the questionnaire to more accurately measure the latent trait of individuals with a high level of sustainable behavior. The dotted line shows the standard error. There is a maximum error of approximately 0.4 for this range and of about 0.15 where the peak occurs; the greater the amount of information, the smaller the error. The zero point denotes the mean level of sustainable behavior of the sample.

Figure 3. Test Information Function (TIF)

• 6.3. Construction of the scale and latent trait estimation

We interpreted the scale considering the anchorage criteria specified in the section of methodological procedures, and based on a qualitative analysis by experts.

The greatest advantage of using IRT is that the scale created is easy to interpret; understanding the meaning of reaching a certain level on the scale by associating the items with these levels (Trierweiller et al., 2013). Based on the scale, we can compare the sustainable behavior of students and different populations.

Six anchor levels were defined: -2, -1, 0, 1, 2 and 3. In interpreting the scale, the characteristics depicted in the items were considered to be cumulative: for example, someone who is positioned at the -1 level of the scale comprises aspects of the -1 level as well as aspects of the previous level -2. The levels were interpreted as follows:

1. At the -2 level were students who cared about their safety and their personal success.
2. At the -1 level were students who sought pleasurable emotions and wanted to take care of themselves while showing concern for others; they believed that plants and animals should be respected and tried to buy environmentally friendly products.
3. At level 0, students believed in and demonstrated their capabilities. They planned their lives rather than rely on luck and take action to prevent environmental problems. They have adopted a religious belief; some have done volunteer work and did not need the help of their superiors to get what they wanted. Furthermore, they believe that their own behavior influences the quality of their friendships.
4. At level 1, students also worried about climate change. They have made their own decisions and have done a variety of things in their lives. Moreover, they have not bought products that are produced using child labor, even if they are cheaper.
5. At level 2, students were predominantly influenced by internal determinants related to the protection of their own interests.
6. At the last level, level 3 – in addition to answering "yes" to the questions mentioned in the previous levels – the students get what they want because they struggle to reach their goals, believe in their capabilities, like to have control over other people, and believe that people should be satisfied with what they have. They demonstrated concern for the environment by buying environmentally friendly products even if they are more expensive than conventional products, and they also hold on to cans and only discard them in appropriate places.

Figure 4 shows the frequency of students at each latent trait level. The highest frequencies occur at levels -1 to 1 of sustainable behavior, with 90% of the students found at these levels. Most students are at level 0 (49%). Note that there were no students with scores corresponding to level 3. In general, there were few students who showed a level of sustainable behavior that was very low. Only two students responded negatively to all 37 items.

Figure 4. Distribution of predisposition level to sustainable behavior of students

We notice that at level 1, the students demonstrate greater concern for social and environmental facets of sustainable behavior rather than a predominance of purely economically motivated behavior. They worry about climate change, they do not buy products that are produced using child labor, even if they are cheaper; they buy environmentally friendly products even if they are more expensive than conventional products, and they also only discard cans in appropriate places. However, no student in the sample used to create the scale is located at the highest level (level 3), which requires more sustainable behavior. Similar findings were obtained by Thapa (1999) among undergraduate students using other techniques and questionnaire, which found that students were not very participative in regard to environmentally responsible behaviors except for recycling issues.

Kuo and Jackson (2014) used the NEP to measure engineering students' attitudes and hence the assumption that at engineering universities, the percentage of males is high, pro-environmental attitudes are likely to be poor and may not change. They measured differences in attitudes before and after an environmental course, and concluded based on the results that engineering majors showed poorer endorsement of the NEP in comparison to non-engineering majors, but the authors identified an increase in engineering students' endorsement was statistically significant after taking an environmental course. Nevertheless, these students believe in the domination of humans over the environment and in the human ingenuity, able to overcome resource limits.

• 7. Conclusion

The results of this study showed the viability of creating a sustainable behavior measurement scale for undergraduate students using IRT through a one-dimensional, two-parameter logistic model.

We created a standardized measurement scale with 37 items that allows understand the significance of reaching a certain level of the scale by associating the items with these levels, and allows comparing the predispositions toward sustainable behavior of different students.

The questionnaire items used in the analysis showed good discrimination capability, a range between 0.63 and 2.04, and levels of difficulty covering the entire latent trait continuum (the range between -3.13 and 3.24). However, six items were removed because they were found to impair the reliability of the results obtained based on the IRT that aimed to measure sustainable behavior. We can reformulate these items and include them to the survey in the future.

The statements related to sustainable behavior "I am concerned about planning my life because I do not like to rely on luck" and "I buy environmentally friendly products even if they are more expensive than conventional products" were the least endorsed by students and showed a higher level of difficulty in relation to other items, demonstrating that economic factors are still predominant when purchasing decisions were made by the students we evaluated. On the other hand, the item "I seek success" and "I try to live in a safe place" had a higher percentage of positive responses. The items "my life is determined by my actions" and "the quality of the friends I have depends on how pleasant I am" had the greatest power to discriminate individuals with and without this behavior.

We identified six anchor levels on this scale. The higher levels are associated with more sustainable behaviors. Analysis of the students' positions on the scale facilitates the identification of characteristics that dominate sustainable behavior and thus the points at which it is possible to sensitize this behavior.

The item bank created can be used for the development of short forms or can be administered as a Computerized Adaptive Test (CAT), allowing a more individualized assessment. A limitation of the study was the sample size, which should be increased for future studies, allowing more accurate results.

Acknowledgements

We would like to thank the Coordination for the Improvement of Personnel of Superior Level (CAPES) and National Council for Scientific and Technological Development (CNPq), Brazil, by financial support.

References

Ajzen, I. (2002); Perceived behavioral control, self-efficacy, locus of control, and the theory of planned behavior, Journal of Applied Social Psychology, 32(4), 665-683.

Almeida, F.J.R.; Sobral, F.J.B.A. (2009); O sistema de valores humanos de administradores brasileiros: adaptação da escala PVQ para o estudo de valores no Brasil. Revista de Administração Mackenzie, 10(3), 101-126.

Andrade, D.F.; Tavares, H.R.; Valle, R.C. (2000); Item response theory: concepts and applications. Associação Brasileira de Estatística, Caxambu.

Baker, F.B. (2001); The basics of item response theory. 2 ed. Eric, University of Wisconsin.

Bamberg, S.; Möser, G. (2007); Twenty years after Hines, Hungerford, and Tomera: a new meta-analysis of psycho-social determinants of pro-environmental behavior, Journal of Environmental Psychology, 27, 14–25.

Birnbaum, A. (1968); Some latent trait models and their use in inferring an examinee's ability, contributed chapters, in: Statistical theories of mental test scores reading; Lord, F.M.; Novick, M.R. (ed); Addison Wesley, Reading, MA, 395-479 p.

Bock, R.D.; Gibbons, R.; Muraki, E. (1988); Full-Information item factor analysis, Applied Psychological Measurement, 12(3), 261-280.

Chalmers, P. (2011); Multidimensional item response theory. Package: mirt, Version:0.1-19.

Cordano, M.; Welcomer, S.; Scherer, R.F.; Pradenas, L.; Parada, V. (2011); A cross-cultural assessment of three theories of pro-environmental behavior: a comparison between business students of Chile and the United States, Environment and Behavior, 43(5) 634-657.

Corral-Verdugo, V.; Pinheiro, J. (2004); Aproximaciones al estudio de la conducta sustentable, Medio Ambiente y Comportamiento Humano, 5(1y2), 1-26.

Daneshmandi, A.W.; MacLauchlan, M. (2006); Toward effective evaluation of environmental education: Validity of the children's environmental attitudes and knowledge scale using data from a sample of Irish adolescents, The Journal of Environmental Education, 37(2), 13-23.

Dimopoulos, D.; Paraskevopoulos, S.; Pantis, J.D. (2008); The cognitive and attitudinal effects of a conservation educational module on elementary school students, The Journal of Environmental Education, 39(3), 47-61.

Elkington, J. (1997); Partnerships from cannibals with forks: the triple bottom line of 21st-century business, Capstone Publishing Ltd., Oxford.

Hambleton, R.K.; Swaminathan, H.; Rogers, H.J. (1991); Fundamentals of item response theory: measurement methods for the social science, v.2. Sage Publications, Inc.

Hines, J.M.; Hungerford, H.R.; Tomera, A.N. (1987); Analysis and synthesis of research on responsible environmental behavior: a meta-analysis, The Journal of Environmental Education, 18(2), 1-8.

Hubbard, G. (2009); Measuring organizational performance: beyond the triple bottom line, Business Strategy and the Environment, 18(3), 177-191.

Juárez-Nájera, M.; Rivera-Martínez, J.G.; Hafkamp, W.A. (2010); An explorative socio-psychological model for determining sustainable behavior: pilot study in German and Mexican Universities, Journal of Cleaner Production, 18(7), 686-694.

Kuo, S-Y; Jackson, N.L. (2014); Influence of an Environmental studies course on attitudes of undergraduates at an engineering university, The Journal of Environmental Education, 45(2), 91-104.

Lozano, R.; Lukman, R.; Lozano, F.J.; Huisingh, D.; Lambrechts, W. (2013); Declarations for sustainability in higher education: becoming better leaders, through addressing the university system; Journal of Cleaner Production, 48, 10-19.

Merkel, J.; Litten, L.H. (2007); The sustainability challenge, Advancing Sustainability in Higher Education, 134(Special Issue), 7-26.

Mislevy, R.J. (1986); Recent developments in the factor analysis of categorical variables, Journal of Educational and Behavioral Statistics, 11(1), 3-31.

Mislevy, R.J.; Bock, R.D. (1990); Bilog: item analysis and test scoring with binary logistic models [Computer Program] Scientific Software, Chicago.

Morris, C.G.; Maisto, A.A. (2004); Introdução a Psicologia, Pearson, São Paulo.

Müderrisoğlu, H.; Altanlar, A. (2011); Attitudes and behaviors of undergraduate students toward environmental issues, International Journal of Environmental Science and Technology, 8 (1), 159-168.

Pappas, E.; Pierrakos, O.; Nagel, R. (2013); Using Bloom's Taxonomy to teach sustainability in multiple contexts, Journal of Cleaner Production, 48, 54-64.

Polli, G.M.; Kuhnen, A. (2011); Possibility of use social representation theory to environmental-person studies, Estudos de Psicologia, 16(1), 57-64.

R Core Team (2013); R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. http://www.rproject.org

Revelle, W. (2012); Procedures for psychological, psychometric, and personality research. Package: psych, Version 1.2.3.

Ryu, H.C.; Brody, S.D. (2006); Examining the impacts of a graduate course on sustainable development using ecological footprint analysis, International Journal of Sustainability in Higher Education, 7(2), 158-175.

Schwartz, S.H. (1977); Normative influences on altruism, en: Advances in Experimental Social Psychology (v.10); Berkowitz, L. (ed.). Academic Press, New York, 221-271 p.

Schwartz, S.H. (1992); Universals in the context and structure of values: theoretical advances and empirical tests in 20 countries, in: Advances in Experimental Social Psychology (v.25); Zanna, M. (ed.), Academic Press, Orlando, pp. 1-65.

Schwartz, S.H. (1994); Are there universal aspects in the structure and contents of human values, Journal of Social Issues, 50(4), 19-45.

Schwartz, S.H. (2005); Basic human values: Intercultural context and structure across countries, en: Valores e comportamento nas organizações; Tamayo, A.; Porto, J. (eds.), Vozes, Petrópolis, RJ, 21-55 p.

Tamayo, A. (2012); Validade fatorial da escala Levenson de Locus de controle, Psicologia: Teoria e Pesquisa, 5(1), 111-122.

Thapa, B. (1999); Environmentalism: The relation of environmental attitudes and environmentally responsible behaviors among undergraduate students, Bulletin of Science, Technology & Society, 19(5), 426-438.

Thomas, M.L. (2011); The value of Item Response Theory in clinical assessment: a review. Assessment, 18, 291-307.

Trierweiller, A.C.; Peixe, B.C.S.; Tezza, R.; Bornia, A.C.; Campos, L.M.S. (2013); Measuring environmental management disclosure in industries in Brazil with item response theory, Journal of Cleaner Production, 47, 298–305.

Appendix A.

Items used to measure the sustainable behavior of students (RETURN)

1.Production Engineering Department, Santa Catarina Federal University (UFSC), Brazil. Email: debospenassato@gmail.com

2. Production Engineering Department, Santa Catarina Federal University (UFSC), Brazil. Email: andreatri@gmail.com

3. Production Engineering Department, Santa Catarina Federal University (UFSC), Brazil. Email: cezar.bornia@ufsc.br

4. Administration Department, Santa Catarina Federal University (UFSC), Brazil. Email: biabizzy@gmail.com

5. Administration Department, Santa Catarina Federal University (UFSC), Brazil. Email: erdmann@cse.ufsc.br

6. Production Engineering Department, Santa Catarina Federal University (UFSC), Brazil. Email: lucila.campos@ufsc.br