Rev Andal Med Deporte. 2021; 14(3): 171-175


Revista Andaluza de

Medicina del Deporte

 

https://ws208.juntadeandalucia.es/ojs

 

Original

Construct validity of Rosenberg Self-esteem Scale in Brazilian Athletic Context

W. R. da Silvaa*, T. E. Medeirosa, A. Duarte Pescab, E. P. Vieiraa, F. Luiz Cardosoa

a Centro de Ciências da Saúde e do Esporte. Universidade do Estado de Santa Catarina. Brasil.
b Departamento de Psicologia. Faculdade Complexo de Ensino Superior de Santa Catarina. Brasil.


ARTICLE INFORMATION: Received 4 november 2019, accepted 19 december 2019, online 9 march 2020

 

 

ABSTRACT

Objective: The purpose of this study was to evaluate the psychometric properties of the Rosenberg Self-esteem Scale in the athletic context of Brazil.

Method: In total, 387 athletes participated in the study, 232 men and 155 women, with a mean age of 22.1+/-4.5 years, practitioners of team modalities in clubs in Southern Brazil. The construct validity was evaluated through exploratory Factorial Analyses with Oblimin Rotation and the factorial weight 0.3 was used to exclude items. Cronbach’s Alpha and Polychoric Correlation evaluated the internal variance consistency.

Results: The Rosenberg Self-esteem Scale uploaded on 2 factors showed 61% of the variance of the construct, with factor 1 adding items about positive self-esteem and factor 2 about negative self-esteem. To confirm the internal consistency of the instrument, we performed polychoric correlation between the items on the scale. All items showed significant positive correlation above 0.3 (p>0.05) confirming the good internal consistency of the questionnaire.

Conclusion: This research identifies good psychometric properties of the Rosenberg Self-esteem Scale in the Brazilian sport context of athletes of team sports. The bifactorial structure was verified, agreeing with the original proposal, suggesting the separate score calculation of each factor on the self-esteem Scale.

Keywords: Psychometrics; Validation studies; Sport psychology; Sports; Brazil.

 

Validez del constructo de la Escala de Autoestima de Rosenberg para el contexto deportivo brasileño

RESUMEN

Objetivo: El propósito de este estudio fue evaluar las propiedades psicométricas de la Escala de Autoestima de Rosenberg en el contexto deportivo de Brasil.

Método: En total, 387 atletas participaron en el estudio, 232 hombres y 155 mujeres, con una edad media de 22.1+/-4.5 años, practicantes de modalidades de equipo en clubes del sur de Brasil. La validez de constructo se evaluó mediante análisis factoriales exploratorios con Rotación Oblimin y se utilizó el peso factorial 0.3 para excluir los ítems. La correlación alfa y policórica de Cronbach evaluó la consistencia de la varianza interna.

Resultados: La Escala de Autoestima de Rosenberg expresada en 2 factores mostró el 61% de la varianza del Constructo, con el factor 1 agregando ítems sobre autoestima positiva y el factor 2 sobre autoestima negativa. Para confirmar la consistencia interna del instrumento, realizamos una correlación policórica entre los ítems de la escala. Todos los ítems mostraron una correlación positiva significativa por encima de 0.3 (p> 0.05) confirmando la buena consistencia interna del cuestionario.

Conclusión: Esta investigación identifica buenas propiedades psicométricas de la Escala de Autoestima de Rosenberg en el contexto deportivo brasileño de atletas de deportes de equipo. Se verificó la estructura bifactorial, de acuerdo con la propuesta original, sugiriendo el cálculo de puntaje por separado de cada factor en la Escala de autoestima.

Palabras clave: Psicometría; Estudios validación; Psicología deporte; Deportes; Brasil.

 

Validade de Construto da Escala de Autoestima de Rosenberg para o Contexto Esportivo Brasileiro

RESUMO

Objetivo: O objetivo deste estudo foi avaliar as propriedades psicométricas da Escala de Autoestima de Rosenberg no contexto esportivo do Brasil.

Método: No total, 387 atletas participaram do estudo, sendo 232 homens e 155 mulheres, com média de idade de 22,1 (DP= 4,5) anos, praticantes de modalidades de equipes em clubes do sul do Brasil. A validade de construto foi avaliada por meio de Análises fatoriais exploratórias com Rotação Oblimin e o peso fatorial 0.3 foi utilizado para excluir itens. O Alfa de Cronbach e a Correlação Policórica avaliaram a consistência da variância interna.

Resultados: A Escala de Autoestima de Rosenberg carregou em 2 fatores e mostrou 61% da variância do construto, com o fator 1 acrescentando itens sobre autoestima positiva e o fator 2 sobre autoestima negativa. Para confirmar a consistência interna do instrumento, foi realizada correlação policórica entre os itens da escala. Todos os itens apresentaram correlação positiva significativa acima de 0.3 (p> 0.05) confirmando a boa consistência interna do questionário.

Conclusão: Esta pesquisa identifica boas propriedades psicométricas da Escala de Autoestima de Rosenberg no contexto esportivo brasileiro de atletas de esportes coletivos. A estrutura bifatorial foi verificada, concordando com a proposta original, sugerindo o cálculo do escore separado de cada fator na escala de autoestima.

Palavras-chave: Psicometria; Estudo Validação; Psicologia esporte; Esportes; Brasil.

 

Introduction

 

Self-esteem is personal feelings and thoughts regarding self-values, competence, and adequacy, whether related to positive or negative attitudes about oneself.1 Hossler et al.2 shows that the central point about self-esteem is the volatility aspect, which influences and shapes how the subject chooses their goals, for example, accepts themselves, cares about others, and projects their expectations about their future.3 Self-esteem has a relation with psychological constructs such as well-being4 and self-concept5 and has been most widely researched in the personality field.6

Some research has observed self-esteem as a trait and state7 while others suggest that the development of this attribute presents discontinuity and is not one stable course in the vital cycle.8 High self-esteem is considered one of the major predictors of favorable results in adolescence and adulthood, having implications in areas such as occupational success, interpersonal relationships, and academic performance.9 On the other hand, the low self-esteem is observed in problems such as aggression, antisocial behavior, and juvenile misdemeanors.10 Due to its relationship with psychosocial adjustment, self-esteem has been observed as an important detector of mental health and a relevant factor in the analysis of growth and progress in developed countries.11 Additionally, the evaluation of this characteristic has been considered an important tool in the identification and prevention of psychological problems.12

Self-esteem can be measured using the Rosenberg self-esteem scale (RSES),1 which is a unilateral instrument that can classify the level of self-esteem as low, average, and high. Low self-esteem can be translated as a feeling of incompetence, inadequacy, and incapacity to fight against challenges. Average self-esteem is characterized by the fluctuation in the subject between the assent feeling and rejection of self; and high self-esteem is self-judgment of value, belief, and competence.

The original scale was developed for teenagers and contains ten closed sentences, five about "self-image" or positive "self-value" and five about "negative self-image" or "self-depreciation". The questions are displayed in a Likert format with four points, varying between "I totally agree" and "I totally disagree". The RSES has been translated into 28 languages and its application in 53 countries has raised questions about its unidimensional structure.6 Several studies performed factorial analysis on the scale items and developed a bidimensional construct, which shows positive and negative images that the subject has about themselves.13,14

However, there is also evidence confirming the 1-factor model of the RSES, raising a controversial question. Regarding the theory approach, the self-esteem dimensionality has also been challenged. Authors2,7 sustained the multidimensional character, in addition to one problematic decomposition of the construct into different factors, as the authors regard it important to evaluate this characteristic in different fields (social, education, family, and others), as the subjects can have a positive perception about them self in some situations and not in others. Despite the existing controversy about the self-esteem concept and its factorial structure, the RSES has shown a good index of internal consistency in international scientific research, the majority of studies realized with teenagers and adults.15 In Brazil, however, the self-esteem construct has been little researched.13 With respect to the factorial structure of the RSES, the studies performed in Brazil found the same problems as studies in other countries, such as the unidimensionality of the scale16,17 and the bifactorial structure.13,18

Be noteworthy the sports scenario, no tool was found with validity to evaluate in this context. However, some studies have already measured this construct without a validity measure for the sports context.19 Therefore, the validation of a widely used tool, such as the RSES, for athletes, would help in the selection and monitoring of the development of these athletes in Brazil, and aid in the advancement of the sports psychology field. Thus, this study aimed to evaluate the psychometric properties of the RSES in the sports context in Brazil.

 

Method

 

This study is an exploratory descriptive research and the protocol was approved by the Ethics Committee on Human Research in the Santa Catarina State University by International and National Guidelines, protocol number: 275.381/2013. All subjects gave written informed consent in accordance with the Declaration of Helsinki.

 

Participants

 

In total, 387 athletes participated in the study, 232 men, and 155 women, with a mean age of 22.1 +/-=4.5 years, practitioners in team modalities (soccer, indoor soccer, volleyball, handball, basketball, athletics, judo, and karate) in clubs in South Brazil. The participants were chosen intentionally according to the criteria: being aged 16 years or more, registered in the national federation for a minimum of 1 year, have been training in a systematic form for 1 year, 3 times per week, and be training during data collection. The data were collected in 2016.

Initially, official contact was carried out with clubs, associations, and sports secretaries that work with sports development in Santa Catarina and Paraná States. The researcher visited the institutions that agreed to participate in the study to clarify questions about the research, and demonstrate the aim, relevance, and data collection procedures, as well as guarantee total confidentiality of data. The athletes who agreed to participate in the study were provided with the agreement term to be signed by them or their parents or guardian, when the athlete was under 18 years of age. The data collection took place at the training club.

 

Procedures

 

The researcher explained the questionnaires before application and the questionnaires were answered singly and in private. The collection performed with the college students followed the same principles and procedures, however, the questionnaires were answered in a college classroom.

The adaptation of the RSES to Portuguese was utilized in this study is based on a study by Hutz.16 It contains 10 items, with 5 positive visions about oneself and 5 self-depreciating visions. The answer scale is a Likert model with variations between 1 and 5 points (1= strongly disagree; 2= disagree; 3=undecided; 4=agree; 5=strongly disagree). Authors have utilized different formats of the RSES with low or high numbers of items and variable answer options between three and six points on a Likert scale, in accordance with the aims and populations of studies. Although the RSES showed good internal consistency in the original study (α=0.92).1 In Brazilian research, the Cronbach’s Alpha varied between 0.6813 and 0.86.16,17 Concerning points on the scale, a high score indicates better self-esteem.

All sentences translated and used in this study are cited here respectively (see Figure 1):

 

English version

01 I feel that I am a person of value, at least as much as other people.

02 I think I have several good qualities.

03 Taking everything into account, I think I'm a failure.

04 I think I can do things as well as most people do.

05 I think I do not have much to be proud of.

06 I have a positive attitude toward myself.

07 All in all, I'm pleased with myself.

08 I wish I could have more respect for myself.

09 Sometimes I feel useless.

10 Sometimes I feel useless.

Portuguese version

01 Sinto que sou uma pessoa de valor, no mínimo, tanto quanto as outras pessoas.

02 Eu acho que eu tenho várias boas qualidades.

03 Levando tudo em conta, eu penso que sou um fracasso.

04 Eu acho que sou capaz de fazer coisas tão bem quanto à maioria das pessoas.

05 Eu acho que eu não tenho muito que me orgulhar.

06 Eu tenho uma atitude positiva com relação a mim mesmo.

07 No conjunto, eu estou satisfeito comigo.

08 Eu gostaria de poder ter mais respeito por mim mesmo.

09 Às vezes eu me sinto inútil.

10 Às vezes eu acho que não presto para nada.

Figure 1. Original and Translated Version of Instrument.

 

 

Data Analyses

 

Descriptive statistics were used to analyze the average, standard deviation, and frequency. The construct validity was evaluated through exploratory factorial analyses with Oblimin Rotation and used a factorial weight of 0.3 to exclude items. The KMO test and Bartlett´s sphericity were used to analyze factorial adequacy. The Scree plot graphic was utilized to confirm the number of factors on the scale. Cronbach’s Alpha and the Polychoric Correlation evaluated internal consistency. The data were analyzed using Microsoft Excel® and STATA®, version 13.1.

The following Confirmatory Factor Analysis parameters were considered: Chi-Square (χ2); Comparative Fit Index (CFI); Tucker Lewis Index (TLI); and Root Mean Square Error of Approximation (RMSEA). The references values for Chi-Square were no p-value significance. The value 0.95 was adopted as a minimum to deduce model adjustment, in accordance with the CFI and TLI indices, while for the RMSEA, values of 0 to 0.08 were utilized as an acceptable adjustment indicative.

 

Results

 

The RSES uploaded on 2 factors showed 61% of the variance of the construct, with factor 1 items being about positive self-esteem and factor 2 about negative self-esteem. The indices of KMO (0.95) and Bartlett sphericity (x2(300) =15637.05; p≤0.001) these are some indices that indicate the suitability of the items for the use of the Exploratory Factor Analysis. The Table 1 too presents a total of Cronbach’s alpha values for the RSE of each factor (positive and negative). Each item was removed from the scale to verify changes in values in the total Cronbach’s Alpha. However, no differences were verified, confirming the necessity of 10 items in the instrument.

 

Table 1. Exploratory Factorial Analysis and Total of Cronbach’s alpha.

Item

Factor 1

Factor 2

01 I feel that I am a person of value, at least as much as other people.

0.570

 

02 I think I have several good qualities.

0.593

 

03 Taking everything into account, I think I'm a failure.

 

0.781

04 I think I can do things as well as most people do.

0.614

 

05 I think I do not have much to be proud of.

 

0.635

06 I have a positive attitude toward myself.

0.786

 

07 All in all, I'm pleased with myself.

0.630

 

08 I wish I could have more respect for myself.

 

0.676

09 Sometimes I feel useless.

 

0.627

10 Sometimes I feel useless.

 

0.630

Total variance explain

61%

Positive aspects of self esteem

0.835

 

Negative aspects of self esteem

 

0.866

Total of Cronbach’s Alpha

0.886

 

On Table 2 show the polychoric correlation between 10 items of the RSES.

 

Table 2. Polychoric Correlation between 10 items of the Rosenberg Self-Esteem Scale.

Item

1

2

3

4

5

6

7

8

9

1

1.00

 

 

 

 

 

 

 

 

2

0.65

1.00

 

 

 

 

 

 

 

3

0.66

0.75

1.00

 

 

 

 

 

 

4

0.71

0.60

0.81

1.00

 

 

 

 

 

5

0.53

0.54

0.65

0.70

1.00

 

 

 

 

6

0.70

0.87

0.66

0.59

0.52

1.00

 

 

 

7

0.79

0.65

0.72

0.74

0.63

0.80

1.00

 

 

8

0.44

0.52

0.58

0.58

0.52

0.49

0.59

1.00

 

9

0.70

0.67

0.75

0.73

0.61

0.75

0.90

0.58

1.00

 

The confirmatory factorial analysis agrees with the bifactorial model. It was verified that the adjusted value of χ2 was not significant, p>0.05, suggesting a strong model adjustment. The values were appropriate to the adjustment index: TLI (0.96), CFI (0.98), RMSEA index (0.036), and SRMR index (0.012), confirming the bidimensionality of the scale. Figure 2 typifies the graphic of the RSES adapted to the Brazilian sports context.

 

Figure 2. Confirmatory factorial analysis of Rosenberg Self-Esteem Scale.

 

 

Discussion

 

This study aimed to verify the psychometric property of the Rosenberg Self-Esteem Scale scale in the Brazilian sports context. Concerning the construct validity, the exploratory factorial analysis shows a bidimensional structure (positive and negative). The magnitude of factorial weights and the variability of factors explain the good index of validity shown for the RSES.

The confirmatory factorial analysis showed the bidimensional model, with great values for the adjustment index, demonstrating agreement with empirical data for the theoretical model used. Thus, the bifactorial model was considered appropriate. Concerning reliability, the RSES showed good internal consistency. The results of Cronbach’s Alpha were like other studies in Chile and Brazil,13,14 and below values found in other countries.6

Furthermore, as the internal consistency highlights, the values obtained in the item correlations demonstrated that the construct index contributes to the description of latency of the construct (global self-esteem).

In general, the evidence on the dimensionality of the RSES has been polemic because the identification of two factors has occurred due to the method. Marsh20 argues that the cluster questions on each factor result in a tendency by the respondent to agree with the positive affirmations and disagree with the negative ones. The author emphasizes that the effect of the method occurs mostly in young people as answering the negative affirmation requires a higher level of cognitive complexity than responding to positive items. On this study with seven items of the RSES (four positive and three negative), the author showed that students with low verbal ability were susceptible to responding to negative items in a manner that was inconsistent with their answer to positive items.

The method effect has received attention not only from research that supports two factors for the RSES, but also from those who support the unifactorial structure. Thus, for some researchers, the scale is unidimensional, but includes the method effect of negative items,6 while other researchers include these effects in positive items.5 In addition, it has been sustained that the best construct adjustment is obtained when considering the method effect on positive and negative items.21,22 On the other hand, authors support that two correlational factors lead to superior construct adjustment.18

The present study increases the bidimensional perspective of the RSES, providing some evidence about one scale with two dimensions, positive and negative.13,14 Thus, the analysis of the total score of the RSES as well as partial scores for each factor could be indicated. On the other hand, it is important to mention that much evidence exists to support the unifactorial structure of the scale.23 The verify of the organization of the items in their respective factors provides both a theoretical and empirical basis for understanding the breadth and structure of the evaluation of this construct.

Authors have emphasized that the contradiction between the results occurs because the questionnaire can be unidimensional in some populations and bidimensional in others.20,24 Schmitt and Allik6 verify that some cultures demonstrate greater bias for negative items on the RSES than others. For some authors this bias could be responsible for the occurrence of two factors in the self-esteem construct in some contexts.

In this perspective, authors25 argue that items of the scale could be reformulated in the same direction, assessing either "positive self-esteem" or "negative self-esteem". The authors compared three kinds of the RSES (the original, one type with all items written in positive form, and one type with all items written in negative form) and evaluated the construct validity with a tool that evaluated depression and refusal (self-deception). The original scale was the only one to present a two-factor structure and, moreover, presented greater overlap with the constructs of depression and negation than the other two versions. For the authors25 it is necessary to reformulate the RSES with the aim of the tool acquiring one unidimensional structure according to Rosenberg. In this way, it is emphasized the difficulty of combined positive and negative items on the same scale.

This research identifies good psychometric properties of the RSES in the Brazilian sports context for athletes of team sports. The bifactorial structure was verified, agreeing with the original proposal, suggesting a separate score calculation for each factor of self-esteem. While it is necessary to adapt self-esteem measures to each context, utilization of modified types make comparisons difficult between the results and this depends on the factorial structure in each context.

Thus, even though many athletes composed the sample, the results cannot be generalized to all populations of Brazilian athletes. Finally, the characteristics of the sample may have accentuated the method effect on items of the RSES. Furthermore, contentious questions exist about the method effect concerning the RSES, and it is believed that this stimulated the production of new research with a sample from specific sports. Studies verifying the construct validity of the scale with other psychological verification instruments, correlating this with positive and negative factors of self-esteem contributed to this discussion. The realization of longitudinal studies is suggested to verify the permanence of self-esteem. Studies with clinical samples are also recommended, since little is known about the efficacy of the RSES in the clinical environment. Also suggest researches that verify the importance of self-esteem to sport performance and with other psychological constructs on the same context.

 


Authotship. All the authors have intellectually contributed to the development of the study, assume responsibility for its content and also agree with the definitive version of the article. Conflicts of interest. The authors have no conflicts of interest to declare. Funding. CAPES (Coordenação de aperfeiçoamento de pessoal de nivel superior) Provenance and peer review. Not commissioned; externally peer reviewed. Ethical Responsabilities. Protection of individuals and animals: The authors declare that the conducted procedures met the ethical standards of the responsible committee on human experimentation of the World Medical Association and the Declaration of Helsinki . Confidentiality: The authors are responsible for following the protocols established by their respective healthcare centers for accessing data from medical records for performing this type of publication in order to conduct research/dissemination for the community. Privacy: The authors declare no patient data appear in this article.


 

 

References

  1. Rosenberg M. Rosenberg self-esteem scale (SES). Soc. Adolesc. self-image. 1965.
  2. Hossler D, Braxton J, Coopersmith G. Understanding student college choice. Higher education: Handbook of theory and research. 1989;5:231-88.
  3. Bednar RL, Peterson SR. editores. Self-esteem: Paradoxes and innovations in clinical theory and practice. American Psychological Association (APA). 1995. Washington, DC, US. 77-93.
  4. Sánchez EM, de Roda Barrón AL. Social psychology of mental health: the social structure and personality perspective. Span J Psychol. 2003;6:3-11.
  5. Martín-Albo J, Núñez JL, Navarro, JG, Grijalvo F. The Rosenberg Self-Esteem Scale: translation and validation in university students. Span. J. Psychol. 2007;10:458-67.
  6. Schmitt DP, Allik J. Simultaneous administration of the Rosenberg Self-Esteem Scale in 53 nations: exploring the universal and culture-specific features of global self-esteem. J Pers Soc Psychol. 2005;89:623-42.
  7. Harter S, Whitesell NR. Beyond the debate: Why some adolescents report stable self worth over time and situation, whereas others report changes in self worth. J Pers. 2003;71:1027-58.
  8. Cole DA, Maxwell SE, Martin JM, Peeke LG, Seroczynski AD, Tram JM, et al. The development of multiple domains of child and adolescent self concept: A cohort sequential longitudinal design. Child Dev. 2001;72:1723-46.
  9. Trzesniewski KH, Donnellan MB, Robins RW. Stability of self-esteem across the life span. J Pers Soc Psychol. 2003;84:205-20.
  10. Donnellan MB, Trzesniewski, KH, Robins RW, Moffitt TE, Caspi A. Low self-esteem is related to aggression, antisocial behavior, and delinquency. Psychol Sci. 2005;16:328-35.
  11. Mruk CJ. editor. Self-esteem and positive psychology: Research, theory, and practice. New York: Springer Publishing Company. 2013. p 169-170.
  12. Fumin F, Jiyuan F. Self-concept and Mental Health of College Students. Chinese Ment Healt J. 2001;15:76-77.
  13. Avanci JQ, Assis SG, dos Santos NC, Oliveira RVC. Cross-cultural adaptation of self-esteem scale for adolescents. Psicol Reflex Crít. 2007;20:397-405.
  14. Rojas-Barahona CA, Zegers B, Förster CE. La escala de autoestima de Rosenberg: Validación para Chile en una muestra de jóvenes adultos, adultos y adultos mayores. Rev Med Chil. 2009;137:791-800.
  15. Aluja A, Rolland JP, García LF, Rossier J. Dimensionality of the Rosenberg Self-Esteem Scale and its relationships with the three-and the five-factor personality models. J Pers Assess. 2007;88:246-9.
  16. Hutz CS. Adaptação brasileira da escala de auto-estima de Rosenberg. Curso de Pós-Graduação em Psicologia do Desenvolvimento, Universidade Federal do Rio Grande do Sul, Porto Alegre. RS. 2000. Monografia.
  17. Santos PJ, Maia J. Análise factorial confirmatória e validação preliminar de uma versão portuguesa da escala de auto-estima de Rosenberg. Psicol Teor Investig Prát. 2003;2:253-68.
  18. Romano A, Negreiros J, Martins T. Contributos para a validação da escala de auto-estima de Rosenberg numa amostra de adolescentes da região interior norte do país. Psicol Saúde Doenças. 2007;8:109-16.
  19. Laborde S, Guillén F, Mosley E. Positive personality-trait-like individual differences in athletes from individual- and team sports and in non-athletes. Psychol Sport Exerc. 2016;26:9-13.
  20. Marsh HW. Positive and negative global self-esteem: A substantively meaningful distinction or artifactors? J Pers Soc Psychol. 1996;70:810-9.
  21. Quilty LC, Oakman JM, Risko E. Correlates of the Rosenberg Self-Esteem Scale Method Effects. Struct Equ Model. 2006;13:99-117.
  22. Tomas JM, Oliver A. Rosenberg’s self esteem scale: Two factors or method effects. Struct Equ Model. 1999;6:84–98.
  23. Sinclair SJ, Blais MA, Gansler DA, Sandberg E, Bistis K, LoCicero A. Psychometric properties of the Rosenberg Self-Esteem Scale: Overall and across demographic groups living within the United States. Eval Health Prof. 2010;33:56-80.
  24. Alessandri G, Vecchione M, Eisenberg N, Łaguna M. On the factor structure of the Rosenberg General Self-Esteem Scale. Psychological Assessment. 2015;27:621-35.
  25. Greenberger E, Chen C, Dmitrieva J, Farruggia SP. Item-wording and the dimensionality of the Rosenberg Self-Esteem Scale: do they matter? Pers Individ Dif. 2003;35:1241-54.

 


Corresponding author.

E-mail-address: walanrobert@hotmail.com (W. R. da Silva).

 

https://doi.org/10.33155/j.ramd.2019.12.008

© 2021 Consejerı́a de Educación y Deporte de la Junta de Andalucı́a. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)