Teachers’ Language Assessment Literacy

LAL is usually conceptualized as the knowledge, skills, and principles needed to conduct and/or understand various assessment-related activities (Davies, 2008; Fulcher, 2012; Inbar-Lourie, 2013). This broad conceptualization has been used to propose or research LAL across stakeholders (Kremmel & Harding, 2020; Taylor, 2013). However, these three major components differ across stakeholders involved in assessment and represent fluid continua rather than fixed states (Inbar-Lourie, 2013; Pill & Harding, 2013). Stakeholders require different LAL profiles, depending on the contexts where they conduct assessment and the tasks in which they engage.

In the case of language teachers, scholars have emphasized the need to draw and interpret LAL in the teachers’ context and their LAL development (Inbar-Lourie, 2017; Scarino, 2013; Yan et al., 2018; Yan & Fan, 2021). As Scarino (2013) argues, understanding teachers’ assessment lifeworlds is a key condition to foster their professional development in this area of their practice. More recently, in fact, scholars have made the call that teachers’ LAL should not be studied from a deficit perspective, that is, teachers’ lack of assessment literacy (Baker, 2021; Yan, 2021). Rather, language teachers contribute to LAL discussions by bringing their beliefs, experiences, and expertise to the forefront of LAL training (Arias et al., 2012; Baker & Riches, 2018).

Regarding language teachers’ beliefs and practices in assessment, research has suggested two interrelated aspects reflecting LAL development. On the one hand, teachers with training in language assessment tend to have positive perceptions of and usually conduct sound assessment practices (Arias et al., 2012; Baker & Riches, 2018; López Mendoza & Bernal Arandia, 2009). Such practices, as reported in the literature, tend to align mostly with theoretical and practical assessment guidelines, thus reflecting LAL’s knowledge and skills. On the other hand, teachers with no training may perceive assessment negatively and resort to summative-only, traditional practices (López Mendoza & Bernal Arandia, 2009; Sultana, 2019). Finally, while researchers acknowledge that teachers’ beliefs and practices in assessment may be difficult to change, the literature clearly indicates the positive impact of LAL training (Giraldo, 2021; Xu & Brown, 2016).

Against this background, teachers’ LAL includes knowledge, skills, principles, as well as their contextual beliefs and practices, to understand and conduct assessment (Davies, 2008; Fulcher, 2012; Scarino, 2013; Taylor, 2013). Knowledge in LAL refers to language use and language ability frameworks, concepts, and a variety of assessment methods. Skills involve the design of assessments and the interpretation of the data they yield. Finally, principles include ethical and fair uses of assessment. Although drawing the construct of LAL for teachers is still in discussion, when designing LAL training, it needs to be contextualized into concrete activities and practices that reflect these three components. To do so, teachers’ voices and knowledge from the local context are needed to help them identify assessment purposes, design context-specific assessment tasks, and create a communication network among stakeholders.

Operationalizing Teachers’ LAL

For our research and course design, we synthesized authors’ ideas on what teachers’ LAL might entail (Brindley, 2001; Fulcher, 2012; Giraldo, 2021; Kremmel & Harding, 2020; Scarino, 2013; Taylor, 2013). Given its distinct contextual nature, we note that the components below represent empirical and conceptual discussions and cannot be considered a checklist to describe and/or evaluate teachers’ LAL (Csépes, 2021; Xu & Brown, 2016).

• knowledge: language ability frameworks, as operationalized in curricula; purposes (assessment of, for, and as learning), methods, and skills to be assessed; central concepts and their use in practice (e.g., validity, reliability, and authenticity)

• skills: analyzing and adapting existing assessments; designing and evaluating sound assessments; implementing assessments successfully; using assessments to positively impact teaching and learning

• principles: critiquing the uses and misuses of assessments; using assessment ethically and fairly; implementing transparent and democratic assessment practices; analyzing the intended and unintended consequences of assessment

• assessment context: knowledge of learners, beliefs, practices, challenges, needs, and skills; institutional assessment policies and practices

Courses to Foster Teachers’ LAL

Fulcher (2020) has urged that LAL discussions move from description to action—what he calls “a pedagogy of LAL.” Furthermore, as scholars have stated, LAL training must be based on a sound understanding of teachers’ contexts and what they bring to the table (Bøhn & Tsagari, 2021; Scarino, 2013). In our review of published research, we found that successful LAL courses are needs-sensitive (see Arias et al., 2012; Baker & Riches, 2018; Koh et al., 2018; Montee et al., 2013). Thus, teachers’ voices are fundamental to LAL course planning and, eventually, implementation. Additionally, previous research reports that assessment courses for language teachers tend to have these similarities:

• An explicit connection between theory and practice: Teachers study language assessment theories and concepts to analyze or design assessments in context (Arias et al., 2012; Giraldo & Murcia Quintero, 2019; Janssen, 2022; Koh et al., 2018; Levi & Inbar-Lourie, 2020; Walters, 2010).

• A hands-on approach to test analysis and development: Teachers are engaged in analyzing the usefulness of assessment instruments and then design assessments based on technical considerations (Baker & Riches, 2018; Janssen, 2022; Kremmel et al., 2018; Montee et al., 2013).

• A growing reflection on assessment: Although the reported LAL courses are mostly based on knowledge and skills, an outstanding impact on teachers is how they reflect on their prior assessment practices and realize the need to improve the status quo (Arias et al., 2012; Baker & Riches, 2018; Janssen, 2022).

In terms of how training courses operationalize the LAL construct, Giraldo (2021) found that there is a clear allusion to knowledge and skills and secondary attention to principles; in other words, training courses focus on theoretical aspects such as purposes, assessment methods, and concepts such as validity, reliability, and authenticity. However, these courses do not deeply study issues such as fairness and ethics in language teachers’ assessment approach. Giraldo also reports that, in terms of assessment skills, the courses focus on the design of rigorous assessment instruments and their relationship with teaching and learning.

The existing studies reporting on teachers’ evaluations of LAL courses suggest that both contents and activities contribute to their LAL and, most importantly, help them to improve their assessment approach. However, when evaluating the outcomes of LAL courses, previous studies tend to utilize the researcher’s evaluation of teachers’ LAL competencies rather than focusing on how teachers self-assess and perceive their own LAL development (Kremmel et al., 2018; Papadima-Sophocleous, 2022). In the present study, we sought to collect teachers’ feedback (i.e., their voices) on their LAL development as influenced by the OLA course.

Context of the Study

Fulbright Colombia fully sponsored the OLA course thanks to a scholarship called Visiting Colombian Researcher, of which the first author was the recipient. This scholarship allows Colombian researchers to visit a university in the United States and conduct research in various areas. The first author’s proposal involved designing and implementing the OLA course, which was designed for English language teachers in Latin America (for details on the design stage, see Giraldo & Yan, 2023). We used three selection criteria: teachers working at state high schools; teaching the English language; and working in Brazil, Colombia, Ecuador, Peru, or Venezuela.

To participate in the OLA course, we sent a call for applicants to a language testing practice emailing list, social and academic networks, the Latin American Association for Language Testing and Assessment, and the International Research Foundation for English Language Education. Fifty teachers could participate in this free online course; 20 applied and participated in the diagnostic stage. From this group, 19 teachers started the course, and 13 finished.

Participants

Thirteen English language teachers participated in the OLA course: Brazil (n = 3), Colombia (n = 4), Venezuela (n = 1), and Peru (n = 5). They taught English at the high school level in state-funded institutions. The teachers taught general English courses from elementary up to intermediate proficiency levels. The diagnostic stage of our research (i.e., design) indicated that the course should address teachers’ challenges in assessment; include spaces for discussion and interaction with colleagues; target knowledge, skills, and principles in assessment; and discuss current issues in assessment (e.g., bilingual assessment and the assessment of learners with SEN).

Materials, Instruments, and Procedures

Teachers’ Perceptions of the OLA Course

Data in the focus group interview and the questionnaire clearly indicated that the participating teachers evaluated the course and its components positively. The teachers commented positively on how the course was delivered and emphasized the usefulness of the course for their practice in assessment. For example, Teacher7 comments on how the course has helped them to view assessment differently and align it with professional practice:

Interviewer: So, our first question is, how do you feel about the course so far?

Teacher7: I think the course has been very helpful because we came from seeing assessing as maybe a tool to get grades, and just maybe give reports to parents, to a tool to make students more aware of learning so it’s not just the process, not just the final step of giving students like a test or, well, evaluating them, but just starting from very basic things, like from setting the goals for the course, making them aware of the goals they will achieve, and how their performance will be evaluated, like all the things that evaluation has, so it’s been very helpful.

Table 1 shows teachers’ general impressions of the course, materials, activities, and the platform through which the course was delivered. The large means and small ranges reiterate the teachers’ positive evaluation of these general aspects. The slightly larger range of 2 in Item 3 may reflect teachers’ opinions on statistics in the course, to which we will return later.

Teachers’ Perceptions of Contents in the OLA Course

All teachers considered the topics in the course to be useful for their assessment practice in their institutions. Particularly, they emphasized how the course contents were organized and sequenced. As they explained, this organization was useful for them in learning about language assessment. Teacher17 commented on the topic sequence and how this was something she expected to have in the course:

Interviewer: What do you think about the contents of the course?

Teacher17: Well, I really like the sequence of the topics because they are related and connected, and all of them are really good for me because that’s the thing that I was looking for at the beginning of the course. So, everything is new.

On the other hand, teachers’ answers about preferred contents in the courses varied; however, answers in the interview and the questionnaire conclusively showed teachers’ interest in two content topics: assessing learners with SEN and integrated-skills assessment. For instance, Teacher19 stated that she had never addressed the topic of assessing learners with SEN but came to value them as she was engaged in an activity that explored this content:

Another topic is assessing learners with special educational needs. I never have idea [sic] how to, or what kind of strategies, or tech, or methods I can use to assess this kind of students, for example. The first author brought two teachers from Colombia who shared with us their experiences, so it led me to value these kinds of students.

In an open item from the questionnaire, Teacher12 states what topic she found most useful and connects it to her assessment practice: “The most useful course content is assessing integrated skills because the learners learn and practice with their real life, motivating [sic] and the washback is more effective.”

Table 2 shows the results of asking teachers about the usefulness of contents in the OLA course. As demonstrated by the large means and narrow ranges, teachers found most topics extremely useful for assessment in their teaching contexts.

Items 16, 17, and 20 have a slightly wider range. Item 16 (statistics for classroom assessment) may have gotten the smallest mean because, as teachers explained, this content was not so relevant to their current assessment practices. The questionnaire asked teachers what content they found least useful, to which Teacher 5 answered, “Statistics for classroom assessment because it is not commonly used on a daily basis.” In addition, some teachers stated that bilingual assessment was not so useful for the same reason Teacher5 expressed regarding statistics. On the other hand, there is no evidence in the data to describe or explain the range of 2 in Item 20: Ethics and fairness in classroom language assessment.

Teachers’ Perceptions of Activities in the OLA Course

Like their perceptions about course contents, all teachers in this study expressed that they found its activities useful, specifically for learning about language assessment. The teachers commented on how the activities in the synchronous workshops of the course fostered interaction among themselves, which in turn led them to learn from each other. The interactive and collaborative nature of the course, as the excerpt below suggests, had a positive impact on teachers’ LAL.

Teacher10: The most important part of this group is that we are from different parts of Latin America, and to have this opinion of evaluation, assessment, and how they teach in their countries, it was really awesome for me because I can have all the things that they can do in order to fulfill this topic, this assessment.

For fine-grained feedback to evaluate course activities, the questionnaire asked teachers about their usefulness. Table 3 shows the descriptive statistics for 12 major activities the teachers conducted in the course. All means were large and ranges narrow, except for Item 33 (Calculating and interpreting basic statistics for classroom language assessment), with a mean of 4 and a range of 3. This feedback confirms that many of these teachers did not seem to find this activity useful in the course.

Table 3 also shows that all 13 teachers found Activity 29 (Analyzing assessment instruments) extremely useful for learning about language assessment. When asked about activities they found most and least useful in the open-ended items of the questionnaire, we saw the same trend we just described. In the open items, the teachers could explain why they had these perceptions about the two activities.

Teacher3 (most useful activity): The most useful activity for learning is analyzing instruments because it is very important to know why: diagnostic, progress, and achievement and what is the method [sic] according to the skills: listening, writing, etc.

Teacher9 (least useful activity): Calculating and interpreting basic statistics because my country’s curriculum is based on the formative assessment [sic] and most of the time, we use symbols and letters to give a learning process result.

Another aspect that became apparent in the questionnaire was the perceptions teachers had about discussions. As mentioned above, teachers found interaction with others useful to learn about each other’s teaching and assessment contexts. This is confirmed in the large mean and low range in Item 26 in Table 3. However, Item 27 had a smaller mean and slightly wider range, which suggests teachers had differing views on small-group discussions. A closer look at the data in the open items of the questionnaire showed that four teachers found small-group discussions to be the least useful. As Teacher11 states, “Working with colleagues in small groups did not promote a lot of interaction, in my opinion. Teachers were quiet most of the time and only one or two engaged.” Based on these data, it seems like the teachers in the OAC were more willing to participate in whole-group discussions than small-group ones.

Teachers’ Perceptions of the Impact of the OLA Course on Their Professional Development

All the teachers commented that the course positively impacted their professional development regarding their LAL. The teachers explored how the course taught them about language assessment, impacted their assessment practice, and raised their awareness of what language assessment implies. Teacher18 commented on how she can better assess communicative competence and what kind of assessments she can use to do so.

Interviewer: How do you feel about the course so far?

Teacher18: I improve [sic] my knowledge related to how to assess the communicative competence of my students. It means that construct for [sic] language skills and what kind of instrument we can use to know the learning progress of our students. Another thing that I improve to recognize or identify [sic] in order to differentiate, for example, how can we evaluate the task of students, and what kind of method we can use.

Although the positive impact of a course is expected in such an educational scenario, we want to emphasize two areas that were apparent across the data. On the one hand, the teachers commented that the course ignited serious reflection about their past language assessment practices. In particular, the teachers reflected on mistakes they made in assessment and now need to improve. Teacher20 commented on how her professionalism in assessment evolved thanks to the OLA course:

I realized that I was applying some methods, some strategies, some techniques, and some instruments, also, but it was not in a professional way. It was empiric or just part of my teaching practice. Now that I’m going to be more professional with my teaching practices also.

On the other hand, the teachers explained how, thanks to the course and their reflections, they felt the need to improve assessment practices to better serve their students. The answers in the interview suggest that teachers see professional assessment practices as fair. For instance, Teacher15 stated how she was not fair in assessment but hopes to change her practices so they are fair in future assessment experiences:

It somehow makes me a better teacher. I think it will help me be more fair to my students. I wasn’t quite fair before with some of the things I was asking from them. They were not very clear, maybe, or something like that. So, that is something that I will improve from now on, I hope.

The data in Table 4 reinforces the finding that the OLA course positively impacted teachers’ professional development. The data suggest that the teachers strongly agreed with the positive outcomes of the course, as shown in Item 38. Further, the impact of the course reflected LAL’s three overall components: knowledge (Item 40), skills (Items 41 and 42), and principles (Item 43). Item 44 shows how all 13 teachers strongly agreed that the course led them to reflect on areas for improvement in their LAL, which is evident and aligns with the interview sample above.

Suggestions for Improving the OLA Course

Notwithstanding the overall positive evaluation that teachers had of the course, in the data, we noticed an area that the course could have improved. Teacher15 and Teacher20 commented that it was challenging for them to move from one technological tool to another (e.g., from Google Meet to Zoom). As Teacher20 comments, this was exacerbated by their slow internet connection: “I would like to have the activities in the same platform, to be honest, because we just [used] Google Meet, and we had to go different links. It was more difficult for me because of my Internet connection.” Thus, we believe the use of technological apps, especially for conferencing, must be considered carefully when offering online assessment courses.