Statistical Literacy

Statistical literacy refers to an individual’s ability to understand and critically evaluate statistical information and results encountered in various contexts (Gal, 2004; Gönülal, 2019). Statistical literacy involves various skills, including critical thinking; the selection of statistical methods; and data management, analysis, interpretation, and accurate reporting (Gönülal, 2018, 2019; Loewen et al., 2014, 2020; Plonsky, 2021). To inform the design of this study’s statistics course, several components of statistical literacy were reviewed, which are described in the following subsections.

Previous Research on Applied Linguistics Researchers’ Statistical Literacy

Previous research syntheses have identified the prevalence of quantitative research in applied linguistics journals, with classical statistical methods such as t-tests, correlations, chi-square tests, ANOVA, and MANOVA being commonly employed (Khany & Tazik, 2019). While parametric tests dominated, non-parametric equivalents were less frequent. More recent reviews also confirmed the prevalence of quantitative research (Farsani et al., 2021) and basic statistical techniques (Khany & Tazik, 2019) in research articles published in prominent applied linguistics journals.

Survey-based studies revealed a gap in statistical literacy among applied linguists. Loewen et al. (2014) surveyed 331 applied linguistics doctoral students and professors from North America, Australia, Europe, and Asia and found positive attitudes toward quantitative research. However, only 14% of students and 30% of professors felt adequately trained in statistics. Gönülal et al. (2017) investigated the development of statistical literacy among 50 graduate students, noting substantial improvements in basic descriptive and common inferential statistics but weaker gains in advanced statistics. Later, Gönülal (2018) explored predictors of statistical literacy among 120 SLA doctoral students in North America, emphasising the significance of taking more statistics courses, having a quantitative research orientation, and engaging in self-training.

Gönülal’s (2019) subsequent investigation with 120 SLA doctoral students in North America identified challenges in interpreting common and advanced statistics, suggesting improvements in course content and format. Loewen et al. (2020) assessed the statistical knowledge of 198 applied linguistics and SLA researchers from North America and Europe, revealing a gap in advanced statistics and emphasising the positive influence on performance of quantitative research orientation and the number of statistics courses taken. More recently, Zhang and Han’s (2023) survey of 211 applied linguistics master’s students in China highlighted a deficiency in overall statistical literacy despite positive attitudes toward statistics. The number of statistics courses taken, a quantitative research orientation, and self-training were identified as crucial predictors of statistical literacy, attitudes, and self-efficacy.

The findings from research syntheses and survey-based studies emphasise the significance of statistical literacy in applied linguistics research. While quantitative research is prevalent, there is a notable gap in statistical knowledge, emphasising the need for targeted training and improvements in course content. Direct evidence of participants’ ability to perform statistical analyses is needed to understand the effectiveness of statistical training. The present study addresses this gap by incorporating performance and perception data, providing insights into the effectiveness and challenges of statistics training.

Context and Participants

This study was conducted within an applied linguistics doctoral programme on English language teaching education at Universidad del Valle, Colombia. The pilot course stemmed from the need to develop the PhD candidates’ statistical abilities and identify the appropriate content and methodology for these participants and the context. The participants were five third-year doctoral candidates—middle-aged language teachers with extensive teaching experience—who were enrolled in the statistics course I taught. I hold a PhD in applied linguistics and have experience with quantitative research. By obtaining online consent, the study secured confidentiality and anonymity. As shown in Table 1, the participants were unfamiliar with statistics and shared common academic and professional backgrounds.

Research Design

Action research was implemented to track the development of the participants’ statistical literacies. Action research is suitable for positive change and improvement in the participants’ social situation from a “socio-constructivist approach in which teachers are seen as agentive actors and investigators within their social contexts” (Burns, 2019, p. 166). As the course instructor and researcher, I collected classroom data and conducted three cyclical phases of action research: diagnostic, intervention (with cycles), and evaluation (see Figure 1).

Figure 1: Research Phases

The diagnostic phase inquired into participants’ initial statistical performance through a pre-test and self-perception data, establishing a baseline for the 16-week intervention phase. The intervention phase consisted of implementing the course, utilising a theoretical-practical approach that featured collaborative learning. The course syllabus (see Appendix) was designed based on reputable textbooks (e.g., Lowie & Seton, 2013; Phakiti, 2014) and prioritised classical statistical methods due to the participants’ needs. Software for quantitative data analyses included Excel, SPSS, Langtest, and JASP. Ten real-life statistics tasks included performing, interpreting and analysing data, reporting results, and evaluating the quality of quantitative research reports. The course was delivered online through Zoom, as most participants reside in different regions of Colombia. During the sessions, the participants’ oral reflections provided valuable insights into their learning process and experiences with the course; however, due to space constraints, these interactions are not reported here. Their reflections related mainly to the pacing of the course content. The evaluation phase employed a post-test, graded assignments, and an oral final reflection to examine participants’ performance, learning experiences, and perceptions of the course.

Data Collection Instruments

To address Research Question 1, I employed video recording, pre-test and post-test results, and three graded assignments. In the video, participants reported their qualifications, professional experience, research designs, familiarity with statistics, and opinions on Loewen et al.’s (2014, 2020) articles about statistical literacy among applied linguistics and SLA researchers. This provided qualitative insights into the participants’ characteristics and their familiarity with quantitative research prior to the course.

The online pre-test and post-test (administered through Google Forms), along with three graded assignments, provided participants’ performance data on the development of statistical literacy. The pre-test and post-test consisted of three sections (see Table 2), each containing 34 multiple-choice questions and one open-ended question. The option “I do not know” ensured thoughtful responses, avoiding random choices. Both tests’ questions encompassed critical thinking, statistical methods selection, data analysis, interpretation, and reporting, and were informed by the research of Gönülal (2018, 2019), Gönülal et al. (2017), and Loewen et al. (2014, 2020). Prior to implementation, the pre-test was reviewed and piloted by a colleague with experience in quantitative research.

The three graded assignments (see Table 3) gauged the participants’ statistical learning. Table 4 shows a sample of Assignment 1. Continuous feedback was provided during the course through practical individual and group tasks, mirroring the structure of the graded assignments.

To answer Research Question 2, data were gathered through a final oral reflection in the last class session. Individual responses were recorded and then transcribed using the tool Otter.AI. This oral reflection included the following questions:

• How familiar were you with statistical methods before the course, and how familiar are you now?

• What are your strengths and weaknesses regarding your understanding of variables, scales, assumptions, and descriptive and inferential statistics; the use of JASP, Excel, and Langtest; the selection of statistical tests; your interpretation of visual and numerical outputs; and the writing of the research results? Can you critically analyse the quality and robustness of a quantitative research report?

• What did you enjoy the most, and what difficulties did you have during your learning process?

• How motivated are you now about doing statistics?

• What are your perceptions of the statistics course (objectives, content, methodology, assessment, resources) and its place in the curriculum?

Data Analysis

Due to the study’s nature and small sample size, inferential statistics were deemed inappropriate (Gönülal et al., 2017). Descriptive statistics (mean and standard deviation) were employed for reporting the results of both tests and the graded assignments. Qualitative data, primarily derived from the participants’ reflections, were analysed through content analysis, identifying the participants’ learning experiences, strengths, weaknesses, and motivation toward statistics. The participants’ perceptions of the course were analysed in relation to its components: objectives, content, methodology, resources, and assessment.

Validity, Ethics, and Researcher Positionality

This study drew on applied linguistics theory and research for conceptualisation and instrument design. The instruments were reviewed and piloted by an expert in the field. To enhance triangulation and the validity of findings, data were collected from multiple sources (pre- and post-tests, assignments, and student reflections), which were compared to identify patterns and discrepancies, ensuring that findings were not solely reliant on any single measure. Member-checking was also incorporated to allow participants to provide feedback to their peers and comment on preliminary interpretations of their performance and views, thereby reducing potential misrepresentations of their perspectives.

To ensure reliability and provide an impartial perspective on participants’ engagement, interrater agreement was employed by having two researchers independently code reflection data from two randomly chosen participants. Discrepancies were resolved through discussion until consensus was reached. For the remainder of the data, I conducted iterative data analysis.

I was granted permission to conduct this study, and participation was voluntary. Given that I was also the course instructor, I mitigated potential biases that could influence data collection and interpretation by engaging in constant reflection through my lesson plans and interactions with participants about my practices and expectations throughout the pedagogical intervention. To minimise desirability bias, where participants might provide overly positive evaluations due to the researcher’s dual role, they were explicitly encouraged to critique the course and its materials. Feedback sessions emphasised the value of constructive criticism.

Self-Perceptions of Statistical Literacy Before the Course: Facing Statistical Literacy With Concerns

At the outset, all participants acknowledged their lack of statistical knowledge and reported perceiving a deficiency in the quantitative component of Colombian undergraduate and graduate applied linguistics programmes. They expressed concerns about the low statistical literacy among applied linguistics researchers globally, referencing the findings of Loewen et al. (2014, 2020). They highlighted a broader concern about the scarcity of statistical literacy research in Latin America, emphasising the need for more studies in the region, as existing research mainly focuses on universities in North America, Europe, and Asia.

For instance, Antonio stated that he had experience with item reliability but was unfamiliar with other statistical concepts. He considers that “our students are unfamiliar with statistics [and that] our academic community’s attitudes are negative toward numbers.” Claire reported that she had never taken statistics courses. She usually “skip[ped] the results of quantitative research articles and jump[ed] to the conclusions.” Atahualpa said she was “a qualitatively oriented researcher but was open to learning statistics,” while Hector claimed that “there is a need for more studies in Latin America and more training on statistics in applied linguistics programmes.”

This shared concern shaped the participants’ initial engagement with the course, with all of them acknowledging the relevance of statistics to applied linguistics in our research context.

Statistics Performance Before and After the Course: Cognitive Struggles and Breakthroughs

Before the course, the participants encountered some struggles and cognitive dissonance, particularly in understanding and reporting quantitative research outputs. Their performance in the pre-test allowed them to realise the need to develop statistical literacies. However, as the course progressed—and through guided practice and peer collaboration—all participants improved their performance and reported key breakthroughs.

As shown in Figure 2, the group’s pre-test results (M = 4.6, SD = 5.3) were lower than the post-test outcomes (M = 32.2, SD = 2.9), indicating improvement in the teacher-researchers’ statistical literacy at the end of the course.

Figure 2: Box Plots of Pre-Test and Post-Test Participants’ Overall Performance

At the individual level, Table 5 also shows substantial gains (above 60 %) in statistical knowledge and performance, indicating advancements after completing the course. Claire and Atahualpa exhibited the greatest improvements (above 90 %) as seen in their pre-test and post-test results. Overall, these results suggest that the course is effective in fostering participants’ statistical literacy at both the group and individual levels.

Table 6 provides a detailed report of the statistical literacy components assessed in the three sections of the pre-test and post-test.

As shown in Table 6, the participants demonstrated improvement in their knowledge of statistics across several components. Regarding Section 1, the teachers demonstrated enhanced comprehension of descriptive statistics in numerical and graphical data, which is essential for recognising patterns and assumptions in the data. In Section 2, the participants progressed in their understanding of common inferential statistics and in their ability to differentiate between parametric and non-parametric tests, choose appropriate statistical tests, and interpret alpha values and effect sizes. Section 3 also shows progress in participants’ ability to report statistical analysis results, indicating an improvement in conveying quantitative findings.

Furthermore, the three course assignments showed notable improvements in the participants’ development of statistical literacy, as evidenced by their high marks (see Table 7). Assignment 1 (critical analysis of a published quantitative study) posed fewer challenges to the participants compared to Assignments 2 and 3, which were related to performing, interpreting, and reporting quantitative data analyses and results. This suggests that it was easier for the participants to review and analyse a quantitative research report than to conduct more specific procedures. Nevertheless, the results from these assignments suggest that the participants increased their knowledge and use of descriptive and inferential statistics, assumptions, quantitative analysis, and use of statistical tools, as well as their ability to generate, interpret, and report data and critically evaluate quantitative studies.

My feedback to the participants on their assignments identified their strengths and weaknesses. Regarding Assignment 1:

These excellent results mean that you were able to evaluate the robustness of a quantitative research report by identifying and critiquing information related to sampling, measurement and constructs, design and procedures, numerical and graphical outcomes, assumptions, descriptive and inferential statistics, significance levels, effect sizes, and confidence intervals.

Regarding Assignment 2, my feedback highlighted the participants’ ability to perform data analysis and generate and interpret numerical and graphical outcomes. In contrast, participants needed to improve the written report, interpretation, and discussion of results, considering the research questions and theoretical and pedagogical implications:

Overall, you can select a parametric test and perform a quantitative analysis based on a research design. You can also check the assumptions and interpret the visual and numerical data. The only aspect to improve is the interpretations of results (contribution to knowledge). You need to go beyond the interpretation of statistical significance. You must interpret the findings considering the theory, context, or participants. You could have provided more theoretical or pedagogical interpretations.

My feedback for Assignment 3 highlighted the participants’ ability to interpret data, although they had to be more accurate in the written report:

You can interpret and report the results of quantitative studies that employ classical statistical tests. The written reports were accurate and similar to the original research reports. The only aspect to improve is the accuracy of the language used in reporting quantitative results. You will need more input and practice to improve that aspect. Nevertheless, you did an excellent job overall and understood the fundamental aspects of quantitative analyses, results, and reports.

Perceptions of Statistical Literacy After the Course: Empowerment and Use of Statistics as a Tool for Inquiry

After the course, most participants claimed their view of statistical literacy had changed, moving from concerns and struggles to confidence. Their reflections indicated a sense of empowerment and a shift in mindset, as they realised they could engage with quantitative data.

All the participants perceived a substantial increase in their statistical knowledge and skills, and were able to identify their strengths and weaknesses. They reported competence in identifying variables and scales; performing and interpreting descriptive and inferential statistics; using Excel, SPSS, and Langtest; selecting appropriate statistical tests; interpreting visual and numerical outputs; writing the results; and critically analysing quantitative research reports. However, they expressed a need for additional practice to strengthen concepts and procedures and enhance report-writing skills:

Now I know about interpreting, evaluating, and writing statistical results. I feel confident about reading an article and understanding the data. My weakness relates to writing the results, such as reminding the reader about the research question. (Antonio)

I feel more confident about interpreting data and critically evaluating a research report. However, the most challenging part is writing. I know there are formulaic expressions to report the results, but it is hard for me. (Hector)

A shift in perception emerged as they moved from limited knowledge to conceptual understanding, recognising patterns in data and developing confidence in their interpretations.

Motivation Towards Quantitative Research: Reflecting on Growth and Future Applications

In their reflections, the participants showed how their evolving statistical literacy reshaped their academic identity as applied linguistics researchers. They emphasised both their improved skills and their motivation and confidence in engaging with quantitative research in the future:

I am a qualitatively oriented researcher, but I want to do quantitative research because it could be helpful in my context. (Atahualpa)

I am highly motivated now because I see the purpose of quantitative research. Before, I used to say that I was not good at numbers, but now I can use them in my research. There are so many areas where statistics can be used. (Antonio)

I am more motivated now than before. I thought it would be challenging and not helpful because we are conducting qualitative research. Now, I see there are many possibilities to conduct mixed-methods or quantitative research. (Hector)

The participants also reported that guidance and the course resources were positive methodological strategies that influenced their learning and motivation. For instance, Claire said she “liked the practical activities that guided [her] step by step.” Similarly, Hector reported that he “loved designing tables using APA 7th style and performing analysis in SPSS and Langtest.” James also “enjoyed performing the analyses in the software programmes because [he] didn’t know about these programmes before.” This newborn confidence suggests that, beyond skill acquisition, the intervention fostered a sense of agency in participants as researchers.

Perceptions of the Statistics Course: Navigating Statistical Literacy in the Curriculum

The participants’ positive experience with the course allowed them to develop a critical view of the relevance of introducing statistics in applied linguistics programmes. In their reflections, they acknowledged the appropriateness of the course components in their formation. Regarding the course content, participants considered it appropriate, although it was advanced. For Antonio, “this was not an introductory course but an advanced one.” Hector said that “the feedback provided, planning, and course methodology allowed [him] to learn the course contents.” The course learning objectives were also appropriate and achieved by the participants, as Hector claimed. Concerning the course methodology, the participants valued the course planning and practical activities, which were based on real-life statistical tasks. The ten practical tasks focused on understanding, performing, and reporting data analyses and results from correlations, t-tests, and ANOVAs. Participants also designed research studies based on real issues, their contexts, and realities and analysed published quantitative articles in their research areas.

The online modality was also appropriate, although participants preferred face-to-face interactions. James, Claire, and Atahualpa agreed that “on-site classes promote interaction, but online courses are necessary for students from other regions.” Antonio mentioned that “the online modality did not negatively impact [their] learning. It’s just a matter of preference.” The course resources (e.g., statistics books in applied linguistics and social sciences, research articles, videos on statistics, and software) were relevant and authentic, as highlighted by the participants. Special guests also complemented their knowledge. Participants preferred Langtest over Excel and SPSS because it is user-friendly and practical. However, SPSS offers good visual features, including histograms, box plots, and outlier detection. The limitation of Langtest is the lack of APA format tables. JASP was introduced at the end of the course and was not sufficiently explored.

I used and felt more comfortable with Langtest. With SPSS, I had to go back to the videos, but I was able to use it. I like the box plots produced by SPSS because they show you the specific outliers. (Antonio)

I started using SPSS and was getting familiar with it. Then, I realised that Langtest was user-friendly. In SPSS, the histograms and box plots look better. I prefer Langtest, although we had to design the tables and arrange the data. (Hector)

The guest speakers also contributed to our knowledge of statistics. The materials were helpful. They can be published as a textbook for a doctoral programme because we need available and easy-to-understand materials. (Claire)

Regarding course assessment, the participants valued the detailed feedback I provided on each practical activity, which helped them scaffold their learning process. The participants expressed that the pre-test, post-test, and three assignments were appropriate for measuring their performance. As they noted, practical activities should have been graded as part of the process. On the other hand, collaborative work and group assessment allowed them to identify their strengths and weaknesses in statistics.

The detailed feedback on our analyses and writing was helpful because we identified our strengths and weaknesses. The collaborative work was positive because we got together to work and understand the activities and assignments. We got to identify our abilities. (Claire)

The feedback was relevant because the teacher highlighted the mistakes and told us how to correct them. I also share Antonio’s opinion that the practical activities could have been graded and added to our final records in case we failed one assignment. (James)

The pre-test and post-test were helpful because we could see the advances regarding the course’s goals and our knowledge of quantitative research and statistics. (Hector)

Based on their positive experiences and analytical views, the participants suggested that at least two statistics courses should be offered in the doctoral programme: one compulsory introductory statistics course in the first year to familiarise candidates with quantitative research designs, and an elective course in the middle of the curricular stage for advancing this knowledge.

This course should have been offered earlier in the curriculum because we could have chosen other research designs. Some of our classmates started with a mixed-methods approach but had to change because they didn’t know about quantitative research. (James)

This course should be offered in the first year so students can make informed decisions about their studies. We were told we could not conduct a mixed-methods study because we were unprepared. Two courses would cover more topics: One compulsory course at the beginning and one elective course in the middle for those conducting quantitative or mixed-methods studies. (Claire)