Scoping JEAP, 2024-2025
Introducing the Project
This is a space to record progress in a scoping review I am working on as a way to better make sense of what kinds of research have been published in the Journal of EAP (JEAP). I am conducting this review partly because of the need to publish as an aspect of my doctoral studies. That is, a review of JEAP published articles makes sense for a doctoral student attempting to become a member of this practitioner community by contributing to the research pool. Doing so might provide direction and inspiration for my next project. I think this research could also be potentially interesting for practitioners in a similar position wishing to conduct independent unfunded research with the potential to publish in JEAP. One point here is that genre analysis and writing guides have helped us develop a strong understanding of the styles and format that tend to characterise publications in our field, but there is less guidance on what epistemological and methodological approaches tend to get published. That is, there is guidance on how to write, but not necessarily what to research.
One inspiration for the project work is the article “The contexts, theoretical and methodological orientation of EAP research: Evidence from empirical articles published in the Journal of English for Academic Purposes” (Riazi et al., 2020). I noticed, however, that the focus on ‘the empirical’ left other types of content uncounted. Also, having recently completed a Reflexive Thematic Analysis project based on methods discussed in Braun & Clarke (2022), I have found myself wondering whether there is a predominance of positivist, rather than interpretive, research within published articles. And, of course, one way to find out what kind of publications are published in JEAP is to conduct my own analysis.
Finally, as a practitioner-researcher with a full-time job, doctoral studies, and the desire to be a fully engaged parent to my young child, I sometimes find myself asking how others find time to complete high quality research. Are there barriers for entry into the conversation based around access to research resources? What kinds of research might be feasible for myself or similar individuals in my position?
Research Questions
The review aims to investigate what kinds of work appear in JEAP, on what epistemological terms and at what cost to produce. I am interested in who is able to participate in the field’s published conversation. My initial plan is to focus on the volumes published (in print) in 2024 and 2025, specifically volumes 67 to 78.
The (draft) research questions are:
- What was published, and what did it claim to do?
- How were items distributed across genres?
- What kinds of contribution did items of each genre claim to make?
- What can be said about the epistemological frameworks of published work?
- How often are epistemological or methodological positions made explicit?
- Where an epistemological or methodological stance is taken, which appear and how often?
- What can be said about the resources engaged in producing the publications?
- What resources, such as team size, funding, and specialist requirements were employed to develop the published work?
- What proportion could feasibly have been produced by a sole, unfunded practitioner-researcher at a single site within a limited time scale?
Approach: Why a Scoping Review?
Systematic secondary research approaches are potentially appealing, at least initially, for two main reasons. Firstly, there is the popularity of these types of synthesis. As predicted in Norris & Ortega (2007) there is growing interest in how such approaches can be applied to the field of applied linguistics (Chong & Plonsky, 2024, p. 1570). Secondly, there is the appeal of being able to produce work without additional complications of access to sites of learning and with no requirement to seek ethical approval for work with human participants. That is, systematic secondary research appears, at least initially, to be relatively more appropriate for a researcher with limited resources.
Although definitions of scoping review vary, numerous sources describe these in similar terms, for example Pollock et al. (2024, p. 1) state that scoping reviews are appropriate to ‘map the breadth of evidence available on a particular topic, field, concept or issue’. Davis et al. (2009, p. 1396) argue that scoping reviews can be described as a form of ‘reconnaissance’; a ‘preliminary investigation in which information is systematically gathered and examined in order to … guide … future decision-making.’ Other features of scoping reviews from the literature include that they can be used to compare a range of types of source material and that they are appropriate when considering how research is conducted in a given field (Pollock et al., 2024). A key early paper, Arksey & O’Malley (2005, p. 30), noted that the key goals of scoping studies were to develop a ‘narrative or descriptive account of available research’ rather than specifically to appraise, evaluate, or synthesise the effects of research papers. Seeking to extend advice around Arksey & O’Malley’s (2005) framework, Levac et al. (2010) recommend the use of a ‘qualitative thematic analysis’ alongside a ‘numerical summary.’ This suggests that a scoping review needs to be equipped with tools for managing both qualitative and quantitative data, which highlights complications in ensuring research quality.
Establishing Trustworthiness
Norris & Ortega (2007) argue that one of the challenges for systematic research is to develop approaches to synthesis that go beyond the dichotomy of ‘art or science.’ The idea here being that systematic approaches tend to be based on more positivist paradigms, but still need to account for qualitative or interpretive work. A review intended to synthesise articles that use different epistemological frames leads to the problem that we are not comparing like with like. This is not a recent concern. Back in the 1980s, Guba & Lincoln (1982) argued that more interpretive (constructivist or naturalistic) studies required a different set of trustworthiness criteria to those typically applied to more rationalistic or positivist research. Picking up this thread for studies on L2 writing Riazi et al. (2023), argue that quality in qualitative research is better described by Guba & Lincoln’s framework of trustworthiness: credibility, dependability, transferability and confirmability rather than on the more positivist ‘conventional terminology’ of ‘reliability and validity’ (Riazi et al., 2023, p. 2). The authors’ discussion highlights that, despite their argument, the predominant method of signposting quality in qualitative work submitted to the Journal of Second Language Writing was to ‘adhere to the classical concepts and terminologies of quantitative research’ (Riazi et al., 2023, p. 9). What this finding points to is the importance of ‘conceptual coherence’ in research: ‘do your research aims and purpose, philosophical and methodological assumptions, and methods cohere together?’ Braun & Clarke (2022, p. 26). My response to this is to lean into the idea that this work does have a significant qualitative component, requiring Braun & Clarke’s (2022, p. 7) ‘qualitative sensibility.’
One strong need to establish my grounds for trustworthiness is that the literature often advises against conducting scoping reviews as a sole researcher. Mak & Thomas (2022, p. 565), for example, state that ‘scoping reviews are not conducted by a single individual.’ However, it is worth asking why a team approach is desirable. One argument for a team comes in the screening phase of research, where an agreed exclusion and inclusion policy needs to be accurately applied. In my research, I have sidestepped this by deciding to include all documents from specific volumes. A second argument comes from the idea that research work involves the discovery of an ‘objective truth’. If this is accepted, then triangulating researcher perspectives may lead to a more objective description of the truth to be described. However, as argued in Guba & Lincoln (1982) this view is not necessarily the best approach when reaching for trustworthiness within a constructivist paradigm. Instead, the authors argue for qualitative confirmability. This can be improved through processes of reflexivity, whereby the lone researcher tries to consider the impact of their own preferences and biases on the research, and through what the authors describe as a ‘confirmability audit’. If the lone researcher can transparently demonstrate how each claim they make is clearly developed from the data, and which data has been used to make their claims, then the reader can decide whether the claims are sound. Guba & Lincoln (1982, p. 247), it should be noted, do recommend a degree of ‘triangulation.’ However, this does not refer to attempts to gauge, for example, inter-rater reliability in coding. Instead, this is where ‘a variety of data sources, different perspectives or theories, and/or different methods are pitted against one another to cross-check data and interpretation’ (Guba & Lincoln, 1982, p. 247).
An example of Guba & Lincoln’s (1982) ideas being developed to argue for trustworthiness for more recent research can be found in Nowell et al. (2017)‘s work on Thematic Analysis. As in the title of their paper, I am ’striving to meet the trustworthiness criteria’ in a number of key ways. Firstly, credibility can be enhanced through prolonged engagement with research data, data collection triangulation (as it was described above) and Guba & Lincoln’s (1982) ‘peer debriefing’, which refers to sharing my work-in-progress and developing insights with uninvolved peers. I believe that developing this research as an online, freely accessible, and fully documented study allows me to do this. Secondly, as in Nowell et al. (2017, p. 3), transferability refers to the extent to which others can apply research clearly based in a specific context or field to their own contexts or requirements. The advice here, also found in Guba & Lincoln (1982, p. 248) is to use ‘thick description’, firstly so the reader fully understands the context of the original research, and secondly so that they can make their own decisions about how much of the findings relate to a ‘second or similar context.’ Dependability involves ensuring that research processes are transparent and that decisions made in the course of the research appear logical and appropriate to the task (Nowell et al., 2017, p. 3). Working towards both dependability and confirmability requires pulling similar levers of clear and coherent documentation of work produced in the processes of producing it. The idea is to ensure that the reader is clear about how findings were developed, and which claims relate to which aspects of the data. In fact, Guba & Lincoln (1982, p. 248) recommend the use of an ‘auditor’ to check and advise the researcher’s ‘working out.’ Similarly, Nowell et al. (2017) argue that a clear audit trail is essential to meeting the trustworthiness criteria. My response to this is, as already noted, to develop this project openly on the internet, using a publicly available git repository to update my work in progress. This should provide an extensive, traceable, and coherent audit trail and while I do not have a specific auditor, I have peers and a doctoral supervisor to offer advice as the project unfolds.
Across multiple sources listed above, the advice to be a reflexive researcher, or be reflexive in conducting research, is a common request. Braun & Clarke (2022, p. 5) describe reflexivity as ‘critical reflection’ on ‘your role as a researcher, and your research practice and process’ and place reflexivity as a defining skill for the development of quality qualitative analysis. The reflexive journal (Braun & Clarke, 2022, p. 293) is a key aspect of reflexive practice and Nowell et al. (2017, p. 3) recommend the researcher ‘record the daily logistics of the research, methodological decisions and rationales and to record personal reflections of their values and interests.’ I intend to keep a log and update it regularly. However, the reflexive journals discussed in the literature are intended as private, personal documents. The fact that I will update my logs online will have an effect on how candid my reflections are.
Developing a research instrument
Following Arksey & O’Malley’s (2005) framework, the research questions have been written and I have identified the relevant studies. In taking the entire corpus of specific volumes, stage three of the framework does not really apply to this study. The next step, stage four, is to develop the ‘data-charting form and determine which variables to extract in order to answer the research question.’ (Levac et al., 2010, p. 4). This will be done by initially piloting the extraction form with Volume 63 of JEAP, published in May 2023.
The pilot version of the form is outlined below:
| Field | RQ | Rationale and Notes |
|---|---|---|
| DOI | Records the permanent identifier of the record. | |
| Title | Records the title of the record. | |
| Volume | Records the JEAP Volume. | |
| Genre | RQ1 | The purpose here is not to provide a full ‘genre analysis’, but to develop a short list of likely genres to identify and select from. This is likely to be an iterative process as I work through the data. |
| Contribution claimed | RQ1 | Here, I plan to take notes on how the article claims to contribute to the field. These may be appropriate for Thematic Analysis at a later stage. |
| Evidence / Location | RQ1 | For confirmability, I plan to link the notes on contribution claimed to a specific section of the article. |
| Epistemology named? | RQ2 | Where the author has explicitly linked to an epistemological framing, this can be counted to contribute to quantitative data. |
| Language used | RQ2 | To record the language used around epistemological framing. This may be appropriate for thematic analysis at a later stage. |
| Methodology or Method named? | RQ2 | Where the author has explicitly linked to a methodological approach, this can be counted to contribute to quantitative data. |
| Language used | RQ2 | To record the language used around methodological approach. |
| Notes and inferences | RQ2 | Where epistemology and methodology are not stated clearly, this is a space for me to interpret the approach and take notes that may be appropriate for thematic analysis at a later stage. |
| Evidence / Location | RQ2 | For confirmability, I need to link my notes on epistemology and methodology to specific sections of the article. |
| Author Count | RQ3 | Quantitative data that can potentially provide useful information on how feasible similar research would be for a small unfunded team. |
| References to additional labour | RQ3 | Allows me to note instances where work has contributed to the publication that is not captured in the author count. |
| Funding | RQ3 | Where declared, funding suggests resource access not easily accessible for a lone researcher. |
| Open Access | RQ3 | While open access arrangements exist when publishing from specific institutions, this may suggest resource access not easily accessible for a lone researcher. |
| Sites | RQ3 | This relates to institutional access to sites of teaching and learning, which could be considered a research resource. |
| Data source | RQ3 | In terms of the feasibility of conducting similar research in different contexts, different data sources may be more or less accessible. |
| Participants | RQ3 | Similar to the above, access to different kinds of research participants could be considered a research resource. |
| Specialist needs | RQ3 | Here I am thinking about advanced software or similar that puts the research out of reach of a novice researcher. Eye tracking studies for example, require the use of specialised laboratory equipment. |
| Notes | Any | To help me take reflexive notes, ask questions of the data, or highlight aspects of the article I may wish to investigate further. |
I should add here that Research Question RQ3b suggests a count of studies to assess which are more feasible for the sole researcher. This will require analysis of the data extracted around RQ3, and is not something I will attempt to code on the first pass.