Moving beyond binaries in research: weaving the tapestry of participants’ experiences

by

Posted:

3 Comments

The image shows a professional portrait of a woman with long, dark brown hair and glasses, smiling warmly. She is wearing a floral-patterned blouse under a black jacket, and the setting is softly lit, with an out-of-focus background featuring natural light and hints of greenery. The photograph conveys a friendly and approachable demeanour, suitable for professional or academic contexts.

Dr Lynette Pretorius

Dr Lynette Pretorius is an award-winning educator and researcher specialising in doctoral education, AI literacy, research literacy, academic identity, and student wellbeing.

In today’s data-driven world, there is a lot of talk about making decisions based on so-called objective data. For example, schools and universities use information about the mix of students and staff to shape how they teach and run things. Information such as age, where people live, how much schooling they have had, or their income is collected to help make these so-called “informed” decisions. But here’s the problem – we sometimes forget that the people collecting these forms of data and those making these decisions have their own biases. Decisions reflect the majority view, which means that other experiences are often sidelined.

We need to understand that different parts of our backgrounds interact and affect the way we experience the world, often in very different ways. This is what is termed intersectionality. Using intersectionality as a lens helps us to recognise that we cannot look at parts of someone’s identity in isolation. We need to see the whole person and how all parts of their identity come together, influencing their experiences and the way the world sees and treats them. It is like saying, “To understand the whole story, you can’t just read one page. You need to read the entire book.”

This highlights that researchers and decision-makers need to work to improve processes for data collection and analysis to better reflect the diversity of people’s experiences. So, why is it so crucial to bring diverse perspectives into the research mix?

  • Firstly, past research has not done a great job of representing everyone. Surveys can often be pretty narrow, missing out on the complete picture of who participants are, which means we are not getting the full story on how to solve problems for everyone.
  • Secondly, by embracing diversity in research, we stand up for fairness and social justice. Imagine surveys that only see concepts in black and white, leaving out people who do not fit neatly into specific boxes. We are missing out on understanding different experiences and perspectives, which can make our research richer and more meaningful.
  • Finally, acknowledging a wide range of experiences helps us dig deeper into our findings, giving us a clearer view of what is really going on in our context. This approach challenges us to think beyond the usual categories and consider the complex ways people identify themselves. By doing so, we can better reflect society’s diversity and push for changes that make society more inclusive and equitable for everyone.

Despite some improvements in how we collect data in recent years, there is still a long way to go. We need to ensure that our research methods allow people to share the full spectrum of their identities, respecting the richness of their experiences. It is all about giving everyone a voice and ensuring research serves us all, not just a privileged few.

The thing is, when we are exploring human experiences, we must embrace the messiness and all the different parts of who people are. But, sadly, many times, research just clumps people into simple categories, missing out on their full stories. This approach does not just overlook the richness of their identities; it can also make them feel like their voices do not matter, with their real-life experiences either ignored or questioned.

In my recent paper, I propose a new way of collecting data about research participants. I propose that we say, “Hey, let people tell us about themselves in their own words” rather than asking specific questions that limit their responses. To do this, I argue that researchers should include a question where people can share their own diversity stories when they fill out surveys. Why? Because it does justice to their experiences and knowledge.

Read the paper

I have seen firsthand how rich and deep data can be when people share their stories this way, especially when this data is combined with other open-ended research questions. My paper makes the case for letting people have a say in how they are represented in research. It is about giving them the power to share their identities in their own words. The main findings from my study include:

  • When I asked open questions, the replies were eye-opening: I decided to ask people to tell me about themselves in their own way, without the usual checkboxes. And wow, did I get a treasure trove of responses! Some people went the traditional route, but others shared stories and parts of their identities I would never have captured with a simple tick box. This approach really highlighted how everyone has their own unique blend of experiences and backgrounds.
  • Self-written diversity statements are gold mines of insights: One aspect that was particularly unique in my study is that I asked people to jot down their thoughts on what makes them, well, them. I did this by asking them to write their own diversity statement. The depth of what I got back was incredible – from personal tales of grappling with ableism to rich descriptions of cultural heritage and everything in between. It is like these self-written snippets opened a window into the real lives and challenges people face, way beyond what any standard survey could capture.
  • Weaving stories together to highlight the tapestry of people’s lived experiences: One of the most exciting findings from my study is how I used all these different bits of info from the surveys and weaved them into what I call holistic introductory stories. Imagine taking a bit from here and a snippet from there to stitch together a complete narrative about someone. It is like getting a full-colour, 3D picture of a person rather than a flat, 2D sketch. This way, I was not just seeing bits and pieces of someone’s identities, but I was developing a better understanding of how all those bits fit together to make my participants who they are.

My findings highlight the importance of encouraging epistemic justice in our research practices. What is epistemic justice, you may ask? Epistemic justice is about fairness: it ensures that everyone’s voice and knowledge are equally respected, no matter where they come from or how they express themselves. It is about ensuring all perspectives are considered, especially those often ignored or undervalued. To really do justice to everyone’s knowledge, we have to be open to different, even incomplete ways of understanding. That is why I am using open questions and these stories to give everyone a platform to share their experiences. I believe stories are how we make sense of our world. As has been highlighted by other researchers, stories help us understand not just the surface-level stuff people share but the deeper, sometimes hidden layers of their lives.

My focus has been on getting people to write down their stories because there is power in writing. But now that this study is finished, I am thinking, why stop there? There are so many other ways to share and understand each other’s experiences. So, looking ahead, I am keen on mixing things up even more, using all sorts of creative methods to make sure everyone feels seen and heard, especially those who have been left out of the conversation for too long.

Questions to ponder

If you had to write a short diversity statement about yourself, what would you say?

How does the incorporation of self-written diversity statements and open-ended questions in surveys challenge traditional methods of data collection in qualitative research?

The paper advocates for epistemic justice through methodological innovations in order to reduce biases and inequalities in research. How does giving participants the agency to define themselves challenge or change the researcher’s role?

The research outlines a more artistic way of understanding participants through holistic introductory stories. What advantages does this creative approach offer, and what challenges might it pose in traditional research environments?

Theoretical and conceptual frameworks in research

by

Posted:

13 Comments

The image shows a professional portrait of a woman with long, dark brown hair and glasses, smiling warmly. She is wearing a floral-patterned blouse under a black jacket, and the setting is softly lit, with an out-of-focus background featuring natural light and hints of greenery. The photograph conveys a friendly and approachable demeanour, suitable for professional or academic contexts.

Dr Lynette Pretorius

Dr Lynette Pretorius is an award-winning educator and researcher specialising in doctoral education, AI literacy, research literacy, academic identity, and student wellbeing.

Frameworks in research play a crucial role in shaping the direction of a research project. They serve as the foundation upon which studies are built and analysed, offering a lens through which researchers can interpret their findings. However, they are also a source of confusion for researchers so, in this blog post, I explain the differences between theoretical and conceptual frameworks, why they are important in research, and how researchers can choose a framework for their study.

Theoretical and conceptual frameworks: what are the differences?

The most common confusion I see when talking to researchers about frameworks is that they do not understand the difference between a theoretical and a conceptual framework. So what are the differences?

  • A researcher uses a theoretical framework when they are using a pre-established theory for their study. A theoretical framework is founded upon a well-established theory, focuses on explaining phenomena using this existing theory, and is chosen based on examining which theories are commonly used in the field.
  • A researcher uses a conceptual framework when they select concepts from several theories and construct them into a framework for a specific study. A conceptual framework is constructed by the researcher, aims to explore relationships between different concepts, and is developed by synthesising different ideas from the field together.

It is important to note that you should choose either a theoretical or a conceptual framework – usually, one project does not have both. This is because they serve the same purpose – they provide the vision for your study. Having two competing frameworks, therefore, would confuse your study. The only time you would have two frameworks is if your project had two distinct parts that were unrelated to each other.

Why do frameworks matter in research?

So why is it important to have a framework in your study? Frameworks are important because they provide a foundation and vision for your project. They help the researcher make sense of the chaos of the data by revealing the theories or concepts that will be used to analyse and explain findings. Frameworks influence:

  • The research design: Theoretical and conceptual frameworks help shape the research design, including the choice of data collection and analysis methods. By setting out the key concepts and their relationships, they provide a roadmap for conducting the study.
  • The data analysis: They offer a lens for interpreting the data collected during the research. This is particularly crucial in qualitative data analysis, where the researcher seeks to understand complex phenomena through the perspectives of participants.
  • The rigour and relevance of a study: The use of a theoretical or conceptual framework enhances the rigour of a study by ensuring that its findings can be linked back to broader discussions in the field.
  • How research builds on existing knowledge: Grounding research in a theoretical or conceptual framework ensures that new studies build upon or challenge the existing body of knowledge.

Frameworks also reveal the beliefs and assumptions of the researcher. This necessitates that the researcher carefully articulates their framework, explaining key concepts within the framework and why they are important for the particular study. Concepts can often mean different things to different people. For example, if you mention to your friend that you just adopted a dog but do not provide further details, the image of your dog in your head could be quite different from what your friend imagines. In the same way, research concepts are often multifaceted and require careful explanation in a research project.

Identifying frameworks in the literature

Identifying frameworks in published articles is a crucial skill for researchers, enabling them to understand a study’s theoretical or conceptual underpinnings, replicate research designs, or apply similar frameworks in their work. However, this is not always easy, and sometimes frameworks are not explicitly stated. So how do you identify a framework in a published study?

  • Look for explicit mentions: This is the easiest strategy, provided that the framework is mentioned explicitly of course. Search for sentences such as “This study is grounded in X theory” or “We employ Y’s conceptual model as a framework.”
  • Consider the research questions: The nature of the research questions or hypotheses can often indicate the type of framework being used.
  • Trace the references: Frameworks can be identified by looking at the citations of foundational works within a published source. Checking these references can provide a deeper understanding of the framework that the authors have used. Below, I have listed some of the most common frameworks used in social sciences, particularly educational research. Searching for words such as these will help you find the framework in a study.
    • Bourdieu’s Theory of Practice
    • Foucault’s theories on power, discourse, and knowledge
    • Bronfenbrenner’s Ecological Systems Theory
    • Cultural-historical theory (Vygotsky) or Cultural-historical activity theory (Vygotsky and Leontiev)
    • Theories of learning (e.g., community of practice, experiential learning, transformative learning)
    • Performativity and/or embodiment (e.g., gender performativity)
    • Intersectionality and/or epistemic (in)justice
    • New literacies theory or digital literacies
    • TPACK (Technological pedagogical content knowledge)

So how do you choose a framework for your study?

Choosing the right framework for your research is a critical decision that shapes your study’s direction, coherence, and trustworthiness. Whether you opt for a theoretical or conceptual framework depends on your research objectives, the nature of your study, and the existing literature in your field.

  • The first step in choosing a framework is to clarify your research objectives: What are you trying to discover, explore, explain, or analyse? Your research objectives will guide your choice of framework by highlighting the key concepts, theories, or models that are most relevant to addressing your research questions.
  • The second step is to conduct an extensive literature review to identify potential frameworks that have been applied to similar research problems. By examining how these frameworks have been used in past studies, you can identify gaps (i.e., look for areas where existing frameworks may not fully address your research problem or where there is room for further exploration), assess applicability (i.e., determine how well the frameworks align with your research objectives and the specific context of your study), and draw inspiration (i.e., even if existing frameworks do not fit your needs exactly, they can provide a starting point for developing your own).
  • The third step is to assess the theoretical alignment by ensuring the framework’s underlying assumptions align with your research philosophy. This involves thinking about the philosophical assumptions (e.g., your ontology, epistemology, and axiology) that underlie different frameworks and whether they are compatible with your view of the world.
  • The fourth step is to consider the methodological fit of each potential framework. The framework you choose should complement your research methodology, as it will influence the design, data collection, and analysis methods you will use.
  • The fifth step is to explore the flexibility of the framework to your specific research context. Some frameworks may be too rigid, limiting your ability to explore the nuances of your research problem fully. Others might offer the adaptability needed to address unique aspects of your study.
  • Finally, the sixth step is to reflect on the potential originality and contribution of the framework. Your chosen framework should enable you to make a meaningful contribution to your field. This might involve applying an existing framework in a new context, combining frameworks innovatively, or developing a new conceptual framework based on your findings.

Remember that context matters. Just because a theory is common in your field does not mean it is appropriate for your particular context. You might have to modify it by integrating theories or concepts relevant to your context, ensuring it more accurately reflects the cultural, political, religious, or additional aspects of your environment.

  • Conduct a context analysis: Understand the broader and specific conditions in which your research is situated.
  • Engage with local knowledge: Leverage insights from local experts and contextual literature to understand the context-specific nuances.
  • Evaluate frameworks for context sensitivity: Assess potential frameworks for their ability to accommodate or adapt to the specificities of your research context.
  • Consider the adaptability of the framework: Determine if the framework can be modified to better align with contextual demands without losing its theoretical integrity.

Incorporating context into the decision-making process for selecting a research framework underscores a commitment to conducting thoughtful, rigorous, and impactful research that resonates with its intended audience and addresses the complexities of the real world.

Some final thoughts…

The adoption of a well-defined framework in research is not merely a formal requirement but a cornerstone for ensuring the rigour, coherence, and depth of your study. A framework acts as a compass, guiding researchers through the complexities of their inquiry, from formulating research questions to interpreting findings. It provides a structured lens through which data can be examined, theories tested, and new insights uncovered, ensuring that research is not conducted in a vacuum but is connected to, and informed by, the wider scholarly community. Moreover, selecting and applying a framework necessitates a deep engagement with the subject matter and the methodological approaches best suited to exploring it. This engagement fosters a more nuanced understanding of the research problem and enriches the research design, ultimately enhancing the quality and impact of the findings.

Questions to ponder

What challenges might arise from applying a framework in a context different from where it was originally developed, and how can these be addressed?

Can a study be considered rigorous without a clearly defined theoretical framework? Why or why not?

Demystifying research paradigms

by

Posted:

31 Comments

The image shows a professional portrait of a woman with long, dark brown hair and glasses, smiling warmly. She is wearing a floral-patterned blouse under a black jacket, and the setting is softly lit, with an out-of-focus background featuring natural light and hints of greenery. The photograph conveys a friendly and approachable demeanour, suitable for professional or academic contexts.

Dr Lynette Pretorius

Dr Lynette Pretorius is an award-winning educator and researcher specialising in doctoral education, AI literacy, research literacy, academic identity, and student wellbeing.

Read the academic version of this blog post

Let’s talk about one of the most complex parts of research – understanding the philosophical underpinnings of your worldview and how this shapes the way your research is done. This is called a research paradigm and is one of the areas I get the most frequent questions about from graduate research students. The popularity of my research paradigm videos also show that this is clearly a concept that is difficult to understand and hard to articulate. So, in this blog post, I demystify what a research paradigm is, explore the most common research paradigms, demonstrate how a research paradigm influences the research process, and explain why it is important to articulate your paradigm in your research.

What is a research paradigm?

A research paradigm explains what you believe reality is and how you think knowledge can be understood. The technical terms used to articulate a research paradigm are ontology, epistemology, and axiology.

Ontology refers to the nature of reality. When people see reality as objective, it is called a realist ontology. When people see reality as subjective, it is called a relativist ontology. In other words, you need to explain whether you think reality is a singular, objective entity waiting to be discovered, or if it is a construct of individual experiences and perceptions.

Epistemology is concerned with the nature of knowledge and how we come to know something. There are three common epistemologies. First, reality can be objectively known if you can measure it with the right tools. Second, reality can only be partially known because it is created in the minds of individual people. Third, reality is always changing, so it is impossible to fully comprehend at any given moment.

Axiology deals with what is valued in research. It prompts us to consider how our own perspectives and values can shape the ways our research is designed and our findings are interpreted. In other words, you need to explain whether you think research should be value-free or whether you think research is value-laden. Do you think that steps should be taken to remove the researcher’s influence from the research, or is subjectivity inevitable?

What are the most common research paradigms?

I preface this discussion by noting that these are certainly not the only research paradigms that exist. Rather, these are just the ones that are most commonly found in research.

Positivism: Positivism is grounded in the belief that reality is singular and can be objectively observed and quantified. From this viewpoint, the researcher is independent of the subject of research, and knowledge is generated through empirical observation and measurement. Positivists believe that knowledge should be derived from empirical experience and logical reasoning, so they traditionally assert that research can and should be value-free.

Post-positivism: As the name suggests, this paradigm is closely related to positivism, having the same ontology and epistemology. However, post-positivists have a different axiology, arguing that complete objectivity is unattainable and that research cannot be entirely free from values, as choices about what to study, how to study it, and how findings are presented inevitably reflect value judgements. Consequently, they take steps to minimise the researcher’s influence in the way they design their studies by being reflexive about their own biases and the potential value implications of their work.

Constructivism: Constructivists believe that reality is subjective and that this reality can only be partially known because it is constructed in the minds of individual people. Constructivism emphasises that individuals’ realities are constructed through social interactions, so they seek to understand how individuals construct their realities and how these realities are influenced by social, cultural, and historical contexts. Constructivists embrace the value-laden nature of research, emphasising reflexivity and ethical responsibility, and valuing the co-construction of knowledge with participants. This paradigm acknowledges the subjective and interpretive nature of knowledge creation, viewing it as a strength that enriches understanding and contributes to the depth and authenticity of research findings. Constructivist researchers are also often concerned with the social implications of their research, seeking not only to understand the world but also to contribute positively towards social change.

Interpretivism: Interpretivists also believe reality is subjective and that reality can only be partially known because it is constructed in the minds of individual people. Interpretivism emphasises that individuals are the experts in their own experiences and focuses on understanding the subjective meanings and interpretations that individuals attach to their experiences. While interpretivism recognises the influence of researcher and participant values, it leans more towards understanding and interpreting the values and meanings inherent within the study context. Consequently, they believe that the researcher’s role is to immerse themselves in the participants’ experiences to gain a deep, empathetic understanding of their perspectives.

Pragmatism: Pragmatism is a flexible paradigm that suggests research methods should be chosen based on what best addresses the research question. Pragmatists believe reality is constantly changing or debated, so it does not commit to a single reality or method of inquiry. This allows for a mix of qualitative and quantitative methods. Pragmatism encourages an integrative approach to values, blending both subjective and objective perspectives. It recognises the importance of researchers’ and participants’ values and experiences in shaping research processes and outcomes. Yet, it also values objective evidence and the outcomes of research practices. This balance reflects pragmatism’s broader philosophical stance that truth and value are found in the practical implications of research and how it can address real-world problems.

Critical realism: Critical realists believe that an objective reality exists, but that it is layered, consisting of different levels that are not always observable. They believe our understanding of the world is subjective because it is influenced by societal and historical contexts and power relations. Critical realism is deeply intertwined with a commitment to understanding the world as it is, while also striving to transform it for the better. Critical realists believe that researchers bring their own beliefs, biases, and values to the research process, and urge researchers to critically reflect on and disclose their values and how these might impact their research.

I also recognise the importance of acknowledging that historically, some perspectives and bodies of knowledge have been overlooked or marginalised in research. This extends to the development and recognition of research paradigms as well. As such, I encourage you to also explore other diverse paradigms, especially those that may be considered ‘non-traditional’ or originate from non-Western contexts. Embracing a broader spectrum of paradigms can enrich our research approaches and outcomes, offering a wider range of valuable insights and perspectives.

How do research paradigms influence the research process?

Understanding and choosing a research paradigm is crucial because it shapes the entire research process—from framing the research question to deciding on methods, and interpreting the results. In essence, the research paradigm not only guides the technical aspects of study design, data collection, and analysis but also influences how researchers perceive and interact with their subject matter. It shapes the ethical considerations, the relationship with participants, and the ultimate goals of the research. By understanding and consciously choosing a research paradigm, researchers ensure that their work is coherent, meaningful, and aligned with their philosophical perspectives on reality, knowledge, and values. Let’s delve into how these philosophical underpinnings can shape each stage of the research process with some examples.

Formulating research questions

The paradigm a researcher adopts fundamentally shapes the nature of the questions they ask. For instance, in positivism, research questions are often framed to test hypotheses or measure variables, seeking to establish causal relationships or correlations. Constructivist, interpretivist, and critical realist paradigms, however, encourage questions that explore the meanings, experiences, and perceptions of participants, aiming to understand the complexity of human behaviour in particular contexts. These questions are then further refined based on the focus of the paradigm, leading to studies that focus on either societal change or a deeper understanding of lived experience. Pragmatism allows for flexibility, meaning research questions can be designed to solve specific problems, often leading to mixed-methods approaches.

Choosing methodologies and methods

The choice of methodologies and methods is also deeply influenced by the researcher’s paradigmatic stance. Positivists might lean towards structured methodologies that mimic the scientific method, employing quantitative measures such as surveys or experiments. Constructivists and interpretivists prefer qualitative methods like in-depth interviews, participant observation, or thematic analysis, which allow for a deep dive into participants’ lived experiences and the meanings they ascribe to them. Pragmatists select methods based on what best answers the research question, often combining qualitative and quantitative approaches in a pragmatic, problem-solving orientation. Critical realists might use a combination of qualitative and quantitative methods to explore both the surface phenomena and the underlying social or structural mechanisms contributing to these phenomena.

Data collection, analysis, and presentation of findings

Paradigms also dictate how data are collected and analysed, influencing the interaction with participants and the interpretation of data. Positivist approaches tend to emphasise objectivity and detachment, aiming for a neutral stance that minimises the researcher’s influence on the data. Findings are usually presented as objective truths or confirmed hypotheses, using statistical analysis to support conclusions. In contrast, constructivist and interpretivist paradigms view the researcher as a key instrument in the research process, engaging in reflective practices to interpret nuanced meanings within data, acknowledging their subjective influence. Findings are usually presented as insights into the participants’ perspectives, often narratively or through rich, descriptive accounts, emphasising the subjective nature of knowledge. The pragmatist paradigm focuses on practical outcomes, guiding the selection of data collection and analysis techniques that are most likely to produce actionable insights. Results are usually presented in a way that highlights their practical implications. The critical realist paradigm involves analysing data to identify not just what is happening but why it is happening, looking for patterns that reveal the influence of hidden structures or power relations. Findings are usually presented in a way which highlights or critiques the underlying societal mechanisms leading to a particular phenomenon.

Why is it important to explain your research paradigm?

Explaining the research paradigm in research publications or theses is pivotal for several reasons, each contributing to the clarity, integrity, and impact of the research. This explanation serves not just as a methodological formality but as a fundamental component that illuminates the researcher’s philosophical stance, guiding principles, and the rationale behind methodological choices. Here’s why delineating this paradigmatic foundation is crucial:

  • Enhancing transparency and trustworthiness: Detailing the research paradigm enhances the transparency of the study, allowing readers to understand the foundational assumptions that underpin the research. This clarity helps peers, reviewers, and readers assess the rigour or trustworthiness of the research process and its findings because it allows for an assessment of how well the research design, methodology, and methods align with the paradigmatic assumptions.
  • Justifying methodological choices: As noted above, the research paradigm informs the researcher’s methodological choices. By explaining this in their publications or theses, researchers provide a rationale for their methodological decisions. This explanation helps readers understand why certain methods were chosen over others and how these choices are consistent with the researcher’s philosophical stance, enhancing the coherence and integrity of the study.
  • Aiding interpretation of findings: Understanding the paradigmatic perspective of a study aids in the interpretation of its findings. Different paradigms can lead to different interpretations of the same phenomenon. By stating their research paradigm, researchers help readers place the findings within the appropriate conceptual and philosophical framework, ensuring that interpretations are made with an understanding of the underlying assumptions.
  • Contributing to disciplinary dialogue: The explicit articulation of research paradigms contributes to ongoing disciplinary dialogue and debate about the nature of reality and knowledge construction in a field. It encourages reflexivity among researchers, prompting them to consider and articulate their own positions. This reflexivity enriches the field by fostering a diversity of perspectives and approaches, leading to a more robust and dynamic academic discourse.
  • Helping others build upon your study: For those looking to replicate or build upon a study, understanding its paradigmatic underpinnings is essential. It ensures that subsequent research is grounded in the same philosophical assumptions, or consciously diverges from them, maintaining a level of methodological consistency and rigour across studies.

So in summary, understanding and explaining your research paradigms in publications or theses is not merely a theoretical exercise but a practical necessity that underpins the trustworthiness, clarity, and impact of research. It serves as a bridge connecting the philosophical foundations of a study with its practical execution and interpretation, enriching both the research process and its contributions to knowledge.

To cite this article, you can use this reference: Pretorius, L. (2024). Demystifying research paradigms: Navigating ontology, epistemology, and axiology in research. The Qualitative Report, 29(10), 2698-2715. https://doi.org/10.46743/2160-3715/2024.7632

Questions to ponder

Should research be value-free or is it value-laden?

In what ways do your ontology, epistemology, and axiology shape the way you conduct research?

Consider a recent study you’ve encountered. What research paradigm does it seem to align with, and how did this influence its findings and conclusions?

Accurately assessing students’ use of generative AI acknowledgements in assignments

by

Posted:

4 Comments

The image shows a professional portrait of a woman with long, dark brown hair and glasses, smiling warmly. She is wearing a floral-patterned blouse under a black jacket, and the setting is softly lit, with an out-of-focus background featuring natural light and hints of greenery. The photograph conveys a friendly and approachable demeanour, suitable for professional or academic contexts.

Dr Lynette Pretorius

Dr Lynette Pretorius is an award-winning educator and researcher specialising in doctoral education, AI literacy, research literacy, academic identity, and student wellbeing.

Lecturers play a pivotal role in shaping the learning of their students. In a metric-focused university environment, this learning necessitates the assessment of students’ learning throughout their educational journey. Assessing assignments not only gauges the understanding of the subject matter but also evaluates the development of critical academic skills. These skills, such as research, analysis, and effective communication, are integral components of a well-rounded higher education.

Assessing transferable skills

The skills assessed must align with what is taught within the unit. When students perceive a direct connection between what is taught and what is assessed, their engagement and comprehension are heightened. Consequently, if we are going to assess students not only on their content knowledge but also their transferable skills, we need to provide them with the tools to succeed.

I believe that transferable skills enhance the applicability of students’ disciplinary knowledge. For years, I have worked to develop a suite of academic skills resources which are now embedded across the units within our Faculty. These resources include a suite of just-in-time online videos freely available on YouTube, as well as two written booklets (Doing Assignments and Writing Theses) that explicitly teach academic communication skills.

Over the years, I have also worked to improve the assignment rubrics within our Faculty to more accurately assess the skills that are taught within individual units. For example, I have worked with another staff member to develop templates for staff to provide feedback on academic language and literacy. We designed these templates to allow assessors to label specific mistakes for students and to provide students with referrals to appropriate support. Giving students specific labels for their errors helps them to see where they can improve. The referrals to appropriate resources and support help the student improve their skills, encouraging self-directed learning.

It is important to note that we usually recommend that these skills account for no more than 10% of the total grade for the assignment. This is because the main focus of the assessment should be the content – students should be able to clearly demonstrate an understanding and critical evaluation of the topic of the assignment. However, the students’ use of academic language and academic literacy can enhance the quality of their disciplinary content, or it can hinder the meaning of their ideas. As such, our templates allow for 5% to be attached to academic language (specifically, the elements listed in blue here) and 5% to academic literacy (the elements listed in purple here).

Assessing AI literacy

In the era of rapid technological advancement, the rise of generative artificial intelligence (AI) introduces a new dimension to education. As students are increasingly exposed to AI tools, it becomes imperative for educators to teach them how to use these tools effectively. As I have highlighted in another blog post, I firmly believe that it is our role as educators to teach students how to collaborate effectively with AI and evaluate the results obtained, a concept termed AI literacy. I see AI literacy as an essential transferable skill.

A key component of using AI ethically is acknowledging it effectively in written work. It is important to highlight, though, that if we are going to require students to demonstrate AI literacy, including the accurate acknowledgement of the use of AI tools, we need to teach it in our units and also assess it accurately. In my units, I teach students that an acknowledgement should include the name of the AI used, a description of how it was used (including the prompt used where appropriate), and an explanation of how the information was then adapted in the final version of the document. I also provide students with the example below so that they can see how an acknowledgement is used in practice.

I acknowledge that I used ChatGPT (OpenAI, https://chat.openai.com/) in this assignment to improve my written expression quality and generate summaries of the six articles I used in the annotated bibliography section. The summary prompt provided to the AI was “Write a 350 word abstract for this article. Include a summary of the topic of the article, the methodology used, the key findings, and the overall conclusion”. I adapted the summaries it produced to reflect my argument, style, and voice. I also adapted the summaries to better link with my topic under investigation. When I wanted the AI to help me improve my writing clarity, I pasted my written text and asked it to rewrite my work “in less words”, “in a more academic style”, or “using shorter sentences”. I also asked it to explain why it made the changes it did so that I could use this collaborative discussion as a learning process to improve my academic communication skills. I take responsibility for the final version of the text in my assignment.

Clear guidelines within rubrics should also be established to evaluate the ethical and responsible use of AI, reinforcing the importance of acknowledging the role of these tools in academic work. Given my previous work developing rubric templates for staff, I have recently developed a template for the acknowledgement of AI use within assignments. In my template, this criterion falls within the “academic literacy” section of the rubric I mentioned earlier. I have included the rubric criteria below so that other educators can use it as needed. The grading scale is the one used in my university, but it can be easily adapted to other grading scales.

  • High Distinction (80-100%): There was an excellent explanation about how generative AI software was used. This included, where appropriate, explicit details about the software used, the prompts provided to the AI, and explanations as to how the output of the generative AI was adapted for use within the assignment.  
  • Distinction (70-79%): There was a clear explanation about how generative AI software was used. This included, where appropriate, sufficient detail about the software used, the prompts provided to the AI, and explanations as to how the output of the generative AI was adapted for use within the assignment. 
  • Credit (60-69%): There was a reasonably clear explanation about how generative AI software was used. The explanation lacked sufficient details regarding one of the following: the software used, the prompts provided to the AI, and/or explanations as to how the output of the generative AI was adapted for use within the assignment.
  • Pass (50-59%): There was some explanation about how generative AI software was used. The explanation lacked several of the following: the software used, the prompts provided to the AI and/or explanations as to how the output of the generative AI was adapted for use within the assignment.
  • Fail (Below 50%): There was little or no explanation about how generative AI software was used.

Questions to ponder

The blog post outlines a rubric for assessing the acknowledgement and use of generative AI in student assignments. Considering the varying levels of detail and adaptation of AI-generated content required for different grades, what are your thoughts on the fairness and effectiveness of this approach?

How might this rubric evolve as generative AI technology becomes more advanced and commonplace in educational environments?

Autoethnography: What is it and how do you do it?

by

Posted:

5 Comments

The image shows a professional portrait of a woman with long, dark brown hair and glasses, smiling warmly. She is wearing a floral-patterned blouse under a black jacket, and the setting is softly lit, with an out-of-focus background featuring natural light and hints of greenery. The photograph conveys a friendly and approachable demeanour, suitable for professional or academic contexts.

Dr Lynette Pretorius

Dr Lynette Pretorius is an award-winning educator and researcher specialising in doctoral education, AI literacy, research literacy, academic identity, and student wellbeing.

Autoethnography has become an increasingly popular research methodology, particularly within the humanities and social sciences. I use it regularly because of its emphasis on personal experiences, reflexivity, and storytelling which allows for a deeper exploration of complex experiences and societies. So what is autoethnography? The name autoethnography comes from three core aspects: self, culture, and writing. So, literally, autoethnography is an approach to research and writing that seeks to describe and systematically analyse personal experience to better understand cultural experiences.

Read more about the theoretical underpinning of autoethnography

As I’ve noted in a recent book chapter, there are several reasons why I find autoethnography a particularly compelling research methodology.

  • First, autoethnography allows researchers to purposely explore personal experiences to understand a particular culture or society. Researching personal experiences is becoming increasingly important as individuals’ stories are recognised as important sources of knowledge. Personal experiences can provide unique insights into social, cultural, and historical contexts and highlight the complexities of human experience.
  • Second, autoethnography considers insider knowledge as a valuable source of data. Researchers are the participants in their own studies and the stories which are told often explore transformative experiences for the researcher, frequently taking the form of epiphanies that significantly influenced the author’s worldview. I believe that this allows researchers to provide more meaningful insights into complex phenomena compared with more traditional objective research methods.
  • Third, autoethnography empowers researchers as it allows them to embrace emotionality and uncertainty and highlight topics that may be considered hidden or taboo. Autoethnography allows researchers to connect with their own emotions and experiences and, in doing so, find their voice. It allows them to challenge the dominant narratives that often dominate research and to tell their own stories in their own words.
  • And finally, autoethnography is a more accessible type of research for those outside of academia because it is written from personal experience in easy-to-understand language. The autoethnographer also does not merely narrate an experience for their audience. Instead, they try to engage the audience in the conversation so that the audience can understand experiences which may be different from their own. By sharing their own experiences, they can create a space for others to share theirs, fostering a more equitable and inclusive research process.

It is important to note that autoethnography does have some challenges. Some researchers critique it as a methodology because it is not scientific enough, while others say it is not artistic enough. I believe, however, that these critiques fail to see the value of combining both science and art when exploring complex phenomena. In this way, autoethnographers can advocate for social change to address perceived societal wrongs.

Read more about the benefits and challenges of autoethnography

So how do you actually do autoethnography in your research project? It is important to remember that there is no one way to do autoethnography. What is most important is to develop systematic data collection and analysis methods that help you deeply explore your personal experience.

First, it is important to have a series of reflective prompts to help you explore your experiences. I use a simple prompt strategy, which gives very open initial prompts to allow me to delve into my personal experiences, analyse my emotions and thoughts during that period, reflect on how I feel now, and determine how my previous experiences have impacted my current philosophy or practice.

  • Describing the experience
    • What happened?
    • What did I do?
  • Analysing the experience
    • What was I thinking and feeling?
    • How do I feel now?
    • What went well?
    • What could I have done better?
  • Creating a step-by-step plan
    • How will this information be useful in the future?
    • How can I modify my practice in the future?
    • What help do I need?

Second, you need a way to record your reflections. I like to start my reflection journey by voice or video recording a conversation I have with myself, thinking about my past experiences. I start by thinking about what happened, what I did, what I was thinking and feeling at the time, and how I feel now. Then, I explore how I think the experience has informed my way of being now. How has it shaped my future practice? Why? After finishing the recording, I transcribe the recording and use this transcription as my initial data. I have also recently used discussions and images created with generative AI as part of my autoethnography data generation process. If you want to see how that is done, you can watch this video or read the paper.

Third, you can also consult relevant artefacts as part of your autoethnography, such as photos and documents from the past to help you think and reflect more deeply about an experience. You can also consult other important figures, such as family or friends from your past, to help you see the experience from multiple viewpoints. A good example of an autoethnography that used artefacts as additional data sources can be found here. It is important to note that you will require ethics approval for your study if you use photos with other people in them or if the significant people you consult are possibly identifiable in your final project.

Fourth, you use the writing process as part of your reflection process.  Through the writing process, you further reflect on what you were thinking and feeling during the experiences you are describing. These reflections can remind you of other experiences that shaped your understanding of that experience. This continuous writing and re-writing of your story become further data sources that allow you to engage more deeply with your experiences. Remember to lean into your story’s more emotive and vulnerable parts, as this will allow you to uncover hidden perspectives in your understanding more effectively. Ask yourself, why did this experience make me feel this way? What does it tell me about the context I found myself?

Finally, as you write about your experiences, you should incorporate your theoretical analysis. Start looking for key concepts you have identified in your reflections and how they link to your overarching research problem. Which theoretical concepts do they reflect? What can others learn from your experience?

In conclusion, good-quality autoethnography explores personal experiences to illuminate a particular cultural context.  Autoethnography is not merely telling your story. It is analysing your story to uncover previously ignored perspectives within a particular research context.

Read a creative autoethnography
Read a narrative autoethnography

One final thing to note is that autoethnography can be done by one researcher or by a group of researchers. When done together, this type of autoethnography is called collaborative autoethnography. Collaborative autoethnography is particularly pertinent when examining complex social phenomena, such as marginalisation and the pursuit of social justice, as it facilitates the inclusion of multiple perspectives and voices. In this way, the individual voices of the researchers work together to illuminate common themes or experiences.

Read a collaborative autoethnography

Questions to ponder

Autoethnography emphasises the value of personal experiences in understanding cultural contexts. Reflect on an experience from your life that could offer unique insights into a particular cultural or societal aspect. How could analysing this personal experience using autoethnography enhance our understanding of broader cultural phenomena?

What are your thoughts on balancing the scientific rigour and artistic expression in autoethnography? Can you think of any specific situations or contexts where this methodology might be particularly beneficial or problematic?