<gh_stars>0
% Options for packages loaded elsewhere
\PassOptionsToPackage{unicode}{hyperref}
\PassOptionsToPackage{hyphens}{url}
%
\documentclass[
]{book}
\usepackage{amsmath,amssymb}
\usepackage{lmodern}
\usepackage{ifxetex,ifluatex}
\ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
  \usepackage[T1]{fontenc}
  \usepackage[utf8]{inputenc}
  \usepackage{textcomp} % provide euro and other symbols
\else % if luatex or xetex
  \usepackage{unicode-math}
  \defaultfontfeatures{Scale=MatchLowercase}
  \defaultfontfeatures[\rmfamily]{Ligatures=TeX,Scale=1}
\fi
% Use upquote if available, for straight quotes in verbatim environments
\IfFileExists{upquote.sty}{\usepackage{upquote}}{}
\IfFileExists{microtype.sty}{% use microtype if available
  \usepackage[]{microtype}
  \UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts
}{}
\makeatletter
\@ifundefined{KOMAClassName}{% if non-KOMA class
  \IfFileExists{parskip.sty}{%
    \usepackage{parskip}
  }{% else
    \setlength{\parindent}{0pt}
    \setlength{\parskip}{6pt plus 2pt minus 1pt}}
}{% if KOMA class
  \KOMAoptions{parskip=half}}
\makeatother
\usepackage{xcolor}
\IfFileExists{xurl.sty}{\usepackage{xurl}}{} % add URL line breaks if available
\IfFileExists{bookmark.sty}{\usepackage{bookmark}}{\usepackage{hyperref}}
\hypersetup{
  pdftitle={Assessment in Higher Education},
  pdfauthor={<NAME>},
  hidelinks,
  pdfcreator={LaTeX via pandoc}}
\urlstyle{same} % disable monospaced font for URLs
\usepackage{longtable,booktabs,array}
\usepackage{calc} % for calculating minipage widths
% Correct order of tables after \paragraph or \subparagraph
\usepackage{etoolbox}
\makeatletter
\patchcmd\longtable{\par}{\if@noskipsec\mbox{}\fi\par}{}{}
\makeatother
% Allow footnotes in longtable head/foot
\IfFileExists{footnotehyper.sty}{\usepackage{footnotehyper}}{\usepackage{footnote}}
\makesavenoteenv{longtable}
\usepackage{graphicx}
\makeatletter
\def\maxwidth{\ifdim\Gin@nat@width>\linewidth\linewidth\else\Gin@nat@width\fi}
\def\maxheight{\ifdim\Gin@nat@height>\textheight\textheight\else\Gin@nat@height\fi}
\makeatother
% Scale images if necessary, so that they will not overflow the page
% margins by default, and it is still possible to overwrite the defaults
% using explicit options in \includegraphics[width, height, ...]{}
\setkeys{Gin}{width=\maxwidth,height=\maxheight,keepaspectratio}
% Set default figure placement to htbp
\makeatletter
\def\fps@figure{htbp}
\makeatother
\setlength{\emergencystretch}{3em} % prevent overfull lines
\providecommand{\tightlist}{%
  \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
\setcounter{secnumdepth}{5}
\usepackage{booktabs}
\usepackage{amsthm}
\makeatletter
\def\thm@space@setup{%
  \thm@preskip=8pt plus 2pt minus 4pt
  \thm@postskip=\thm@preskip
}
\makeatother
\ifluatex
  \usepackage{selnolig}  % disable illegal ligatures
\fi
\usepackage[]{natbib}
\bibliographystyle{apa}

\title{Assessment in Higher Education}
\author{<NAME>}
\date{2021-10-04}

\begin{document}
\maketitle

{
\setcounter{tocdepth}{1}
\tableofcontents
}
\hypertarget{welcome}{%
\chapter*{Welcome}\label{welcome}}
\addcontentsline{toc}{chapter}{Welcome}

This site hosts the open development of my PhD dissertation at University of Victoria.

I'm working on getting the reference list to look a bit better and need to get my images on track.

\hypertarget{introduction}{%
\chapter*{Introduction}\label{introduction}}
\addcontentsline{toc}{chapter}{Introduction}

Assessing learning is both a critical component of the work of teaching in higher education (HE) and also a major factor in learners' experiences of HE \citep{biggsWhatStudentDoes1999, birenbaumAssessmentInstructionPreferences2007, woldeab21stCenturyAssessment2019}. Instructors in HE often inherit the assessment practices of their supervisors and instructors \citep{lipnevichWhatGradesMean2020}, potentially leading to over-reliance on practices that may not align with modern assessment theories \citep{boudChallengesReformingHigher2020, masseyAssessmentLiteracyCollege2020, shepardRoleAssessmentLearning2000}. These traditional practices tend to emphasize summative, selected-response tests \citep{floresPerceptionsEffectivenessFairness2015, harlenSystematicReviewImpact2002, lipnevichWhatGradesMean2020} which may be characterized by low reliability, leading to inferences which may not be valid reflections of learner achievement \citep{knightSummativeAssessmentHigher2002, smithglasgowStandardizedTestingNursing2019}. Also, the influence of an increasingly technological society is pushing higher education institutions (HEI) to integrate digital tools into the teaching and learning process \citep{pellegrinoPerspectivesIntegrationTechnology2010}. This trend saw a massive increase in activity as a result of the COVID-19 pandemic that forced most HEI in Canada to pivot to emergency remote teaching in the spring of 2020 through at least the summer of 2021 and perhaps beyond. While this increased use of technology may be understood by some as progress, research suggests that digital technologies typically are used in ways which mimic and re-inscribe traditional assessment practices and simply increase the efficiency with which they can be administered \citep{broadfootAssessmentTwentyFirstCenturyLearning2016}. Additionally, the needs of 21st century employers, who are looking for employees with demonstrated ability in collaboration, creative problem-solving, analytical thinking, and the ability to learn, competencies not easily measured in traditional testing formats, as opposed to those who have simply demonstrated high levels of domain-specific knowledge, have changed \citep{worldeconomicforumFutureJobsReport2020}. Finally, assessment practices have significant effects on learners including anxiety about high-stakes testing and their prospects for future studies and employment \citep{harlenSystematicReviewImpact2002}, and also the approaches they take to learning activities \citep{biggsTeachingQualityLearning2011, delucaDifferentialSituatedView2019}. The confluence of these influences suggests that traditional approaches to assessment in HEI are becoming inadequate to meet the needs of modern HEI and ought to be reconsidered.

\hypertarget{topic-of-the-research}{%
\section*{Topic of the Research}\label{topic-of-the-research}}
\addcontentsline{toc}{section}{Topic of the Research}

A starting point for this process of examining assessment in higher education ought to be intentionally considered as there are myriad examples of formal and informal initiatives in higher education which have not realized significant local or systemic change \citep{broadfootAssessmentTwentyFirstCenturyLearning2016, earlAssessmentLearningUsing2013}. Assessment practices are highly resistant to change, in part, as Broadfoot claims, because they are so important, but also, as will be discussed later, because the approaches that individual instructors take to assessing the work of learners are driven by complex forces \citep{blackAssessmentClassroomLearning1998, delucaDifferentialSituatedView2019, stigginsAssessmentLiteracy1991, willisConceptualisingTeachersAssessment2013}. Accordingly, it is incumbent upon researchers who wish to influence assessment practice to begin with a nuanced investigation of how HE instructors approach their assessment practice and what are the impacts of those practices on learners.

\hypertarget{problem-to-be-researched}{%
\section*{Problem to be Researched}\label{problem-to-be-researched}}
\addcontentsline{toc}{section}{Problem to be Researched}

In her important article, Shepard \citeyearpar{shepardRoleAssessmentLearning2000} argues that traditional assessment structures originated in past models of curriculum and instruction which were popular in the early 1900s. These curricular models emphasized the work of psychologists like Thorndike and Skinner who viewed the process of learning as being grounded in the mechanistic view of behaviourism where learning is the result of the precise and controlled input of `knowledge' and reinforced with rewards for correct responses. As such, instructors (appropriately) designed their assessments to align with the curricular goals of the time and assessed learning by determining whether or not a learner could provide the single correct response to a given question at a time removed from the instruction. However, in the latter half of the 20th century, when western psychologists discovered the ideas of <NAME> (who was actually a contemporary of the previously mentioned psychologists), curricula began to take a more social-constructivist approach that emphasized higher-order thinking, problem-solving in social contexts, and metacognitive skills over rote memorization. Unfortunately, it seemed that the efficiencies of testing memory, recognition, and recall through selected-response tests were too deeply embedded in the practices of HE instructors who resisted changing their assessments to match the new curricular goals. Shepard argues for the need to integrate assessment and instruction in such a way as to engage learners in authentic performance tasks more suited to modern understandings of cognition. It appears now that, in the twenty years since Shepard wrote her paper, the goals of early 21st century curricula have continued to diverge from those of the 20th century with the World Economic Forum identifying competencies in collaboration, analytical thinking, creative problem-solving, and the continual learning as being priorities for 21st century employers \citep{worldeconomicforumFutureJobsReport2020}. Consequently, models of assessment which prioritize testing skills in a manner aligned with 20th century curricular models are no longer adequate because they no longer align with the priorities of modern HE \citep{crooksImpactClassroomEvaluation1988}.

\hypertarget{purpose-of-the-research}{%
\section*{Purpose of the Research}\label{purpose-of-the-research}}
\addcontentsline{toc}{section}{Purpose of the Research}

Following previous research by DeLuca and colleagues \citep{delucaTeachersApproachesClassroom2016, delucaExploringAssessmentCultures2021, delucaApproachesClassroomAssessment2016} in the K-12 sector, and Massey et al \citeyearpar{masseyAssessmentLiteracyCollege2020} in the higher education sector, the purpose of this research is to investigate current assessment literacies and practices among higher education instructors and the impacts of those approaches on learners. In order to better respond to actual assessment practices, it is critical to understand the conceptions of HE instructors with respect to assessment \citep{delucaExploringAssessmentCultures2021, offerdahlChangesInstructorsAssessment2011}. Similarly, due to the significant influence assessment practices have on learners and learning, understanding the relationship between instructors' assessment conceptions and practices and the experience of learners will be important in order to provide a foundation for moving into the remainder of the 21st century with assessment practices aligned with both pedagogical models and learner contexts.

\hypertarget{research-questions}{%
\section*{Research Questions}\label{research-questions}}
\addcontentsline{toc}{section}{Research Questions}

Assessment in HE is a tremendously complex enterprise with myriad forces influencing the approaches instructors take in their assessment practice, which, in turn, becomes one of the myriad forces influencing the approaches learners take in their learning activities. Instructors conceptions of assessment are often derived from their past experiences as learners who were the objects of assessment, leading to an over-reliance on summative assessment strategies which may not obtain sufficient levels of reliability or validity. Increasing assessment literacy among HE instructors is a necessary, but not likely sufficient, step in advancing fair and balanced assessment practices in an increasingly technology-mediated HE landscape. These concepts lead to the following research questions:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  Are there distinct patterns in higher education instructors' approaches to assessment in Canada?
\item
  Does the prevalence of these patterns differ by:

  \begin{itemize}
  \tightlist
  \item
    instructors' levels of experience in teaching face-to-face versus online?
  \item
    instructors' levels of experience using technology?
  \end{itemize}
\item
  What factors influence instructors' approaches to assessment?
\item
  How do different assessment strategies affect learners' experiences?
\end{enumerate}

\hypertarget{significance-of-the-research}{%
\section*{Significance of the Research}\label{significance-of-the-research}}
\addcontentsline{toc}{section}{Significance of the Research}

This research will form the basis for an ongoing research agenda focused on research-supported strategies for assessment reform in Canadian HEI. Understanding both how instructors think about assessment and how learners are impacted by assessment decisions will be critical to informing assessment practices and policies as higher education emerges from the pandemic and moves forward into the 21st century.

\hypertarget{review-of-the-literature}{%
\chapter*{Review of the Literature}\label{review-of-the-literature}}
\addcontentsline{toc}{chapter}{Review of the Literature}

\hypertarget{structure-of-the-paper}{%
\section*{Structure of the Paper}\label{structure-of-the-paper}}
\addcontentsline{toc}{section}{Structure of the Paper}

This paper will begin with establishing a clear definition of assessment followed by an analysis of the literature related to the concept of \emph{assessment literacy} (AL) \citep{delucaDifferentialSituatedView2019} as it relates to the approaches to assessment \citep{delucaApproachesClassroomAssessment2016} taken by HE instructors and the perceptions and experiences of HE learners. The last section will introduce a framework for understanding assessment in HE, situated within Biggs' \citetext{\citeyear{biggsWhatStudentDoes1999}; \citeyear{biggsTheoryPracticeCognitive1993}} 3P model of teaching and learning, and research questions which emerge from the literature.

\hypertarget{defining-assessment}{%
\section*{Defining Assessment}\label{defining-assessment}}
\addcontentsline{toc}{section}{Defining Assessment}

There are deep and rich bodies of literature addressing educational assessment \emph{writ large}, both from a summative, psychometric perspective, and from a formative perspective. There is notably more research in the K-12 context, especially in relation to teacher preparation, compared to HE.

Among the more influential publications related to modern views of assessment (then usually called ``evaluation'') was Scriven's \citeyearpar{scrivenMethodologyEvaluation1967} article in which he drew distinctions between ``formative'' and ``summative'' evaluation. Formative evaluation was described as evaluation for the purposes of improvement, and summative evaluation was seen as a validation of the quality of work at the end of a process. This distinction was quickly incorporated into Bloom's \citeyearpar{bloomLearningMasteryInstruction1968} ideas related to mastery learning and began to be promoted as a model for educational reform. However, by the late 1990s, when Black and Wiliam \citeyearpar{blackAssessmentClassroomLearning1998} published their thorough review of the literature, the idea of formative assessment was still not well-defined or implemented. Black and Wiliam framed formative assessment as ``encompassing all those activities undertaken by teachers, and/or by their students, which provide information to be used as feedback to modify the teaching and learning activities in which they are engaged'' \citeyearpar[p.~7-8]{blackAssessmentClassroomLearning1998}. Although Black and Wiliam came to very strongly-stated conclusions about the value of formative assessments (e.g.~``The research reported here shows conclusively that formative assessment does improve learning. The gains in achievement appear to be quite considerable, and as noted earlier, amongst the largest ever reported for educational interventions.'' \citeyearpar[p.~61]{blackAssessmentClassroomLearning1998}), reliance on summative assessments in HE has remained high \citep[\citet{lipnevichWhatGradesMean2020}]{harlenSystematicReviewImpact2002}.

The National Research Council's (NRC) 2001 report \emph{Knowing what students know,} advanced understanding of assessment with their definition of assessment as ``a process of drawing reasonable inferences about what students know on the basis of evidence derived from observations of what they say, do, or make in selected situations'' \citep[p.~112]{pellegrinoKnowingWhatStudents2001} or, more simply, ``reasoning from evidence'' \citep[p.~43]{pellegrinoKnowingWhatStudents2001}, based on Mislevy's assertion that ``test theory is machinery for reasoning from students' behavior to conjectures about their competence, as framed in a particular conception of competence.'' \citeyearpar[ p.~4]{mislevyTestTheoryReconcieved1994}. Such a parsimonious description, however, may hide some of the complexities of fairly and equitably coming to know what learners know and can do in relation to particular outcomes. Since knowledge of a particular domain cannot be directly observed in a learner, and therefore cannot be quantified, instructors must rely on data gathered during the teaching process to support a particular inference about what a learner \emph{probably} knows. The data gathered from performance tasks such as exams, essays, portfolios, labs, etc, become \emph{evidence} when they support an inference about what a learner knows and can do. Hence, \emph{all} summative assessments are probablistic, not deterministic.

More recently, Earl \citeyearpar{earlAssessmentLearningUsing2013} further clarified the role assessment can play in learning by highlighting a distinction between assessment \emph{of} learning (summative assessment) and assessment \emph{for} learning (formative assessment by way of feedback) and also distinguishing both of those from assessment \emph{as} learning (a subset of assessment \emph{for} learning in which learners employ metacognitive skills to regulate their own learning tasks). Earl's delineation between types of assessment reflects the modern view that assessment and learning are, or ought to be, tightly integrated. This relationship will be explored in more detail in relation to Biggs' 3P Model of Teaching and Learning \citetext{\citeyear{biggsEnhancingTeachingConstructive1996}; \citeyear{biggsWhatStudentDoes1999}}.

The definitions of assessment above are typically understood as being \emph{classroom assessment}, language which is more readily applied to face-to-face K-12 learning environments as opposed to HE environments mediated by technology. For the purposes of this paper, I will consider assessment \emph{of} learning and summative assessment to be essentially synonymous, and I will differentiate between assessment \emph{for} and \emph{as} learning. I will use the term \emph{classroom assessment} to differentiate from \emph{large-scale assessment}, understood to be assessments deployed at levels above individual classrooms, such as school-, system-, or provincial/federal-levels, and I will use \emph{online assessment} or \emph{technology-mediated assessment} to refer specifically to classroom assessment in learning environments mediated by technology whether the learners are remote or not.

\hypertarget{conceptions-of-assessment}{%
\section*{Conceptions of Assessment}\label{conceptions-of-assessment}}
\addcontentsline{toc}{section}{Conceptions of Assessment}

Instructors in HE typically receive little formal preparation in either teaching practices or assessment during their graduate studies \citep{lipnevichWhatGradesMean2020, masseyAssessmentLiteracyCollege2020}. Consequently, their own practice tends to follow from what they experienced as learners, which likely emphasized high-stakes summative tests which were either in alignment with outdated pedagogical practices or out of alignment with modern pedagogical practices. These prior conceptions of assessment carry significant weight in how HE instructors approach the assessment of learners in their own courses.

Instructors bring to an educational environment a host of influences related to their institutional context, their past experiences with assessment, their own course policies, and affective beliefs or conceptions about assessment and its purposes \citep{biggsTheoryPracticeCognitive1993, brownQueenslandTeachersConceptions2011}. These influences play a significant role in determining the approaches taken by both instructors with respect to assessment and learners with respect to how they approach learning tasks \citep{brownQueenslandTeachersConceptions2011}. DeLuca et al. \citeyearpar{delucaEstablishingFoundationValid2013} argue that there are categories of conceptions exhibited by K-12 preservice teachers: assessment as testing, assessment as format, assessment purpose, and assessment as process. These conceptions are seen as increasingly complex, with those who see assessment as testing believing that assessment is primarily concerned with summative assessment of learning, usually using teacher-created selected-response tests. Those who see assessment as format tend to focus on whether the assessment is a ``performance, product, or objectively-scored assessment'' (p.~110). Assessment as purpose is delineated according to the summative/formative binary or Earl's \citeyearpar{earlAssessmentLearningUsing2013} assessment \emph{of/for/as} learning model. DeLuca et al.~also identify other purposes of assessment such as accountability, gatekeeping, and teacher evaluation. Lastly, assessment as process, which is based on the National Research Council's description of assessment being a process of reasoning from evidence \citeyearpar{pellegrinoKnowingWhatStudents2001}.

Fletcher et al. \citeyearpar{fletcherFacultyStudentsConceptions2012} used Brown's \citeyearpar{brownTeachersConceptionsAssessment2017} abridged \emph{Conceptions of Assessment} (CoA) questionnaire to measure learners' and instructors' conceptions as follows: ``assessment makes institutions accountable, assessment makes students accountable, assessment describes improvements in student abilities, assessment improves student learning, assessment improves teaching, assessment is valid, assessment is irrelevant and bad, assessment is irrelevant and ignored, and assessment is irrelevant and inaccurate'' (p.~122). They report that instructors were more likely than learners to view assessment as consistent and trustworthy methods to understand and improve learning and that learners were more likely to have negative views of assessment and viewed it as a measure of student and institutional accountability.

Massey et al. \citep{masseyAssessmentLiteracyCollege2020} used DeLuca et al.'s \citeyearpar{delucaEstablishingFoundationValid2013} framework of conceptions in their study of HE instructors' conceptions of assessment before and after an instructional development course focussed on assessment. They also considered the idea that there are two general orientations towards assessment in HE, an ``assessment culture'' and a ``testing culture'' \citep[as cited in \citet{masseyAssessmentLiteracyCollege2020}]{birenbaumAssessment2000Pluralistic1996, birenbaumNewInsightsLearning2003}. They report that they saw significant shifts in participants' conceptions of assessment from more simplistic views of assessment as testing pre-treatment, to more complex and nuanced views or assessment as process post-treatment.

From the literature, it is clear that instructors' conceptions of assessment are deeply influenced by many internal and external factors and, especially in HE, where there are few constraints on assessment practice \citep{lipnevichWhatGradesMean2020}, there are many ways to describe or delineate different conceptions of assessment. Accordingly, there are multiple ways to conceptualize the skills and dispositions that comprise the idea of \emph{assessment literacy}.

\hypertarget{assessment-literacy}{%
\section*{Assessment Literacy}\label{assessment-literacy}}
\addcontentsline{toc}{section}{Assessment Literacy}

The idea of AL is relatively recent in the K-12 literature and is nascent and under-theorized with respect to HE contexts \citep{medlandExaminingAssessmentLiteracy2015}. AL has been defined variously as ``the skills and knowledge teachers require to measure and support student learning through assessment'' \citep{delucaApproachesClassroomAssessment2016}, ``a basic understanding of educational assessment and related skills to apply such knowledge to various measures of student achievement'' \citep{xuTeacherAssessmentLiteracy2016}, ``an individual's understandings of the fundamental assessment concepts and procedures deemed likely to influence educational decisions'' \citep{pophamAssessmentLiteracyOverlooked2011} and ``a dynamic context-dependent social practice that involves teachers articulating and negotiating classroom and cultural knowledges with one another and with learners, in the initiation, development and practice of assessment to achieve the learning goals of students'' \citep{willisConceptualisingTeachersAssessment2013}. Key to these definitions are the ideas that AL is a complex, multi-faceted construct, that AL requires \emph{adequate} (not high) levels of psychometric or statistical analyses, and that it is intended to enable learner success.

The recognition of AL as a critical competency for educators was influenced by the growing demands for teacher and school accountability in the post-WWII era in the USA and Canada, particularly the \emph{Elementary and Secondary Education Act} (ESEA), passed in 1965, and the \emph{No Child Left Behind} (NCLB) act, passed in 2002 \citep[see][ for a detailed discussion]{delucaPreparingTeachersAge2012}. As such, conceptualizations of AL have tended to be based on sets of standards to which K-12 teachers are obligated. The first set of standards was the \emph{Standards for Teacher Competence in Educational Assessment of Students} (the \emph{Standards}), published by a committee of representatives from the American Federation of Teachers, the National Council on Measurement in Education, and the National Education Association \citep{StandardsTeacherCompetence1990}. The \emph{Standards} are a list of seven skills expected of teachers:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  Teachers should be skilled in \emph{choosing} assessment methods appropriate for instructional decisions.
\item
  Teachers should be skilled in \emph{developing} assessment methods appropriate for instructional decisions.
\item
  The teacher should be skilled in administering, scoring, and interpreting the results of both externally-produced and teacher-produced assessment methods.
\item
  Teachers should be skilled in using assessment results when making decisions about individual students, planning teaching, developing curriculum, and school improvement.
\item
  Teachers should be skilled in developing valid pupil grading procedures which use pupil assessments.
\item
  Teachers should be skilled in communicating assessment results to students, parents, other lay audiences, and other educators.
\item
  Teachers should be skilled in recognizing unethical, illegal, and otherwise inappropriate assessment methods and uses of assessment information.
\end{enumerate}

Shortly after the publication of the \emph{Standards}, the term \emph{assessment literacy} appeared in the literature with Stiggins' \citeyearpar{stigginsAssessmentLiteracy1991} article called \emph{Assessment Literacy}. Stiggins initial article was an account of his observation that teacher education programs at the time spent very little time training teachers in the methods and dispositions of educational measurement. Stiggins followed this with another article \citep{stigginsAssessmentLiteracy21st1995} where he outlined five characteristics of sound assessments, which:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  arise from and serve clear purposes;
\item
  arise from and reflect clear and appropriate achievement targets;
\item
  rely on a proper assessment method, given the purpose and the target;
\item
  sample student achievement appropriately; and
\item
  control for all relevant sources of bias and distortion. \citeyearpar[p.~240]{stigginsAssessmentLiteracy21st1995}
\end{enumerate}

At around the same time, a group of Canadian educators published the \emph{Principles for Fair Student Assessment Practices for Education in Canada}, Part A of which was a list of 37 guidelines related to five principles of fair student classroom assessment and was based on the 1990 \emph{Standards} (see Appendix A). Part B was focused on externally-developed standardized tests.

While these three sets of recommendations varied widely in their granularity, all tended to reflect an emphasis on the 20th century conceptions of curriculum which prioritized linear and sequential teaching of knowledge followed sometime later by selected-response tests of knowledge. This required teachers to be literate in the psychometric skills required to administer and interpret these tests \citep{delucaTeacherAssessmentLiteracy2016, shepardRoleAssessmentLearning2000, xuTeacherAssessmentLiteracy2016} or assessment \emph{of} learning. One notable distinction between the sets of standards is that the Canadian committee specifically noted their principles could be applied to K-12 as well as higher education, although the latter context would require changes in how assessment data are reported.

Twenty years following the publication of the \emph{Standards}, Brookhart \citeyearpar{brookhartEducationalAssessmentKnowledge2011} argued that the \emph{Standards} had become outdated because they did not address either the growing practices and ideas of formative assessment (assessment \emph{for} and \emph{as} learning) or standards-based assessment and that they needed to be revised. Brookhart suggested a list of 11 skills (see Appendix A) to adjust the focus of the 1990 \emph{Standards} to be in greater alignment with more modern conceptions of assessment.

Finally, in 2015, the Joint Committee on Standards for Educational Evaluation (JCSEE), with key representatives from both Canada and the USA, published the most recent set of standards, called the \emph{Classroom Assessment Standards for PreK-12 Teachers} (see Appendix A). The JCSEE standards are grouped into three broad domains (foundations, use, and quality), each with five or six related standards. Despite the similarities to the \emph{Principles for Fair Student Assessment Practices for Education in Canada}, including at least one common committee member, the JCSEE Standards are specifically not intended for use in HE.

As traditional conceptions of assessment and the standards expected of teachers, grounded in behaviourism and the need for objectivity tended to focus on assessment as a set of skill-based competencies to be employed by instructors, so AL could be defined as a set of sequential tasks in which instructors should engage to ensure objectivity and fairness (e.g. \citep{natrielloImpactEvaluationProcesses1987}). Recently, as curriculum and pedagogy have changed, several researchers have proposed models related to AL grounded in socio-constructivist views of learning \citep{delucaPreparingTeachersAge2012, pastoreTeacherAssessmentLiteracy2019, xuTeacherAssessmentLiteracy2016}. DeLuca's \citeyearpar{delucaPreparingTeachersAge2012} model, developed in the context of the \emph{No Child Left Behind} accountability mandate in K-12 schools in the USA, is a coherent lens through which to understand how a pre-service teacher could develop assessment expertise throughout their teacher education program. DeLuca frames AL within Fostaty Young and Wilson's \citeyearpar{fostatyyoungAssessmentLearningICE2000} ICE model which presents an integrated progression of learning through three levels of complexity, ideas, connections, and extensions. Underlying the three levels is a foundational level, where teacher candidates come to know and understand assessment as being situated within a model of teaching and learning, experiential and inclusive pedagogies, and an orientation towards continual professional learning. At the \emph{ideas} level, teacher candidates gain expertise in the big ideas related to assessment (theories of learning, planning educational experiences, classroom assessment, and issues of diversity and inclusion). At the \emph{connections} level, teacher candidates begin to construct their own cognitive conceptions of assessment as they integrate their own past experiences with their new knowledge and add their experience in practica. At the \emph{extensions} level, teacher candidates begin to hone their practice as they deepen their understanding of assessment.

Willis et al. \citeyearpar{willisConceptualisingTeachersAssessment2013} describe assessment literacy in alignment with Bernstien's \citeyearpar{bernsteinVerticalHorizontalDiscourse1999} idea that there are ``horizontal'' and ``vertical'' discourses (p.~159) with respect to assessment. A horizontal discourse is the local, contextualized discourse around assessment which influences local practice, while a vertical discourse is the formalized, structured discourse on assessment in the literature and other more authoritative venues. They argue that any given instructor's conceptions and practice of assessment will be informed by and negotiated within multiple horizontal and vertical discourses. Similarly, Xu and Brown's \citeyearpar{xuTeacherAssessmentLiteracy2016} model, teacher AL in practise (TALiP) presents a pathway for pre-service teachers to gain expertise in assessment. Based on a synthesis of 100 peer-reviewed publications between 1985 and 2015, they present a six-component model:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  the knowledge base
\item
  teacher conceptions of assessment,
\item
  institutional and socio-cultural contexts,
\item
  teacher assessment literacy in practice,
\item
  teacher learning, and
\item
  teacher identity as assessor.
  Finally, Pastore and Andrade \citeyearpar{pastoreTeacherAssessmentLiteracy2019} developed their model through a Delphi inquiry of 35 international experts in educational assessment and teacher education. They propose a model with three dimensions, conceptual, praxeological, and socio-emotional.
\end{enumerate}

The authors of each of these three models recognize that AL is conceptualized as a multi-dimensional construct encompassing psychometric skills, affective beliefs and values, external and regulatory environments, and socially negotiated practices. However, these models are largely specific to the K-12 environment in general, and more specifically related to the preparation of K-12 teachers. Given that there are very few parallels between the preparation of K-12 teachers for their role and the preparation of HE instructors for theirs, it seems that a framework for understanding AL among HE instructors should consider their general lack of formal preparation for teaching or assessing learning \citep{lipnevichWhatGradesMean2020, masseyAssessmentLiteracyCollege2020}.

Bearman et al. \citeyearpar{bearmanSupportAssessmentPractice2016} proposed a model for assessment decision-making in Australian HE, the Assessment Design Decisions Framwork (ADDF) in which the authors acknowledge the difficulty in translating idealized beliefs about assessment into actual practice as well as the lack of literature regarding \emph{how} HE instructors go about designing assessments. While not specifically an AL framework, there are overlaps in terms of the dimensions they identified. They argue, like those previously mentioned, that assessment is a complex and messy process. Unique to their model is the idea that there is often a disconnect between what instructors know or believe to be true about assessment, and how instructors' practice is impacted (or not) by their beliefs. Additionally, they argue, in alignment with Price et al. \citeyearpar{priceIfWasGoing2011}, that AL ought to be considered from both instructor and learner perspectives. Their framework is comprised of six dimensions:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  purposes of assessments
\item
  contexts of assessments
\item
  learner outcomes
\item
  tasks
\item
  feedback processes
\item
  interactions
\end{enumerate}

Herppich et al. \citeyearpar{herppichTeachersAssessmentCompetence2018} frame a model of assessment competence beginning with the idea that the purpose of educational assessment is to inform both formative (ongoing learning) and summative (credentialling or certification) decisions locating the educational decision subsequent and subordinate to the judgement. They use an example where a teacher observes a learner struggling with a test and might come to an appropriate judgement of a learner's knowledge, but make an inappropriate instructional decision based on that information. This distinction is useful when, like Herppich and colleagues, the construct under investigation is \emph{assessment competence}, which is observed \emph{after} instructional activities. However, in the present paper, \emph{conceptions of assessment} and \emph{AL} are the relevant constructs and they are embedded in a model of teaching and learning where conceptions of assessment \emph{precede} instructional activities.

DeLuca et al.'s model \citeyearpar{delucaApproachesClassroomAssessment2016}, \emph{approaches to classroom assessment} defines and conceptualizes the \emph{assessment} component of Biggs' 3P model \citetext{\citeyear{biggsWhatStudentDoes1999}; \citeyear{biggsTheoryPracticeCognitive1993}} and so aligns well with a concise model of teaching and learning in higher education. The \emph{Approaches to Classroom Assessment} model is based on the JCSEE standards \citep{klingerClassroomAssessmentStandards2015} and describes four themes of AL, each with three dimensions. The model represents somewhat of a break from previous models in that it references \emph{approaches to assessment} rather than \emph{assessment literacies}. This is a reflection of the authors' view that language around \emph{literacies} and \emph{competencies} may indicate a reliance on ``correct'' views or methods rather than the complex array of influences that lead to multiple legitimate approaches as identified in the literature \citep{delucaDifferentialSituatedView2019, willisConceptualisingTeachersAssessment2013}.

The themes DeLuca et al \citeyearpar[p.~10]{delucaExploringAssessmentCultures2021} describe along with their associated dimensions are listed below and illustrated in figure 2:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  Assessment purposes.\\
\end{enumerate}

\begin{itemize}
\tightlist
\item
  Assessment \emph{of} learning

  \begin{itemize}
  \tightlist
  \item
    Teachers' use of evidence to summate student learning and assign a grade in relation to students' achievement of learning objectives\\
  \end{itemize}
\item
  Assessment \emph{for} learning

  \begin{itemize}
  \tightlist
  \item
    Teachers' and students' use of evidence to provide feedback on progress towards learning objectives (i.e., inform next steps for learning and instructions).\\
  \item
    Involves both teacher-directed and student-centred approaches to formative assessment.\\
  \end{itemize}
\item
  Assessment \emph{as} learning

  \begin{itemize}
  \tightlist
  \item
    Focuses on how the student is learning by providing feedback or experiences that foster students' metacognitive abilities and learning skills (e.g., self-assessment, goal-setting, learning plans).\\
  \item
    Involves teachers but is primarily student-centred.\\
  \end{itemize}
\end{itemize}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\setcounter{enumi}{1}
\tightlist
\item
  Assessment process\\
\end{enumerate}

\begin{itemize}
\tightlist
\item
  Design

  \begin{itemize}
  \tightlist
  \item
    Focuses on the development of reliable assessments and items that measure student learning in relation to learning objectives.\\
  \end{itemize}
\item
  Use/scoring

  \begin{itemize}
  \tightlist
  \item
    Focuses of the adjustment and use of scoring protocols and grading schemes to respond to assessment scenarios.\\
  \end{itemize}
\item
  Communication

  \begin{itemize}
  \tightlist
  \item
    focuses on the interpretation of assessment results and feedback through communication to students and parents.\\
  \end{itemize}
\end{itemize}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\setcounter{enumi}{2}
\tightlist
\item
  Assessment fairness\\
\end{enumerate}

\begin{itemize}
\tightlist
\item
  Standard

  \begin{itemize}
  \tightlist
  \item
    Maintains the equal assessment protocols for all students.\\
  \end{itemize}
\item
  Equitable

  \begin{itemize}
  \tightlist
  \item
    Differentiates assessment protocols for formally identified students (i.e., special education or English language learners)\\
  \end{itemize}
\item
  Differentiated

  \begin{itemize}
  \tightlist
  \item
    Individualizes learning opportunities and assessments that address each student's unique learning needs and goals.\\
  \end{itemize}
\end{itemize}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\setcounter{enumi}{3}
\tightlist
\item
  Assessment theory\\
\end{enumerate}

\begin{itemize}
\tightlist
\item
  Consistent

  \begin{itemize}
  \tightlist
  \item
    Works to ensure consistency in results within assessments, across time periods, and between teachers.\\
  \end{itemize}
\item
  Contextual

  \begin{itemize}
  \tightlist
  \item
    Works to ensure assessment or evaluation measures what it claims to measure (i.e., learning objectives) and promote valid interpretations of results.\\
  \end{itemize}
\item
  Balanced

  \begin{itemize}
  \tightlist
  \item
    Works to ensure consistency in measuring what an assessment or evaluation intends to measure, and degree to which an assessment or evaluation measures what it claims to measure.
  \end{itemize}
\end{itemize}

\includegraphics{images/approaches-to-assessment.png}
Figure 2. Approaches to Classroom Assessment. \citep[p.~10]{delucaExploringAssessmentCultures2021}

\hypertarget{assessment-and-technology}{%
\section*{Assessment and Technology}\label{assessment-and-technology}}
\addcontentsline{toc}{section}{Assessment and Technology}

Educational technologies are often viewed and reported on with a distinct positivity bias \citep{irvineLandscapeMergingModalities2020} wherein `new' technologies are presumed to represent progress and will inevitably have a positive effect on learning. This can be seen in the titles given to some initiatives, such as ``Technology-Enhanced Assessment'' \citep{oldfieldAssessmentDigitalAge2012, timmisRethinkingAssessmentDigital2016}, ``IT-enabled assessment'' \citep{webbAssessmentTwentyFirstCentury2018}, or ``technology-rich'' \citep{linAssessingLearningTechnologyrich2020}. As such, I will use the more neutral term ``technology-mediated'' to indicate that adding digital technology to an assessment environment does not necessarily improve that environment. Similar to assessment practices being grounded in (both philosophically, as in `based upon', and figuratively, as in `stuck in') behaviourist conceptions of pedagogy leading to practices that rely heavily on summative approaches to assessment, so too, many educational technologies are grounded in (based upon and stuck in) behaviourist conceptions of pedagogy leading to practices that rely heavily on summative approaches to assessment. This can be seen in the progressively more advanced technologies beginning with Pressey's teaching machines \citep{benjaminHistoryTeachingMachines1988, presseyMachineAutomaticTeaching1927, wattersTeachingMachinesHistory2021} which was built to automate the process of ``drilling'' learners in an effort to teach them some concept. The machine needed to be pre-programmed with a series of selected-response questions along with distractors and correct answers. As a learner answered each question, the machine was programmed to match the response to the programmed correct response, and if it matched, the learner was determined to have ``mastered'' that question and it was dropped from the bank of questions the learner had not yet mastered. If it did not match, the question was cycled back into the bank to be repeated. Clearly, this technology was promoted as a tool to be used to modernize and increase the efficiency of tasks that aligned with the dominant pedagigical paradigm at the time. A second example, although not one marketed directly to schools, but to parents, was the \emph{Speak \& Spell}, released in 1978 by Texas Instruments \citep{braguinskiArchiveCommunicationInteractive2018, frantzSpeakSpell2014}, which represented an advance in technology and an increase in efficiency, as the \emph{Speak \& Spell} could be programmed to store and reproduce voice recordings of words as well as multiple recordings of feedback messages \citep{frantzSpeakSpell2014}. While the \emph{Speak \& Spell} was a leap forward in processing power, memory storage, and therefore complexity, the underlying pedagogy remained identical to that of Pressey's teaching machine \citep{wattersSpeakSpellHistory2015}. Moving forward again, and modern technologies are vastly more powerful than teaching machines or the \emph{Speak \& Spell} and power very complex adaptive tests, such as the NCLEX-RN, the national licensing exam for Registered Nurses in Canada and the USA \citep{smithglasgowStandardizedTestingNursing2019}. These advances in both hardware and software allow for still greater efficiencies in testing, yet the NCLEX-RN must still be programmed with selected-response questions, their distractors, and correct responses, all still in alignment with behavioural models of pedagogy. The NCLEX-RN is an example of a large-scale, standardized assessment (LSA), so is not parallel to the classroom assessment practices which are the subject of this paper, but I mention it here to draw the distinction between professionally-created LSAs and most instructor-created classroom assessments. The NCLEX-RN is continually revised and updated and reflects very robust psychometric properties (validity and reliability) \citep{smithglasgowStandardizedTestingNursing2019}. More importantly, however, the NCLEX-RN has been updated to ``shift away from a primary focus on content and the indirect testing of clinical judgment to a major focus on clinical judgment'' \citep[p.~2]{caputiReflectionsNextGeneration2019}. Caputi, in her reflections on the next-generation NCLEX-RN (NGN) asks 2 questions (the second of which is most relevant here): ``1) Are students ready for this type of NCLEX? 2) If our students already pass the NCLEX, can we keep doing the same type of preparation for the NGN?'' (p.~2). Her answer to both questions is ``No.'' She goes on to argue,

\begin{quote}
So, what can faculty do? I propose that nurse faculty, at all levels of nursing education, revise their curricula to teach a detailed thinking process that students must employ over and over throughout the nursing curriculum. Just as students practice psychomotor skills until they are perfected, they must do so with thinking skills and strategies A new model for teaching clinical judgment is needed", (p.~2, emphasis in original).
\end{quote}

In this case, the technology-mediated assessment instrument has been designed to measure 21st century skills, and Caputi recognizes that if pre-service nurses are going to have to pass the NGN, nursing instructors will need to realign their pedagogy. This example illustrates how LSAs exert pressure on instructors and schools to adjust their pedagogy, in the case of nursing, to encourage 21st century pedagogy that teaches thinking skills. The opposite is also true, however, in that LSAs which emphasize the lower-level cognitive skills prized by the behaviourists cause instructors to match their pedagogy to that model \citep{caputiReflectionsNextGeneration2019, clarke-miduraAssessmentTechnologyChange2010, delucaExploringAssessmentCultures2021, pellegrinoPerspectivesIntegrationTechnology2010}. Further, the American Educational Research Association (AERA), the National Council on Measurement in Education (NCME), and the American Psychological Association (APA) argue that LSAs tend to have other negative effects on education systems, namely the narrowing of curricula (teaching to the test), reduced instructional strategies (previously mentioned), higher dropout rates, and the enactment of policies and practices that increase test scores without increasing learning \citeyearpar{aeraStandardsEducationalPsychological2014}. The NCLEX-RN notwithstanding, many implementations of technology in assessment remain focused on increasing the efficiencies of summative test administration \citep{broadfootAssessmentTwentyFirstCenturyLearning2016, pellegrinoPerspectivesIntegrationTechnology2010, webbAssessmentTwentyFirstCentury2018}.

\hypertarget{impact-on-learners}{%
\section*{Impact on Learners}\label{impact-on-learners}}
\addcontentsline{toc}{section}{Impact on Learners}

Aside from the effects on learners' approaches to their learning, assessment, particularly summative assessment, has profound effects in other ways, such as determining status or progression in a course or program, determining eligibility for scholarships and awards, determining career paths and on other affective constructs such as motivation \citep{crooksImpactClassroomEvaluation1988} and anxiety \citep{birenbaumAssessmentInstructionPreferences2007, harlenSystematicReviewImpact2002}. Jones et al. \citeyearpar{jonesStudentWellbeingAssessment2021} report that assessment practices impact learners' well-being, which they define as including physical and mental health as well as the ability for learners to ``fully exercise their cognitive, emotional, physical and social powers, leading to flourishing'' (p.~439). They note that summative assessment practices are associated in the literature with ``anxiety, depression, disordered eating, self-harm, panic attacks, burnout, \ldots{} thoughts of suicide \ldots{} disordered sleep, loss of appetite, physical inactivity, poor physical health, \ldots{} substance misuse \ldots{} poorer productivity, motivation and test scores'' and that changing from norm-referenced scoring (where learners are ranked relative to each other) to criterion-referenced or pass-fail scoring is associated with lower levels of stress and anxiety among medical students in the USA.

\hypertarget{theoretical-framework-overview}{%
\section*{Theoretical Framework Overview}\label{theoretical-framework-overview}}
\addcontentsline{toc}{section}{Theoretical Framework Overview}

Given the wide variety of models, mostly in K-12 contexts but increasingly in HE, and the degree to which there appears to be overlap between those models, I take the position that introducing a new model into an already crowded landscape would serve little purpose. Furthermore, models do exist which have been validated empirically \citep{delucaApproachesClassroomAssessment2016} and the authors of which have indicated that, despite the model being developed for K-12 contexts, adjusting the model to fit the HE context would be a worthwhile contribution to the literature (personal communication).

Here, I present an argument that DeLuca et al.'s idea of \emph{approaches to classroom assessment} \citeyearpar{delucaApproachesClassroomAssessment2016} aligns well with the landscape of teaching, learning, and assessment in HE and may provide a blueprint for assessment reform in HE. In order to situate \emph{Approaches to Classroom Assessment} within the context of teaching and learning in HE, I begin with Biggs' \citep{biggsWhatStudentDoes1999, biggsTheoryPracticeCognitive1993} 3P model of teaching and learning (see Figure 3) which presents three stages of the teaching and learning process. In the first stage, \emph{presage}, Biggs models both learners and the teaching context. The learners' contexts include their previous learning, cognitive abilities, and their beliefs about the learning context, and all of these influence their preferred approaches to learning. The teaching context includes the instructors beliefs and conceptions of the curriculum and assessment, and also institutional and sometimes regulatory requirements, and each of these influence the instructor's approach to teaching, and more to the point for this paper, their approach to assessment. The \emph{process} stage represents the activities in which learners engage in order to meet the learning outcomes. Biggs notes that learners tend to either take \emph{surface} approaches, where they use low-level cognitive skills (memorization of facts) in activities that require high-level cognitive skills (analysis or critique), or they take deep approaches, where they use high-level cognitive skills for activities which require them. Learner activity in this stage is influenced by their own backgrounds, abilities, and affective views of the purpose of the task and it is influenced by the choices the instructor makes based on their presage influences. In this way, Earl's \citeyearpar{earlAssessmentLearningUsing2013} delineation between assessment \emph{of}, \emph{for}, and \emph{as} learning can be seen to influence student learning approaches. For example, researchers note that when instructors' approach prioritizes assessment \emph{of} learning, learners tend to take a surface approach to their learning \citep{birenbaumAssessmentInstructionPreferences2007}. Finally, the \emph{product} stage represents the learners' achievement in relation to the outcomes. Moving left to right, each of the first two stages represents an input into the system, and the third, an output. However, as the arrows between the components are two-way, the product stage also becomes an input, the results of which are fed back into the system (right to left) informing learners of their status in relation to the outcomes as well as the utility of their learning activities. It also informs the instructor about the nature of the learners' achievement and the utility of the learning tasks. The \emph{achievement} component provides formative feedback directly to both instructors and learners, informing both about next steps in the learning process. It also provides insight into the learning activities and, when that is fed back to learners, provides metacognitive cues as to the strategies learners might use in future tasks. It also provides information to instructors about the nature of the learning activities and how their approaches to assessment are impacting learning.

\includegraphics{images/3p-model.png}
Figure 3. 3P Model of Teaching and Learning. \citep{biggsWhatStudentDoes1999, biggsTheoryPracticeCognitive1993}

\hypertarget{method}{%
\chapter*{Method}\label{method}}
\addcontentsline{toc}{chapter}{Method}

\hypertarget{overview-of-the-problem-purpose-and-questions}{%
\section*{Overview of the Problem, Purpose, and Questions}\label{overview-of-the-problem-purpose-and-questions}}
\addcontentsline{toc}{section}{Overview of the Problem, Purpose, and Questions}

Research suggests that higher education faculty receive little formal preparation for assessing learning \citep{lipnevichWhatGradesMean2020} and that there may be misalignment between modern pedagogical practices and the approaches to assessment taken by higher education instructors \citep{knightSummativeAssessmentHigher2002, masseyAssessmentLiteracyCollege2020, shepardRoleAssessmentLearning2000}. The purpose of this paper will be to outline a methodological approach to investigating higher education instructors' assessment literacies and practices as well as the impact of those practices on learners. Assessment literacy is the result of a complex interplay of skills, beliefs, and values about four domains of assessment practice: (1) the purposes of assessment; (2) the processes associated with designing, administering, scoring, and communicating the results of assessment tasks; (3) fairness in assessment practices; and (4) assessment theory, each with three priority areas. \citep{delucaDifferentialSituatedView2019}.

Specific research questions to be addressed are:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  Are there distinct patterns in higher education instructors' approaches to assessment in Canada?
\item
  Does the prevalence of these patterns differ by:

  \begin{itemize}
  \tightlist
  \item
    instructors' levels of experience in teaching face-to-face versus online?
  \item
    instructors' levels of experience using technology?
  \end{itemize}
\item
  What factors influence instructors' approaches to assessment?
\item
  How do instructors' assessment approaches affect learners' experiences?
\end{enumerate}

In reviewing the extant literature concerning these topics and questions, and how researchers approach them, I searched the Educational Resource Information Center (ERIC), Google Scholar, and the University of Victoria Library Summon 2.0 databases. I also searched specific journals, such as Frontiers in Education, International Review of Research in Open and Distributed Learning, the Journal of Mixed Methods Research, Educational Measurement: Issues and Practice, and Assessment and Evaluation in Higher Education. Search terms included ``assessment literacy'', ``assessment'', ``online'', ``higher education'', ``learning'', and ``learner impact'' in various combinations. I also used \href{https://researchrabbit.ai}{researchrabbit.ai} to assist with tracking citations of seminal and other important articles and for visualizing the relationships between found articles.

\hypertarget{methodological-approaches}{%
\section*{Methodological Approaches}\label{methodological-approaches}}
\addcontentsline{toc}{section}{Methodological Approaches}

All research is grounded in a particular tradition or worldview, often called a \emph{paradigm}. A research paradigm is a set of philosophical assumptions and values about the universe and the ways in which we can learn and know about it \citep{johnsonEducationalResearchQuantitative2017}. According to Johnson and Christenson \citeyearpar{johnsonEducationalResearchQuantitative2017}, research paradigms are generally concerned with the answers to questions related to:
- how we learn about something (methodology),
- how we know something is true (epistemology)
- how we know reality, or if something exists (ontology),
- how we know if something is valuable or ethical (axiology), and;
- how we communicate and present arguments (rhetoric).

The three primary research paradigms are \emph{quantitative}, \emph{qualitative}, and \emph{mixed} research (often called \emph{mixed methods} research). Quantitative researchers traditionally followed the epistemological position of \emph{positivism}, characterized by the idea that knowledge is singular, objective, and can be discovered through the empirical analysis of numerical data \citep{heldDecolonizingResearchParadigms2019}, although most modern quantitative researchers take a \emph{post-positivist} view which recognizes that the values and beliefs of the researcher play a role in research and challenge the idea of pure objectivity \citep{tashakkoriFoundationsMixedMethods2020}. Quantitative research is often confirmatory in nature, with the researcher gathering data to either confirm or disconfirm a specific hypothesis \citep{johnsonEducationalResearchQuantitative2017}. Qualitative researchers, on the other hand, tend to take the epistemological view of \emph{constructivism} which posits that meaning is constructed by the researcher as they observe individuals and groups in relation to each other and particular phenomena. Qualitative research is exploratory in nature and uses non-numerical data such as texts, transcripts, images, and categories \citep{johnsonEducationalResearchQuantitative2017} in response to open-ended questions \citep{tashakkoriFoundationsMixedMethods2020}. During the latter half of the 20th century, when qualitative approaches were becoming more popular the research community tended towards a dualistic, `either/or' view of these two paradigms \citep{niglasMultidimensionalModelResearch2010, tashakkoriFoundationsMixedMethods2020} to the extent that the two paradigms were seen to be fundamentally incompatible. For example, Guba \citetext{\citeyear[p.~81 as quoted in]{gubaParadigmDialog1990}; \citealp[p.~14]{johnsonMixedMethodsResearch2004}} emphasized ``accommodation between paradigms is impossible \ldots{} we are led to vastly diverse, disparate, and totally antithetical ends''. Early in the 21st century, however, Johnson and Onwuegbuzie countered that argument and proposed that paradigms are better visualized on a continuum, that both paradigms are useful, and that mixing methods can ``draw from the strengths and minimize the weaknesses of both in single research studies and across studies'' \citeyearpar[p.~13-14]{johnsonMixedMethodsResearch2004}. So mixed research is characterized by an epistemologically \emph{pragmatic} view that both quantitative and qualitative approaches are valuable and that combining numerical and non-numerical data, analyses, and results can lead to deeper understanding \citep{bazeleyIntegratingAnalysesMixed2018, tashakkoriFoundationsMixedMethods2020}. More specifically, Johnson, et al. \citeyearpar{johnsonDefinitionMixedMethods2007} provide the following definition of mixed research:

\begin{quote}
Mixed methods research is the type of research in which a researcher or team of researchers combines elements of qualitative and quantitative research approaches (e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the broad purposes of breadth and depth of understanding and corroboration. (p.~123)
\end{quote}

I will use the term \emph{mixed research} in alignment with Johnson and Christensen \citeyearpar{johnsonEducationalResearchQuantitative2017} who advocate for that term rather than \emph{mixed methods research} because it is more than the method of research that is mixed. Indeed, mixing occurs at the level of paradigms, questions, methods, analyses, and reporting of results \citep{bazeleyIntegratingAnalysesMixed2018}.

Niglas \citeyearpar{niglasMultidimensionalModelResearch2010} contends that while the epistemological and philosophical stances of the researcher are important in informing the method of research, the more important considerations are the problem, purpose and questions the researcher intends to investigate. Similarly, Bazeley \citeyearpar{bazeleyIntegratingAnalysesMixed2018} argues that the most important factor that researchers should consider when choosing a methodology is ``whether the methods chosen and the strategies used \ldots{} serve the \emph{purpose} of the research'' (p.~8, emphasis in original). Following Bazeley's argument, and considering the purpose of this research project, a mixed approach is justified. Further justification is indicated in the diversity of approaches taken by researchers investigating assessment literacy or the impact of assessment on learners using quantitative \citep{delucaTeachersApproachesClassroom2016, delucaExploringAssessmentCultures2021, masseyAssessmentLiteracyCollege2020, nayaginPreserviceTeachersApproaches2020, pereiraHowUndergraduatesPerceive2021}, qualitative \citep{boudWhatFeedbackLiterate2021, coombsSeaSeaCanadian2020, delucaPedagogySlowSignificant2021, earleBalancingDemandsValidity2020, fivesNavigatingComplexCognitive2020, medlandAssessmentIlliterateShared2019, watsonSmallDataOnline2017}, and mixed \citep{delucaStudentPerspectivesAssessment2018, nicholsonEnhancingStudentEngagement2018, iannoneImpactHighStakes2020, tekirAlignmentIntendedEnacted2021} approaches.

\hypertarget{quantitative-approaches}{%
\subsection*{Quantitative approaches}\label{quantitative-approaches}}
\addcontentsline{toc}{subsection}{Quantitative approaches}

The construct of assessment literacy has a history of being grounded in sets of standards published by researchers, governments, or regulatory agencies and of being assessed through the collection and analysis of numeric and categorical data \citep{delucaTeachersApproachesClassroom2016, gotchSystematicReviewAssessment2014}. Gotch and French \citeyearpar{gotchSystematicReviewAssessment2014}, in their systematic review of the literature on assessment literacy inventories, found 36 instruments, all but one of which \citep{jarrEducationPractitionersInterpretation2012} gathered only quantitative (numeric and categorical) data. Studies based on these instruments (for example, Alkharusi, et al. \citeyearpar{alkharusiEducationalAssessmentAttitudes2012} based on Plake, et al. \citeyearpar{plakeAssessmentCompetenciesTeachers2005}) generally included quantitative analyses because the data gathered are quantitative. Early instruments \citep{mertlerMeasuringTeachersKnowledge2005, plakeTeacherAssessmentLiteracy1993} were based on the 1990 \emph{Standards for Teacher Competency in Educational Assessment of Students} \citep{StandardsTeacherCompetence1990}. Since then, Brookhart \citeyearpar{brookhartEducationalAssessmentKnowledge2011} published a critique of the 1990 standards arguing that they were out of date and not in alignment with modern pedagogy. This was followed a few years later by Gotch and French \citeyearpar{gotchSystematicReviewAssessment2014} publishing the aforementioned systematic review, during which they found that the psychometric properties of the 36 instruments published between 1991 and 2012 were low. Finally, Klinger et al. \citeyearpar{klingerClassroomAssessmentStandards2015} published the \emph{Classroom Assessment Standards for PreK-12 Teachers} \citep{klingerClassroomAssessmentStandards2015}. These cumulative advances led to DeLuca et al. \citeyearpar{delucaTeacherAssessmentLiteracy2016} to develop the \emph{Approaches to Classroom Assessment Inventory (ACAI)} and enabled researchers to base their quantitative investigations on an instrument shown to produce valid results in relation to a modern model of assessment literacy.

There are numerous examples of research teams using quantitative approaches to investigate assessment literacy, some of which are mentioned here. DeLuca et al. \citeyearpar{delucaTeachersApproachesClassroom2016} surveyed 404 K-12 teachers in Canada and the USA using DeLuca et al.'s \citeyearpar{delucaApproachesClassroomAssessment2016} \emph{Approaches to Classroom Assessment Inventory (ACAI)}, an instrument based on the 2015 Standards. DeLuca et al. \citeyearpar{delucaExploringAssessmentCultures2021}, recently used the same inventory in an international study comparing assessment practices in the USA, Canada, and China. Nayagi and Rajendran \citeyearpar{nayaginPreserviceTeachersApproaches2020} also used the ACAI to investigate approaches to assessment, although they modified it for their context of pre-service teachers in India. Massey et al. \citeyearpar{masseyAssessmentLiteracyCollege2020} surveyed higher education instructors to investigate their approaches to and confidence with assessment. Lastly, with respect to the impact of instructors' assessment practices on learners, Pereira et al. \citeyearpar{pereiraHowUndergraduatesPerceive2021}, Pereira et al. \citeyearpar{pereiraPerceptionsPortugueseUndergraduate2017}, and Flores et al. \citeyearpar{floresPerceptionsEffectivenessFairness2015} used surveys to collect quantitative data about the perceptions of Portuguese undergraduate learners regarding the assessment practices of their instructors. While there are more studies of assessment literacy in the K-12 sector \citep{medlandAssessmentIlliterateShared2019}, there is a growing number in the higher education sector (e.g. \citep{masseyAssessmentLiteracyCollege2020, nayaginPreserviceTeachersApproaches2020, pereiraHowUndergraduatesPerceive2021}).

Research questions 1 and 2 in the present study are intended to answer questions about the assessment literacies of higher education instructors in Canada, a construct which has been shown to be amenable to quantitative survey approaches in diverse settings, including K-12 teachers, pre-service teachers, and higher education instructors both in Canada and internationally.

\hypertarget{qualitative-approaches}{%
\subsection*{Qualitative approaches}\label{qualitative-approaches}}
\addcontentsline{toc}{subsection}{Qualitative approaches}

Assessment literacy has been the focus of significant attention in quantitative research, but it is also becoming clear that the construct is much deeper and more complex than models based on sets of standards and competencies \citep{delucaPoliciesProgramsPractices2019, delucaPoliciesProgramsPractices2019, willisConceptualisingTeachersAssessment2013, xuTeacherAssessmentLiteracy2016}. Willis \citeyearpar{willisConceptualisingTeachersAssessment2013} framed assessment literacy within multiple ``horizontal and vertical structures of educational discourse'' following Bernstein \citeyearpar{bernsteinVerticalHorizontalDiscourse1999}, who described vertical discourses as structured disciplinary knowledge passed down from specialists, and horizontal discourses as more contextualized knowledge practiced and passed along at a local level. Following on this idea, DeLuca et al. \citeyearpar{delucaDifferentialSituatedView2019} describe assessment literacy as being a ``situated and differential practice predicated on negotiated knowledges'' (p.~7) and elsewhere, DeLuca et al. \citeyearpar{delucaPoliciesProgramsPractices2019} argue that ``Assessment capability involves situated professional judgement, that is the ability to draw on learning and assessment theories and experiences to purposefully design, interpret, and use a range of assessment evidence in the service of student learning'' (p.~2). Similarly, Xu and Brown's \citeyearpar{xuTeacherAssessmentLiteracy2016} model, \emph{Teacher Assessment Literacy in Practice (TALiP)}, explicitly includes socio-cultural factors, teacher judgement, and teacher identity.

The methodological implication of this shift to a more complex understanding of assessment literacy is that there seems to have been an attendant shift in researchers engaging in qualitative investigations of assessment literacy \citep{bearmanHowUniversityTeachers2017, delucaPedagogySlowSignificant2021, fivesNavigatingComplexCognitive2020, medlandAssessmentIlliterateShared2019, watsonSmallDataOnline2017}. Qualitative methodologies exist because of the view that, according to Guba and Lincoln \citeyearpar{gubaFourthGenerationEvaluation1989}, reality is constructed through social processes, and is not simply observed in the world. In my view, this epistemological stance within the community of qualitative researchers aligns well with the nature of assessment literacy as a phenomenon impacted by horizontal discourses. In my own review of the literature on assessment literacy, there seems to be a marked increase in the number of qualitative studies published since 2016, which may reflect a deepening understanding of the characteristics of assessment literacy.

While the trend to use qualitative approaches to investigate assessment literacy and the impact of assessment on learners is more recent compared to quantitative approaches, there are good examples of qualitative studies, some of which I briefly describe here. Watson et al. \citeyearpar{watsonSmallDataOnline2017} published a case study of how instructor assessment practices impacted a single learner who struggled through a graduate-level learning module. Benediktsson and Ragnarsdóttir \citeyearpar{benediktssonImmigrantStudentsExperiences2020} used focus group interviews followed by semi-structured interviews with individual learners to investigate the assessment experiences of immigrant learners in Icelandic universities. Bearman et al. \citeyearpar{bearmanHowUniversityTeachers2017} interviewed thirty-three higher education instructors from a variety of disciplines and institutions to investigate how they designed assessments and found that there is a need for professional development activities about assessment to have a relational focus. In the UK context of having external peers provide regulatory feedback, Medland \citeyearpar{medlandAssessmentIlliterateShared2019} used an exploratory case study approach to investigate the assessment literacy of external examiners in UK higher education. In another case study with a single subject, Fives and Barnes \citeyearpar{fivesNavigatingComplexCognitive2020}, recognizing the complexity of classroom assessment, investigated the assessment-related routines of an experienced teacher. Their goal was to engage in a close examination of how an experienced teacher enacted assessment practices in their classroom in order to ``make explicit {[}the{]} cognitive tasks involved in their work'' (p.~3). More recently, DeLuca et al.~(\citeyearpar{delucaPedagogySlowSignificant2021}) used a case study methodology to analyze data gathered from interviews, reflections, and course work to investigate the learning trajectories of 35 pre-service teachers during their program. Each of these studies was exploratory in nature, and/or designed specifically to uncover deeper meanings than could be discerned with the use of a survey or other quantitative instrument, which would require the researcher to anticipate and direct respondents toward a limited set of possible answers.

Research questions 3 and 4 of the present study are intended to explore the qualitative and subjective experiences of instructors, what factors influence their approaches to assessment and the experiences of learners as they engage in assessment activities. Both of these questions will require responses to open-ended questions and will be exploratory in nature, aligning with a qualitative paradigm and epistemology.

\hypertarget{mixed-approaches}{%
\subsection*{Mixed approaches}\label{mixed-approaches}}
\addcontentsline{toc}{subsection}{Mixed approaches}

Researchers have argued that using mixed approaches to research is a way to strengthen and increase confidence in the validity of results by maximizing the strengths of each paradigm while minimizing the weaknesses \citep{creswellResearchDesignQualitative2009, mckimValueMixedMethods2017}. Creamer \citeyearpar{creamerIntroductionFullyIntegrated2018} argues that mixing of qualitative and quantitative methods is the defining characteristic of mixed research. She defines mixing as ``\emph{the linking, merging, or embedding of qualitative and quantitative strands of a mixed methods study}'' (p.~6, emphasis in original). In a departure from the historical paradigm wars between quantitative and qualitative approaches, Niglas \citeyearpar{niglasMultidimensionalModelResearch2010} contends for the idea that mixed research studies fall on a continuum between qualitative and quantitative approaches with varying degrees of epistemological stances and methods. Given the abundance of literature exploring instructor assessment literacy and practices and the impact of those practices on learners from both quantitative and qualitative paradigms, it is not surprising that there is also a body of literature investigating these constructs using mixed research approaches \citep{esfandiariMixedmethodsCrosssectionalStudy2016, ogan-bekirogluPreServiceTeachers2014, solomonidouStudentsConceptionsAssessment2017}. Creamer \citeyearpar{creamerIntroductionFullyIntegrated2018} suggests evaluating mixed research projects using a table to make the structure of the study more plainly visible. The following tables are overviews of selected mixed research projects investigating assessment literacy in K-12 and higher education.

\textbf{Table 1}

\emph{Overview of }``Students' conceptions of assessment purposes in a low stakes secondary-school context: A mixed methodology approach''* \citep{solomonidouStudentsConceptionsAssessment2017}*

\begin{longtable}[]{@{}
  >{\raggedright\arraybackslash}p{(\columnwidth - 6\tabcolsep) * \real{0.25}}
  >{\raggedright\arraybackslash}p{(\columnwidth - 6\tabcolsep) * \real{0.25}}
  >{\raggedright\arraybackslash}p{(\columnwidth - 6\tabcolsep) * \real{0.25}}
  >{\raggedright\arraybackslash}p{(\columnwidth - 6\tabcolsep) * \real{0.25}}@{}}
\toprule
Study Component & Details & Yes/No & Comments \\
\midrule
\endhead
Rationale/Purpose for Mixing & Enhancement & & \\
Priority & QUAN → qual & & \\
Timing of Data Collection & Sequential & & \\
Timing of Data Analysis & Sequential & & \\
Mixing & & & \\
& Design & Yes & Qualitative data were used to confirm tentative inferences from quantitative analysis. \\
& Data Collection & Yes & Semi-structured interview questions were revised in light of quantitative inferences, and participants could view their survey answers during their interviews. \\
& Data Analysis & Yes & Blended in the report. \\
& Inferences & Yes & Blended in the report. \\
& Fully Integrated & Yes & \\
Qualitative Inference & Learners value assessment information but it takes work to use the information formatively. & - & - \\
Quantitative Inference & Learners agreed more with the idea that assessment is a tool for improvement and less with the idea that assessment is negative. & & \\
Meta-inference & Teacher education and development should systematically include alternative assessment practices. & & - \\
Value Added & Linking data from both surveys and interviews provided opportunity for learners to elaborate on their views and provide a richer picture of their conceptions of assessment. & & \\
\bottomrule
\end{longtable}

\textbf{Table 2}

\emph{Overview of }``A Mixed-methods, Cross-sectional Study of Assessment Literacy of Iranian University Instructors: Implications for Teachers' Professional Development''* \citep{esfandiariMixedmethodsCrosssectionalStudy2016}*

\begin{longtable}[]{@{}
  >{\raggedright\arraybackslash}p{(\columnwidth - 6\tabcolsep) * \real{0.25}}
  >{\raggedright\arraybackslash}p{(\columnwidth - 6\tabcolsep) * \real{0.25}}
  >{\raggedright\arraybackslash}p{(\columnwidth - 6\tabcolsep) * \real{0.25}}
  >{\raggedright\arraybackslash}p{(\columnwidth - 6\tabcolsep) * \real{0.25}}@{}}
\toprule
Study Component & Details & Yes/No & Comments \\
\midrule
\endhead
Rationale/Purpose for Mixing & Complementarity & & \\
Priority & QUAN → qual & & \\
Timing of Data Collection & Sequential & & \\
Timing of Data Analysis & Sequential & & \\
Mixing & & & \\
& Design & Yes & The qualitative phase was used to elaborate on the quantiative phase. \\
& Data Collection & No & \\
& Data Analysis & Yes & Qualitative analysis filled gaps in quantitative inferences. \\
& Inferences & Yes & Blended in the report. \\
& Fully Integrated & No & \\
Qualitative Inference & Subgroups of participants assessed learners differently, used assessment for different purposes, had different levels of training in assessment, and different views on the psychometric properties of tests. & & \\
Quantitative Inference & Results showed support for a three-component model of assessment literacy. & & \\
Meta-inference & Assessment literacy is a multi-dimensional concept affected my many factors, including teaching experience, professional development and training, and the local teaching context. & & \\
Value Added & Qualitative analysis was critical to helping the researchers understand the differences seen between subgroups in the quantitative analysis. & & \\
\bottomrule
\end{longtable}

\textbf{Table 3}

\emph{Overview of ``Pre-service teachers' assessment literacy and its implementation into practice'' \citep{ogan-bekirogluPreServiceTeachers2014}}

\begin{longtable}[]{@{}
  >{\raggedright\arraybackslash}p{(\columnwidth - 6\tabcolsep) * \real{0.25}}
  >{\raggedright\arraybackslash}p{(\columnwidth - 6\tabcolsep) * \real{0.25}}
  >{\raggedright\arraybackslash}p{(\columnwidth - 6\tabcolsep) * \real{0.25}}
  >{\raggedright\arraybackslash}p{(\columnwidth - 6\tabcolsep) * \real{0.25}}@{}}
\toprule
Study Component & Details & Yes/No & Comments \\
\midrule
\endhead
Rationale/Purpose for Mixing & Triangulation and Enhancement & & \\
Priority & QUAN → qual+qual & & \\
Timing of Data Collection & Sequential & & \\
Timing of Data Analysis & Side-by-side comparison (concurrent) & & \\
Mixing & & & \\
& Design & Yes & The qualitative phase was used to validate and elaborate on the quantitative phase. \\
& Data Collection & No & \\
& Data Analysis & Yes & Qualitative and quantitative data were analyzed concurrently by presenting them side-by-side for comparison. \\
& Inferences & Yes & Blended in the report. \\
& Fully Integrated & No & \\
Qualitative Inference & Qualitative analysis showed alignment with quantitative analysis & & \\
Quantitative Inference & Pre-service teachers' assessment literacy was determined to be ``close to constructivist'' (p.~352) & & \\
Meta-inference & Although participants were able to demonstrate knowledge of assessment literacy, they had difficulty translating that knowledge into practice. & & \\
Value Added & Qualitative analysis was important for determining the difficulty in translating knowledge of assessment literacy into practice of good assessment. & & \\
\bottomrule
\end{longtable}

\textbf{Table 4}

\emph{Overview of ``A mixed method exploration of student perceptions of assessment in nursing and biomedicine'' \citep{garveyMixedMethodExploration2021}}

\begin{longtable}[]{@{}
  >{\raggedright\arraybackslash}p{(\columnwidth - 6\tabcolsep) * \real{0.25}}
  >{\raggedright\arraybackslash}p{(\columnwidth - 6\tabcolsep) * \real{0.25}}
  >{\raggedright\arraybackslash}p{(\columnwidth - 6\tabcolsep) * \real{0.25}}
  >{\raggedright\arraybackslash}p{(\columnwidth - 6\tabcolsep) * \real{0.25}}@{}}
\toprule
Study Component & Details & Yes/No & Comments \\
\midrule
\endhead
Rationale/Purpose for Mixing & Triangulation & & \\
Priority & QUAN+QUAL & & \\
Timing of Data Collection & Concurrent & & \\
Timing of Data Analysis & Consecutive & & \\
Mixing & & & \\
& Design & Yes & The qualitative phase was used to validate and elaborate on the quantitative phase. \\
& Data Collection & Yes & Open-ended (qualitative) responses were gathered along with quantitative responses and were designed to elicit clarifying views. \\
& Data Analysis & No & \\
& Inferences & Yes & Blended in the report. \\
& Fully Integrated & No & \\
Qualitative Inference & Qualitative analysis deeper levels of complexity compared to quantitative analysis. & & \\
Quantitative Inference & First-year university learners indicated positive views of assessment, regardless of their program. & & \\
Meta-inference & Universities should recognize that first-year learners tend to have positive views of assessment and should communicate often with them about the purposes of assessment. & & \\
Value Added & Qualitative analysis showed areas of misalignment with the quantitative analysis. & & \\
\bottomrule
\end{longtable}

Each of the preceding four studies shows different ways in which mixed research can enhance validity or provide avenues for deeper and more complex understanding of instructors' assessment literacy and practices and the impact of those on learners. The tables illustrate the complexity of mixed studies and highlight the challenges associated with integrating quantitative and qualitative approaches at each stage of the research. However, even those studies which do not demonstrate all of the qualities of fully integrated mixed research contain valuable insights based on the integration of the two analyses.

\hypertarget{epistemological-alignment-with-purpose-of-the-research}{%
\subsection*{Epistemological Alignment with Purpose of the Research}\label{epistemological-alignment-with-purpose-of-the-research}}
\addcontentsline{toc}{subsection}{Epistemological Alignment with Purpose of the Research}

The purpose of this project is to investigate a multi-dimensional construct, assessment literacy, that is influenced by factors both internal (for example: beliefs, values, and past experiences) and external (for example: assessment culture \citep{delucaExploringAssessmentCultures2021, masseyAssessmentLiteracyCollege2020} or government regulations) to instructors \citep{delucaDifferentialSituatedView2019}. In addition, the project will investigate the factors that influence instructors in higher education to take particular approaches to their assessment practice and how those practices impact learners. In envisioning this multi-dimensional research project and formulating the research questions, I recognize value in both the discovery and the creation of meaning in the research process, a pragmatic epistemology that aligns with the ethos of many mixed researchers. In alignment with the examples from the literature described in the preceding sections, this project will draw from both quantitative and qualitative paradigms, will use both quantitative and qualitative data connected in multiple ways, engage in analyses in alignment with both traditions, and will present results as inferences from both quantitative and qualitative analyses in addition to meta-inferences \citep[p.~62]{bazeleyIntegratingAnalysesMixed2018} from the integration of analyses. As such, I propose a mixed research investigation, which will be described in greater detail in the following sections.

\hypertarget{visualization-of-a-mixed-research-project}{%
\section*{Visualization of a Mixed Research Project}\label{visualization-of-a-mixed-research-project}}
\addcontentsline{toc}{section}{Visualization of a Mixed Research Project}

Researchers who use mixed approaches often include visual overviews of their research project as mixed research projects can be very complex \citep{creamerIntroductionFullyIntegrated2018}. Figure 1 below is a visual representation of the present project. It is helpful to understand some of the conventions of visualizing mixed research projects as seen in figure 1. According to Tashakkori et al. \citeyearpar{tashakkoriFoundationsMixedMethods2020}, quantitative components are depicted as ovals and qualitative components are depicted as rectangles. Additionally, in figure 1, barred rectangles indicate mixed inferences. Solid arrows indicate the general direction of the workflow throughout the project and dashed arrows represent potential feedback loops where the results and inferences from one phase inform one or more components of a subsequent phase. Mixed researchers use upper- and lower-case letters to indicate the priority in any given phase. QUAN (all-caps) indicates that quantitative data and analyses are prioritized, while qual (lower-case) indicates that qualitative data and analyses are secondary. Further, a `+' indicates that two or more types of data are collected and/or analyzed concurrently, while a ' → ' indicates that two or more types of data are collected or analyzed consecutively or sequentially. So, the notation `QUAN+qual' indicates that both quantitative and qualitative data are collected and analyzed concurrently with quantitative data and analysis being prioritized. The notation `QUAN → QUAL' indicates that quantitative data are being collected and/or analyzed before qualitative data, and both types of data and analysis have equal weight.

\textbf{Figure 1}

\emph{A Visualization of a QUAN+qual → QUAL Sequential Exploratory Mixed Research Study.}

\includegraphics{images/methods-viz.png}

Following a pilot phase, I intend to engage in a QUAN+qual → QUAL sequential exploratory research project comprising two separate data collection and analysis phases. In phase 1a, I will use a survey of higher education instructors to collect and analyze primarily quantitative data to investigate research questions 1 and 2.

\begin{quote}
\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  Are there distinct patterns in higher education instructors' approaches to assessment in Canada?
\item
  Does the prevalence of these patterns differ by:
\end{enumerate}

\begin{itemize}
\tightlist
\item
  instructors' levels of experience in teaching face-to-face versus online?
\item
  instructors' levels of experience using technology?
\end{itemize}
\end{quote}

This will be supplemented in phase 1b by the \emph{concurrent collection} and \emph{subsequent analysis} of qualitative data to answer research question 3.

\begin{quote}
\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\setcounter{enumi}{2}
\tightlist
\item
  What factors influence instructors' approaches to assessment?
\end{enumerate}
\end{quote}

The second phase will begin following the analysis and integrationof phase 1a+b data and will collect and analyze qualitative data in semi-structured interviews with learners nominated by instructors in phase 1. Phase 2 analysis is intended to answer research question 4.

\begin{quote}
\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\setcounter{enumi}{3}
\tightlist
\item
  How do instructors' assessment approaches affect learners' experiences?
\end{enumerate}
\end{quote}

Since research question 4 is an integrative question \citep{creamerIntroductionFullyIntegrated2018}, it cannot be answered fully until the inferences from phase 1 and phase 2 are integrated into one or more meta-inferences.

\hypertarget{participants-and-sampling-strategy}{%
\section*{Participants and Sampling Strategy}\label{participants-and-sampling-strategy}}
\addcontentsline{toc}{section}{Participants and Sampling Strategy}

The target population for the research will be instructors who have taught at least one semester-length, accredited course in the previous 12 months at an English-speaking {[}\^{}1{]} public higher education institution in Canada. Ideally, inferences from phase 1 of the study would be generalizable to the total population of English-speaking higher education instructors in Canada, which would require a random sample in which every member of the target population would have equal opportunity to be selected and no selection would have been influenced by a previous selection \citep{hibbertsCommonSurveySampling2012, rencklyAirUniversitySampling2002}. Care will be taken to ensure the sample is representative of the whole population by engaging in a stratified random sampling technique at the institutional level \citep{hibbertsCommonSurveySampling2012}. This would involve using regions within Canada as strata (for example, British Columbia, the prairie provinces, the north, Ontario, Quebec, eastern Canada), randomly choosing 2-3 higher education institutions from within those strata, and sending the survey to all instructors at those institutions. Recruitment of participants for phase 2 will involve phase 1 participants (instructors) nominating 8-10 learners from their courses, I will randomly select 2-3 respondents from phase 1 and then randomly select 4-5 of their nominated learners to receive invitations to the semi-structured interviews. This procedure will ensure that instructors will not know which of their nominated learners participated in the research and it will create ``linked data'' \citeyearpar[p.~126]{bazeleyIntegratingAnalysesMixed2018}, allowing for greater explanatory power due to being able to match specific instructor approaches with learner impacts.

\hypertarget{data-collection-strategy}{%
\section*{Data Collection Strategy}\label{data-collection-strategy}}
\addcontentsline{toc}{section}{Data Collection Strategy}

\hypertarget{phase-1}{%
\subsection*{Phase 1}\label{phase-1}}
\addcontentsline{toc}{subsection}{Phase 1}

I propose that the phase 1 data collection instrument be DeLuca et al.'s \citeyearpar{delucaApproachesClassroomAssessment2016} ACAI (see Appendix 1 for the full ACAI v3.0 and its specifications). The ACAI was designed by a team of researchers, led by Dr.~<NAME> at Queen's University in Canada to investigate the approaches that K-12 educators take to assessment in their classrooms. It is based on an analysis of assessment standards published in five majority English-speaking countries since 1990. The instrument is based on a four-dimensional framework of assessment literacy (assessment purposes, assessment processes, assessment fairness, and measurement theory), and includes one open-ended question and several closed-ended questions about how teachers form their approaches to assessment. It also includes a section regarding educator beliefs about assessment based on the \emph{Beliefs about Assessment} scale \citep{smithPreparingTeachersUse2014}. The ACAI has been used in a wide variety of studies within the instrument's author's research group \citep{coombsChangingApproachesClassroom2018, coombsPersoncenteredAnalysisTeacher2020, delucaDifferentialSituatedView2019, delucaTeachersApproachesClassroom2016, delucaPoliciesProgramsPractices2019, schneiderLinkingPersonalityTeachers2020}, and also by other researchers \citep{nayaginPreserviceTeachersApproaches2020}. In several of these studies, the ACAI has been modified somewhat to fit the particular context or research design.

As the ACAI was designed for use in K-12 contexts and has been used less often to investigate higher education instructors' approaches to assessment, I will modify the language in the instrument to be more suitable for the higher education context and to include more open-ended questions. Following these modifications, I will undertake a pilot phase to determine whether the modified ACAI is suitable for the higher education context. The pilot phase will inform the version of the ACAI used in Phase 1 of the study and will include both expert review and the recruitment of a small sample from the target population to preview the survey in its intended mode and offer specific and guided feedback \citep{hibbertsCommonSurveySampling2012, rencklyAirUniversitySampling2002}.

\hypertarget{phase-2}{%
\subsection*{Phase 2}\label{phase-2}}
\addcontentsline{toc}{subsection}{Phase 2}

The semi-structured interviews in phase 2 will include questions similar to the following:

\begin{itemize}
\tightlist
\item
  Please describe what you understand assessment to be.
\item
  Did the assessment strategies that your instructor used help you to learn the course material?
\item
  Did {[}instructor{]} use technology to administer assessments?

  \begin{itemize}
  \tightlist
  \item
    How did their use/not use of technology impact your performance on the assessments?
  \end{itemize}
\item
  Please describe your \emph{favourite} assessment from {[}instructor{]}'s course {[}course name{]}?

  \begin{itemize}
  \tightlist
  \item
    What made that assessment your favourite?
  \end{itemize}
\item
  Please describe your \emph{least favourite} assessment from {[}instructor{]}'s course {[}course name{]}?

  \begin{itemize}
  \tightlist
  \item
    What made that assessment your least favourite?
  \end{itemize}
\item
  What factors made it \emph{easier} to do well on the assessments in {[}course{]}?
\item
  What factors made it \emph{more difficult} to do well on the assessments in {[}course{]}?
\item
  How did the assessment strategies your instructor used make you feel?
\item
  Do you receive any support or accommodations to allow you to complete assessments?
\end{itemize}

\hypertarget{types-of-data-and-analysis}{%
\section*{Types of Data and Analysis}\label{types-of-data-and-analysis}}
\addcontentsline{toc}{section}{Types of Data and Analysis}

As shown in the visualization of the research project (Figure 1), both quantitative and qualitative data will be gathered for analysis. The following sections will describe the data I intend to gather and how it may address the research questions.

\hypertarget{quantitative-data}{%
\subsection*{Quantitative data}\label{quantitative-data}}
\addcontentsline{toc}{subsection}{Quantitative data}

During phase 1a, quantitative data will be gathered using the ACAI and will primarily be in the form of responses to 6-point Likert scale items. Similar to Flores et al. \citeyearpar{floresPortugueseUniversityStudents2020}, the first step will be to engage in a confirmatory factor analysis to determine the fit of the data with the \emph{Approaches to Classroom Assessment} model. This step will address research question 1. Then, as the data will be \emph{ordinal} data (i.e.~a score of five is greater than a score of three), but not \emph{interval} data (i.e.~it is not possible to determine \emph{how much greater} a score of five is compared to a score of three) \citep{bulutLikertScalesFriend2021, jamiesonLikertScalesHow2004}, it will not be appropriate to use the data to calculate common statistics such as mean or standard deviation (i.e.~there is no way to calculate the mean of ``highly likely'' and ``highly unlikely''). Consequently, more advanced statistical techniques which rely on the mean and standard deviation (i.e.~\emph{t} test, analysis of variance), otherwise known as \emph{parametric} tests, are not appropriate for ordinal data \citep{fieldDiscoveringStatisticsUsing2018}. Fortunately, there are non-parametric tests that enable analysis of data that do not meet the requirements for parametric tests. One example, which will be useful in comparing two different demographic groups within my data (e.g.~instructors with more than two years experience teaching with technology compared to instructors with less than two years of experience teaching with technology), is the \emph{Mann-Whitney-Wilcoxon} test \citep{gaoNonparametricStatistics2010} (MWW). Instead of calculating based on the mean scores of different samples, the MWW test requires the researcher to first rank the scores and then calculate the differences between the samples based on the ranked scores. If I am to compare 3 or more groups, the \emph{Kruskal-Wallis} test \citep{gaoNonparametricStatistics2010}, based on the same ranking procedure, would be more appropriate. These two tests will be used to address research question 2.

\hypertarget{qualitative-data}{%
\subsection*{Qualitative data}\label{qualitative-data}}
\addcontentsline{toc}{subsection}{Qualitative data}

During phase 1b, I will gather and analyze qualitative data through the use of open-ended questions embedded in the ACAI. Part B of the ACAI currently contains open-ended questions asking respondents to describe how they would respond to the scenario if their response isn't listed as an option in the instrument \citep{delucaACAIInstrumentSpecificationsND}. Below are some open-ended questions which might be added to Part A (demographics) of the ACAI:

\begin{itemize}
\tightlist
\item
  In a few sentences or point form notes, please describe the most important factors you consider when planning how you will assess learners in an upcoming course.
\item
  In a few sentences or point form notes, please describe what you understand assessment to be.
\item
  In what ways do you use digital technology in your assessment practice?
\end{itemize}

The current and proposed open-ended questions in the ACAI are intended to provide qualitative data which will be analyzed using a theory-driven thematic coding process \citep{delucaStudentPerspectivesAssessment2018, nameyDataReductionTechniques2008} to explore in greater depth the factors that influence instructors to approach assessment as they do in their courses (research question 3).

Namey et al. \citeyearpar{nameyDataReductionTechniques2008} describe two general approaches to the analysis of qualitative data: content analysis and thematic analysis. In content analysis, the researcher ``evaluates the frequency and saliency of particular words or phrases in a body of original text data in order to identify keywords or repeated ideas'' (p.~138), but with the drawback that there is limited interpretation or consideration of context. In thematic analysis, researchers consider not only specific words and phrases, but also ``implicit and explicit ideas'' (p.~138). They further describe ``data-driven'' (themes emerge from the data) and ``theory-driven'' (the researcher identifies themes \emph{a priori}, without consulting the data)(p.~138) approaches \citep[see also:][]{tashakkoriFoundationsMixedMethods2020}. In phase 1b of the proposed project, I will take a theory-driven approach to a thematic analysis where data will be coded in alignment with the four dimensions of the \emph{Approaches to Classroom Assessment} framework \citep{delucaTeachersApproachesClassroom2016} (the dotted line connecting ``Phase 1a Inferences'' to ``Phase 1b Analysis'' in figure 1). In addition, I will be particularly mindful that emergent themes may become clear in the data and I will be especially interested in those not represented in, or which are in contradiction to, the quantitative analysis \citep{tashakkoriFoundationsMixedMethods2020}.

After completing both phase 1a and 1b, I will be able to synthesize inferences from the two phase 1 analyses, and also one or more meta-inferences from the integrated analysis of the two sub-phases. These inferences will inform the phase 2 semi-structured interview questions, providing an integration point (the dotted lines from ``Phase 1a Analysis'', ``Phase 1b Analysis'', and ``Phase 1 Meta-inferences'' to ``Semi-structured Interviews'').

Raw qualitative data from phase 2 of the project will be in the form of audio recordings of semi-structured interviews, which will be transcribed verbatim and stored as plain text files. As this data is less connected to phase 1 data because it is a different sample population, I will use a data-driven thematic approach to uncover emergent themes in the data while not precluding the possibility that theory-driven themes may emerge.

Following the phase 1 and phase 2 analyses, I will synthesize the inferences from phase 1a and 1b, the meta-inferences from phase 1, and the inferences from phase 2 to draw project-level meta-inferences from the whole of the data, thereby addressing research question 4.

\hypertarget{research-procedures}{%
\section*{Research Procedures}\label{research-procedures}}
\addcontentsline{toc}{section}{Research Procedures}

Following approval from the University of Victoria Human Research Ethics Board (HREB) and a brief pilot phase, I will use a stratified random sampling technique to randomly select 2-3 higher education institutions from each of six regions in Canada (British Columbia, the prairie provinces, Ontario, Quebec, eastern Canada, and northern Canada), for a total of 12-18 institutions. I will contact the faculty associations at each of these institutions to request that they forward my letter of invitation and embedded link to a web-based survey. Each participant will be asked to self-identify and to nominate 8-10 learners in one of their courses and include the nominees' contact information. To begin phase 2, I will randomly select 2-3 instructors from phase 1, then randomly select 4-5 of their nominated learners to invite to semi-structured interviews. The interviews will be conducted via web-conferencing software that allows recording and the automatic generation of transcripts. Table 5 shows a projected timeline for the research.

\textbf{Table 5}

\textbf{\emph{Projected Timeline}}

\begin{longtable}[]{@{}ll@{}}
\toprule
Projected Date & Project Stage \\
\midrule
\endhead
December 2021 & UVic HREB Approval \\
January 2022 & Pilot Phase \\
February 2022 & Phase 1 Data Collection \\
July 2022 & Phase 1a and 1b Data Analysis \\
September 2022 & Phase 2 Data Collection \\
November 2022 & Phase 2 Data Analysis \\
January 2023 & Integration and Write-Up \\
\bottomrule
\end{longtable}

\hypertarget{chapter-summary}{%
\section*{Chapter Summary}\label{chapter-summary}}
\addcontentsline{toc}{section}{Chapter Summary}

In this chapter, I have discussed the three major approaches to research, \emph{quantitative}, \emph{qualitative}, and \emph{mixed}, along with their underlying epistemologies. Through a review of the literature on assessment literacy and its impact on learners in higher education, I show that researchers use all three approaches and that the most important consideration in determining a method to pursue an investigation in this area is the purpose of the investigation. As the purpose of this investigation is to gain understanding of the multi-dimensional construct of assessment literacy \emph{and} its effects on learners, a highly subjective phenomenon, a mixed research approach is justified. I discuss a possible method to gather and analyze quantitative and qualitative data from higher education instructors and their learners, along with strategies for drawing inferences and meta-inferences based on the integration of the separate analyses.

{[}\^{}1{]} If funding for translation services is secured, I would extend the sample population to include French-speaking instructors and learners.

\hypertarget{references}{%
\chapter*{References}\label{references}}
\addcontentsline{toc}{chapter}{References}

\hypertarget{results}{%
\chapter*{Results}\label{results}}
\addcontentsline{toc}{chapter}{Results}

We have finished a nice book.

\hypertarget{discussion}{%
\chapter*{Discussion}\label{discussion}}
\addcontentsline{toc}{chapter}{Discussion}

We have finished a nice book.

\hypertarget{conclusion}{%
\chapter*{Conclusion}\label{conclusion}}
\addcontentsline{toc}{chapter}{Conclusion}

We have finished a nice book.

\hypertarget{approaches-to-classroom-assessment-inventory-v3.0}{%
\chapter{Approaches to Classroom Assessment Inventory v3.0}\label{approaches-to-classroom-assessment-inventory-v3.0}}

The ACAI is part of an educational research project. By completing the ACAI you will be consenting to your anonymous responses being including in our research study. Additional details on research ethics are presented in this letter of information and consent page.

\textbf{Project title:} Teachers' Approaches to Classroom Assessment

\textbf{Ethical clearance:} This study has been granted clearance according to the recommended principles of Canadian ethics guidelines and Queen's policies.

\textbf{Purpose:} The purpose of this research is to develop and test a survey to examine teachers' approaches to classroom assessment. Results from this survey will serve to describe teachers' assessment practices within various educational systems. Findings from this study will provide a basis for future teacher professional development activities in assessment.

\textbf{Participation:} If you agree to participate, you will be asked to complete the online survey. In total, completion of the survey should take approximately 20 minutes. The survey includes questions about your approach to assessment practice, your perceived skill in assessment, and professional learning interests in assessment. At the end of the survey you will be asked if you are interested in participating in future research on assessment. This question is optional, and you may choose not to answer it. However, if you are interested, you will be asked for your name and contact information.

\textbf{Withdrawal Procedures:} There are no known risks to participating in this study. Participation is voluntary and you are free to withdraw at any time by simply closing your browser window. However, once you have submitted your responses, you will be unable to withdraw your data from the study. If you have elected to provide your contact information at the end of the survey, you may request removal of your data. In order to withdraw, please contact the principal researcher, <NAME>, via email (\href{mailto:<EMAIL>}{\nolinkurl{<EMAIL>}}) or phone (613-533-6000 x.77675).

\textbf{Privacy:} Your confidentiality will be maintained to the extent possible. Presentation or publication of research results will not identify you in any way. Your name will not be used in the publication or presentation of results. All of the data will be stored on password-protected computers at Queen's University and, in accordance with Queen's policies, will be retained for at least five years before being destroyed. Contact information, should you choose to provide it, will be stored separately from your survey responses. Only the researcher and research assistants will have access to the data. The survey data obtained in this research will be encrypted using SSL and stored on secure servers located in Canada and your IP address will not be known or identifiable.

\textbf{Consent \& Agreeing to Participate:} If you are interested in participating in this study, please click on the NEXT button below to complete the survey. By completing the survey you freely agree to participate in this study.
Questions: Any questions about study participation may be directed to Dr.~<NAME> (613 533 6000 x77675 or by email at \href{mailto:<EMAIL>}{\nolinkurl{<EMAIL>}}). Any ethical concerns about the study may be directed to the Chair of the General Research Ethics Board at 613-533-6081 or \href{mailto:<EMAIL>}{\nolinkurl{<EMAIL>}}.

Do you consent?
Yes No

\hypertarget{part-a-questions-about-you-and-your-teaching-background}{%
\section{PART A: QUESTIONS ABOUT YOU AND YOUR TEACHING BACKGROUND}\label{part-a-questions-about-you-and-your-teaching-background}}

Please enter the last 5 digits of your phone number. This information will be used to link your responses if you take this survey multiple times and not for identification purposes.
With which gender do you self-identify?
Female
Male
Other
Prefer not to respond

My age is: (open response)

I currently live in: (scroll down menu for all countries, Canada with provinces, America with states)

In a few sentences or point form notes, please describe experiences or people who have influenced your understanding and practice of classroom assessment. (open response)

I am currently complete a teacher education program.
Yes (stream 1) No (stream 2)

\hypertarget{stream-1}{%
\section{Stream 1}\label{stream-1}}

At what institution are you completing your teacher education program? (open response)

At what stage are you in your teacher education program?

At the beginning Around the middle Near the end

I will be certified/licensed as a (select all that apply):
- Primary teacher
- Early elementary teacher
- Upper elementary teacher
- Middle grades/intermediate/junior school teacher
- Senior grades/ high school teacher
- K-12 teacher

Thinking of your teacher education program, how important were these experiences in shaping your thinking or practice of assessment?

\begin{longtable}[]{@{}llllll@{}}
\toprule
& Not at all Important & Unimportant & Important & Very Important & Not Applicable \\
\midrule
\endhead
Practicum & & & & & \\
Assessment Course(s) & & & & & \\
Curriculum Course(s) & & & & & \\
Self-study and/or Personal Learning & & & & & \\
Conversations with Peers & & & & & \\
Conversations with Teacher Educators/Instructors Outside of Courses & & & & & \\
\bottomrule
\end{longtable}

\hypertarget{stream-2}{%
\section{Stream 2}\label{stream-2}}

I currently work as a
- Full-time teacher
- Part-time teacher
- Substitute teacher
- School administrator
- District administrator
- Instructional coach or similar position
- Teacher's aide or paraprofessional
- Not applicable/not working
- Other-please specify

How long have you been a professional educator? (open response)

I currently work in a(n):
- Preschool Elementary school
- Middle school
- Junior high school
- High school
- Other-please specify

During my pre-service teacher education program, I completed the following preparation in assessment:
- Two or more full-semester courses in assessment
- A one-semester equivalent course in assessment
- A short or mini course in assessment of less than one semester equivalency
- A module in assessment (e.g., one face-to-face workshop, one online session)
- One or more teaching methods courses with assessment topics integrated
- No formal preparation in assessment
- I don't remember

How important are the following sources in developing your knowledge or practice of classroom assessment?

\begin{longtable}[]{@{}llllll@{}}
\toprule
& Not at all Important & Unimportant & Important & Very Important & Not Applicable \\
\midrule
\endhead
Professional Learning Communities & & & & & \\
Workshops & & & & & \\
Classroom Experiences & & & & & \\
Self-study and/or Personal Learning & & & & & \\
Conversations with Peers & & & & & \\
Conversations with Administration & & & & & \\
\bottomrule
\end{longtable}

\hypertarget{part-b-scenario-based-questions}{%
\section{PART B: SCENARIO-BASED QUESTIONS}\label{part-b-scenario-based-questions}}

You will be presented with four scenarios in this section. Each scenario has 12 actions. For each action, please identify how likely you are to do each of the following as part of your overall response to the scenario (1=highly unlikely; 2=unlikely; 3=somewhat unlikely; 4=somewhat likely; 5=likely; 6=highly likely).

Please interpret the scenario in relation to your current teaching context (i.e., grade, school, community); if you teach across contexts, select one context and keep this context in mind when completing the entire survey. There are no right or wrong answers. We are interested in your honest responses to the scenarios. If you do not understand the statement, select `Don't Know.'

\hypertarget{scenario-1}{%
\subsection{Scenario 1:}\label{scenario-1}}

\begin{quote}
You give your class a paper-pencil summative unit test with accommodations and modifications for identified learners. Sixteen of the 24 students fail.
\end{quote}

As a teacher in this situation, how likely are you to do each of the following actions as part of your overall response to the scenario? If you have experienced this scenario in your teaching context, base your response on the actions you took.

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  Keep and record the test grade for all students and move on to the next unit.
\item
  Disregard the test grade, then reteach and retest the material.
\item
  Have students generate a plan to relearn the material.
\item
  Analyze test results to determine if failure was the result of poor test design.
\item
  Remove test questions that most students struggled with and re-calculate student scores without those questions.
\item
  Schedule student conferences to discuss test performance and next steps.
\item
  Allow the entire class to retake a similar test.
\item
  Allow only students with exceptionalities who struggled with the test to take a new assessment.
\item
  Provide a new assessment to every student who struggled with the test.
\item
  Identify test questions that most students consistently answered incorrectly, then provide students with new questions to re-test those concepts.
\item
  Adjust student grades based on previously collected formative assessment information.
\item
  Analyze test results to identify test design errors and consider student circumstances for failure, then adjust individual student grades accordingly.
\end{enumerate}

If your response to this scenario was not listed above, how would you most likely respond? (open response)

\hypertarget{scenario-2}{%
\subsection{Scenario 2:}\label{scenario-2}}

\begin{quote}
You discover that a student has plagiarized some of their assignment (e.g., an essay, lab report).
\end{quote}

As a teacher in this situation, how likely are you to do each of the following actions? If you have experienced this scenario in your teaching context, base your response on the actions you took.

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  Reduce the student's grade.
\item
  Have them re-write the plagiarized section in their own words, then re- grade the assignment.
\item
  Ask them to reflect on why plagiarism is a problem and what they would do differently next time.
\item
  As the teacher, reflect on how the assignment could have been structured differently to deter plagiarism.
\item
  Adjust the student grade to reflect the portion of work that was plagiarized.
\item
  Discuss with the student the reasons for the plagiarism, severity of plagiarism, and negotiate potential next steps for their learning.
\item
  Apply the same consequence you would for other students to ensure all students are treated equally.
\item
  Consider if the student has identified learning needs and exceptionalities before determining response to plagiarism.
\item
  Discuss why the student plagiarized and agree upon an appropriate alternative assignment.
\item
  Apply all aspects of school policy on plagiarism to ensure consistency across all students.
\item
  Consider the original aspects of the assignment and the plagiarized text to determine what the student knows and does not appear to know related to content expectations.
\item
  Apply the school policy on plagiarism unless there are extenuating circumstances that led to the plagiarism.
\end{enumerate}

\textbf{If your response to this scenario was not listed above, how would you most likely respond?} (open response)

\hypertarget{scenario-3}{%
\subsection{Scenario 3}\label{scenario-3}}

\begin{quote}
Out of 28 students in your class, 4 students are classified/identified with an exceptionality and have an Individual Education Plan (IEP) (i.e., each student requires accommodations but not a modified curriculum) as well as several other unidentified students with differentiated learning needs. You must decide how to accurately assess learning in your class.
\end{quote}

\textbf{As a teacher in this situation, how likely are you to do each of the following actions? If you have experienced this scenario in your teaching context, base your response on the actions you took.}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  Provide all identified students with accommodations on all summative assessments.
\item
  Use formative assessments with all of your students based on their individual learning needs to support their learning.
\item
  Allow each student to develop a learning plan based on their strengths and goals to support their own learning.
\item
  Design a variety of assessment methods to target each learning objective.
\item
  Develop and use different rubric criteria to assess identified students.
\item
  Explain to students and parents the purpose of accommodations and how they will be implemented to support student learning and assessment.
\item
  Grade students using the same assessments and without accommodations.
\item
  Assume that accommodated assessments provide an equally valid measure of identified students' learning.
\item
  Provide a variety of assessment options and allow students to choose how they will demonstrate their learning.
\item
  Use the same scoring rubric for all students.
\item
  Consider individual student learning needs when assessing student work.
\item
  Use the same scoring rubric for all students but use professional judgment to apply criteria differently based on individual student capability.
\end{enumerate}

\textbf{If your response to this scenario was not listed above, how would you most likely respond?} (open response)

\hypertarget{section}{%
\subsection{}\label{section}}

\hypertarget{scenario-4}{%
\subsection{Scenario 4:}\label{scenario-4}}

You are planning a unit for your class.

\textbf{Question 1: As a teacher in this situation, how likely are you to do each of the following actions? If you have experienced this scenario in your teaching context, base your response on the actions you took.}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  Start by designing a summative assessment based on curriculum expectations then plan your lessons.
\item
  Use formative assessments to guide your subsequent lesson planning.
\item
  Co-construct learning goals with students then have each student develop a personal learning plan for the unit.
\item
  Design a summative unit assessment based on what was taught to students.
\item
  When planning a unit, consider when assessments and feedback will be provided to students.
\item
  Discuss assignments and co-construct grading criteria for the unit with your students.
\item
  Plan class lessons and assessments that are the same for all students and encompass the curriculum expectations.
\item
  Ensure that assessments throughout unit are accommodated for formally identified learners (i.e., learners with exceptionalities).
\item
  Give all students a diagnostic assessment at the beginning of the unit and have students use their results to select personalized appropriate learning and assessment activities.
\item
  Work with other teachers to develop a reliable summative assessment for the unit.
\item
  In designing your assessments, prioritize the alignment between the assessment and the intended learning objectives.
\item
  Adapt assessments that have worked well with other students like yours to take into consideration the context and activities of your enacted lessons.
\end{enumerate}

\textbf{If your response to this scenario was not listed above, how would you most likely respond?} (open response)

\hypertarget{part-c-questions-about-classroom-assessment-practices-and-professional-learning}{%
\section{PART C: QUESTIONS ABOUT CLASSROOM ASSESSMENT PRACTICES AND PROFESSIONAL LEARNING}\label{part-c-questions-about-classroom-assessment-practices-and-professional-learning}}

Below are a number of statements about assessment. Please indicate your level of agreement with each statement (1=strongly disagree; 2=disagree; 3=somewhat disagree; 4=somewhat agree; 5=agree; 6=strongly agree). There are no right or wrong answers. We are interested in your honest opinions about assessment. If you do not understand the statement, select `Don't Know.'

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  The primary purpose of classroom assessment is to assign a grade or level to student work.
\item
  Classroom assessment should be used to determine if students have met provincial standards.
\item
  Feedback from classroom assessments improves student learning.
\item
  By using assessment, teachers can track the progress of students.
\item
  Students whould use assessment to evaluate their own work.
\item
  Students are able to provide accurate and useful feedback to each other.
\item
  Teachers have the skills and knowledge to construct good assessments.
\item
  For good classroom assessment, teachers need extensive knowledge of the curriculum.
\item
  Classroom assessment involves judging a student's performance in relation to a set of goals/standards/criteria.
\item
  Classroom assessment is integral to developing lesson plans and implementing curriculum.
\item
  Classroom assessment should be used to provide evidence of student progress for administrative purposes.
\item
  Classroom assessment is useful when reporting a student's achievement/progress to parents and caregivers.
\item
  An important component of classroom assessment is students taking large-scale tests (e.g.~provincial assessment, EQAO).
\item
  Provincial assessments (e.g.~EQAO) are a meaningful form of assessment.
\item
  All classroom assessments should be adapted to suit the learning needs of identified students (e.g.~English language learners).
\item
  Students with exceptionalities should be provided with different classroom assessment than other students.
\item
  For good assessment, teachers need to know how their students learn.
\item
  Classroom assessment helps teachers identify the particular learning needs of any student.
\item
  Classroom assessment results is a good indicator of the quality of a school.
\item
  Assessment results reflect the quality of teaching.
\item
  Assessment is an imprecise process.
\item
  Observing students is a valid form of assessment.
\item
  Classroom assessment is of little use to teachers on a day-to-day basis.
\item
  Classroom assessment interrupts students' learning.
\item
  Assessment is a stressful activity for students.
\item
  Assessment takes time away from teaching.
\item
  Teachers use too many assessments.
\item
  Assessment is a positive force for improving the social climate in a class.
\item
  Classroom assessments motivate students to do their best.
\item
  Assessment is an engaging and enjoyable experience for students.
\end{enumerate}

\hypertarget{approaches-to-classroom-assessment-inventory-v3.0-1}{%
\chapter{Approaches to Classroom Assessment Inventory v3.0}\label{approaches-to-classroom-assessment-inventory-v3.0-1}}

The ACAI is part of an educational research project. By completing the ACAI you will be consenting to your anonymous responses being including in our research study. Additional details on research ethics are presented in this letter of information and consent page.

\textbf{Project title:} Teachers' Approaches to Classroom Assessment

\textbf{Ethical clearance:} This study has been granted clearance according to the recommended principles of Canadian ethics guidelines and Queen's policies.

\textbf{Purpose:} The purpose of this research is to develop and test a survey to examine teachers' approaches to classroom assessment. Results from this survey will serve to describe teachers' assessment practices within various educational systems. Findings from this study will provide a basis for future teacher professional development activities in assessment.

\textbf{Participation:} If you agree to participate, you will be asked to complete the online survey. In total, completion of the survey should take approximately 20 minutes. The survey includes questions about your approach to assessment practice, your perceived skill in assessment, and professional learning interests in assessment. At the end of the survey you will be asked if you are interested in participating in future research on assessment. This question is optional, and you may choose not to answer it. However, if you are interested, you will be asked for your name and contact information.

\textbf{Withdrawal Procedures:} There are no known risks to participating in this study. Participation is voluntary and you are free to withdraw at any time by simply closing your browser window. However, once you have submitted your responses, you will be unable to withdraw your data from the study. If you have elected to provide your contact information at the end of the survey, you may request removal of your data. In order to withdraw, please contact the principal researcher, <NAME>, via email (\href{mailto:<EMAIL>}{\nolinkurl{<EMAIL>}}) or phone (613-533-6000 x.77675).

\textbf{Privacy:} Your confidentiality will be maintained to the extent possible. Presentation or publication of research results will not identify you in any way. Your name will not be used in the publication or presentation of results. All of the data will be stored on password-protected computers at Queen's University and, in accordance with Queen's policies, will be retained for at least five years before being destroyed. Contact information, should you choose to provide it, will be stored separately from your survey responses. Only the researcher and research assistants will have access to the data. The survey data obtained in this research will be encrypted using SSL and stored on secure servers located in Canada and your IP address will not be known or identifiable.

\textbf{Consent \& Agreeing to Participate:} If you are interested in participating in this study, please click on the NEXT button below to complete the survey. By completing the survey you freely agree to participate in this study.
Questions: Any questions about study participation may be directed to Dr.~<NAME> (613 533 6000 x77675 or by email at \href{mailto:<EMAIL>}{\nolinkurl{<EMAIL>}}). Any ethical concerns about the study may be directed to the Chair of the General Research Ethics Board at 613-533-6081 or \href{mailto:<EMAIL>}{\nolinkurl{<EMAIL>}}.

Do you consent?
Yes No

\hypertarget{part-a-questions-about-you-and-your-teaching-background-1}{%
\section{PART A: QUESTIONS ABOUT YOU AND YOUR TEACHING BACKGROUND}\label{part-a-questions-about-you-and-your-teaching-background-1}}

Please enter the last 5 digits of your phone number. This information will be used to link your responses if you take this survey multiple times and not for identification purposes.
With which gender do you self-identify?
Female
Male
Other
Prefer not to respond

My age is: (open response)

I currently live in: (scroll down menu for all countries, Canada with provinces, America with states)

In a few sentences or point form notes, please describe experiences or people who have influenced your understanding and practice of classroom assessment. (open response)

I am currently complete a teacher education program.
Yes (stream 1) No (stream 2)

\hypertarget{stream-1-1}{%
\section{Stream 1}\label{stream-1-1}}

At what institution are you completing your teacher education program? (open response)

At what stage are you in your teacher education program?

At the beginning Around the middle Near the end

I will be certified/licensed as a (select all that apply):
- Primary teacher
- Early elementary teacher
- Upper elementary teacher
- Middle grades/intermediate/junior school teacher
- Senior grades/ high school teacher
- K-12 teacher

Thinking of your teacher education program, how important were these experiences in shaping your thinking or practice of assessment?

\begin{longtable}[]{@{}llllll@{}}
\toprule
& Not at all Important & Unimportant & Important & Very Important & Not Applicable \\
\midrule
\endhead
Practicum & & & & & \\
Assessment Course(s) & & & & & \\
Curriculum Course(s) & & & & & \\
Self-study and/or Personal Learning & & & & & \\
Conversations with Peers & & & & & \\
Conversations with Teacher Educators/Instructors Outside of Courses & & & & & \\
\bottomrule
\end{longtable}

\hypertarget{stream-2-1}{%
\section{Stream 2}\label{stream-2-1}}

I currently work as a
- Full-time teacher
- Part-time teacher
- Substitute teacher
- School administrator
- District administrator
- Instructional coach or similar position
- Teacher's aide or paraprofessional
- Not applicable/not working
- Other-please specify

How long have you been a professional educator? (open response)

I currently work in a(n):
- Preschool Elementary school
- Middle school
- Junior high school
- High school
- Other-please specify

During my pre-service teacher education program, I completed the following preparation in assessment:
- Two or more full-semester courses in assessment
- A one-semester equivalent course in assessment
- A short or mini course in assessment of less than one semester equivalency
- A module in assessment (e.g., one face-to-face workshop, one online session)
- One or more teaching methods courses with assessment topics integrated
- No formal preparation in assessment
- I don't remember

How important are the following sources in developing your knowledge or practice of classroom assessment?

\begin{longtable}[]{@{}llllll@{}}
\toprule
& Not at all Important & Unimportant & Important & Very Important & Not Applicable \\
\midrule
\endhead
Professional Learning Communities & & & & & \\
Workshops & & & & & \\
Classroom Experiences & & & & & \\
Self-study and/or Personal Learning & & & & & \\
Conversations with Peers & & & & & \\
Conversations with Administration & & & & & \\
\bottomrule
\end{longtable}

\hypertarget{part-b-scenario-based-questions-1}{%
\section{PART B: SCENARIO-BASED QUESTIONS}\label{part-b-scenario-based-questions-1}}

You will be presented with four scenarios in this section. Each scenario has 12 actions. For each action, please identify how likely you are to do each of the following as part of your overall response to the scenario (1=highly unlikely; 2=unlikely; 3=somewhat unlikely; 4=somewhat likely; 5=likely; 6=highly likely).

Please interpret the scenario in relation to your current teaching context (i.e., grade, school, community); if you teach across contexts, select one context and keep this context in mind when completing the entire survey. There are no right or wrong answers. We are interested in your honest responses to the scenarios. If you do not understand the statement, select `Don't Know.'

\hypertarget{scenario-1-1}{%
\subsection{Scenario 1:}\label{scenario-1-1}}

\begin{quote}
You give your class a paper-pencil summative unit test with accommodations and modifications for identified learners. Sixteen of the 24 students fail.
\end{quote}

As a teacher in this situation, how likely are you to do each of the following actions as part of your overall response to the scenario? If you have experienced this scenario in your teaching context, base your response on the actions you took.

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  Keep and record the test grade for all students and move on to the next unit.
\item
  Disregard the test grade, then reteach and retest the material.
\item
  Have students generate a plan to relearn the material.
\item
  Analyze test results to determine if failure was the result of poor test design.
\item
  Remove test questions that most students struggled with and re-calculate student scores without those questions.
\item
  Schedule student conferences to discuss test performance and next steps.
\item
  Allow the entire class to retake a similar test.
\item
  Allow only students with exceptionalities who struggled with the test to take a new assessment.
\item
  Provide a new assessment to every student who struggled with the test.
\item
  Identify test questions that most students consistently answered incorrectly, then provide students with new questions to re-test those concepts.
\item
  Adjust student grades based on previously collected formative assessment information.
\item
  Analyze test results to identify test design errors and consider student circumstances for failure, then adjust individual student grades accordingly.
\end{enumerate}

If your response to this scenario was not listed above, how would you most likely respond? (open response)

\hypertarget{scenario-2-1}{%
\subsection{Scenario 2:}\label{scenario-2-1}}

\begin{quote}
You discover that a student has plagiarized some of their assignment (e.g., an essay, lab report).
\end{quote}

As a teacher in this situation, how likely are you to do each of the following actions? If you have experienced this scenario in your teaching context, base your response on the actions you took.

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  Reduce the student's grade.
\item
  Have them re-write the plagiarized section in their own words, then re- grade the assignment.
\item
  Ask them to reflect on why plagiarism is a problem and what they would do differently next time.
\item
  As the teacher, reflect on how the assignment could have been structured differently to deter plagiarism.
\item
  Adjust the student grade to reflect the portion of work that was plagiarized.
\item
  Discuss with the student the reasons for the plagiarism, severity of plagiarism, and negotiate potential next steps for their learning.
\item
  Apply the same consequence you would for other students to ensure all students are treated equally.
\item
  Consider if the student has identified learning needs and exceptionalities before determining response to plagiarism.
\item
  Discuss why the student plagiarized and agree upon an appropriate alternative assignment.
\item
  Apply all aspects of school policy on plagiarism to ensure consistency across all students.
\item
  Consider the original aspects of the assignment and the plagiarized text to determine what the student knows and does not appear to know related to content expectations.
\item
  Apply the school policy on plagiarism unless there are extenuating circumstances that led to the plagiarism.
\end{enumerate}

\textbf{If your response to this scenario was not listed above, how would you most likely respond?} (open response)

\hypertarget{scenario-3-1}{%
\subsection{Scenario 3}\label{scenario-3-1}}

\begin{quote}
Out of 28 students in your class, 4 students are classified/identified with an exceptionality and have an Individual Education Plan (IEP) (i.e., each student requires accommodations but not a modified curriculum) as well as several other unidentified students with differentiated learning needs. You must decide how to accurately assess learning in your class.
\end{quote}

\textbf{As a teacher in this situation, how likely are you to do each of the following actions? If you have experienced this scenario in your teaching context, base your response on the actions you took.}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  Provide all identified students with accommodations on all summative assessments.
\item
  Use formative assessments with all of your students based on their individual learning needs to support their learning.
\item
  Allow each student to develop a learning plan based on their strengths and goals to support their own learning.
\item
  Design a variety of assessment methods to target each learning objective.
\item
  Develop and use different rubric criteria to assess identified students.
\item
  Explain to students and parents the purpose of accommodations and how they will be implemented to support student learning and assessment.
\item
  Grade students using the same assessments and without accommodations.
\item
  Assume that accommodated assessments provide an equally valid measure of identified students' learning.
\item
  Provide a variety of assessment options and allow students to choose how they will demonstrate their learning.
\item
  Use the same scoring rubric for all students.
\item
  Consider individual student learning needs when assessing student work.
\item
  Use the same scoring rubric for all students but use professional judgment to apply criteria differently based on individual student capability.
\end{enumerate}

\textbf{If your response to this scenario was not listed above, how would you most likely respond?} (open response)

\hypertarget{section-1}{%
\subsection{}\label{section-1}}

\hypertarget{scenario-4-1}{%
\subsection{Scenario 4:}\label{scenario-4-1}}

You are planning a unit for your class.

\textbf{Question 1: As a teacher in this situation, how likely are you to do each of the following actions? If you have experienced this scenario in your teaching context, base your response on the actions you took.}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  Start by designing a summative assessment based on curriculum expectations then plan your lessons.
\item
  Use formative assessments to guide your subsequent lesson planning.
\item
  Co-construct learning goals with students then have each student develop a personal learning plan for the unit.
\item
  Design a summative unit assessment based on what was taught to students.
\item
  When planning a unit, consider when assessments and feedback will be provided to students.
\item
  Discuss assignments and co-construct grading criteria for the unit with your students.
\item
  Plan class lessons and assessments that are the same for all students and encompass the curriculum expectations.
\item
  Ensure that assessments throughout unit are accommodated for formally identified learners (i.e., learners with exceptionalities).
\item
  Give all students a diagnostic assessment at the beginning of the unit and have students use their results to select personalized appropriate learning and assessment activities.
\item
  Work with other teachers to develop a reliable summative assessment for the unit.
\item
  In designing your assessments, prioritize the alignment between the assessment and the intended learning objectives.
\item
  Adapt assessments that have worked well with other students like yours to take into consideration the context and activities of your enacted lessons.
\end{enumerate}

\textbf{If your response to this scenario was not listed above, how would you most likely respond?} (open response)

\hypertarget{part-c-questions-about-classroom-assessment-practices-and-professional-learning-1}{%
\section{PART C: QUESTIONS ABOUT CLASSROOM ASSESSMENT PRACTICES AND PROFESSIONAL LEARNING}\label{part-c-questions-about-classroom-assessment-practices-and-professional-learning-1}}

Below are a number of statements about assessment. Please indicate your level of agreement with each statement (1=strongly disagree; 2=disagree; 3=somewhat disagree; 4=somewhat agree; 5=agree; 6=strongly agree). There are no right or wrong answers. We are interested in your honest opinions about assessment. If you do not understand the statement, select `Don't Know.'

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  The primary purpose of classroom assessment is to assign a grade or level to student work.
\item
  Classroom assessment should be used to determine if students have met provincial standards.
\item
  Feedback from classroom assessments improves student learning.
\item
  By using assessment, teachers can track the progress of students.
\item
  Students whould use assessment to evaluate their own work.
\item
  Students are able to provide accurate and useful feedback to each other.
\item
  Teachers have the skills and knowledge to construct good assessments.
\item
  For good classroom assessment, teachers need extensive knowledge of the curriculum.
\item
  Classroom assessment involves judging a student's performance in relation to a set of goals/standards/criteria.
\item
  Classroom assessment is integral to developing lesson plans and implementing curriculum.
\item
  Classroom assessment should be used to provide evidence of student progress for administrative purposes.
\item
  Classroom assessment is useful when reporting a student's achievement/progress to parents and caregivers.
\item
  An important component of classroom assessment is students taking large-scale tests (e.g.~provincial assessment, EQAO).
\item
  Provincial assessments (e.g.~EQAO) are a meaningful form of assessment.
\item
  All classroom assessments should be adapted to suit the learning needs of identified students (e.g.~English language learners).
\item
  Students with exceptionalities should be provided with different classroom assessment than other students.
\item
  For good assessment, teachers need to know how their students learn.
\item
  Classroom assessment helps teachers identify the particular learning needs of any student.
\item
  Classroom assessment results is a good indicator of the quality of a school.
\item
  Assessment results reflect the quality of teaching.
\item
  Assessment is an imprecise process.
\item
  Observing students is a valid form of assessment.
\item
  Classroom assessment is of little use to teachers on a day-to-day basis.
\item
  Classroom assessment interrupts students' learning.
\item
  Assessment is a stressful activity for students.
\item
  Assessment takes time away from teaching.
\item
  Teachers use too many assessments.
\item
  Assessment is a positive force for improving the social climate in a class.
\item
  Classroom assessments motivate students to do their best.
\item
  Assessment is an engaging and enjoyable experience for students.
\end{enumerate}

  \bibliography{book.bib,packages.bib}

\end{document}
