Implementing Learning Analytics for Learning Impact: Taking Tools to Task

Learning analytics has the potential to impact student learning, at scale. Embedded in that claim are a set of assumptions and tensions around the nature of scale, impact on student learning, and the scope of infrastructure encompassed by ‘learning analytics’ as a socio-technical field. Drawing on our design experience of developing learning analytics and inducting others into its use, we present a model that we have used to address five key challenges we have encountered. In developing this model, we recommend: A focus on impact on learning through augmentation of existing practice; the centrality of tasks in implementing learning analytics for impact on learning; the commensurate centrality of learning in evaluating learning analytics; inclusion of co-design approaches in implementing learning analytics across sites; and an attention to both social and technical infrastructure.


Introduction
Increasing use of data and technology in education has paralleled developing areas of learning analytics, artificial intelligence in education, and educational data mining, with a specific focus on how data can be used to research and inform learning.Undoubtedly, there has been widespread interest in the potential of learning analytics.However, amongst the hype, reviews of the field indicate that meaningful translation of this interest into impact on learning is less clear, with a limited number of studies demonstrating impact on learners (Dawson et al., 2014;Ferguson et al., 2016;Ferguson & Clow, 2017;Hoel et al., 2015;Na & Tasir, 2017;Papamitsiou & Economides, 2014;Sclater et al., 2016;Viberg et al., 2018;Zilvinskis et al., 2017).
Across stakeholders -at the institutional, departmental, and individual academic and student levelthere is interest in how data can inform learning (Buckingham Shum, 2012).This interest has been paid heed by major technology vendors, who are increasingly marketing analytics packages for their products, particularly those based in Learning Management Systems (LMS).Alongside this focus on the LMS, researchers have been interested in analysis of data 'beyond the LMS' in social media data and other sites that students learn 'in the wild' (see, for example, Kitto et al., 2015;Pardo & Kloos, 2011).Across this work, the argument is that Learning analytics has the potential for tangible positive impact on student learning by supporting effective teaching and learning strategies.However, despite the potential, and emerging evidence of impact, adoption has not yet been widespread (Ferguson et al., 2016).Indeed, beyond impact and adoption, at the level of implementation even where the data analytics of the commercial vendors can be related to learning outcomes, institutions tend to focus on technical implementation factors (Macfadyen & Dawson, 2012).Meanwhile, the importance of teacher inquiry processes in using data are underexplored; that is, how educators use learning analytics to support students and engage in inquiry processes that support design for learning (Alhadad et al., 2018).Thus, there is a disconnect between adoption of learning analytics (including those implemented in Learning Management Systems), research showing the potential for student level impact, and implementation in educator practice.Thus, despite notable individual research and institutional exceptions, and the promise of broader projects such as the European SHEILA and LACE projects, as recently as 2017 Rienties, Cross, and Zdrahal (2017) suggest that the field of learning analytics research is not yet an evidence based field.Indeed the measurement of impact on learning is a known challenge, with emerging frameworks identifying indicators of quality in learning analytics applications (Greller & Drachsler, 2012;Scheffel et al., 2014).Although such evidence of impact exists across selected small-scale studies, there is a gap in translating this impact into adoption and implementation beyond research contexts.

Task Focussed Impact
In this paper, we address this gap and the related question: How do we increase impactful uptake of learning analytics in education?A particular focus of our approach to addressing this question is the consideration of common features of pedagogical contexts (at whatever scale): specific learning needs and tasks that address those needs.Learning analytics needs learning theory to achieve learning impact (Knight & Buckingham Shum, 2017;Wise & Shaffer, 2015).In this paper, we will argue for the centrality of the task for this theory, drawing on Goodyear's (1999) description.
We define the task as: "a specification for learner activity.Its design draws on the best of what we know about how people learn, on a deep knowledge of academic subject matter and/or vocational competences, and on knowledge of the learners.Essays, laboratory exercises, a structured discussion session or debate, a diagnostic exercise, a topic to research, an artefact to build, a program to writeall these are examples of kinds of learning task.A task needs to be sufficiently well-specified that the chances of a learner engaging in unproductive activity are kept within tolerable limits.Its specification may also need a degree of openness in order to meet variable learner needs and initiate a creative response" (Goodyear, 1999, p. 4).
That is, we argue that the reason we can scale across learners is precisely because they are engaged in some shared enterprise of learning.Thus, our efforts to generalise should be driven by theories around what that shared enterprise (and differences in it) might look like, why particular tasks are related to particular outcomes, differences in performance in such tasks, and so on.Indeed, insofar as there are differences, the large majority of those differences arise from the different aims, backgrounds, and interests of the students -i.e., differences in their pedagogical context.
It is in tasks that instructors design for learning within wider institutional and program-level contexts.And it is in tasks and their descriptions that fine grain trace data take on instructional meaning.Thus, we argue that learning analytics must come into alignment with -at times to augment -task designs to support learning (Knight, 2020;Knight, Shibani, et al., 2018;Shibani et al., 2019).Generalisation then, in the cases that are our focus involves the process of identifying similar features of tasks over contexts, and contextualising analytics for them.We suggest that a deliberately design (or co-design) oriented approach can lead to scaled adoption and impact by involving stakeholders in implementation and diffusion of learning analytics.
In the following sections we expand on this perspective by introducing a model, developed through our own learning analytics implementations, and the challenges we have tackled.Our approach to addressing the question of developing meaningful uptake of learning analytics is to distil key lessons from our own design-based implementation research (Penuel et al., 2011).In so doing, we recognise that both our context and interpretative lens are specific, while aiming to exemplify a design approach in which both practice and theory are developed (see, Collins et al., 2004).In this approach, the research consists in both developing practical impact through implementation at specific sites, and theoretical development through distillation of principles and approaches to foster and evaluate effective adoption.Thus, while the model we present is grounded in our own design work, this distillation of the model principles is a research contribution to theory development in its own right.
For each part of the model we consider the practical impact of the challenge by drawing on our work with writing analytics -learning analytics as applied to the processes and products of student writing tasks as detailed in the two example cases below.Each section provides a part of that model detailed in Figures 1.1-1.5, culminating in Figure 1.6 (which the reader may refer to for detailed elaboration).For each challenge, we provide both an example from our work, and responses we have made to those challenges, summarised as recommendations in the conclusions based on this learning.Over the course of the five challenges we develop a model for task oriented learning analytics that we have found productive in thinking about learning analytics strategic development and implementation, and which we believe can inform the development of other sub-fields in learning analytics.In this work we have developed a number of learning analytics applications to gain insight into student writing, and provide feedback to students on their writing.In particular the work focuses on the scholarly features of students' writing such as their use of particular argumentative moves, and their use of reflection on their own learning, as the example cases below discuss.

Case 1: Writing Analytics implementations
Over a number of design cycles, we have implemented writing analytics in classroom contexts.This work began with investigation of a parser to identify 'rhetorical moves' (parts of text that express some meaning to a reader), in the kinds of scholarly writing students must do.We then sought to implement a preliminary tool to provide automated writing feedback, which identified those moves to the students, in a classroom context both by providing access for them to use adhoc, and through implementing the tool on exemplars (that the students used).Following this, we developed a learning design in which students engaged with the analytics in tasks designed to develop their evaluative judgement (Boud, 2000) of their own work.These tasks allowed us to evaluate the impact of the writing analytics, by engaging with students with improving the same draft text, such that we could assess improvements made to that same text by all students individually, with and without the writing analytics, and with and without other task scaffolding.After positive evaluation of these designs, the learning task design was transferred to another disciplinary context in which similar augmentation of their existing practice occurred.

Case 2: Data Carpentry Workshops
The challenges and model that we present have been developed based on our experience of trying to tackle a specific problem: How do we engage both non-technical teachers and noneducational technologists in implementing writing analytics for impact?To this end, a series of workshops (Knight et al., 2017;Shibani et al., 2018) was run that developed a specific instantiation of the general model presented in this paper.In these workshops we adopted a 'data carpentry' approach (Teal et al., 2015), intended to provide a basic grounding (a toolbox) in computational approaches to analysis, within a particular disciplinary context.This approach is similar to 'recipe' based workshop metaphors, which seek to apply analytic ingredients in routinized ways (recipes) to problems (for example, Jaakonmäki et al., 2017).These workshops begin by mapping lower level text features to simple pedagogic aims (are key citations present, are there low level errors such as long sentences, etc.), and developing these to understand the potential of more advanced NLP for feedback to students.This approach was taken as a way to induct educators into computational thinking of text analysis, and those with more technical backgrounds into the application of this text analysis to learning contexts.

Challenge 1: How do we understand
scale of impact in learning analytics?

The Challenge
Learning analytics and other data informed educational decision-making hold significant potential to impact learning.However, the kind of intended impact is not always clear.Indeed, in learning contexts, features of the learning environment that are specific to particular sites, including the learners themselves, make achieving impact challenging.Learning Analytics thus faces a challenge: Challenge 1: How do we understand scale of impact in learning analytics?
Ambiguity in understanding scale of impact is compounded by a tension in approaches to developing learning analytics.Specifically, we see two core tensions in implementing learning analytics for impact: First, population versus learning gain scale: In designing learning analytics, we often face conflicts between a desire to 'reach' as many students as possible (i.e., large scale cohorts), and that of making large scale impact on learning (i.e., that learning gains increase compared to other possible interventions).For example, in our own work we have balanced this tension in reporting the number of users of particular tools, versus indicators of impact on users' learning (and the size of that impact).Similarly, a focus on data that is relatively easy to obtain by targeting users that are easy to reach in large-scale settings, predominantly online, can be in opposition to an approach that derives impact by actively looking for sites in which there is clear potential to improve learning (in particular target users, or targeting particular misconceptions or skill development, for example).Recognising this tension between numbers of users and learning impact gain encourages us to consider this balance, and the particular cohorts that our learning analytics target (perhaps ones who are already privileged in learning systems).
Second, generalisability and adoption: Because of the desire to develop learning analytics with robust models for larger numbers, a portion of work in the field (and allied fields such as Learning at Scale) has focused on learning in large scale settings, rather than on the features of learning that exist -and need supporting -across large numbers of smaller sites, as highlighted in the introduction (Dawson et al., 2014;Ferguson et al., 2016;Ferguson & Clow, 2017;Hoel et al., 2015;Na & Tasir, 2017;Papamitsiou & Economides, 2014;Sclater et al., 2016;Viberg et al., 2018;Zilvinskis et al., 2017).Thus, work has focused on big classes with more general models, rather than on supporting educators to develop their own analytical tools, and to adopt and adapt tools over multiple sites.Indeed, even in such large scale learning contexts, we should be cautious of 'one size fits all' approaches to learning analytics deployment (Gašević et al., 2016).For example, in our own work we have balanced this tension in the desire to identify tasks that may be adaptable across multiple sites, rather than tackling implementation through identifying highly specific tasks or seeking to find the largest cohorts.
These tensions are exacerbated by a preponderance within education analytics research communities to focus on scalability, some exclusively so, such as Learning at Scale (L@S) and Educational Data Mining (EDM).Given that a chief affordance of computational technology is quantitative scalability beyond what is possible with manual human calculation, this focus is not unexpected.However, we contend that while certainly it is possible to achieve scale in learning gains while also scaling populations, a focus on the latter can lead to a disconnect between learning analytics implementations and impact of scale on student learning.This tension has also been observed in the parallel field of Artificial Intelligence in education (AIED), with a discussion of low levels of adoption of AIED by Baker (2016) suggesting that research should shift away from a technical focus on improving AI, to one that uses the same underlying data to develop intelligence augmentation (IA) that instead supports humans in their decision making processes.Thus, learning analytics can be developed to augment existing good pedagogic practices by ensuring that they can be flexibly deployed by educators in their contexts (Knight, 2020).Such an approach has the advantage that it does not fix the tool-context relation in assuming that future students will undertake the same tasks in a closed environment, and act in the same way (Baker, 2016).Moreover, where learning analytics can align closely to existing culture, practice, and resources, they are likely to achieve higher levels of adoption (Ferguson et al., 2014;Zhao et al., 2002).

Practical Impact
What, then, is the practical impact of these tensions?Learning analytics are developed in institutional contexts -whether in university and other educational provider settings, or commercial entities -and as such there is a natural tendency to ask how many students use our tool, and what sites (classes, courses) offer us access to the largest numbers and most data?A consequence of this line of questioning is to build analytics that are either very general (to be applied in multiple large contexts), or that are highly specified, typically to larger contexts.This lends itself to the concern raised in the introduction, that there is a paucity of studies that measure impact in learning analytics research, and examples of instructional adoption.
For learning analytics to achieve impact at scale, we propose a greater focus on augmenting existing practices of teaching and learning, with implementation and impact central from the outset in learning analytics projects.As we will discuss below, such an approach requires evaluation of impact not just in terms of scale, but in terms of individual student learning, and concern for how interventions at one site might be transferred to others (as a method to increase scale of impact).
We suggest that for some research, this would mark a shift away from single-site, site-specific approaches, in which the tool is developed and tested first, with implementation a second and distinct step.The focus on implementation and impact can be seen in centrality in our proposed model (see Figure 1Error!Reference source not found..11 ).
A key consideration in the approach we have taken to developing learning analytics (including the writing analytics tools we have collectively contributed to), is to balance such questions, by noting that when we talk about impact we should not forget we mean impact on learning something specific, supported and assessed through a specific task (or set of tasks).That is, tasks are the site at which targeted learning needs -particular skills, knowledge, or constructs relevant to a particular pedagogical context -are operationalised and measured.Thus, although impact has been discussed in a range of ways in the literature (including retention, engagement, course learning outcomes, etc.), our own focus results in a specific interest in how learning analytics produce change as an intervention, and thus how we evaluate learning analytics in terms of learning that can be related to tasks.
Similarly, to develop sustainable approaches to analytics use, as researchers we need to be able to implement and integrate analytics in such a way that the developer or researcher is no longer required to support the analytics' use.In this approach to implementation, learning analytics is necessarily seen as encompassing more than solely the analytics -it is the analytics put to work in a specific context in a meaningful way.Thus, when taking this definition, scale is required to bring with it a scaling of application in context with meaning.This is a far more challenging endeavour than adding users, as it requires us to continually assess if each added user's learning is indeed being positively impacted by the analytics.Therefore, our response to the challenge of understanding scale has been: Response 1: Implementation and integration into or augmentation of pedagogic practice should be core to the definition of impact, and central to the implementation of learning analytics at scale.Case 1: Writing Analytics Implementations In our implementations we have sought to scale application with meaning in context.We have done this by including capacity building for deployment of analytics in particular sites in the design of learning analytics, in order to understand the analytics and their use.This has involved co-developing resources to on-board other academics into using effective task designs (and contextualised analytics tools) in their own teaching (as we discuss in the following sections).Thus, rather than focusing on technical concerns alone, a key question for us has been how we design these tools and tasks and resources to build a learning analytics infrastructure, so others can adopt them, and building scale through multiple implementation sites that augment the work of the individual academics.Our work thus investigated sites of existing good practice (practice that supports learning), and potential to augment these sites with learning analytics (see, Knight, 2020;Knight, Shibani, et al., 2018).For example, by identifying the reflection process currently used and alignment of this with writing analytics tools (see, Gibson et al., 2017); and in identifying draft submission and revisions processes and augmenting these with formative feedback, and a reflective statement on the feedback received as a part of that process (see papers for details).This approach aims to both develop the sites of augmentation, and provide support for them, both as encouragements to further adoption of both the original pedagogy and learning analytics.

Case 2: Data Carpentry Workshops
Data carpentry workshops are explicitly intended to on-board a broad set of practitioners into use of a particular toolbox of analytic approaches, and to build a community of practice around that learning.The workshops we have run have been about implementation of a design process rather than designed product; that is, they are intended to induct participants into using natural language processing for insight, and the particular design process that we use in our own work, rather than any specific application or pedagogic task (such as standardised assessments).

3.Challenge 2: How do we operationalise improvement in learning analytics?
3.1 The Challenge In the previous section, we suggested that there are tensions between differing approaches to building scalable impact, recommending that increasing attention be paid to transfer of implementations across multiple sites as opposed to targeting large scale single site deployments.A core focus of many, although by no means all, learning analytics approaches has been the drive for predictive validity (Liu & Koedinger, 2017).However, as we argue above, a more important consideration is the impact of our learning analytics on learning.As such, it is important that there is greater focus on the kinds of tasks that are likely to measure and support learning in authentic contexts, including practices that might be augmented with learning analytics.Indeed, there is little consensus regarding how we should evaluate the success of learning analytics and their impact (Kop et al., 2017;Scheffel et al., 2017).As we argued above, such a focus is likely to increase adoption, and thus lead to scaled impact.Moreover, it prompts a shift in focus to make the task the central object in learning analytics design.A key challenge we encountered in our learning analytics work, then is: Challenge 2: How do we operationalise improvement in learning analytics?
For example, in our own work in Case 1, working with teachers forced us to consider how to operationalise improvement for those educators in terms of the students engaging in the task.When working on writing analytics generally, it is possible to assume that this is happening without being confronted by the reality of students actually using the analytics as part of the task.Similarly, in Case 2, in engaging both technical users and teachers in the workshops, we were confronted with the reality that operationalising the LA was more than just producing the analytics and asking the teachers -what can you do with this?It involved thinking carefully about how the analytics could be used in their tasks.
Positioning the task as the central object of learning analytics design is a natural consequence of adopting a pragmatic approach to learning analytics research (Gibson & Lang, 2018).Pragmatism is characterised by a focus on the practical impact of ideas and actions.When applied to learning analytics, this leads us to consider impact in the practical application of the learning analytics, which in turn requires us to consider how (and where, when and with whom) the analytics will be used.In other words, taking a pragmatic approach requires a starting hypothesis that our learning analytics will have potentiality for impact for learning within a particular learning context.Consideration of task is therefore central, as without it, the potentiality cannot be hypothesised.We note that while this approach is common in design based approaches where the artefact (task) is of central concern as a designed object, as well as some areas of the learning sciences like game based learning where the game (task) is primal),it is not yet characteristic of most learning analytics research.
In creating a hypothesis, we are in fact reasoning that this potentiality might in the process of application become actuality such that we observe the impact of our learning analytics on a real learner undertaking a real learning task in a real learning context.Gibson and Lang (2018) suggest that when Peirce's pragmatic maxim is adopted, the practical consequence becomes the whole understanding, and when applied to learning analytics that means our whole understanding of learning analytics is determined by the practical consequences of the analytics: the impact of the analytics on the learning.They suggest five steps are required when adopting this approach: 1. Contextualise requires us to identify the context, the learner and the task, and the salient characteristic of each which may be important to the practicality of the learning analytics.2. Clarify requires us to identify the potentiality, that it may indeed be possible to collect data on, and analyse and make a difference to, those aspects of the learning that we care about.3. Hypothesise requires that we formulate a hypothesis on how the learning analytics we result in real change that matters.4. Apply requires us to put our hypothesis (3) to the test in the right context (1) and in the right manner (2). 5. Evaluate requires that we evaluate the extent to which the hypothesis holds.If so, then we have real impact on learning.If not, we are required to either abandon our hypothesis or reassess it repeating the steps above.
We draw attention to how this pragmatic approach differs from a purely data exploration approach which implies that finding patterns in educational data might help improve learning without making any upfront commitment to do so.We also note that making an upfront commitment requires some basis of making the hypothesis in the first place in which case like challenge one, some measure of theoretical understanding of what is going on at the task level will be necessary.

Practical Impact
Learning analytics design is incomplete when viewed separately from the tasks that they are designed to address or augment.In our own work, these tasks have been developed and iterated along with the tool, with evaluation developed in this context.As such, in designing writing analytics (indeed, any learning analytics) we have not sought to develop tools to be deployed 'in the wild' without context, but instead sought to create close alignment between a flexible tool design, task requirements, learning outcomes, and the pedagogical context (including particular students, educator capacity, program and institutional context, and so on).
To implement learning analytics effectively to achieve impact on learning, the task is central.Tasks go beyond platforms or situations of learning (for example, discussion forums, or online quizzes), to express intentions.It is in learning tasks that the pragmatic maxim is expressed in the relation between activities (or mediating processes) at the implementation site, and their targeted intended learning needs.This is encapsulated in the extension of our model to include the pedagogical domain (see Figure 1Error!Reference source not found..2).
We can also understand how analytics developed around collections of tasks can address particular learning needs, in the form of a theory of change (i.e., that interaction with this tool in these contexts, will create these changes, leading to that outcome).Thus, our second response, and the component of the model, that to achieve implementation and impact there must be alignment with learning needs, and learning tasks within particular pedagogical contexts (institutional settings, cohort dynamics, and so on).Case 1: Writing Analytics Implementations Across our implementation sites, one tool we have used is conjecture mapping (Sandoval, 2014) to align our assumptions about learning outcomes and processes, to task design, and learning analytics integration with that (reported in, Knight, Shibani, et al., 2018).These conjecture maps allow us to create a testable, improvable, model by articulating design and theoretical conjectures through maps that describe how a design is embodied in tools, materials, task and participant structures, and the practices and mediating processes that are observable to us and can be connected to learning outcomes.Abstractions such as these can help us to describe our expectations of how particular tasks lead to particular mediating processes (design conjectures), and how those processes lead to learning outcomes (theoretical conjectures), as in the simplified example below in Figure 2. By developing abstractions such as this, we make concrete assumptions about learning, that can represent existing practices, as well as novel methods of addressing complex problems (Goodyear et al., 2006).
Figure 2: Conjecture map of a revision task for students using automated feedback from writing analytics (adapted from an extended conjecture map in, Knight, Shibani, et al., 2018) In common with other more generic approaches (such as logic models or driver diagrams), what these diagrams foreground is the relationship between the designed structure of tasks, intended activities or observable interactions of the learner, the material objects, and learning outcomes.
Figure 2 describes a task design we have implemented in which students engage in revising a text that we provide them with, and our conjectures regarding the learning in this task.The task has been implemented both with and without analytics augmentation, with mediating processes amended as appropriate, to allow us to evaluate the impact of that augmentation.These kinds of abstractions help us to separate specifics of an instantiation, from the kind of logical pragmatic maxim described above.

Case 2: Data Carpentry Workshops
Our workshop designs explicitly embed the pragmatic maxim by developing approaches that provide meaningful analytics for both the technical developer and the educator in the context of a specific task.Say the assessment criteria of an academic writing requires content coverage and the inclusion of key authors in the field.We can find if those authors are covered by the student in their writing by using Named Entity Recognition, and we can draw on the most appropriate available technology for this particular task of author identification.However, this is incomplete in terms of the learning task.We need to also present the results of the analysis in a pedagogically meaningful way to the student -for example identifying which important authors may be missing from their writing, and drawing their attention to the requirements of the task.This kind of approach jointly satisfies the objectives of both the technical developer and the teacher, but more importantly it positions the approach in terms of its impact on the learning task.This in turn helps both parties to understand both the Named Entity Recognition (especially for those not from that background) and the task (especially for those who are more technically oriented).In our workshops it has been the case that a significant number of attendees have lacked expertise regarding the importance of the field that is not their own (i.e.technical developers with little educational knowledge, and educators with little technical knowledge).In some contexts (e.g. the Learning Analytics and Knowledge (LAK) conference), attendees have been more aware of the importance of both fields.However, even in these contexts the centrality of the task as a unifying focus has generally not been people's starting point, with a typical focus instead being on labelled data for machine learning task.Thus, the workshop designs aim to build community bridges and perspective, and a shared vision for implementation.

The Challenge
We have argued for the centrality of the task in developing learning analytics to be implemented and evaluated across sites.However, a challenge for such a claim lies in the validation of tools to support learning, particularly without gold standard human raters (in classification tasks), and/or large numbers of students (in various prediction tasks).As Wise, Knight, and Ochoa put it recently, how are we to understand "when are learning analytics ready, and what are they ready for?" (Wise et al., 2018), thus our third challenge is: Challenge 3: How do we evaluate the 'readiness' of learning analytics?
As we discuss in our cases below, there is a temptation to withhold LA work from implementation in learning contexts until it reaches a level of maturity.However, doing so means that 'maturity' must be determined independently of the learning task.By contrast, developing LA as close as possible to the learning task allows for readiness to be determined by the potential to positively impact the task itself.Note, this approach corresponds to the kind of shift that has occurred in modern software development.Rather than pre-specifying the software, developing it, then implementing it, a more agile approach is adopted where a minimum viable product is put to work (in the task), and is developed in response to user needs.
As Kitto, Buckingham Shum, and Gibson (2018) note, many of the interesting potentials of learning analytics, relate to asking students to develop their own ability to judge the quality of their work and their practice.Reviews indicate that many current evaluations focus on computational approaches to evaluating educational data mining (Liu & Koedinger, 2017) and leaning analytics (Kitto et al., 2018), with relatively less focus on impact on learning.However, what these approaches to evaluation miss is that improvements in algorithms (however measured) do not necessarily result in an increase in the impact on learning.Indeed, impact may be increased through changes entirely peripheral to the algorithm.In their paper Kitto, Buckingham Shum and Gibson thus suggest that we should "embrace imperfection", as both a necessary part of learning analytics (i.e., perfection is not possible), and a desirable feature of working with learning tools.By this latter claim they mean that it is possible that students learn more by engaging with systems that they must have critical interaction with, precisely because these systems -like human graders -do get things wrong.
As such, rather than algorithmic improvement, we want to aim for improvement in learning.Learning may not be improved by presenting all available features; careful consideration must go into how we map features to feedback (see next section).By overemphasising computational evaluation, we might delay release of tools that could have practical impact in supporting good pedagogy.Similarly, a reliance on computational accuracy can result in uncritical use of an algorithm even if it lacks positive impact on learning.
We need a focus on the kinds of tasks students engage in when learning, and how analytics will be used to augment activity to support that learning.As Lockyer, Heathcote, and Dawson (2013) point out, data captured through learning analytics can help us to test and develop assumptions around how our students learn.Thus, the need for large-scale and highly specified environments may be relaxed in the context of well theorised tasks and measure of learning.Such an approach adopts an iterative design stance, suggesting that both tool and implementation may be developed over time.

Practical Impact
By focusing on the pragmatic maxim -impact on learning -our evaluation of tools shifts from mostly computational approaches, to more socio-technical approaches that keep learning at the heart of our work.For example, within our own work significant focus has been on the implementation of parsers that detect particular features in student writing to provide feedback to students.Given the complexity of language and the context in which it is used, features of text are sometimes hard for even humans to agree on.For instance, the sentence "It is apparent that stronger brand perception is linked to customer loyalty", can be read as expressing background work in one context, whereas it can be thought of expressing an argued claim from a summary of a paper's findings in another.The parser detecting one over the other is a problem when computationally assessing its accuracy against human standard.However, if the end goal of the tool is to bring to attention key features of text to the user, then we may take a cautious approach to balancing focus on small increases in accuracy against investing in tasks that address learning outcomes.As described above, we have engaged in iterative evaluation of these tools, using taskbased evaluation to do so.Our cases draw contrasting accounts of discussions we have frequently encountered to illustrate the significance of this challenge.
Learning analytics are ready when we have good reason -based on the pragmatic maxim -to suppose that they will support learning in some context.We have suggested that by focusing on tasks and their learning outcomes, we can achieve impact in learning analytics.That is, in developing learning analytics we should consider both the task and the tool, and ask in our design: "what impact will this change have on learning?"As we discuss further below, we suggest that a 'design for learning' approach will support this kind of work, by supporting learning analytics designers in connecting the low level features to particular analytic tasks -in the form of user experience, and specific feedback systems -that are aligned with the needs of the task.Thus, our third response focuses on the way that user interfaces intersect with practical sites of implementation, as the closest point of contact to learning tasks, to map features (applied computation), that are limited only by our computational capacity (what the computer can detect) and context (what tools are being used to collect data, data storage, etc.).This is shown as an extension to our model in Figure 1Error!Reference source not found..3, and leads to our third response: Response 3: Embrace imperfection & an approach to computational elements of learning analytics that is targeted at achieving impact on learning by connecting to task oriented approach.Case 1: Writing Analytics implementations Our first example comes from calls to move to machine learning approaches to detect the moves, which sometimes indicate we should place effort in human annotation of the text features for training purposes.Such a shift has implications for both features and feedback.At a feature level, machine learning approaches may use features that are not readily interpretable to humans, or that are rough proxies -such as word counts -about which feedback would not be readily interpretable, nor useful.Moreover, rather than optimising to detect features, we could instead optimise to detect learnable moments.This alternative starting point would be to look for features that might be described as "near misses", for example, we may have more impact on learning by identifying sentences that give background that is intended to highlight contrast with a prior claim, than by increasing accuracy of detection of such contrast moves.That is because these kinds of text have pedagogic interest -if students are failing to instantiate features of interest in their writing, but have other features present, then knowing this allows us to provide feedback.In other words, the focus is on identifying options to provide meaningful feedback that can enhance learning, which might come from relatively less sophisticated approaches, with imperfect algorithms.

Case 2: Data Carpentry Workshops
Second, a common question at workshops and conferences is for people to ask about the accuracy of the tools and parsers; i.e., does it identify the features that it is supposed to (true positives), and not incorrectly identify other features (false positives).However, as we outline above, while of course this is a reasonable question -and one that needs addressing -a focus on accuracy can lead us away from our core interest in student learning.Instead, our focus has been on creating feedback specific to tasks, for example, through working with academics to understand the requirements of texts within their particular contexts, and asking students to reflect -using the automated feedback -on how well they have addressed those needs.The tasks address imperfections in the analytics both by being well grounded in pedagogic theory, that allows us to develop our conjectures regarding mediating learning processes, and by inviting learners to disagree and critique the feedback, empowering the critical use of such tools in learning.

The Challenge
Our first response was to augment practice for impact, but how can we most effectively work with stakeholders to do that?Our second response was to evaluate learning analytics with a task specific focus on learning impact, but how can that be done when contexts vary?Our third challenge requires that we acknowledge imperfection in our tools, while tailoring them to our specific learning tasks, but what is the best approach for doing this?Our responses have left us with questions associated with how we should design learning analytics for task level factors across pedagogical contexts.This brings us to our fourth challenge: Challenge 4: How do we design for implementation across sites?
This is a challenge that must be addressed given the importance of aligning learning analytics with designs for learning.Here again the centrality of the task is key, as tasks provide sites for contextualisation through design for learning processes.This is because tasks set out the expectations for student's activity (expectations that may or may not be met).Tasks define the pragmatic intent for analytics on particular activities (such as writing), defining the learning need they address, and conjectures regarding interventions (such as feedback) to achieve them, based on theory.They are thus crucial in understanding how to align analytics across sites for particular tasks that share some features, but perhaps not others.By exploring these similarities and differences, and the intents behind learning tasks, learning analytics coupled with learning design provides us with the tools to test theories (Lockyer et al., 2013), and support wider pedagogical interventions (Wise, 2014).This potential comes through the combination of scale over sites, and a deep pedagogic understanding of those sites as described through learning designs.
A number of approaches have been proposed to support the alignment between design of the pedagogy and design of the analytics.For example, Alhadad and Thompson (2017) highlight the role of teacher inquiry in data informed practice, while Bakharia et al., (2016), proposed a framework to support aligning learning analytics and design through their understanding of the learning context.In a systematic review (Mangaroska & Giannakos, 2018) of 43 studies that connect learning design and analytics, the need for a framework was highlighted, in order to (1) capture relations between learning analytics and design; and (2) investigate these designs and their impact on learning.However, while there have been a number of recent calls for such a design approach, and indeed a LAK keynote call (Järvelä, 2017) for publication outlets with this focus, there are few examples of such design process in the learning analytics literature.
We propose that it in considering teachers as designers, it is similarly important to consider the role of technology design.Both as a designed artefact in its own right, and the designed affordances of technologies to be flexibly adopted and adapted across contexts; I.e. a dual focus on design of technology and design with technology.How, then, to bring together stakeholders to implement across sites?In our work we have highlighted the potential of participatory co-design that aims at contextualisation of analytics tools and tasks; or, as in Figure 3 below, design in which tasks, assessments, computational features, and the way those features are presented to learners in the user interface (UI) and feedback, are brought into alignment.

Practical Impact
The specific approach we have taken brings together learning analytics and learning design elements to provide contextualized support across learning contexts.In this Contextualizable Learning Analytics Design (CLAD) model (see Figure 3), (1) elements of learning analytics (features and feedback), and (2) elements of learning design (assessment and task design) flexibly interlock and shape each other (Knight, 2020;Knight, Shibani, et al., 2018;Shibani et al., 2019).The CLAD model fosters the development of flexible LA that is relevant and context-sensitive across learning sites, in order to maximise its impact on learning.This design approach emphasizes the connection of technology with pedagogy mediated by the feedback-UI component.In our own work, we have developed designs across multiple contexts that allow us to map instructor's rubrics and learning designs to create tasks in which learning analytics are embedded, with feedback provided that is specific to the learning task.This has in part been facilitated by a 'middleware layer' in our technology, that explicitly maps features to feedback for particular tasks.
Design based approaches support our ability to research, develop, and implement learning analytics in tandem.They thus allow us to align theory and practice to build both, in collaboration with educators.Design abstractions allow these task descriptions to be shared and tested over contexts (Goodyear & Retalis, 2010).The CLAD model emphasises specific concerns for learning analytics; ensuring that these task descriptions and their mediating artefacts (including the learning analytics technology) are brought into alignment, to support contextualisation and adaptation across sites.Design abstractions or patterns are an important feature both for the pedagogic and technological designs, because they emphasise the need for designing for flexibility that supports adaptable alignment over contexts.Design thus holds significant potential in learning analytics, hence our fourth response: Response 4: Design should permeate the pedagogic and technical context of learning analytics, with design patterns providing a design language for this.Case 1: Writing Analytics implementations We have engaged in work to develop learning analytics, and transfer of those across multiple sites, by designing analytics on scholarly and reflective styles of writing, and implementing these across multiple disciplinary contexts (see, Knight, Buckingham Shum, et al., 2018;Knight et al., in submission;Shibani et al., in submission).In these implementations, the educators are brought to the forefront of design process by co-designing feedback and writing interventions for their learning contexts, with agency in bringing LA to authentic practice (Shibani et al., in submission).By using the kinds of abstractions described in Challenge 2, these designs become shareable task descriptions that can be adopted and adapted in new contexts (Knight, 2020;Knight, Shibani, et al., 2018;Shibani et al., 2019).Indeed, we have undertaken such work in order to transfer a co-designed learning analytics augmented task which was developed in the context of legal education, in to an accounting education context (described in detail in, Shibani et al., in submission), exemplifying the application of contextualizable LA at scale.

Case 2: Data Carpentry Workshops
By adopting a principle of sharing resources, the data carpentry workshops presented an opportunity for practitioners (both teaching and technical) from many different organisations to take the workshop methods and materials, and adapt them for their own local context.Within the technical domain, software was made open source with the Apache 2.0 licence and made available via a GitHub repository.The open source license permits copying and modifying and implementing the software with very few restrictions, fostering flexible deployment across varied pedagogic and technical contexts.Similarly, within the pedagogical domain supporting resources have been provided openly under a creative commons licence and made available via the research group's website.Data carpentry workshops further facilitated adapting of the resources by presenting both runnable software and pedagogical documentation in a Jupyter notebook environment, which is also open and shareable.This notebook environment allows for live modification of the software, and editing of pedagogical information (including embedding of media), minimising the friction for adapting for different contexts.Finally, the more complex writing analytics software was provided as a live running service which could be called from the notebooks.This allowed participants without technical expertise to access the tech resources without setting up their own analysis platform, enabling them to focus on the pedagogical aspect of the workshops.

Challenge 5: How do we build institutional capacity for learning analytics?
6.1 The Challenge So how do we build systems to scale across sites, using well-conceived learning tasks that are supported by tools that tackle that learning?And how do we share what we are doing -in the design of our tasks and tools -to build scale, and to improve our approaches?Or, to put it another way: Challenge 5: How do we build institutional capacity for learning analytics?
The challenge of institutional capacity comes down to the kinds of infrastructure required to support implementing learning analytics for impact.As, for example, two Australian reports note (Colvin et al., 2016;Siemens et al., 2013), developing technical infrastructure and expertise to reap the benefits of learning analytics takes time.Importantly, the simple sharing of technical infrastructure is not enough to ensure impactful adoption of learning analytics as a socio-technical field, as the earlier sections have discussed.Our claim, then, is that two kinds of infrastructure must be developed in tandem: 1. Social infrastructure -shared approaches to learning, task design, evaluation strategies, and strategies for integration of the tech (and its adaptability), etc. 2. Technical infrastructure -shared technical platforms that are adaptable to context, allowing for sharing of features, mapping of features to feedback and tasks, and sharing of datasets, and end user applications, etc.
Developing these infrastructures fosters three key benefits, described in the logic model displayed in Figure 4, for internal and external collaboration: 1.It integrates shared evaluation of tools and tasks across contexts 2. It accelerates contextualisation to new contexts 3. It supports creation of shared research and teaching datasets, that can be used to develop new learning analytics, test new models, and as sample materials in teaching.

Practical Impact
The approaches we have outlined above necessitate a greater attention to particular features of technical and social infrastructure.To achieve impact at scale, we have argued that social and technical infrastructure must be developed in tandem, using design patterns that are based around tasks related to specific learning needs to contextualise to particular pedagogical contexts.Thus, our final response: Response 5: A clear attention to technical and social infrastructure to achieve learning analytics for learning impact In our own work, building on these approaches, we draw on two key examples: Co-design and learning patterns; and data carpentry workshops that emphasise discourse between technical and pedagogic communities and practices in learning analytics.

Case 1: Writing Analytics implementations
In the first, we used co-design, and design patterns, to work with academics to understand their practice, and abstract task designs that are augmented with learning analytics, to transfer these designs across pedagogical contexts with a shared learning need (see, Shibani et al., in submission).
In detailed interviews with instructors who have undertaken this work the ability to align the analytics and task design for the specific context was highlighted as an important concern in their adoption of the learning analytics (Shibani et al., in submission).These design patterns have furthermore been shared alongside open source software, in a form that shares both research background, and patterns that describe both the learning design and analytics design for the particular learning context (see discussion in, Knight et al., in submission).In addition to the technical infrastructure that supports the use of an enterprise-level tool across the university, the social infrastructure underlying its usage is also built.Consolidated resources and best practices are shared by academics across disciplines so that a community of users can build upon each other's work.One example from our own work involved bringing together academics from different disciplines, prior to implementation, to share how they might use the software in their own contexts, and encouraging them to share their experiences post implementation through the institution's teaching and learning showcase.

Case 2: Data Carpentry Workshops
In the second, we have run a series of workshops (for example, Knight et al., 2017;Shibani et al., 2018) 2 , in which we have sought to bridge technical and pedagogic communities and practices, through workshops that make use of a version of the model.In those workshops we have used Jupyter notebooks to "demonstrate the process through which pedagogy and analytics come together in order to provide formative feedback to students on their writing."Noting that the key element of this process are:  A clearly defined learning task  An understanding of the students response to the task (the writing)  One or more hypotheses on how aspects of the learning task may be related to computationally identifiable features in the student writing  Throughout the process, the aim is to balance what is possible with the technology (i.e. Natural Language Processing -NLP) with what is valuable in terms of student learning.
It is from the planning of these workshops that the model we present in this paper was derived, as depicted in Figure 5 below.

7.Conclusions and Recommendations for the Field
Learning analytics has potential to be implemented to achieve impact on learning at scale.However, there is a risk that in attempting to implement at scale we reduce our understanding of pedagogical context.For this potential to be achieved we have argued, based on our own design experience, that implementation should focus on integration with practice, including through augmenting existing design.This approach emphasises the task as central to learning, as indicated in the overarching model in Figure 1Error!Reference source not found..6.This is in contrast to a tool-centric approach to increasing the use of a specific tool across contexts or within large-scale sites, or activity-centric approaches which relate activity to outcomes without a well-founded understanding of the context in which that activity occurred.
Our responses to the challenges that we encountered in our learning analytics work present opportunities for the field more generally, providing the basis for a model (summarised in Appendix 1).Based on these responses, we make the following five recommendations for consideration by other learning analytics practitioners.
1. Recommendation 1: Implementation and integration into pedagogic practice should be core to the definition of impact, and central to the implementation of learning analytics at scale that augments existing good practices.2. Recommendation 2: Learning tasks are the central object in developing and evaluating learning analytics, and should receive due attention at both the pedagogic and technical levels 3. Recommendation 3: Embrace imperfection & an approach to computational elements of learning analytics that is targeted at achieving impact on learning by connecting to task oriented approach.The task-centric approach we have argued for has potential to reframe discourse in learning analytics.For example, considering the task and conjectures about its learning impact as central, frames both the design of systems such as dashboards, and the gathering and analysis of low-level trace data; in both cases, our proposals would suggest that analytics avoid abstraction from task and learning need.
The model and recommendations are derived from our own work, and have been instrumental in how we have navigated challenges in this space.Across learning analytics work there are a number of challenges to implementation in addition to those we present here.The recommendations and model that we present here provide one way to frame that implementation across areas of learning analytics.Further work is required to establish the utility of the model across other sites and in differing contexts for learning analytics, for example through user-testing of the model in such contexts.For learning analytics to achieve its potential to impact learning at scale, concerns of implementation and integration into practice must be addressed.How we navigate the challenges implicit in "learning analytics at scale" is crucial to this.In this paper we have used examples from our own work to present recommendations for practice, and a model that supports navigating these challenges.

Figure 1 . 1 :
Figure 1.1:The Centrality of Implementation and Impact in Learning Analytics (Top: General, Bottom: Instantiated model representing our own work)

Response 2 :
Figure 1Error!Reference source not found..2: The importance of the pedagogical domain to implementation and impact.(Top: General, Bottom: Instantiated model representing our own work)

Figure 1Error !
Figure 1Error!Reference source not found..3: The computational context needs to be balanced with the pedagogical context for impact.(Top: General, Bottom: Instantiated model representing our own work)

Figure 1Error !
Figure 1Error!Reference source not found..4: The significance of design patterns.(Top: General, Bottom: Instantiated model representing our own work)

Figure 4 :
Figure 4: Logic model describing impacts of focus on shared technical and social infrastructure

Figure 1Error !
Figure 1Error!Reference source not found..5: Social and Technical infrastructure is required.Error!Reference source not found.(Top:General, Bottom: Instantiated model representing our own work)

Figure 5 :
Figure 5: A writing analytics instantiated model of learning analytics directed toward implementation and impact.
4. Recommendation 4: Design should permeate the pedagogic and technical context, with patterns.5. Recommendation 5: A clear attention to technical and social infrastructure to achieve learning analytics for learning impact Although our model was developed from the perspective of writing analytics, the general nature of the challenges (and our responses to them) suggest that the model could hold value for other areas of learning analytics design practice.To this end we present a generalised version in figure 1.6.

Figure 1Error !
Figure 1Error!Reference source not found..6: A model of learning analytics directed toward implementation and impact.