A major issue with ITS is the cost in terms of money and effort of development. Nye (2014) states that “cost does not appear to be a primary barrier” to the adoption of ITS by teachers, but the context in his research is teachers using ITS and not from the authoring perspective. For a higher education institution like UP, the customization cost will always be an issue given how fast a program’s discipline or field changes, preference of faculty for certain perspectives and/or pedagogy, culture, and obscurity (highly specialized field with very few students) of some fields that it does not attract major developers of ITS. There are free open source authoring tools for developing ITS but it does not address the cost in time and effort as development of a tutor is intricate. Domain ontology generators have been developed to automatically create semantic webs of domain but it does not equate to the instruction and feedback needed by the system; the ontology only provides an outline at best.
The problem for the ODeL institution is whether to commit scarce resources to ITS development or customization; or commit the resources to other less costly educational technology with a faster return on investment (at least in terms of student performance). However, one needs to ask if the other technology will be able to adaptively tutor and assess specific competencies that are required to be demonstrated (and assessed as it is being demonstrated) at a distance. Does personalized tutoring and adaptive assessment by a machine at a large scale justify the higher cost of development? What topics and learning goals is it worth investing on ITS? But undoubtedly for a leading ODeL institution like UP Open University, it cannot ignore research in this field, being the leader in distance education in the country.
Another argument for ITS development is like course development: the long front end development has a trade-off with relatively hands-free tutoring and assessment especially in large-scale classes like MOOCs. In other words, there are certain ODeL learning circumstances where partial automation of assessment tasks is unavoidable, and ITS is a good option for accomplishing those tasks.
Teachers may dismiss the technology as a complex flashcard used for rote memorization but ITS has been used to assess not only cognitive knowledge but also metacognition, affect, and practical know-how like troubleshooting electronics and machinery. Also, it can be integrated with online simulations and game-based learning to provide an immersive experience and have those experiences assessed. The experiences may be contrived but assessing performance in an ITS while a student is solving problems on realistic tasks is a lot more authentic than having them answer multiple-choice questions. For some learning goals that require performance, ITS is claimed to be a valid assessment method.
There may also be fear, uncertainty, and doubt about ITS and anything associated with artificial intelligence particularly over the idea that a machine will replace human tutors. Note that this is similar to the fear of distance education (DE) due to the risk of professors losing tenure and being replaced by temporary affiliate faculty. The fear of DE may have basis due to policies of “for-profit universities” in the past but with ITS, it would be foolhardy to replace human assessors and evaluators for all assessment tasks in ODeL. Admittedly, there are a lot of student outcomes that ITS would not be able to properly assess. ITS has assumptions (like decomposition of knowledge, binary correctness and incorrectness of answers, or learning/unlearning, and the precedence rule) that if broken, will render the ITS assessment unreliable and invalid. For instance, ITS cannot tutor nor assess the dissertation writing process.
Learning activities that are difficult to decompose due to the holistic approach (going back and forth in writing) and the need for insight and serendipity breaks the assumptions of ITS. But ITS can tutor and assess how students format their bibliography in a prescribed citation format. ITS can free faculty members from the onerous task of assessing some outcomes in ODeL. Generally, faculty members dislike assessing student works that are tedious and repetitive. Year in, and year out, the students repeat the same errors and the faculty is left to copy-paste feedback mutatis mutandis. The task of assessment is more difficult in ODeL due to the large number of enrolments and the large volume of student outputs. Timely human assessment and feedback become impossible in a Massive Open Online Course (MOOC) with thousands of students. ODeL practitioners should find a creative way to assess by design using ITS in the same manner that instructional design is used to teach by design.
This chapter explores the diagnostic and formative assessment of learning in Intelligent Tutoring Systems (ITS). Due to the high cost and complexity of developing an ITS, the technology has not been systematically implemented yet at the UP Open University. Careful consideration is needed as to why and where ITS should be deployed in an open and distance e-Learning (ODeL) setting. To help ODeL stakeholders deliberate on this issue the chapter offers an analysis of how assessment feedback, continuous assessment, and adaptive assessment are implemented in an ITS. It looks at how different ITS student models are used to diagnose students’ learning difficulties or misconceptions; and how adaptive feedback is automated to correct misconceptions or to assist students in overcoming difficulties. This chapter also provides a survey of the implementation of ITS in the past,considers possibilities for adaptation in the ODeL mode of teaching and learning, and posits potential issues from an educator’s perspective. It is hoped that this work will contribute to the discussion on this educational technology and provide a direction for further research and identification of projects for ODeL where ITS would be most effective.
An Intelligent Tutoring System (ITS) is a type of Intelligent Computer Aided Instruction (ICAI) that possesses knowledge of the domain (expert model), the learner (student model), and the teaching strategies (tutor model). ITS can diagnose errors and tailor remediation based on the diagnosis (Shute & Psotka, 2001). This diagnostic ability of ITS is the concern of this chapter.
Three topics on assessment in open and distance e-Learning (ODeL) are relevant to ITS: adaptive assessment, continuous assessment, and assessment feedback. Underlying all three is automation; something that undoubtedly may be addressed by ITS.
The four principles of assessment that are a concern of ITS are: validity, reliability, purpose, and proportionality (Nye, Mitros, Schunn, Foltz, Gašević, & Katz, 2017). The first three principles are well-known in assessment in ODeL and need not be elaborated here. Proportionality, on the other hand, is a term rarely used in the literature on the principles of assessment. It has been described by Nye and colleagues (2017) as an issue of “whether the assessment measures balance each construct relative to their level of importance to the goals for their intended use.” The same idea must have been expressed by researchers on the principles of assessment as a concern for the weighting of assessment tasks—a balanced distribution and progression of assessment tasks among topics based on complexity, difficulty, and purpose (Nicol & Draper, 2007; Gibbs & Simpson, 2004; James, McInnis, & Devlin, 2002). There is also a recognition of the scarcity of the amount of time that students can allocate to assessment tasks such that assessment workload should be managed across courses and not just within a course (National Union of Students, 2015; James, McInnis, & Devlin, 2002).
ITS shares a common characteristic with computerized adaptive testing (CAT): it adapts assessment to individual students and stops assessing when a metric threshold is reached indicating that a student has demonstrated a learning competency satisfactorily. For example, in a test, there is no need for all students to answer all questions (something akin to “covering” the topics). In addition, at a higher level, ITS allocates more assessment on what have been identified as more important competencies. Unimportant skills like random rote memorization of facts need not be assessed.
A bold conjecture is that ITS-based assessment may even do away with sequestered testing of students, i.e., summative assessment events like final examinations where learning is not allowed (Van Lehn, 2008). The claim is that assessment is done at the same time students are learning in an ITS. A characteristic of adaptive assessment in ITS is the provision of hints, feedback, and the final answer in every step of problem-based learning. This characteristic is also a cause of problems but we will return to that after describing how assessment is done in the step-by-step process of tutoring in an ITS.
Continuous assessment, as understood in ODeL in the course level, is defined by the distribution of final grades among faculty marked assignments and examinations as opposed to end-of-term examination as sole basis of passing or failing a course. Clearly, this is not what we mean by continuous assessment in ITS. In ITS, the continuity of assessment is at the task level, that is, the student is assessed every time she or he completes a step in a learning task. Take note that the path taken by a student for the entire topic being tutored differs per student. So, every time a student answers an embedded question or provides the outcome of a step in a problem, the intelligent tutor assesses the student and uses the result to determine the next step or content that will be presented. In this case, the ITS determines the custom path of learning for that student step-by-step. Another concept of continuous assessment may be the processing of log-data from a learning platform (like a Learning Management System) to find patterns in how students learn, then using that pattern to diagnose students’ difficulties (e.g., risk of dropping the course). This approach is done in learning analytics that uses dashboards and traffic light visualization for teachers. However, there is less structure in the log-data as it records every click of the student and interpreting the pattern in that data in terms of what to do next for the student is problematic. In the ITS, the alternate paths are designed in advance and the predictive validity of assessment leads to action on diagnosed issues with student learning. In other words, the so-called labels or target variables (as they are called in predictive analytics) are designed into the structure of the content of the tutoring system.
What are the ITS components that require this step-by-step assessment? Do note that learning in ITS usually involves problem-based learning which is why we speak of steps to solving the problem.
The four components of ITS are the domain model, learner model, tutor model, and interface.
The domain model is represented in some ITS as a semantic web of some unit of knowledge. This model contains the learning objectives, competencies or outcomes that will be measured in an assessment, and the content of assessment tasks (e.g., instructions for tasks, questions, and answers to questions). The abstract model is usually implemented as a database structure in the ITS software.
The learner model or student model is like a dynamic class record that defines what the student already knows. Some ITS implement this model as a Bayesian Knowledge Tracing (BKT) table in a database that is constantly updated whenever a student works on a step.
Data are then used by the tutor model to determine the next content or problem to be presented to the learner. The tutor model is basically a computer program that evaluates the results of the assessment and performs a pre-programmed decision on how to help the student learn.
The last component is the interface where the format of the assessment is determined such as the type and behavior of questions, as well as the timing of hints and feedback. In the early history of ITS, the interface was just a console with text (Carbonell, 1970). Currently the interface may include multimedia web pages, talking avatars in dialogue-based assessments (Olney, 2008), digital whiteboards that can interpret handwriting (Korea Advanced Institute of Science and Technology Artificial Intelligence and Probabilistic Laboratory [KAISTAIPR], 2009), and even robots (Lee, 2016; Lee, & Kim, 2010).
In a face-to-face (F2F) classroom, all the assessment tasks of preparing the content of a test, printing, administering, scoring, and evaluating are done by a teacher; however, in an ITS, the assessment process is distributed among its components.
The process of the assessment of the learner happens in a double loop fashion. There is an inner loop of assessment, hints, and feedback on steps, and an outer loop for tasks (Van Lehn, 2006).
Note how the inner loop in Figure 1 will continue to give hints until either the student does the step correctly or all the hints are used up, and the bottom-out hint is displayed. The bottom-out hint or last hint tells the student what to do. This can also apply to tutoring systems that present exercises after a non-interactive material like videos is given to a student. In some implementation of this approach, a student is continuously asked questions about the video until a threshold of consecutive correct answers is reached.
When all the steps are completed, the system looks into the learner model for the record of the student then into the domain model for the next topic following rules given in the tutor model.
A gradebook or class record, even on paper, is an abstract model of student performance in a class. Teachers use that record of scores formatively to evaluate what remedial work to give individual students and what changes are needed in the next content of instruction. It is used for summative evaluation to grant grades and review the content of the entire course. ITS learner models can be thought of as a “fancy” automated gradebook (Van Lehn, 2006). Not only does the ITS automatically mark student performance (not just test answers but also physical (Lee, 2016; Lee, & Kim, 2010) performance and emotion (Feidakis, Daradoumis, & Caballé, 2011; Malekzadeh, Mustafa, & Lahsasna, 2015) but also does automatic evaluation.
The ITS learner model is more complex than paper gradebooks that are tables of scores evaluated with a grading scale with an arbitrary passing threshold. Examples of evaluation in a learner model are number of consecutive correct responses (N-CCR), knowledge tracing, constraints-based models, and knowledge space models. These models have been used in deployed online ITS, but there are others that will not be included in this chapter (See Sottilare, Graesser, Hu, & Holden, 2013 for a more comprehensive introduction on learner modelling).
- Number of Consecutive Correct Responses (N-CCR) – similar to computerized adaptive testing (CAT) in that the test does not have a fixed number of questions,but it differs in the way competency on a skill is determined i.e. when the system stops questioning. Rather than using a statistical formula to determine when to stop, N-CCR simply uses a threshold of consecutive correct response. That is if the threshold for a specific skill is three consecutive correct responses (3-CCR), and the learner consecutively answers three questions correctly the system will predict that the student will get the answer to the next question correctly at least 80% of the time (Kelly, Wang, Thompson, & Heffernan, 2015). It then gives the next set of questions for the next skill, otherwise it will loop or continue to give questions on the current skill until the threshold is reached. Mastery of a task is determined simply by the completion of all skill question sets for the task.
The N-CCR method is reported to be used by ASSISTments (Kelly & Heffernan, 2016; ASSISTments, 2012; Feng, Heffernan, & Koedinger, 2009; Feng, & Heffernan, 2010.). Similarly, Khan Academy is reported to have used 5-CCR (Kelly & Heffernan, 2016), but KA-Lite, the offline open source web server for Khan Academy videos and exercises, appears to use 10-CCR (Tibbles, 2015; Hu, 2011; Faus, 2014).
- Model, Knowledge, and Example Tracing – ITS that uses model tracing has a domain model that contains generalized rules or procedures on how to solve a problem (think of the list of rules as the answer key, and the tutor checks the student’s solutions against this answer key). The rules may have been manually created by experts or generated by the ITS. Note, though, that erroneous solutions may also be modeled as rules (called buggy or mal-rules thus giving the ITS the ability to diagnose misconceptions about steps). We assume that the rules are associated with certain skills expected to be demonstrated by students. The action of students, correct and erroneous solutions as matched against the rules and request for hints are recorded (Pavlik Jr., Brawner, Olney, & Mitrovic, 2013). In model tracing, the students would loop through the steps and given hints until one of their solutions matches a rule.
Knowledge tracing (see Corbett & Anderson, 1995) looks at the history of the performance of the students in doing the steps. The performance data of students are then used to update a table of probabilities that would determine if they learned, did not learn, guessed, or forgot a step. The table of probabilities is updated with Bayesian Knowledge Tracing methods (see Baker, 2014; Martori, Cuadros, & Gonzalez-Sabate, 2016). The probabilities are used to determine the probability that the student would understand the next question from that skill question set correctly in the same manner that N-CCR tries to predict the student’s ability to answer the next correct answer; and the probability that the student has learned the skill.
One of the problems with these approaches is that generating or manually writing the rules (which determine what is a correct or incorrect step) is painstaking as even simple acts (e.g., dividing a number) could have many rules. To help mitigate this authoring problem, example-tracing has been implemented in some ITS like Cognitive Tutor Authoring Tools (CTAT) (Aleven, Mclaren, Sewall, & Koedinger, 2009). The approach uses an example solution for a given problem, and the system generalizes the solution automatically into a graph of solutions (called behavior graph; we speak here of a tree graph with nodes connected by edges). The path taken by the student (steps are the nodes) is compared to the solution paths in the graph such that it will allow for alternate solutions other than the one specified in the example by the author or teacher.
- Constraint-based models – if in model, knowledge, and example tracing, the tutor tries to replicate how an expert would solve a problem, in a constraint-based model one only needs to elaborate on the constraints of a problem, e.g., what syntax and semantics of the answers are allowed (see Ohlsson, 1994 for examples of constraints). When the system sees the student violating the constraints, it assumes there is a misconception and a hint or positive or negative feedback is given to the student. If the answer or solution satisfies the constraints then it is deemed correct (Pavlik Jr., Brawner, Olney, & Mitrovic, 2013). In other words, the system tutors only know the correct approach to the solution and does not know the details of the misconceptions. ASPIRE is an online tutor that uses constraint-based modeling (Mitrovic, 2009; Mitrovic, Suraweera, Martin, Zakharov, Milik & Holland, 2006)
- Knowledge Space models – an example of an online tutor that uses knowledge space models is the Assessment and Learning in Knowledge Spaces (ALEKS) (McGraw-Hill Education, 2017). ALEKS provides practice questions on a given topic. An artificial intelligent system selects the sequence of questions a student can work on based on answers to previous questions. In fact, before a student can start learning in ALEKS, she or he is asked at least 20 to 30 questions in an orientation activity used to build her or his initial knowledge space. Afterwards, the space is updated during the actual learning period. ALEKS does not provide a grade but a pie chart that visualizes what the system estimates the student knows about the curriculum as far down as concepts. It will also recommend the next topic for the student.
What is a knowledge space? If a knowledge state and its complement are in the knowledge structure, then the union of the state and its complement state is also a knowledge state of that structure. This knowledge structure is said to be a knowledge space (Doignon, & Falmagne,1999). The knowledge state is the “complete set of problems that an individual is capable of solving in a particular topic” (Falmagne, Cosyn, Doignon, & Thiéry, 2006). The combinations of knowledge states are the elements of a knowledge structure that is visualized as a precedence graph or described in set roster method. The assessor-tutor system’s job is to uncover the “state of the student being assessed, among all the feasible states” (Falmagne, Cosyn, Doignon, & Thiéry, 2006). The result of the assessment has two lists: a list of problems that the student can do, and another list of problems that the student is ready to learn (Falmagne, Cosyn, Doignon, & Thiéry, 2006). For a precise mathematical explanation of knowledge space theory please see Jean-Paul Doignon and Jean-Claude Falmagne’s books Knowledge Spaces (1999) and Learning Spaces: Interdisciplinary Applied Mathematics (2011).
The given examples of learner modeling show that they have something in common when the ITS performs the task of a marker and assessor of an assessment. They all need to have or generate some answer key to be able to determine the correct answers or approaches to solving the problems posed by the system. They all need to record the result of the assessment per step and summarize it to decide on what would be the next task to present or recommend to the student. All of them also provide hints and personalized and timely feedback.
Personalization and timeliness are extremely important principles of assessment in ODeL. But one may think that the machine may provide imperfect feedback because it will fail to comprehend the difficulties of the student. But the same concern is raised against peer feedback (an alternative approach to supplement the limitations of teacher feedback in ODeL). Gibbs and Simpson (2004) conjecture that, “There may be a trade-off between the rapidity and quality of feedback so that, for example, imperfect feedback from a fellow student provided almost immediately may have much more impact than more perfect feedback from a tutor four weeks later.” The same may be said of feedback from a machine.
An additional benefit of feedback from a machine is affective. The student may feel less embarrassed to receive negative feedback from a machine as opposed to a human teacher and thus help her or him try again. The recognition of persistence in the face of failure (a design built-in the system), and even the acceptance of failure is something ITS shares with game-based learning. The art form of finding the Goldilocks zone of motivating students to do repetitive tasks without frustrating them is an active area of research in ITS.
If feedback even from a machine is something ODeL practitioners may find positive, hinting from a machine is a double edged sword. ITS researchers know that there are students who will game the system if hints are offered by request (Baker, Walonoski, Heffernan, Roll, Corbett, & Koedinger, 2008). These students will just keep on clicking until they reach the bottom-out hint that tells them what to do or show the answer to a question. This issue needs further research on effective timing and mode of hinting.
Considering the description of ITS behavior so far, we may ask if tutoring systems are limited to task-level assessment. The next section looks at how tutoring systems adapts and interconnects with other larger assessment systems.
A related concept that can be considered an outer-outer loop of the assessment system is the design of the Intelligent Tutoring Interface for Technology Enhanced Learning (INTUITEL) adaptive Learning Management System (LMS) (Henning, 2014). In an INTUITEL LMS the units of the domain are not tasks but larger units called learning objects– the LMS resources (e.g., readings, videos, games) and activities (e.g., assignments, quizzes, etc.) that most ODeL practitioners are familiar with. What INTUITEL adapts is the sequence called “learning pathways” by which a student works on the resources and activities (Heberle, Henning, Streicher, Swertz, Bock, & Zander, 2013) based on “didactic factors,” (e.g., topic, progress, age, culture, learning styles). In addition, learners can override the recommended learning pathway and the system will recalculate a new set of learning pathways from that decision. Hence, this approach may be combined with ITS as embedded learning objects such that adaptive assessment can be done at the task level and at the course level.
Systems like INTUITEL, CTAT, and ASSISTments try to provide a platform that provides non-programmers reusable tools or component implementation of an ITS to author tutors for multiple domains. The problem with early ITS is that they were customized for specific domains that they tutor. An ITS built to teach Mathematics cannot be reused to teach Language and vice versa.
An open source project that explicitly tries to support multiple learner modeling and hence, different ITS assessments for multiple domains is the Generalized Intelligent Framework for Tutoring (GIFT) (see https://gifttutoring.org/). An interesting process framework suggested for adoption in the design of GIFT is the Assessment Process Framework (APF) (Orvis, et al. 2017). This is a guide that may be used to develop assessments in an ITS. In ODeL, there are clear process frameworks for developing e-examinations and e-portfolio but not for ITS yet. So, this may be a good starting point for developing a similar framework.
Aside from APF the GIFT community also suggests the use of evidence-centered assessment design (ECD) (Sottilare, Graesser, Hu, & Goodwin, 2017) and the experience application programming interface (xAPI) of the Total Learning Architecture (TLA) technology framework (Goodwin, Folsom-Kovarik, Johnson, Schatz, & Sottilare, 2017). ECD is a framework used to support the development of assessments that evaluate performance where paper-and-pencil tests are unsuitable. It also provides processes that look at assessment results as evidence to support or rebut claims about students’ learning.
TLA is similar to a student’s transcript of records (TORs). TLA goes beyond TORs in that its xAPI may be able to include learning experiences from MOOCs, non-formal training, and any informal learning activity under the banner of lifelong learning. An ITS may be an automated provider of experience points for such a framework.
A successful ITS is beneficial to ODeL. It supplements the ability of teachers to provide assessment and feedback by designing the ITS assessment up-front in the same manner that instruction is designed up-front. For large-scale classes like MOOCs, partial automation of the assessment process is critical.
In addition, personalization is an important principle for ODeL. ITS’s adaptiveness personalizes assessment for each student. Not everyone takes the same assessment, at the same length of time and at the same time, but the assessment can still be valid and reliable.
ITS supports continuous assessment while the student is learning. Timely automation and frequent detailed feedback support students’ self-regulation and self-directedness. They do not have to wait for the teacher to find out where they made a mistake in an assessment; they can try again to correct their mistake or to explore alternative solutions. ITS can be used for formative assessments in online courses.
In ODeL, one must constantly look for ways to tip the balance of success or failure of students in their favor. The promise of automated feedback to support self-regulation and reduce alienation of students in distance education is a worthwhile area for empirical research.
The research on this educational technology is increasing probably due to the increasing interest in artificial intelligence. Moreover, under the theme of assessment there appears to be a trend towards convergence of ITS, game-based learning, educational data mining, and learning analytics. Round-table discussions, workshops, and seminars in this area of research in ODeL should be done. National conferences on the topic would help network researchers from different universities in the country determine the state-of-the art of ITS in the country.
There is also the need for an open and free (as in free software) general framework for authoring ITS as the current open source authoring solutions fall short of the openness that open source LMS like MOODLE and Sakai have achieved. Most ITS authoring software would open source part of their tool but there would be an online part left that would prevent users from deploying their own online platform. This is understandable as those who invested in developing these software applications would at least want data from users to improve the software. Therefore, ODeL should consider how it can use open data with other users of an open and free authoring platform without endangering the privacy of students.
Lastly, there is also need to identify priority domains to use as a testbed for piloting intelligent tutors in an ODeL setting. These pilot ITS should be offered as Open Educational Resources (OERs) so that they can be refined by other users.
ITS thinkers are concerned with the same issues as ODeL assessment thinkers — principles, processes, frameworks, and assessment products. They are just using a different vocabulary that may afford ODeL practitioners a way to visualize the assessment process and how students think during and after that process. It can provide a language for posing precise hypotheses about assessment that are testable.
ITS goes beyond automation because it exposes abstract and messy ideas about how students think and learn particularly during the assessment. It allows users to instantiate models of assessment for testing and demonstration. Some of these models are drawn from learning theories; others are stochastic. They provide an alternative vocabulary for explaining why students pass or fail, and how they may be able to overcome failure and correct themselves.
ODeL practitioners can look at ITS through the lens of the distance education theories of equivalency (Simonson, Smaldino, Albright, & Zvacek, 2012) and interaction. Interaction theory states that there are three forms of interaction in distance education: student-teacher interaction, student-student interaction, and student-content interaction (Anderson, 2003). An increase in one form of interaction in an ODeL course may offset a decrease in other forms of interaction, provided they offer equivalent learning experiences. The idea in equivalency theory can be applied. The theory states that one does not need to provide identical learning experiences found in F2F classroom in ODeL, and by extension to the relationships of interactions, i.e. we do not need to provide identical assessment experiences for all students all the time. Equivalent assessment experiences may be used and that does not preclude machines providing the experience. Clearly, ITS is an example of student-content interaction that may offset a drop in teacher presence. But one should look at the relationships of the form of interaction as dynamic over time. There are times when teacher presence and peer presence is high and student-content interaction may be low, and at other times during the course when teacher presence or peer presence is low and student-content interaction is high with ITS.
We ought to see if the claims of ITS on personalization, adaptiveness, and support for self-directedness will find merit in ODeL and MOOCs.
Anderson, T. (2003). Modes of interaction in distance education: recent developments and research questions. In M.G. Moore, & W.G. Anderson (Eds.). Handbook of distance education (pp. 129-144). New Jersey: Lawrence Erlbaum.
Corbett, A. T. & Anderson, J. R. (1995). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4, 253-278.
ASSISTments. (2012, September). This is ASSISTments. [Online digital video]. Retrieved from https://youtu.be/faA4OY1ccPc.
Baker, R. (2014, March 17). Big data: Week 4 video 2 – Knowledge inference Bayesian knowledge tracing. [Online digital video]. Retrieved from https://youtu.be/y0jXUuChTsg.
Baker, R., Walonoski, J., Heffernan, N., Roll, I., Corbett, A. & Koedinger, K. (2008). Why students engage in “gaming the system” behavior in interactive learning environments. Journal of Interactive Learning Research, 19(2), 185.
Carbonell, J.R. (1970). Mixed-initiative man-computer instructional dialogues. [Doctoral dissertation]. Massachusetts Institute of Tech-nology.
Doignon, J.P., & Falmagne, J.C. (1999). Knowledge spaces. Berlin, Heidelberg: Springer.
Falmagne, J.C., Cosyn, E., Doignon, J.P., & Thiéry, N. (2006). The assessment of knowledge, in theory and in practice. In R. Missaoui, & J. Schmid (eds.), Formal Concept Analysis. Berlin, Heidelberg: Springer.
Falmagne, J.C., & Doignon, J.P. (2011). Learning spaces: Interdisciplinary applied mathematics. Berlin, Heidelberg: Springer.
Faus, M. (2014, May 8). Improving Khan Academy’s student knowledge model for better predictions. [Web Log]. Retrieved from http://mattfaus.com/2014/05/improving-khan-academys-student-knowledge-model-for-better-predictions/.
Feidakis, M., Daradoumis, T., & Caballé, S. (2011). Emotion measurement in intelligent tutoring systems: What, when and how to measure. 2011 Third International Conference on Intelligent Networking and Collaborative Systems, 807-812.
Feng, M., Heffernan, N. T., & Koedinger, K. R. (2009). Addressing the assessment challenge in an intelligent tutoring system that tutors as it assesses. The Journal of User Modeling and User Adapted Interaction, 19, 243–266. Retrieved from https://www.sri.com/sites/default/files/publications/can_we_get_better_assessment_from_a_tutoring_system_compared_to_traditional_paper_testing.pdf
Feng, M. & Heffernan, N. (2010). Can we get better assessment from a tutoring system compared to traditional paper testing? Can we have our cake (Better assessment) and eat it too (Student learning during the test), In Baker, R.S.J.d., Merceron, A., Pavlik, P.I. Jr. (eds.), Proceedings of the 3rd International Conference on Educational Data Mining, Pittsburgh, PA., (pp 41–50). Retrieved from https://www.sri.com/sites/default/files/publications/can_we_get_better_assessment_from_a_tutoring_system_compared_to_traditional_paper_testing.pdf.
Gibbs, G. & Simpson, C. (2004) Conditions under which assessment supports student learning. Learning and Teaching in Higher Education, 1, 3-31.
Goodwin, G., Folsom-Kovarik, J.T., Johnson, A., Schatz, S., & Sottilare, R. (2017). Exploring assessment mechanisms in the Total Learning Architecture (TLA). In R. A. Sottilare, A.C. Graesser, X. Hu, & G.A. Goodwin (eds.), Design Recommendations for Intelligent Tutoring Systems: Assessment Methods (Vol. 5). US Army Research Laboratory.
Heberle, F., Henning, P.A., Streicher, A., Swertz, C., Bock, J., & Zander, S. (2013). Advancement of MOOCs with learning pathways. Retrieved from http://homepage.univie.ac.at/christian.swertz/texte/2014_hbmeu/advancement_of_MOOCs_with_Learning_Pathways.pdf.
Henning, P.A. (2014, November 28). INTUITEL introduction video. [Online digital video]. Retrieved from https://youtu.be/rUYHRzfWPWo.
Hu, D. (2011, November 2). How Khan Academy is using machine learning to assess student mastery. [Web log]. Retrieved from http://david-hu.com/2011/11/02/how-khan-academy-is-using-machine-learning-to-assess-student-mastery.html.
James, R., McInnis, C. & Devlin, M. (2002). Assessing learning in Australian Universities. Ideas, strategies and resources for quality in student assessment. Centre for the Study for Higher Education for the Australian Universities Teaching Committee, p.9. Retrieved from http://www.ntu.edu.vn/Portals/96/Tu%20lieu%20tham%20khao/Phuong%20phap%20danh%20gia/assessing%20learning.pdf.
KAISTAIPR. (2009, August 17). Pen-based intelligent tutor for simultaneous equations. [Online digital video]. Retrieved from https://youtu.be/EW1Rg0kJRv4.
Kelly, K.M., & Heffernan, N.T. (2016). Optimizing the amount of practice in an on-line learning platform. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale (L@S ‘16). New York: ACM.
Kelly, K., Wang, Y., Thompson, T., & Heffernan, N. (2015). Defining mastery: Knowledge tracing versus n-consecutive correct responses. In Santos, O.C., et al (eds.), Proceedings of the 8th International Conference on Educational Data Mining, Madrid, Spain, pp. 630-631.
Lee, D.H. (2016, January 31). Interactive robot-based tutoring system. [Online digital video]. Retrieved from https://youtu.be/USkInbtlALw.
Lee, D. H. & Kim, J.H. (2010). A framework for an interactive robot-based tutoring system and its application to ball-passing training. In 2010 IEEE International Conference on Robotics and Biomimetics, Tianjin, China, pp. 573-578.
Malekzadeh, M., Mustafa, M. B., & Lahsasna, A. (2015). A review of emotion regulation in intelligent tutoring systems. Educational Technology & Society, 18 (4), 435–445.
Martori, F., Cuadros, J. & Gonzalez-Sabate, L. (2016). Introduction to student modeling and Bayesian Knowledge Tracing. Retrieved from https://www.researchgate.net/publication/305473599_Introduction_ to_student_modeling_and_Bayesian_Knowledge_Tracing.
McGraw-Hill Education. (2017). Overview of ALEKS. Retrieved from https://www.aleks.com/about_aleks/overview.
Mitrovic, A., Suraweera, P., Martin, B., Zakharov, K., Milik, N., & Holland, J. (2006). Authoring constraints-based tutors in ASPIRE. In M. Ikeda, K.D. Ashley, & T.W. Chan, (eds.), Intelligent Tutoring Systems. Berlin, Heidelberg: Springer.
Mitrovic, et al. (2009). ASPIRE: An authoring system and deployment environment for constraint-based tutors. International Journal of Artificial Intelligence in Education, 19, pp. 155-188.
Mislevy, R.J. (2003). On the structure of educational assessments. CSE technical report 597. Los Angeles, CA: University of California. Retrieved from https://archive.org/details/ERIC_ED480557 .
Nicol, D., & Draper, S. (2009). A blueprint for transformational organisational change in higher education: REAP as a case study. In M. Mayes, D. Morrison, H. Mellar, P. Bullen & M. Oliver (eds.), Transforming Higher Education through Technology-Enhanced Learning. York, U.K.: The Higher Education Academy. Retrieved from https://www.heacademy.ac.uk/knowledge-hub/transforming-higher-education- through-technology-enhanced-learning.
National Union of Students. (2015). Assessment and feedback benchmarking tool. Retrieved from http://www.nusconnect.org.uk/resources/assessment-and-feedback-benchmarking- tool.
Nye, B.D. (2014). Barriers to ITS adoption: A systematic mapping study. In S. Trausan-Matu, K.E. Boyer, M. Crosby, & K. Panourgia, (eds.), Intelligent Tutoring Systems: 12th International Conference, ITS 2014, Honolulu, HI, USA, June 509, 2014 Proceedings. Switzerland: Springer.
Nye, B.D., Mitros, P., Schunn, C., Foltz, P.W., Gašević, D., & Katz, I.R. (2017). Why assess? The role of assessment in learning science and society. In R. A. Sottilare, A.C. Graesser, X. Hu, & G.A.
Goodwin (eds.), Design Recommendations for Intelligent Tutoring Systems: Assessment Methods (Vol. 5). US Army Research Laboratory.
Ohlsson, S. (1994). Constraint-based student modeling. In J.E. Greer & G.I. McCalla (eds.), Student Modelling: The Key to Individualized Knowledge-Based Instruction. Springer Berlin Heidelberg.
Olney, A. (2008, October 16). AutoTutor demo. [Online digital video]. Retrieved from https://youtu.be/aPcoZPjL2G8.
Orvis, K.L., et al. (2017). Towards systematic assessment of human performance interventions in the US army: An assessment process framework. In R. A. Sottilare, A.C. Graesser, X. Hu, & G.A.
Goodwin (eds.), Design Recommendations for Intelligent Tutoring Systems: Assessment Methods (Vol. 5). US Army Research Laboratory.
Pavlik Jr., P.I., Brawner, K., Olney, A, & Mitrovic, A. (2013). A review of student models used in intelligent tutoring systems. In R. Sottilare, A. Graesser, X. Hu, & H. Holden, Design recommendations for intelligent tutoring systems: Learner Modeling (Vol. 1). U.S. Army Research Laboratory.
Shute, V.J., & Psotka, J. (2001, August 3). Intelligent tutoring systems: Past, present and future. In Jonassen, D.H., (ed.), The Handbook of Research for Educational Communications and Technology.
The Association for Educational Communications and Technology. Retrieved from http://www.aect.org/edtech/ed1/19/19-03.html. Boston, MA: Pearson.
Simonson, M., Smaldino, S., Albright, M. & Zvacek, S. (2012). Teaching and learning at a distance. Foundations of distance education. (5th Ed.)
Sinatra, A.M., Ososky, S, & Sottilare, R. (2017). Assessment in intelligent tutoring systems in traditional, mixed mode, and online courses. In R. A. Sottilare, A.C. Graesser, X. Hu, & G.A. Goodwin (eds.), Design Recommendations for Intelligent Tutoring Systems: Assessment Methods (Vol. 5). US Army Research Laboratory.
Sottilare, R., Graesser, A., Hu, X., & Holden, H. (2013). Design recommendations for intelligent tutoring systems: Learner Modeling (Vol. 1). U.S. Army Research Laboratory.
Sottilare, R.A., Graesser, A.C., Hu, X., & Goodwin, G.A. (eds.). (2017). Design Recommendations for Intelligent Tutoring Systems: Assessment Methods (Vol. 5). US Army Research Laboratory.
Tibbles, R. (2015, June 17). KA Lite: Student data records. Retrieved from https://github.com/learningequality/ka-lite/wiki/Student-Data-Records.
Van Lehn, K. (2006). The behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16(3), pp. 227-265. Retrieved from http://learnlab.org/opportunities/summer/readings/06IJAIED.pdf.
VanLehn, K. (2008). Intelligent tutoring systems for continuous, embedded assessment. In C. Dwyer (ed.), The Future of Assessment: Shaping Teaching and Learning. Mahwah, NJ: Erbaum, pp. 113-138.
Tutors Assessment and Learning in Knowledge Spaces (ALEKS). Available at https://www.aleks.com/.
ASPIRE. (2010). Available at http://aspire.cosc.canterbury.ac.nz/ASPIRE-login.php.
ASSISTments. (2016). Available at https://www.assistments.org/.
Cognitive Tutor Authoring Tools (CTAT). Available at http://ctat.pact.cs.cmu.edu/. Its online LMS TutorShop is available at https://tutorshop.web.cmu.edu/.
Generalized Intelligent Framework for Tutoring (GIFT). (nd) Overview Available at https://gifttutoring.org/. Its online LMS GIFT Cloud / Virtual Open Campus is available at https://cloud.gifttutoring.org/.
Khan Academy. (2017). Available at https://www.khanacademy.org/.
KA Lite. (2017). KA Lite: An offline version of Khan Academy Available at https://learningequality.org/ka-lite/.
INTUITEL. Available at http://www.intuitel.de/public-software/.
Cantada, R. P. (2018). Intelligent Tutoring Systems as an Approach to e-Learning Assessment. In M. F. Lumanta, & L. C. Carascal (Eds.), Assessment Praxis in Open and Distance e-Learning: Thoughts and Practices in UPOU (pp. 25-44). Los Baños, Laguna, Philippines: UP Open University
