Cristina Conati

There has been increasing interest in non-invasive predictors of Alzheimer’s disease (AD) as an initial screening for this condition. Previously, successful attempts leveraged eye-tracking and language data generated during picture narration and reading tasks. These results were obtained with high-end, expensive eye-trackers. We explore classification using eye-tracking data collected with a webcam, where our classifiers are built using a deep-learning approach. Our results show that the webcam gaze classifier is not as good as the classifier based on high-end eye-tracking data, meaning its AU-ROC, Sensitivity and Specificity are significantly lower. However, the webcam-based classifier still beats a majority-class baseline classifier in terms of AU-ROC, indicating that predictive signals can be extracted from webcam gaze tracking. Our results provide an encouraging proof of concept that webcam gaze tracking should be further explored as an affordable alternative to high-end eye-trackers for the detection of AD.

View record

There is increasing evidence that, when interacting with an AI system, users may benefit from having personalized explanations of the system’s behavior. Providing such personalized explanations requires that the AI system can assess user properties that are relevant for personalization. In the absence of prior information on such user properties, the system must rely on being able to predict them during the course of the interaction, in order to deliver personalized explanations as needed. In this thesis, we investigate the feasibility of predicting three user traits – conscientiousness, need for cognition and reading proficiency – that have been shown to impact the effectiveness of explanations when users interact with an Intelligent Tutoring System (ITS). We discuss results on training machine learning models on eye-tracking data, action data and a combination of both as users interact with the ITS. For the eye-tracking data, we test features generated from summative statistics of a user’s gaze, and those from patterns in sequences of areas of interest generated from a user’s gaze. We provide a detailed analysis of the relative efficacy of such models, and show that prediction above baseline classifiers is possible even during early stages of the interaction, which is crucial for the timely personalization of explanations. Lastly, we examine the feature permutation importance of our best models to gain insight into how they work and relate to literature.

View record

Previous work leveraged eye-tracking to predict a user’s levels of cognitive abilities and performancewhile reading magazine style narrative visualizations (MSNV), a common type of multimodal documentwhich combines text and visualization to narrate a story. The eye-tracking data, used for training theclassifiers, came from a user study, called control, where subjects simply read through MSNVs withoutreceiving any type of adaptive guidance, otherwise known as the control condition. The goal was tocapture the relationship between users’ normal MSNV processing and their levels of cognitive abilitiesand performance and use that to drive personalization. In addition to the control study, two other userstudies were also previously conducted to investigate the benefits of adaptive support, also known asadaptive studies. In these studies, subjects were provided with gaze-based interventions to facilitate theirprocessing of the MSNVs.In the control study, there was no intervention, and the MSNVs did not adapt to the users in any waybecause the idea was to make the predictions based on users normal, unguided MSNV processing andthen use those predictions to deliver the appropriate adaptations. In the adaptive studies, however,interventions influenced the way subjects processed the MSNVs. As a result, their gaze behavior did notrepresent how they would have behaved in the intended control condition, and a classifier merely trainedon the eye-tracking data from these studies would not learn the proper relationship.In this thesis, we propose different strategies for combining the additional eye-tracking data from theadaptive studies with our original data from the control study to mitigate the potential differences and toform more consistent combinations conducive to improved performance. Our results show that theadditional eye-tracking data can significantly improve the accuracy of our classifiers.

View record

Encouraged by the success of deep learning in a variety of domains, we investigate the effectiveness of a novel application of such methods for detecting user confusion with eye-tracking data. We introduce an architecture that uses RNN and CNN sub-models in parallel to take advantage of the temporal and visuospatial aspects of our data. Experiments with a dataset of user interactions with the ValueChart visualization tool show that our model outperforms an existing model based on a Random Forest classifier, resulting in a 22% improvement in combined sensitivity & specificity. This is a larger improvement in performance than that achieved by either the CNN or RNN when considered alone, though all three deep learning models outperform the Random Forest baseline. To investigate this effect and understand the performance increase achieved by the deep learning models, we carried out preliminary investigations using explainable AI methods, from which we derive future directions for exploring performance gains from combining deep learning models.

View record

Our research is a step toward understanding when explanations of AI-driven hints and feedback are useful in Intelligent Tutoring Systems (ITS). We added an explanation functionality for the adaptive hints provided by the Adaptive CSP (ACSP) applet, an intelligent interactive simulation that helps students learn an algorithm for constraint satisfaction problems. We present the design of the explanation functionality and the results of an exploratory study to evaluate how students use it, including an analysis of how students’ experience with the explanation functionality is affected by several personality traits and abilities. Our results show a significant impact of a measure of curiosity and the Agreeableness personality trait and provide insight toward designing personalized Explainable AI (XAI) for ITS.

View record

Previous work has shown that some user cognitive abilities relevant for processing information visualizations can be predicted from eye tracking data. Performing this type of user modeling is important for devising user-adaptive visualizations that can adapt to a user’s abilities as needed during the interaction. In this thesis, we contribute to previous work by extending the type of visualizations considered and the set of cognitive abilities that can be predicted from gaze data, thus providing evidence on the generality of these findings. We also evaluate how quality of gaze data impacts prediction. Finally, we further extend previous work by investigating interaction data as an alternative source to predict our target user characteristics. We present a formal comparison of predictors based solely on gaze data, on interaction data, or on a combination of the two.

View record

Interactive simulations can foster student driven, exploratory learning. However, students may not always learn effectively in these unstructured environments. Due to this, it would be advantageous to provide adaptive support to those that are not effectively using the learning environment. To achieve this, it is helpful to build a user-model that can estimate the learner’s trajectories and need for help during interaction. However, this is challenging because it is hard to know a priori which behaviors are conducive to learning. It is particularly challenging in complex Exploratory Learning Environments (like in PhET’s DC Circuit Construction Kit which is used in this work) because of the large variety of ways to interact. To address this problem, we evaluate multiple representations of student interactions with the simulation that capture different amounts of granularity and feature engineering. We then apply the student modeling framework proposed in [1] to mine the student behaviors and classify learners. Our results indicate that the proposed framework is able to extend to a more complex environment in that we are able to successfully classify students and identify behaviors intuitively associated with high and low learning. We also discuss the trade-offs between the differing levels of granularity and feature engineering in the tested interaction representations in terms of their ability to evaluate learning and inform feedback.[1] Samad Kardan and Cristina Conati. 2011. A Framework for Capturing Distinguishing User Interaction Behaviours in Novel Interfaces. Proceedings of the 4th International Conference on Educational Data Mining, 159–168.

View record

A user-adaptive information visualization system capable of learning models of users and the visualization tasks they perform could provide interventions optimized for helping specific users in specific task contexts. This thesis investigates the accuracy of predicting visualization tasks, user performance on tasks, and user traits from gaze data. It is shown that predictions made with a logistic regression model are significantly better than a baseline classifier, with particularly strong results for predicting task type and user performance. Furthermore, classifiers built with interface-independent are compared with classifiers built with interface-dependent features. Interface-independent features are shown to be comparable or superior to interface-dependent ones. Adding highlighting interventions to trials is shown to have an effect on the accuracy of predictive models trained on the data from those trials and these effects are discussed. The applicability of all results to real-time classification is tested using datasets that limit the amount of observations that are processed into classification features. Finally, trends in features selected by classifiers and classifier accuracies over time are explored as a means to interpret the performance of the tested classification models.

View record

User-adaptive visualization can provide intelligent personalization to aid the user in the information processing. The adaptations, in the form as simple as helpful highlighting, are applied based on user’s characteristics and preferences inferred by the system. Previous work has shown that binary labels of user’s cognitive abilities relevant for processing information visualizations could be predicted in real time, by leveraging user gaze patterns collected via a non-intrusive eye-tracking device. The classification accuracies reported were in the 59–65% range, which is statistically more accurate than a majority-class classifier, but not of great practical significance. In this thesis, we expand on previous work by showing that significantly higher accuracies can be achieved by leveraging summative statistics on a user’s pupil size and head distance to the screen measurements, also collected by an eye tracker. Our experiments show that these results hold for two datasets, providing evidence of the generality of our findings. We also explore the sequential nature of gaze movement by extracting common substring patterns and using the frequency of these patterns as features for classifying user’s cognitive abilities. Our sequence features are able to classify more accurately than the majority-class baseline, but unable to outperform our best classification model with the summative eye-tracking features.

View record

Student modeling has been gaining interest among researchers recently. A lot of work has been done on exploring value of interface actions on predicting learning. The focus of this thesis is on using eye-tracking data and action logs for building classifies to infer a student’s learning performance during interaction with MetaTutor, an Intelligent Tutoring System( ITS) that scaffolds self-regulated learning (SRL). Research has shown that eye tracking can be a valuable source for predicting learning for certain learning environments. In this thesis we extend these results by showing that modeling based on eye-tracking data is a valuable approach to predicting learning for another type of ITS, a hypermedia learning environment. We use data from 50 students (collected by a research team at McGill University, which also designed MetaTutor) to compare the performance of actions and eye-tracking data (1) after a complete interaction, and (2) during interaction when different amounts of gaze and action data are available. We built several classifiers using common machine learning algorithms and techniques, with feature sets that are based on (1) eye-tracking data only, (2) actions data only and (3) eye-tracking and actions data combined. The results we found show that eye-tracking data brings important information in predicting student’s performance for an ITS supporting SRL in both overall and over time analysis. The features used for training classifiers suggest that usage of SRL tools available in MetaTutor can be a good predictor of learning.

View record

In this thesis we investigate the usefulness of various data sources for predicting emotions relevant to learning, specifically boredom and curiosity. The data was collected during a study with MetaTutor, an intelligent tutoring system (ITS) designed to promote the use of self-regulated learning strategies. We used a variety of machine learning and feature selection techniques to predict students‘ self-reported emotions from eye tracking data, distance from the screen, electrodermal activity, and an ensemble of all three sources. We also examine the optimal amount of interaction time needed to make predictions using each source, as well as which gaze features are most predictive of each emotion. The findings provide insight into how to detect when students disengage from MetaTutor.

View record

Research has shown that individual differences can play a role in information visualization effectiveness. Unfortunately, results are limited given that there are so many individual differences that exist, and information visualizations are commonly designed without taking into account these user differences. The aim of this thesis is to investigate the impact of a specific set of individual differences (i.e., user characteristics) in order to identify which of these user differences have an impact on various aspects of information visualization performance. Eye tracking is also employed, in order to see if there is an impact of individual differences on user gaze behavior, both in general and for specific information visualization elements (i.e., legend, labels). In order to gather the necessary data, a user study is conducted, where users are required to complete a series of tasks on two common information visualizations: bar graphs and radar graphs. For each user, the following set of user characteristics are measured: perceptual speed, visual working memory, verbal working memory, and visualization expertise.Using various statistical models, results indicate that user characteristics do have a significant impact on visualization performance in terms of task completion time, visualization preference, and visualization ease-of-use. Furthermore, it is also found that user characteristics have a significant impact on user gaze behavior, and these individual differences can also influence how a user processes specific elements within a given information visualization.

View record

Educational games are designed to leverage students’ motivation and engagement in playing games to deliver pedagogical concepts to the players during game play. Adaptive educational games, in addition, utilize students’ models of learning to support personalization of learning experience according to students’ educational needs. A student’s model needs to be capable of making an evaluation of the mastery level of the target skills in the student and providing reliable base for generating tailored interventions to meet the user’s needs. Prime Climb, an adaptive educational game for students in grades 5 or 6 to practice number factorization related skill, provides a test-bed for research on user modeling and personalization in the domain of education games. Prime Climb leverages a student’s model using Dynamic Bayesian Network to implement personalization for assisting the students practice number factorization while playing the game. This thesis presents research conducted to improve the student’s model in Prime Climb by detecting and resolving the issue of degeneracy in the model. The issue of degeneracy is related to a situation in which the model’s accuracy is at its global maximum yet it violates conceptual assumptions about the process being modeled. Several criteria to evaluate the student’s model are introduced. Furthermore, using educational data mining techniques, different patterns of students’ interactions with Prime Climb were investigated to understand how students with higher prior knowledge or higher learning gain behave differently compared to students with lower prior knowledge and lower learning gain.

View record

Prime Climb is an educational game that provides individual support for learning number factorization skills in the form of hints based on a model of student learning. Previous studies with Prime Climb indicated that students may not always be paying attention to the hints, even when they are justified (i.e. based on a student model’s assessment). In this thesis we will discuss the test-bed game, Prime Climb, and our re-implementation of the game which allowed us to modify the game dynamically and will allow for more rapid prototyping in the future. To assist students as they play the game, Prime Climb includes a pedagogical agent which provides individualized support by providing user-adaptive hints. We then move into our work with the eye-tracker to better understand if and how students process the agent’s personalized hints. We will conclude with a user study in which we use eye-tracking data to capture user attention patterns as impacted by factors related to existing user knowledge, hint types, and attitude towards getting help in general. We plan to leverage these results in the future to make hint delivery more effective.

View record