Redesign of Cognitive Tutor Authoring Tools

Author: Hao Li

Overview

Cognitive Tutor Authoring Tools (CTAT) is a tool for authoring Intelligent Tutors. It is used extensively in research projects at CMU, as well as in the Personalized Online Learning course. With this tool, authors are able to build highly customized learning by doing exercises, and scale them with the mass-production function. It is able to provide step by step hints, as well as error specific feedback. Adaptivity is the sole objective.

However, these features come at a cost. To really achieve the promised adaptivity, a lot of effort is needed in the authoring process - the behavioral graph can be too complex to handle, and the hints and feedback requires a lot of manual input. This is a huge barrier for its broad adoption.

Above all, are Intelligent Tutors really valuable? What value do teachers see in them? Will it eventually change the game of classroom instruction?

Throughout this summer, a team of four METALS students, with the guidance of Prof. Vincent Aleven and Jonathan Sewall, explores the perspectives of different categories of CTAT users. With valuable data gathered from the interviews, they plan to make improvements to this powerful tool for broader adoption beyond research projects.

Process

1. Brainstorming - Roses, Buds and Thorns
2. Interview
3. Storyboards
4. Prototyping
5. Redesign

Interview

We divided our desired interview participants into three categories:

• current authors,
• potential authors,
• current teachers

For each category, we came up with a set of questions to ask, targeting at specific aspects about tutor authoring and/or usage.

Heuristic Analysis

Along with the interviews, the team members each conducted a heuristic analysis on the features of the behavioral graph editing in CTAT. Referring to the web article on heuristic analysis, we identified issues with the graph editor and categorized them into the 10 categories listed in the article. The categories with the highest frequencies are: Visibility of System Information, Recognition rather than Recall, and Flexibility and Efficiency of Use.

Storyboards

Based on the findings synthesized from the interview and the heuristic analysis, we designed 10 storyboards, presenting a problem and a solution to it. We conducted speed dating sessions, where we describe the storyboards to the participants and asked them to quickly provide a short evaluation and assign a score from 1 to 5 on how well they thought the ideas would work. We then picked the two most popular ideas - graph editing and community building, and started designing a low-fi prototype.

Prototyping

Check out the prototype!

Future Work

1. One thing we recommended to start is to connect teachers with authors. This could be done both inside the Personalized Online Learning course or outside in a broader community.
2. Make iterations on the prototypes

Acknowledgements

Special thanks to Prof. Vincent Aleven and Jonathan Sewall, who guided us through the whole project, providing valuable feedback and suggestions. Thanks to all participants who provided us with great information that helped us proceed through the project.