Paper from UIST 2008

Comments:
Comment #1
Comment #2
Comment #3

An Application-Independent System for
Visualizing User Operation History
Toshio Nakamura & Takeo Igarashi

Link to UIST paper.

Overview
This paper describes an application created to record and annotate visual representations of operation history. Its purpose is to allow users to find application states easily and quickly.

Related research:
• Chimera system- sequence of small snapshots
• Su's system- history as an annotated diagram
• Mondrian- screen snapshots before and after operations are executed
  

The downside to these applications is that the history-recording system is deeply integrated in the application. These systems can't be easily applied to other existing software. In this paper, the researchers developed an application-independent system that can be used without modifying the existing application.

Visual Design

• Operations with motion are denoted using an arrow
• Operations without motion are denoted using an icon (left-, right-, other-clicks)
  
• Keyboard operations are identified using word balloons
• Snapshots: strobe style (shows movement), inset style (bounding boxes show active regions)

Implementation
GUI events are first sent to an event queue where they are later sent to the specific application to be handled. Instead of modifying the existing application to monitor actions, this system acts as a "middle man" (proxy) between the event queue and the application. First, events are performed and sent to the event queue. Then, before the queue can send them to the application, they are intercepted by the recording system. The recording system tracks all events in an event log.

The system first breaks up these recorded events into basic operations such as "press," "release," and "click." These operations may consist of several lower-level recorded events (click = click + action + repaint). After these are determined, sequences of the same event are converted into "general events." For example, using the mouse wheel to scroll down a page would generate many "wheel" events. These events would be consolidated into one general "iterative operation event." Finally, "semantic operations" can be determined from the different operations. The mouse dragging an object across the screen could consist of clicking the mouse, then dragging, then releasing the mouse button.

Filtering rules can be customized to limit what type of actions the system records. Some user actions, such as erratically moving the mouse around the screen, are pointless to record. Other interactions can be clustered together to be considered a single action.

Finally, the snapshots are annotated using the collected event information using one of the styles (normal, strobe, inset).

Prototype System

• Two areas: upper shows details of particular scene and lower shows the storyboard
  
• Right-click on scene to toggle focus on entire window or only region of interaction
• Specify an area to search for all related user interactions in that area
• Execution states can be restored from each scene in the storyboard

User Study
A study was performed to determine if users could understand the annotated history generated by the system, and whether viewing the history would improve the performance of the users.

The users were first allowed to view a 150sec video (as many times as they wanted) about how to use the system. Then, the users viewed an operation history (sequence of annotated images). Once they started, a dialog containing an image that represented an operation (drawn manually and not system-generated) popped up. They were tasked to find that image in the operation history. Users did this for ten different images. Two groups of six people were formed. Each group consisted of three expert users and three inexperienced users. The experiment was performed using three different types of applications: sketch-based modeling system, icon manipulation, misc GUI widgets.

Results

• No significant differences between completion times among different applications
• Some interfaces were much easier to use with the annotated history
• Annotated history generally outperformed the snapshot history
• Annotated history could be useful for applications that have sketches or gestures
• Using the annotated history resulted in lower error rates for the modeling and GUI systems
• Though the number of errors associated with using the annotated history was higher for the icon manipulation application, the users still finished faster overall
• A survey revealed that all users preferred the annotated history and thought it was easy to understand