Paper from CHI 2008

Comments:
Comment #1
Comment #2
Comment #3

Electronic Voting Machines versus Traditional Methods:
Improved Preference, Similar Performance
Sarah Everett, Kristen Greene, Michael Byrne, Dan Wallach,
Kyle Derr, Daniel Sandler, Ted Torous

Link to CHI 2008 paper.

Overview:
An estimated 66 million U.S. citizens submitted their vote in the 2006 Presidential Election using a direct recording electronic (DRE) system. Though they are widely used, little research has been done to prove they are more usable than more traditional methods of voting. This paper compares usability data from a DRE with that of other methods (level machines, punch cards, paper ballots).

Experiment 1:
The first experiment presented the usual DRE voting system. Subjects were presented first with an instruction screen. After that, they were able to vote on each item/proposition in the ballot. Navigating through each page was done using a "Next Page" and "Previous Page" button. After all items had been voted on, all of their selections were summarized on a screen so users could double-check their choices. From there, the ballot could be submitted. After submitting their votes, participants completed a survey (System Usability Scale, measuring satisfaction). The first experiment composed of two separate parts.

Experiment 1A:
In this experiment, the subjects were divided into two groups. The first group was instructed to vote whichever way that they wanted. The second group was given a piece of paper that instructed them how to vote on each issue. Each participant voted using the DRE and an additional method (lever machine, paper ballot, punch card).
Undirected, Directed with no roll-off.

Experiment 1B:
This experiment was similar to the previous, except for one aspect. The subjects were divided into three groups, instead of two. The third group was also given a piece of paper that instructed them how to vote on each issue, but on some issues they were told to not vote.
Undirected, Directed with no roll-off, Directed with moderate roll-off.

Results:
There wasn't a large distinction among the average voting times between the DRE and other methods. The only distinct improvement was with the lever machine. Also, people with more computer-experience took less time to vote on the DRE than other participants with less experience. Based on the results of the SUS, they found that participants preferred using the DRE to all other voting methods, regardless of any individual characteristics (age, computer-experience, etc). They also found that the DRE didn't reduce the number of voting errors that occurred. In conclusion, they found that changing the voting technology didn't result in less voting errors.

Experiment 2:
Voters rarely choose to vote on every issue that they are presented with on a ballot. The further they go down the ballot, the less likely it is that a voter will vote in a given race. Experiment 2 focused on comparing the traditional "sequential" voting design with a system where users could start on an overview page and jump to the individual elections that they wanted to vote on. Though it obviously would reduce the time users spent voting, would it also affect usability or increase voting errors?

Again, participants were divided into two groups. The first group voted using the "sequential" voting system. The second group used the new webpage-like system. The same information-conditions were used: undirected, directed with no roll-off, directed with moderate roll-off. In addition, a fourth condition was added--directed with additional roll-off. In this experiement, all participants voted three times: first using the DRE, then using one of the three traditional methods, then again with the same type of DRE. This time, no surveys were given after voting.

Results:
Of course, they found that the average completion time for the direct DRE (269.9s) was lower than that of the sequential DRE (442.3s); however, no reliable differences were found between the direct DRE and the other traditional voting methods. No differences were determined in any of the three directed information conditions. For the undirected condition, they found that the sequential voting method (910s) was much slower than the direct voting method (205s). This is because participants that used the direct DRE system voted in far fewer races than those using the sequential DRE.

They also found that voting errors occurred more often on the direct DRE than on the sequential DRE. This is due to the fact that almost 25% of participants using the direct DRE prematurely submitted their ballots before they intended. Regardless of the voting method used, over 34% of all ballots that were cast contained at least one error.

Though the direct DRE was much faster than the sequential DRE, the subjective usability ratings for the sequential system were higher. This was attributed to the fact that the direct DRE systems were inaccurate and resulted in more voting errors.