A Jacaard two-way agreement score is a measure of the similarity or differences within a set of data. It ranges from 0 to 1. If all the items within a dataset differ, the score will be a 0. If all the items within a dataset are exactly the same, the score will be a 1. Thus, higher scores indicated greater similarity of items in the dataset, and thus higher agreement.

Table 1. Agreement scores for novices and experts

As can be seen in this dataset, Novices and Experts had very similar agreement scores, with a few exceptions (outlined in blue).

We also conducted a visual inspection of the data itself . We looked at the detailed results for both Novices and Experts, as shown in Data Analysis (Part 2: Winners and Tails), and looked for differences between the two groups that were explainable based on experience. We found no strong pattern to any of the differences we saw that was explainable based on experience (aside from the pattern already noted above for Zoom In and Zoom Out).

As mentioned in Data Analysis (Part 3: Cultural Differences), however, we investigated the cause of the seemingly cultural influence on Scroll Up, Scroll Down, Back, and Forward. These actions are directional in nature (people tended to view the Back Action as retrieving a page to the left of the visible page and the Forward action as retrieving a page to the right of the visible page). In each of these, roughly half the participants scrolled one way while the other half scrolled the other way. In looking at this data a little more closely, it shows France and Spain tending to go one direction with many of the other countries going the other direction. At first, we thought this may be a cultural difference. However, when looking at the data closely, we discovered it was actually an experiential difference.

Swiping up to scroll down is a behavior found in most touchscreen devices that do not show scroll bars and that are “finger-driven” touchscreen interfaces. Swiping down to scroll down is a behavior found in most non-touchscreen devices (those that use navigation keys) and in touchscreen interfaces that do show a scroll bar, most commonly found on “stylus-driven” touchscreen interfaces. We hypothesized that experience with finger-driven interfaces may provide a better explanation of the differences that we saw in these actions

So, we classified every user’s phone as a “finger-driven” touchscreen phone, a “stylus-driven” touchscreen phone, or not (“none”) a touchscreen phone at all (made up of the Novices in this study). We then looked at the number of people who own a finger-drive touchscreen phone and the direction that they scrolled, and the number of people who own a stylus-driven touchscreen phone or didn’t own a touchscreen phone at all and the direction that they scrolled. In the end, we found that those who own a finger-driven touchscreen phone were far more likely, in general, to swipe up to Scroll Down, swipe down to Scroll Up, swipe left to go Forward, and swipe right to go Back. Thus, it seems what was once thought to be a cultural difference was in actuality an experiential difference.

Tables 1-4 below show the results of this analysis.

Table 1. Back

Table 2. Forward

Table 3. Scroll Down

Table 4. Scroll Up

>> Follow this research at http://www.humancentric.com/GestureResearch

The first analysis we ran is: did one country use certain gestures more than any other country? To answer this question, we analyzed the top 38 gestures (gestures that had more than 40 people use them) and ran a chi-squared analysis to identify if one country used that gesture more than another. There were no significant differences found in that analysis. Thus, this analysis did not identify any significant differences between countries in their use of the top gestures.

The second analysis we ran was to assess whether one country used certain types of gestures more than another. We classified all the gestures into 2 categories:

1. Direct manipulation
2. Symbolic

A direct manipulation gesture is a gesture where the participant using gestures that relate at least loosely to the real world. So, to move an object, the participant touches the object and then physically drags it to a new location. Or, to scroll, the user will touch the screen and push it one direction or another.

A symbolic gesture is a gesture where the participant creates a symbol that conveys a desired action, such as drawing a question mark for help or making an “x” over an object to delete it.

We ran an Analysis of Variance (ANOVA) to evaluate the use of direct manipulation versus symbolic gestures. We ran an 8 x 2 x 2 ANOVA to analyze the differences shown by 8 Countries (China, Finland, France, Germany, Finland, India, Spain, UK, USA),  2 Genders (Male, Female), and 2 levels of Experience (Expert, Novice). This analysis showed the main effect of Country was significant (p <0.01), as was the interaction between Country and Gender (p < 0.05). The main effect of Country was very interesting and we looked at it in more detail. Subsequent analyses showed the following differences depicted in Figure 1.

Figure 1:

The third analysis we conducted was actually more of a visual inspection of the data itself. We looked at the detailed results from each country, as shown in Data Analysis (Part 2: Winners and Tails), and looked for differences between countries that were explainable based on culture. We found no pattern to any of the differences we saw that was explainable as based on culture (aside from the pattern already noted above that Chinese participants used more symbolic gestures than any other country). We did find a pattern that at first looked like a cultural difference (see the results for Scroll Up, Scroll Down, Back, and Forward), but in further analyses discovered that it was actually an experiential difference, which we will discuss next.

>> Follow this research at http://www.humancentric.com/GestureResearch

To find out the answer to this question, we first sorted the data from most frequent to least frequent. This resulted in a data table that shows the gestures created, the number of people in each country that created that gesture, the number of experts and novices who created that gesture, and the total number of participants overall who created that gesture, all sorted from most frequent to least frequent.

When looking at the data, we wanted to know, statistically, which gesture(s) were created the most often (which we termed the “winner”) and which gestures were created the least often (which we termed the “tail”).

To identify the statistical winners, we ran a chi square test and compared the top gesture to the next top gesture and if there was no statistical difference between them, we grouped them together in the winner group. We then compared the top gesture to the third-top gesture and if there was no statistical difference between them, we added the third-top gesture to the winner group. We continued in this way until we found a statistical difference and then we stopped. Figure 1 describes this process.

Figure 1. Chi-Square Testing

To identify the tail, we did the same thing only we used a Fisher’s exact test (due to the low number of observations) and compared the lowest gesture to the next lowest gesture. If there was no statistical difference, we grouped them together into the “tail.” We then compared the lowest gesture to the third-lowest gesture and again if there was no statistical difference, we added the third-lowest gesture to the tail. We continued in this way until we found a statistical difference and then we stopped. Figure 2 shows this process:

Figure 2. Fisher Exact Testing

Below are all the results for all 30 actions, with the winners at the top and tail at the bottom in gray. The top winners are highlighted in yellow.

[ Accept/Verify | Back | Close Folder | Continuous Scroll | Copy | Cut | Delete | Forward | Help | Home | Magnify | Minimize | Move | Multi-Select | Open Folder | Open Menu | Paste | Pitch | Print | Redo | Rotate | Save | Scroll Down | Scroll Up | Stop Scroll | Undo | Zoom In | Zoom Out | Place Focus | Select Text ]

Accept/Verify

Back

Close Folder

Continuous Scroll

Copy

Cut

Delete

Forward

Help

Home

Magnify

Minimize

Move

Multi-Select

Open Folder

Open Menu

Paste

Pitch

Print

Redo

Rotate

Save

Scroll Down

Scroll Up

Stop Scroll

Undo

Zoom In

Zoom Out

Place Focus

Select Text

Follow this research at http://www.humancentric.com/GestureResearch

The result of the data collection yielded a detailed spreadsheet of all unique gestures (Gesture Glossary) and a results spreadsheet of all the gestures each participant made for each action in every country.

In all, we had 40 participants in 8 countries and 20 participants in 1 country create gestures for 28 actions, yielding over 9,500 gestures in total. Each of these gestures were catalogued according to the lexicon defined by the research team.

As noted in Data Structure, the lexicon collected very detailed information about each gesture.  This detail will prove useful in many analyses. However, for certain types of data analyses, not all of the detail is needed. At times, the detail of the information resulted in gesture differences that were not meaningful. For example, when someone selects a word to cut or copy, it isn’t significant which direction they swipe to highlight that word. Another example: when someone is performing a page-level command (like scrolling), it doesn’t necessarily matter if they perform that gesture over an object or over whitespace. So, we undertook an effort to combine gestures when the differences between them are not meaningful for a particular action.

To combine gestures where the differences between them are not meaningful, we first defined a series of guideline definitions of what is meaningful and what is not. We then had 3 independent researchers apply those guidelines to the unique gestures in the glossary. We then reconciled the different results of those 3 people into a final combination set.

We defined 2 sets of guideline principles, based on the type of action. There are 2 types of actions: 1) those that require action on an individual object on the page (termed Object Level Actions) like delete and close folder, and 2) those that require an action on the page itself (termed Page Level Actions), like scroll and print.

For Object Level Actions, we decided to apply the following guidelines:

• Swipe direction is not meaningful, so combine all swipe actions regardless of direction
• Location of action is meaningful, so leave all detail regarding location
• Combine when the participant performs an action outside the scope of the task, and removing that action leaves an identical gesture to an existing gesture
• Combine repeated gestures into one group regardless of the number of times it was repeated (e.g. “tap”)
• Combine letter/word symbols into two categories: 1) Letter/word other than “X”, and 2) “X”

The actions classified as “Object Level Actions” are:

• Delete
• Open Menu
• Close Folder
• Open Folder
• Accept/Verify
• Paste
• Move
• Place Focus
• Multi Select
• Select Text

For Page Level Actions, we decided to apply the following guidelines:

• Swipe direction is meaningful, so leave all detail regarding direction of swipe
• Location of action is not meaningful, so combine all gestures where location was the only difference.
• Combine when the participant performs an action outside the scope of the task, and removing that action leaves an identical gesture to an existing gesture
• Combine repeated gestures into one group regardless of the number of times it was repeated (e.g. “tap”)
• Combine letter/word symbols into two categories: 1) Letter/word other than “X”, and 2) “X”

The actions classified as “Page Level Actions” are:

• Forward
• Undo
• Cut
• Copy
• Redo
• Back
• Home
• Save
• Print
• Pitch
• Rotate
• Scroll Down
• Scroll up
• Zoom Out
• Magnify
• Zoom In
• Continuous Scroll
• Help
• Minimize
• Stop Scroll

After the gestures were combined, we took one additional step. For five actions (cut, copy, paste, magnify, and open menu), there were actually two steps. For cut, copy, and magnify, the user first had to select text and then perform a command that would cut, copy or magnifiy. Some users combined the two steps into one gesture, which is fine, but many took two distinct steps. We decided the step of selecting the text (cut, copy, magnify) should be split from the command that performed the action, so we split them. We created a new gesture category, called “select text” and put all of the gestures intended to select text into that category, leaving only the gestures that were intended to perform the original action (cut, copy, magnify) in the action’s category.

For paste and open menu, the user first had to place focus on a particular spot on the screen and then give the command that would paste or open the menu. Again, some users combined the two steps into one gesture, but many took two distinct steps. We again decided to split the step of placing focus from the step of performing the command (paste and open menu) and moved the gestures intended to place focus into a “place focus” category, leaving only the gestures that were intended to perform the original action (paste, open menu) in the action’s category. These steps together resulted in a cleaner view of those 5 actions.

Our next step was to analyze the data.

>> Follow this research at http://www.humancentric.com/GestureResearch

All-in-all, the iPhone has delivered on that promise. Nothing demonstrated that better than a recent trip to Paris. I have done a bit of international travel, but this was to be my first time to France. France is famous for not wanting to speak English to tourists and I don’t know a lick of French and I was traveling alone, so I did more than my usual preparations for this trip.

The best thing I did, though, was to take my iPhone and to find 3^rd Party apps that would be useful. After calling AT&T and purchasing an international data plan for one month, I downloaded a number of great applications to help me get around.

Metro Paris – this one was perhaps my favorite. It contained a map of all the metro stations, their names, and schedules. It also determined the best route to get from one station to another. I was able to get around to just about anywhere in Paris on the Metro

Paris – This is an offline map of Paris. It contains names of the main roads and names of all the Metro stations superimposed on the map. Best of all, it was GPS enabled and could plot my location on the map.

Between these two apps, I was able to get around Paris very easily. Using Paris, I could plot where I was and find the nearest Metro station. Once at a Metro station, I could use Metro Paris to get anywhere else in Paris. And, I could do all this without carrying around a map and looking like a tourist the entire time. When I needed greater accuracy or needed to know which direction I was facing, I could use Google Maps. But since that used data and the other apps didn’t, I used Google Maps rarely. With these apps, I never got lost at all and always knew where I was and how to get to anywhere else.

French Pro – This was a great app for translating English into French. It is cleverly laid out to quickly find the French phase you need and even speaks it so you can simply imitate the sounds. It’s not great for learning how to hold a conversation, but it is great for asking a simple question.

With these 3 apps, I had no trouble getting anywhere or asking for anything I needed in Paris. This made my trip to Paris that much more fun and demonstrated the power and value of 3^rd party apps when cleverly designed.

Oh yea, it placed phone calls too.

To help maintain consistency across researchers, all unique gestures were catalogued and uploaded to an online gesture glossary. Tables 1 and 2 below show the data collected for each unique gesture in the gesture glossary.

Early researchers uploaded many unique gestures to the online gesture glossary, while later researchers were able to compare the gesture made by their participants to the gesture pictures in the glossary and re-use the same gesture ID and description.

All gestures were reviewed and entered into the results spreadsheet, the structure of which is shown in Table 3. Because most of the detail describing the gesture is contained in the online gesture glossary, the values entered into the results spreadsheet were limited to the minimum number of items to classify the gesture and enable the data analyses.

Table 1. Lexicon for describing unique gestures.

Table 2.  Lexicon values.

Table 3. Gesture results spreadsheet

>> Follow this research at http://www.humancentric.com/GestureResearch

Here are a few key instructions to all moderators who ran this study:

- The moderator needs to make sure that the participant performs an actual “gesture”! If participants say that they would want to tap a softkey for a particular action then we have to document this finding. But we would then inform the user that there is no such softkey and encourage them to find a gesture, with which they would trigger this action.

- The moderator should encourage the participant to use more than one finger if they think that is best for the action.

- It is important that the participants hold the rig inline with their line of sight. If participants hold the rig at an angle or twisted in one direction, the marks may not be accurate.

- If participants want to perform more than one gesture for a certain action, the moderator asks them to choose the most adequate one.

Action: Multiselect

“Please perform a gesture, with which you would select different items at once.”

Action: Move object

“Please perform a gesture, with which you would move an object (drag & drop).”

Action: Delete

“Please perform a gesture, with which you would delete a file.”

Action: Scroll down

“Please perform a gesture, with which you would scroll down to the next page.”

Action: Scroll up

“Please perform a gesture, with which you would scroll up to the previous page.”

Action: Continuous scroll

“Please perform a gesture, with which you would continuously scroll down a very long list of items.”

“For example, if you were looking at the first item in a list of 1,000 items, how would you scroll to the 500^th item without repeating a gesture?”

Action: Stop scroll

“Please perform a gesture, with which you would stop to continuously scroll a page.”

Action: Open menu

“Please perform a gesture, with which you would open up a menu; e.g. on the computer, you would be using the right click of your mouse and a context menu opens.”

Action: Open folder

“Please perform a gesture, with which you would open a folder to see what files are inside.”

Action: Close folder

“Please perform a gesture, with which you would close a folder.”

Action: Accept / Verify

“Please perform a gesture, with which you would accept /verify a certain action, e.g. you would like to accept that a text message can be deleted.”

Action: Home

“Please perform a gesture, with which you would return to your home screen.”

Action: Help

“Please perform a gesture which you would use to access help.”

Action: Zoom in

“Please perform a gesture, with which you would zoom into a map.”

Action: Zoom out

“Please perform a gesture, with which you would zoom out of a map.

Action: Magnify

“Please perform a gesture, with which you would magnify a text fragment.

Action: Rotate image

“Please perform a gesture, with which you would rotate an image.”

Action: Pitch

“Please perform a gesture, with which you would tilt the view forward and backward when looking at Google Maps.”

Action: Save

“Please perform a gesture, with which you would save changes that you have made on a screen.”

Action: Print

“Please perform a gesture, with which you would print a certain document.”

Action: Minimize

“Please perform a gesture, with which you would minimize an application.”

Action: Back

“Next we will be looking at things you might do in a Web Browser. Please perform a gesture, with which you would move back a page in your browser window.”

Action: Forward

“Please perform a gesture, with which you would move forward a page in your browser window.”

Action: Cut

“Please perform a gesture, with which you would cut out a text fragment”.

Action: Paste

“Please perform a gesture, with which you would paste a text fragment into an existing text”

Action: Undo

“Please perform a gesture, with which you would undo a certain action.”

Action: Redo

“Please perform a gesture, with which you would redo / repeat a certain action.”

Action: Copy

“Please perform a gesture, with which you would copy a text fragment.”

>> Follow this research at http://www.humancentric.com/GestureResearch

As mentioned in the Gesture Research – Protocol (Part 1: Equipment) blog, participants were asked to define their own gesture for 28 actions. The 28 actions were:

• Multiselect
• Move object
• Delete
• Scroll down
• Scroll up
• Continuous scroll
• Stop scroll
• Open menu
• Open folder
• Closer folder
• Accept/Verify
• Home
• Help
• Zoom in
• Zoom out
• Magnify
• Rotate Image
• Pitch
• Save
• Print
• Minimize
• Back
• Forward
• Cut
• Paste
• Undo
• Redo
• Copy

These actions were described pictorially by showing participants a “before” screen and an “after” screen.

Figure 1. The Before screen and After screen for the multiselect gesture

The “before” screen was placed in the gesture rig we created. Participants were asked to dip their fingers in charcoal and make the gesture that would result in the “after” screen. The charcoal left a trace that was recorded by the webcam

Figure 2. The gesture rig

40 people participated in this research from 8 different countries. 20 of the participants were experienced with multitouch interfaces, and 20 of the participants were not experienced with them.  The 8 countries were:

• China
• Finland
• France
• Germany
• India
• Spain
• UK
• US

In addition to 40 people from each of these countries, 20 people from Italy participated.

An example of the tracing that was recorded by the webcam can be seen here:

>> Follow this research at http://www.humancentric.com/GestureResearch

The study concept: Ask 40 people (20 experienced in using multitouch touchscreens, 20 inexperienced) to create a gesture that will elicit software to perform a particular action (e.g. scroll, rotate, multiselect, cut, copy, paste). Then, see what gestures people commonly make and if there is a difference across cultures in those common gestures.

The challenge: Ensure repeatability of the study protocol across 9 different teams.

The solution: A pictorial-based protocol where participants were shown a before screen and an after screen and asked to perform a gesture that would result in the after screen. This protocol relied very little on verbal communication for trying to describe an action where the mere description itself could influence the participant’s gesture and translation of that description across multiple languages would increase the chance of that occurring.

Figure 1. Example Before and After screen for Multiselect Action

The next challenge: How to record the gesture generated. Also, how can the gesture be made on the before picture so we can identify location-specific gestures? And, how can the gesture be recorded and viewed later?

The solution: We created a gesture rig where a webcam was mounted on a neck over a base where the gesture would be performed. The before screen was placed on the base itself. Every country used a gesture rig for recording the gestures made to ensure a consistent video output.

Figure 2. Gesture rig

The gesture rig consists of three primary components:

1. Base
2. Neck
3. Camera

Base

The base is designed to simulate a handheld touchscreen device. Each base held a printed image of the before screen and a protective layer of transparent plastic (i.e., Plexiglas). The Plexiglas fit securely within the base so gestures can be captured on its surface. Users performed the gestures on the Plexiglas after dipping their finger(s) in charcoal.  The charcoal left a trace of the gesture which could then be referenced in pictures and video in later analyses. The Plexiglas was then wiped clean after the gesture was made and a new before screen placed underneath it.

The base was constructed from sturdy lightweight plastic. The base can be held or placed on a surface to provide balance for the attached webcam or better simulate an electronic handheld device.

Figure 3. Base

Neck

The neck, which was made of 3mm thick aluminum, secured the Webcam to the base, managed the Webcam’s cable and maintained a consistent camera position to record the gestures. The neck can be adjusted by carefully bending it toward or away from the base to allow for alternate camera distances.

Figure 4. Neck

Camera

The camera used was any readily-available USB Webcam capable of recording video in 640/480 resolution at 10 frames per second or better.

Figure 5. Sample screen capture

>> Follow this research at http://www.humancentric.com/GestureResearch

A lot.

Some believe we have entered a new era in interaction design.  For many years we’ve been using the mouse, pointstick, touchpad, and of course, the keyboard.  A new interaction convention is gaining popularity now that makes use of the whole human body.

This is the first in a series of blogs about gesture research conducted by the International Usability Partners (IUP). As this research is conducted, analyses and conclusions will be posted here as they are completed. The first couple blogs will provide the background, protocol, and data collection methods. Future blogs will contain the analyses and results.

Touchscreen user interfaces are gaining popularity seemingly exponentially.  Many touchscreens are incorporating gestures, and they are rising just as rapidly.  Touchscreens are most popular on mobile phones, but many products are under development and on the market using touchscreens such as ovens, refrigerators, combines, laptops, automobiles, insulin pumps and many others.  Touchscreens are appearing on an increasing number of consumer products.  And the desire to enable gestures will follow.

Dan Saffer states in his book Designing Gestural Interfaces that “the next several years will be seminal years for interaction designers and engineers who will create the next generation of interaction design inputs, possibly defining them for decades to come.  We will design new ways of interaction with our devices, environment, and even each other. We have an opportunity that comes along only once in a generation, and we should seize it.”

Very few studies appear in the literature on the identification and understandability of gestures on touchscreens.  It is truly an emerging convention. Even fewer, and perhaps even none, have studied the identification and understandability of gestures across cultures.  Yet, many of the products that enable gestures will be intended for a global audience.  Designers have a definite need to know how understandable gestures are, and how universal they are, while the field is still young.

The International Usability Partners (IUP) has run a global study aimed at identifying the most common user-generated gestures for control of a touchscreen within a given country and across cultures.  The (IUP) is an established network of 12 independent usability companies based in 12 different countries (including Belgium, China, Chile, Finland, France, Germany, India, Italy, Spain, Sweden, UK, and USA) who have joined  to provide user experience services worldwide. This research involved collecting user-defined gestures for 28 common commands (like scroll, open, rotate, zoom in, cut, copy, paste) across 9 different countries and analyzing the resultant data for consistency of responses and cultural influence. 40 participants in each country were presented with a “before” screen image and an “after” screen image and requested to make a gesture that will result in that “after” screen image. The gesture made was recorded and catalogued.

The analysis begins soon.

>> Follow this research at http://www.humancentric.com/GestureResearch

About the International Usability Partners:
IUP is a network of equals, run by all partners for their clients’ benefit. Each partner is an expert in global user-experience research and design, specializing in the cultural and linguistic characteristics of their home country. “IUP quality” stands for companies that offer world-class user experience services of consistently high quality in a reliable and efficient manner. All IUP members have a common understanding of the methods, processes and scientific background of the field of user experience (human-computer interaction).

IUP members have effective quality assurance and process improvement programs in place as well as regular professional training for their staff. Members also have state of the art equipment and tools that facilitate in-house and mobile user research. Facilities are conveniently located around the world in 14 cities including: Chicago, Helsinki, Hyderabad, London, Paris, Torino, Munich, and Beijing.