User Testing Civic Design

instructionsWhat happens when you take 25 public and community servants, put them in a room with an authority on user research in the civic sector for a day, and ask them to make origami giraffes using flawed instructions?*

Last month, OpenPlans and Code for America convened a testing and user research workshop, with the goal of getting as many public sector participants as possible. Cyd Harrell, UX Evangelist for Code for America, led the workshop. Workshop participants came from the cities of Philadelphia, Bridgeport, CT, and New York. From NYC agencies, there were representatives from the departments of Information Technology and Telecommunications, Education, Housing, Housing Preservation & Development, and Parks & Recreation; 311; and the Mayor’s Office of Operations.

While not necessarily part of their official job description, all the participants understood that as city agencies put more information and services online, thinking about the end users of that information and services is an important part of service delivery.

The Workshop

bad giraffeCyd’s presentation was a mix of general philosophy, practical tips, and hands-on activities (e.g., the aforementioned giraffe origami, which provided an opportunity for participants to see how test subjects react to an imperfect design task).

Prepare

It should be clear to you why you are doing your research. What exactly are you are asking for–how much time will it take, where will it be? Who are the participants you need? Who will ask them to participate? Who will vouch for you (i.e., an official sponsor or trusted leader)? Will there be a reward or incentive for participation? What are you actually asking them to do?

What is the tech set-up you need? Is it the participant’s smartphone or a computer? Will drawings of a smartphone on sticky notes accomplish what you need? Think about who you are designing for, and what technology they are likely to have.

Find the right participants

If you need feedback from a specific demographic group, Cyd tells us, having an appeal for participants come from the leader of a particular community organization–like a minister, or a principal–can have a lot of impact.

If you’re having trouble recruiting enough participants, Cyd says, remember that zero users will give you zero insights. If you can even get one person to participate in your research, or three people, you’ll get some useful insights. You get about 80% of pure user problems by observing 5 people, and you’ll almost always get to 100% with 15.

Set expectations

Moderators should make it clear that the participants themselves aren’t being tested. Say this upfront: “This isn’t a test of you, it’s a test of the system. There are no right or wrong answers. Positive and negative feedback is equally valuable. Nothing you say that is going to hurt my feelings.”

Customize your script

Write a script for your sessions, and customize it depending on what you want to learn. If you’re testing a shoe-buying website, you’d ask “Please look for a pair of shoes you would like to buy” rather than “Would you pick out a pair of pink pumps?” In more civic technology terms, to test a travel planning app, you’d ask the user to plan his commute, rather than asking “do you take the subway?”

Meet the user with empathy

User research is about people. You should be a warm, neutral observer, Cyd tells us, not a cold scientific researcher. It is a touching and important thing that people have shared this time and these observations with you, especially when it’s something important to them. Though you may want to offer incentives to participants, like a gift card to a coffee shop, just thanking someone is often enough reward for someone helping you. (Chocolate is also nice.)

good giraffeReassure users that their participation isn’t going to adversely affect their experience with their existing technology–and then make sure that it doesn’t.

Stay out of their way

Keep your users talking. You don’t want to lead your users to specific conclusions, so these phrases are useful to keep the conversation going when you’re conducting qualitative research: mm-hm, oh interesting, I hear you, right, okay, uh-huh. Can you tell me a little more about that? When you did X a minute ago, can you tell me how you made that choice?

Finally, Cyd’s advice about reporting user testing results to the design team, management, or client, is to share any kind of success at the beginning of the report. And remember, whatever the results: Always fix the machine, not the person.

You can see the slides from the workshop here.

What’s next?

Everyone who responded to a follow-up survey declared that the workshop gave them ideas for their next user research project and nearly all of them went back to their workplaces and shared what they’d learned with their colleagues, multiplying the productive effect of having spent a day away from the office this way. All of them would like additional opportunities to learn more about design and user testing. As one person put it, she would like to learn “anything that helps city government more nimbly interact with the tech community to better serve the public and increase access to information and services.”

Given this strong interest in other design-related trainings, Code for America and OpenPlans are talking with other partners about bringing Civic Design Camp to NYC this fall. Code for America convened the first Civic Design Camp in San Francisco in April. You can read about it here. If you are interested in attending or sponsoring Civic Design Camp NE, please get in touch.

(Thank you to our hosts, ThoughtWorks, who were extraordinarily welcoming, and to all the workshop participants, which in addition to civil servants also included staff from Kinvolved, Ask Them, Democracy Works, Cool Culture, Transportation Alternatives.)

*You get a bunch of funny-looking giraffes.