How To Run User Tests At A Conference

How To Run User Tests At A Conference

• By Daniel Sauble


• November 5th, 2014


• PrototypingUsabilityUser Research


• 0 Comments

User contrast is hard. In a universe of flexible program development, there’s a unchanging vigour to iterate, iterate, iterate. It’s challenging adequate to find time to design, let alone get unchanging feedback from genuine users.

For many of us, a thought of doing grave user testing, is a challenging challenge. There are many reasons why: we don’t have adequate lead time; we can’t find adequate participants, or a right form of participant; we can’t remonstrate your trainer to spend a money.

In annoy of this, user contrast is a best proceed to urge your designs. If we rest on anecdotal data, or your possess experience, we can’t pattern a good resolution to your user’s problems. User contrast is vital. But how do we make a box for it and indeed do it?

What Is User Testing?

Let me start by defining what user contrast is, and what it is not.

User Testing Is…

• Formal
  
  Your thought is to get qualitative feedback on a singular pattern iteration from mixed participants. By gripping a sessions matching (or as identical to one another as possible), you’ll be means to suss out a commonalities between them.


• Observational
  
  Users don’t know what they need. Asking them what they wish is frequency a winning strategy. Instead, you’re softened off being a wordless observer. Give them an interactive pattern and watch them perform genuine tasks with it.


• Experimental
  
  At a core of any user investigate is a tiny set of 3 to 5 pattern hypotheses. The thought of your investigate is to countenance or nullify those hypotheses. The subsequent iteration of a pattern will change accordingly.

User Testing Is Not…

• Ad-hoc
  
  Don’t accept what a singular chairman says during face value. Until we get vigilance from several people that a pattern is flawed, secrete judgment. Once 5 or 6 participants have given unchanging feedback, change a design.


• Interrogative
  
  Interviews are useful for training about users, their roles, and their experiences. But keep it brief. Interviews tend to put a concentration on what people contend they do, not what they indeed do.


• Quantitative
  
  Because a representation distance is small, we can’t make clever statistical extrapolations formed on numbers alone. If we caring about numbers, demeanour into surveys, telemetry, and self-guided usability tests instead.

What Is A User Study?

A user investigate is a investigate project. It starts with a tiny set of pattern questions. You take those questions, reformulate them as hypotheses, digest a devise for validating a hypotheses, and control 5 or 6 user tests. Once done, we summarize a formula and confirm on subsequent steps. If a commentary were clear, we competence make improvements to a design. If a commentary were unclear, we competence control an additional study.

¹

You won’t get it right a initial time. Test your design, iterate, and repeat.” (Image credit²)

A Good User Study Has Clear and Measurable Outcomes

If we have transparent expectations, it will be many easier to take movement on what we learn. This is mostly achieved with hypotheses: testable statements we assume to be loyal for a functions of validation. Examples of good hypotheses include:

• Users can supplement an object to their selling transport and check out within 5 minutes.


• Users wish to click on server-related blunder messages to see additional details.


• Users are not undone by a miss of a dashboard in a product.

A Good User Study Is Easy to Facilitate

This is generally vicious if we are not a facilitator. If a monitor is fresh with user testing, you’ll need to yield a exam book that is easy to understand, keeps a exam on track, and explains what we are perplexing to learn from a test.

A Good User Study Must Be Sufficiently Detailed and Interactive

If we wish to magnitude a user’s greeting to an on-screen animation, we substantially need a coded prototype. If we need to confirm either a sold shade can be wanting from a final design, a set of PSD mockups will do. Needless to say, this is a lot of relocating pieces. Effective user studies are rigorous, and rather costly to lift off as a result. If we cut corners, we competence second-guess your formula and need to run another investigate to be sure.

Self-Evaluation

That’s what user contrast is. Now, ask yourself a following questions:

• Do we control user tests?


• Are they a unchanging partial of your practice?


• Would we like to do some-more of them?


• What’s gripping we from doing some-more of them?

I ask these questions often. It’s extraordinary how few of us do user contrast with any consistency, myself included. Everyone wishes they did some-more of it. That’s both a problem and an opportunity.

User Testing In An Agile World

The flexible mantra is “fail fast, destroy early”. The faster we fail, a faster you’ll intersect on a right solution. This equates to a lot of parsimonious iterations. Agile teams traditionally have two-week sprints, with a thought of releasing a using (read: testable) build during a finish of any sprint.

Great, right? The problem is that this leaves really tiny time to countenance a design, promulgate a results, and do just-in-time pattern for a subsequent iteration. Recruiting can take a week in itself, to contend zero of a testing.

And that’s not tenable. At most, you’ll have a few days to get some actionable insights before a subsequent iteration starts. How competence we solve this problem?

Let’s make a few assumptions:

• Five iterations from a start to a finish of a pattern process.


• Five participants in any user exam (25 participants for all iterations).


• Four designs in moody concurrently (five iterations each, 100 participants in total).

One proceed to solve a problem of removing out in front is to validate mixed iterations before any program is built. Not any pattern needs a live-code antecedent to countenance it. Sometimes, a clickable Balsamiq PDF is enough. Now, we’ve shifted a problem. The array of pattern iterations (and a array of exam participants) is a same as before, yet we can get a lot serve before engineering starts building anything. You customarily need a lot of participants, fast.

User Testing At Conferences

Unless you’re propitious adequate to pattern a product that millions of people use, recruiting can be a challenge. Since we pattern program for complement administrators, a best place to get qualitative feedback in a matter of days is during an IT conference.

The elementary stairs are:

1. Pick a conference


2. Write some studies


3. Set adult your booth


4. Analyze a formula (in genuine time)


5. Iterate on a design


6. Rinse and repeat

Obviously, you’ll need help, so pierce some volunteers with you. Also, don’t pattern to spike this a initial time we try it. Give yourself a possibility to make mistakes and learn from them.

³

Conferences: a best place to control a lot of user tests in a really brief volume of time. (Image credit⁴) (View vast version⁵)

The array of times we can iterate depends on what you’re learning. If you’re training a lot, keep going. If you’re using into apparatus limitations, it competence be time to stop and have your growth organisation build we a live-code prototype.

Bonus: if we have program growth skills, we competence be means to build a antecedent yourself. Better yet, pierce some developers with you.

Disclaimer: I’ve finished conference-based user contrast twice, and haven’t wholly nailed these stairs (even yet we’ve finished good strides in a right direction). It competence take a few tries to get it right.

Attempt #1: PuppetConf 2012

Once a year, Puppet Labs hosts PuppetConf, a tech discussion for IT professionals. In 2012, it was reason during a Mission Bay Conference Center in San Francisco and 750 people attended.

Two of us prepared 5 studies and set adult 3 user contrast stations in a jammed hallway. Each user contrast hire consisted of a laptop, a smoke-stack of exam scripts and NDAs, and a proffer to assistance promote a tests. We had about 16 volunteers, and ran 50 user tests.

⁶

Mission Bay Conference Center during UCSF, a site of a 2012 user testing. (Image credit⁷) (View vast version⁸)

This was a good experience, yet we didn’t get many actionable investigate out of it. Our concentration was on information gathering. We didn’t worry to investigate that information until weeks after a conference, that meant it had collected dust. In addition, a things we tested weren’t on a product roadmap, so a investigate wasn’t timely anyway.

Attempt #2: PuppetConf 2013

In 2013, we steady a user contrast experiment. That year, it was reason in a Fairmont San Francisco hotel and 1,200 people attended.

Five of us prepared 6 studies and set adult 3 user contrast stations in a room adjacent to a jammed hallway. We combined dedicated lapel mics and three-ring binders to keep a scripts organized. With a same array of volunteers (16), we ran roughly twice as many user tests (95).

This year was vastly some-more successful than a prior year. We pulled investigate into a eventuality itself, so we got actionable information some-more fast than before.

⁹

Fairmont San Francisco, a site of a 2013 user testing. (Image credit¹⁰) (View vast version¹¹)

Unfortunately, we didn’t go a additional step of iterating on what we schooled during a conference. Our product wasn’t influenced until months later. It was a step in a right direction, yet too delayed to be deliberate agile.

What Did We Learn?

In 2012, we finished a vast array of mistakes, yet we schooled from those mistakes, softened a tests and contrast process, and doubled both a peculiarity and apportion of a tests in 2013. So, don’t be fearful of failing. A bad user contrast knowledge will customarily assistance we learn and urge for subsequent time.

Here are some of my observations from those experiences.

Conferences Let You Cut a Fat out of Recruiting

Recruiting is really time-consuming. We have a full-time position on a investigate organisation during Puppet for that really purpose. But during conferences, people are already benefaction and peaceful to rivet with you. All we need to do is uncover up.

In a standard user study, we send out a screener email to 50–100 people in a contrast pool. A lot of people won’t respond, and of those who do, customarily some will accommodate a mandate for a test. It takes time to get adequate current responses, and infrequently we have to dilate a net, that takes some-more time.

Conferences Let You Validate Your Entire Roadmap

In both years we had some-more seductiveness in contrast than we could facilitate. In 2013, a 95 participants who tested with us were distant some-more than we needed.

If we confirm to control self-guided, quantitative usability tests, we can run even some-more tests. In 2014, a investigate organisation had over 200 people take a single usability test.

Conferences Are Chaotic, But Process Can Help

In 2012, we had a elementary four-stage process: greet, recruit, test, and swag.

1. Greet
   
   Every time someone came to a booth, we had a greeter proffer who pronounced Hi and told them what we were doing.


2. Recruit
   
   Next, we asked if they wanted to join a Puppet Test Pilot pool for contrast opportunities around a year. If so, we scanned their badge.


3. Test
   
   If we had a exam hire available, we asked if they wanted to take a 15–20 notation user test. If so, a greeter introduced a member to a monitor during one of a stations.


4. Swag
   
   At a finish of a testing, we thanked any participant, and gave them a singular book T-shirt and a sealed duplicate of Pro Puppet.

This routine worked well, yet there were a integrate of apparent holes. First, we didn’t have a good screening process, so there was no pledge that a member was a good compare for a tests. Second, we didn’t have a devise to fast learn from a tests and act accordingly (see: agile).

To scold these shortcomings, we introduced dual additional stairs in a 2013 contrast process: greet, recruit, screen, test, swag and analyze.

• Screen
  
  At a commencement of a contrast process, a monitor asked a member 6 questions, one for any user test. If a answer was yes, we knew they’d be a good compare for a test.


• Analyze
  
  At a finish of a contrast process, a monitor filled out a brief form. Each user exam was allocated a content field, with a investigate hypotheses alongside. The monitor entered their notes, and noted a effect of any hypothesis.

Conferences Allow Your Competitors to Snoop

We used NDAs to negate this. As an unintended side-effect, they finished a contrast seem some-more disdainful and special, so participants were fervent to pointer them.

In 2013, we switched from paper to digital forms, around DocuSign. From a logistical standpoint, this was a good move. We didn’t have to keep lane of lax stacks of paper after a conference. On a other hand, a signing workflow was rather cumbersome. People had to pointer their initials 3 times and click mixed times to finish a NDA.

Conferences Are a Great Way to Build User Empathy

Ultimately, user contrast is about people, not testing. Both years, we recruited volunteers from non-UX departments within a company: engineering, product, marketing, and sales. It was good to give these people an eventuality to rivet with a users over genuine problems.

And it goes both ways. People adore to speak about their job, their pain points, and how your product or use falls brief of easing that pain. No, anecdotal information isn’t terribly useful in a pattern context, yet it can assistance we build a mental indication of real-world problems.

Conferences + User Testing Is a Scary Combination

As we mentioned, we recruited volunteers from non-UX teams. Many of those volunteers had never conducted user tests before. It was a nerve-wracking knowledge for many of them.

In 2013, we instituted a training routine to get a volunteers adult to speed some-more quickly. To do this, we instituted a array of training meetings.

In a initial meeting, we got everybody in a same room and talked by a contrast routine and a tests during a high level. Next, we pennyless adult into tiny groups of dual or 3 people apiece. In these groups, we had volunteers use facilitating a tests with any other. The exam author attended these as well, to mark areas in need of alleviation or clarification.

If a volunteers were still shaken about a awaiting of user testing, we met with them personally. In some cases, we assured them to lift brazen and run user tests anyway. In other cases, we changed them to a reduction perfectionist role, customarily a purpose of a greeter.

Conferences Are a Black Hole for Data

In a initial year, one of a 3 exam laptops was mysteriously wiped of data. The second year, dual of a laptops were stolen. We mislaid all of a exam recordings on those machines.

The china backing was a post-test investigate we did in 2013. Because a facilitators took such severe notes, and saved those records to a cloud, we defended a data, even yet a tangible recordings were lost.

Process Is King, But Organization Is Queen

Keeping things digital as many as probable helps. If we contingency use paper, don’t use manilla folders. Instead, use three-ring binders with labels to keep your papers collated.

On a digital side of things, cruise carrying a singular folder where all conference-related papers and information live. Use collection like DropBox or Box to keep all synchronized opposite machines. Having internal copies is critical, in box a network goes down, that it substantially will.

Use Retrospectives to Learn and Improve

After a conference, reason a assembly with a core contrast team. For a initial 5 or 10 minutes, write ideas on gummy notes. These ideas should take a form of things to stop doing, keep doing, or try doing. Put these stickies on a whiteboard, underneath a suitable mainstay (keep, stop, or try).

Once everybody runs out of ideas, collect a volunteer. This chairman groups a stickies by thesis (e.g. “communication”, “process”, “people”). Ideally, all boils down to 3 to 5 groups. For any group, find an actionable proceed to urge that area, afterwards allot any movement object to a member of a group. It becomes that person’s shortcoming to possess it.

Should You Add Conferences To Your Toolbox?

Having finished this a integrate of times, it’s transparent that there are pros and cons. No user contrast apparatus or technique is a cure-all, and conference-based contrast is no exception.

Pros

• Lots of participants
  
  Hundreds during a tiny conference, thousands during a center conference, tens of thousands during a vast conference. Take your pick.


• Easy recruiting
  
  Build it and they will come. It helps if we indicate your laptops into a room, and have a designs clearly displayed on their screens.


• Enables fast iteration
  
  You can simply finish 5 or 6 tests in an hour or two. Faster if we have mixed exam stations.

Cons

• Chaotic contrast environment
  
  You know those still usability contrast bedrooms with a mirrored glass? You won’t find those during a conference.


• Travel required
  
  Unless you’re propitious adequate to have a applicable discussion in your city, you’ll substantially need to fly somewhere. This can be expensive.


• Difficult timing
  
  Remember those roadmaps we mentioned earlier? If a pattern proviso doesn’t line adult with a conference, find a opposite proceed to get a investigate we need.

In general, this proceed works good when we have a predicted product roadmap. If we know what you’re going to be building, and when, we can time a pattern proviso to coincide with one or some-more conferences.

On a other hand, if we need a coherence to run tests during a moment’s notice, this proceed won’t work well. In that situation, we suggest carrying a dedicated room for contrast during your company, containing all a apparatus you’ll need.

Tips To Make This Work For You

If you’ve review this distant and consider conference-based user contrast is right for you, great! Here are some tips to assistance we succeed.

• Pick a discussion 5 months in advance
  
  You don’t have to know accurately what you’ll be testing, yet it’s a good thought to have a aim date and venue in mind, so we can start meditative about it.


• Pick a discussion with people who don’t know we exist
  
  Because we ran contrast during a possess conferences, everybody knew about us. This self-selection disposition prevented us from removing a good cross-sample of a intensity market.


• Don’t collect a counter in a busiest hallway
  
  As tantalizing as it competence be to get limit visibility, ask youself if a additional disharmony is value it. In 2013, we picked a counter in a room distant by a half wall from a bustling hallway. As a result, we had good prominence though being in a center of a chaos.


• Don’t write any investigate yourself
  
  The initial year, we wrote 4 of a 5 user studies. As a result, they were challenging to promote and didn’t outcome in actionable data. It’s time immoderate to write a good user exam that validates your hypotheses and is easy to facilitate.


• Don’t report people in advance
  
  When your contrast stations fill up, it’s tantalizing to start a watchful list. Don’t do that. You’ll spin gratified to a list and have to spin people away, even when there seem to be dull exam stations. Be serendipitous about it.


• Practice using any exam on any appurtenance before a conference
  
  Murphy’s law. Need we contend more?


• Go onward and user test
  
  The customarily thing worse than a bad user contrast knowledge is not doing it during all. If we fail, during slightest you’ll learn how to do it softened subsequent time. If we don’t do anything, you’ve schooled nothing.

And that’s it. If we have any questions, greatfully get in hold by Twitter or leave a note in a comments below. Thank we for reading.

Resources

When we initial due conference-based user contrast to my team, we was an novice true out of school. If we could lift this off, so can you. If you’re still intimidated, start small. You can grow your efforts, yet we have to start somewhere.

Here are some of a resources we used in testing:

Tools

• DocuSign¹² for e-signing of NDAs


• Silverback¹³ for recording user tests

Articles and examples

• Everything created by Jakob Nielsen¹⁴


• Puppet Labs’ signup form for Puppet Test Pilots¹⁵

Since doing this, I’ve schooled of others who have finished user contrast during events. Here’s a list of articles with somewhat opposite takes on a process:

• User Testing in a Wild: Research during Conferences and Other Events¹⁶


• 5 Considerations for User Testing during Events¹⁷


• Get Out of a Lab! Run Live User Testing for Omnichannel. We Dare You.¹⁸


• 2 Days, 200 Customers, and Conclusive Results: The New User Testing¹⁹


• Usability Testing during Conferences²⁰

(il, og, ml)

Front page picture credits: Rosenfeld Media
²¹.

Footnotes

1. 1 https://www.flickr.com/photos/raneko/4204026836/


2. 2 https://www.flickr.com/photos/raneko/4204026836/


3. 3 http://www.smashingmagazine.com/wp-content/uploads/2014/10/03-conference-opt.jpg


4. 4 https://www.flickr.com/photos/leweb3/6498827487/


5. 5 http://www.smashingmagazine.com/wp-content/uploads/2014/10/03-conference-opt.jpg


6. 6 http://www.smashingmagazine.com/wp-content/uploads/2014/10/04-puppetconf2012-opt.jpg


7. 7 https://www.flickr.com/photos/greentechmedia/5730027311/


8. 8 http://www.smashingmagazine.com/wp-content/uploads/2014/10/04-puppetconf2012-opt.jpg


9. 9 http://www.smashingmagazine.com/wp-content/uploads/2014/10/05-puppetconf2013-opt.jpg


10. 10 https://www.flickr.com/photos/bradfordcoy/4400862442/


11. 11 http://www.smashingmagazine.com/wp-content/uploads/2014/10/05-puppetconf2013-opt.jpg


12. 12 http://www.docusign.com


13. 13 http://www.silverbackapp.com


14. 14 http://www.nngroup.com/topic/user-testing/


15. 15 https://puppetlabs.com/community/puppet-test-pilots-program


16. 16 https://www.gv.com/lib/user-testing-in-the-wild-research-at-conferences-and-other-events


17. 17 https://www.youeye.com/blog/user-testing-at-events/


18. 18 http://joylab.co.uk/blog/run-live-user-testing-we-dare-you/


19. 19 https://blogs.atlassian.com/2014/10/user-testing-atlassian-summit/


20. 20 http://web.archive.org/web/20090321153629/http://www.dexodesign.com/2007/07/29/usability-testing-at-conferences/


21. 21 https://www.flickr.com/photos/rosenfeldmedia/7171775806/

↑ Back to topShare on Twitter

How To Run User Tests At A Conference