At EdUI conference watching Dana Chisnell’s workshop.
3 things we want to get out of the day
1 – remote user testing
2 – what makes a successful test
3 – quick & dirty testing
3 things that are scary about user testing
1 – am I reporting the results correctly?
2 – do I have enough participants?
3 – is quick & dirty testing valid?
Today’s agenda:
1 – quick testing – putting sites together quickly in an educational setting; want to be able to feel confident in decisions;
2 – valid testing
3 – concerned about how user testing fits into the process
4 – get continuous feedback from users; constant feedback
4 essential steps:
art and craft involved; there is a rigorous process, but it doesn’t have to be scientifically rigorous.
talk about classic methodology for user testing
classic can take a long time to conduct testing; few teams have the luxury of doing this; lots of pressure to get everything exactly right
you can do classic usability testing; however, you don’t have to do that; you can conduct really good user testing that are super informative without the longer, more classic testing. go to where the users are
4 steps today
1 – planning
2 – recruitment
3 – doing the sessions
4 – communicating with team on results- debrief with observers and decide together
what you need
- someone to test
- somewhere to test
- something to study
get alot out of field studies – going to where the users are
“sit next to someone and watch them do something”
***
demonstration
“son is looking for an agricultural engineering degree. the parent is looking for the dates for getting your application in. and what the fees are”
start at google
search was “university of pennsylvania application deadline”
next search (after not finding the information) “application deadline for university of pennsylvania”
questions asked by dana: 1- what do you think the site is about? 2 – what do expect to find 3 – what’s the definition of early decision? what would you do now that the information is not available? what’s your assumption about fees and whether there is one? what was the best thing about this site? what one thing should be improved?
app fees and application deadlines not together on the site
jargon – inside information – who we are as an institution; very little interest in who the user is and who might come there
interview based tasks instead of scenario based tasks
lots of biases that result from user testing; not a psych experiment; this is learning what the design problems are in an interface
how well does the interface support the goal?
***
why test?
why are designs hard to use?
> teams focus on technology
> audiences change and adapt
> creating a usable design is difficult
> team specialists don’t often coordinate
> design and implementation don’t match
supporting great design
> some testing is better than no testing at all; even 1 person is better than not asking anyone
> ideally input from users at each phase and you’re working on a team that works together
> need support from leadership; if management doesn’t support it, you’ll never fit it
> willingness to learn as you go; have to be flexible about unexpected things
> defined usability goals and objectives
***
where’s the value in testing?
70% watching someone use the design
start getting feedback as soon as possible
focus on:
> what can make the most difference to the most users
> what can be implemented easily with resources available
***
guidelines are good, but…
> these don’t guarantee usability and accessibiliy
***
informed designs come from data
> go to the user and watch them
***
what makes great design?
> usefulness – founders have no idea how ppl use twitter (e.g.); what is valuable to the user
> efficiency – how does the user accomplish their goals
> effectiveness – accomplishing the goal
> learnability – pay attention to this if ppl are not using the application all the time; ppl need to relearn it all the time
> satisfaction – how happy is someone about using the site
> accessibility – easy as possible
***
goals of usability testing
identify problems that lead to :
> misinformation
> incomplete transactions
> need for support from administration or staff
make it more likely that users can reach their goals
***
what should he design do?
how does it work?
does it do what we want it to do?
types of testing
> exploratory, formative
> assessment, summative
> validation, verification
big ideas:
you don’t have to do it by the book
essentials: someone, somewhere, something
simple is difficult
testing supports great design
value comes from observing users
usability testing fits in each phase
****
developing a test plan
anatomy – (lite version)
a. goals and objectives
what obstacles in obtaining goals; not research questions; not data measures;
objective for usability testing of our new design:
1. assess whether the navigation makes sense for different types of users
2. how well does the site work for different types of users?
b. research questions (should be measurable; should have in mind what tasks we have)
1. how well does the user navigate the site
2. does the search box make any difference in how the users reach their goals
3. how well does the site navigation work for end users
4. what obstacles do users encounter when coming to the site
5. does the starting point make any differenc ein the success
c. participant characteristics -who are they?
d. task descriptions
Group 1 -
1. start at the website itself
2. look for the book Maus
3. how long can i checkout a book?
e. description of method (how you’re going to gather the data)
highlevel participant; are you going to sit in the room with them? think aloud? include some reference to where you are in the design cycle
techniques -
think aloud:
> tell me what you’re doing
> tell me what you’re thinking
review at end
> walk me through what you did
> how did that go
>was it confusing or frustrating
within tasks themselves – what are you watching for?
how are you measuing that – leads back to research questions
evidence:
> behavoir
> quotes
> error and success rates
> debriefing discussions
“user participant never fails; the design fails”
can do hints or prompts – geographic; telling them specifically to go to next question
rankings or ratings – doesn’t recommend – complicated
for formative tests – you’re not learning anything when you do time on task
measuring time is good for known tasks; summative testing; benchmark testing;
big ideas
know why you’re testing
gather data to answer specific questions
match user goals to testing
identify evidence of success/failure
determine method, techniques
****
finding participants
develop/document characterics
goals of the study
identify key behavoirs
locate sources
***
focus on functionality
know the goals of the study
what suppports behavoir
implies motivation
behavoir not demographics – they do not matter
what you care about is what the person does; age doesn’t matter
recruit for a specific behavoir
***
where to find participants:
education – not difficult to find
invite users
compensation is an incentive
go where users are (e.g. starbucks on the first floor)
tap communities – facebook, twitter, etc. (reminder: send survey to the libraries facebook group)
***
screening & selecting participants
be involved as the researcher in the recruiting process
don’t hand off to an agency or to an admin
can be sure that your expectations are being met
contact with participants is the best way to get someone to show up
market segmentation – represent preferences and opinions about the business; not behavoir
you can’t represent segments in study b/c you can’t do a big enough study
it’s about identifying design problems, not generalizing preferences to a larger audience
pull characteristics from personas; develop profiles; archetype descriptions
if you want to split ppl into groups, then you can think about demographics
look at requirements and classifiers in recruiting participants
***
screening questionnaires – not always necessary
types of questions that would be useful
> tell me about your job
> when was the last itme you conducted training
how many? it depends on degree of confidence; where you are in the design process; available resources; availability of the type of participants needed; duration of the test session
***
based on user testing above:
user profile: researcher needing an article on x; requirements & classifiers: have been to the site before and downloaded articles on at least1 occasion.
big ideas:
visualize the person & her goals
recruit on behavoir (not demographics)
use every community you can think of
screen and select yourself, if possible
ask open-ended questions
