Evaluation techniques for interactive systems

Tablet in hand photo created by rawpixel — from freepik

Imagine you worked really hard in designing a usable interactive system. You used all the design patterns and principles ever known to man. And in your eyes, it’s the most perfectly designed system out there. Then someone comes along and tells you that it is garbage and they will never use it.

So how do you make sure that your design appeals to other people? You would need to evaluate it — but how?

That’s where evaluation comes in handy.

So what is evaluation?

The role of evaluation is to ensure that our system actually behaves the way we want and it meets user requirements.

Now imagine you are running a marathon. You thought you trained very hard for this one race. Practiced every single day. But you never bothered to evaluate yourself prior to the race. And on the actual race day, everyone else passed you easily. And you got your evaluation like a slap in the face. At that point, the evaluation is useless. That is because evaluation should be done each and every step of the way and not just at the end. That way the results of the evaluation can be used to improve and adjust accordingly.

Similarly, in designing a system, analytic and informal techniques should be used to test the design throughout the process. This way we can ensure that effort and resources are not wasted on unnecessary implementations.

It is much easier to change a design in the early stages of development than in the later stages.

Now there are multiple evaluation techniques out there, but before that, we need to know why we do evaluations and what we are trying to achieve through it.

Goals of evaluation

Evaluation has three main goals,

1) Assess system functionality and usability

If the design of the system doesn’t let the user perform the intended task, what is even the point of it. The functionality of a system is the most crucial thing we need to keep in mind. In order to evaluate this, we may measure the user’s performance with the system. As a result, we can assess the effectiveness of the system in supporting the tasks.

2) Assess users’ experience of the interaction

Even if the user is able to do their task effectively, their user experience in completing that task could be problematic. We need to make sure that user is having a positive experience throughout their tasks. And that they do not feel overloaded.

For this we need to consider,

1. How easy the system is to learn.
2. Its usability.
3. The user’s satisfaction with it.

3) Identify specific problems with the system

Other problems that are related to both the functionality and the usability of a design can arise. Sometimes some aspects of the design can cause unexpected side effects. Or it may confuse users.

Evaluation through expert analysis

Since we need to conduct evaluations throughout the design process, and since we haven’t started the implementation work at the very beginning, we need to evaluate without direct involvement by users. Instead, we can have experts analyze our design.

Let’s discuss a few methods used to evaluate interactive systems through expert analysis. Keep in mind that methods do not assess the actual use of the system, only whether or not a system upholds accepted usability principles.

1) Cognitive Walkthrough

Walk-throughs involve “walking” through a task with the product while identifying problematic usability features.

A cognitive walk-through takes a cognitive perspective. Here the focus is on evaluating designs for ease of learning. This involves simulating how users go about problem-solving at each step in human-computer interaction.

In order to do a Cognitive Walkthrough we need,

1. A prototype of the system. (More detailed, the better)
2. A description of the task the user is to perform.
3. A complete set of actions needed to complete the said task.
4. User’s details. (Who are they? What kind of experience and knowledge do they have?..etc)

Here the evaluators have to try to answer the following four questions for each step in the action sequence.

1. Is the effect of the action the same as the user’s goal at that point?
2. Will users see that the action is available?
3. Once users have found the correct action, will they know it is the one they need?
4. After the action is taken, will users understand the feedback they get?

2) Heuristic Evaluation

Heuristic Evaluation is done according to a set of usability principles known as heuristics. They are called “heuristics” because they are broad rules of thumb and not specific usability guidelines.

I’ll be discussing all 10 of Jakob Nielsen’s 10 general principles for interaction design.

1. Visibility of system status
2. Match between system and the real world
3. User control and freedom
4. Consistency and standards
5. Error prevention
6. Recognition rather than recall
7. Flexibility and efficiency of use
8. Aesthetic and minimalist design
9. Help users recognize, diagnose, and recover from errors
10. Help and documentation

3) Review-based evaluation

A final approach to expert evaluation is using previous results as evidence to support aspects of the design. Since it is expensive to repeat experiments continuously, we can use expert reviews instead. The reviewer should take both the similarities and the differences between the experiment and the design under consideration. The reviewer’s expertise will help us make the correct assumptions.

You can read more about Interaction design evaluation through expert analysis in another article of mine,

Interaction design evaluation

Evaluation through user participation

Although Evaluation through expert analysis is useful, they are not a substitute for actual usability testing with the people for who the system is intended: the users.

Styles of evaluation

Now there are two distinct evaluation styles. Those performed under laboratory conditions and those conducted in the work environment.

In Laboratory studies, users are taken out of their normal work environment and are used as lab rats to take part in controlled tests. But these lab studies might create situations that will never happen in the real world. However, in some cases, such as if the system is to be placed in an unattainable area, such as a space station, laboratory observation is the only alternative.

In Field studies, we go into the user’s work environment to observe the system in action. Normally, Field observations are preferred to laboratory studies because it allows us to observe the interaction in its natural setting.

Now let’s move on to the actual methods of evaluation.

Empirical methods: experimental evaluation

A controlled experiment is one of the most powerful ways to evaluate a design. It can be used to study a wide range of different issues at different levels of detail.

In our experiment, these factors play an important role,

1. Participants — The participants should be chosen to match the expected user population as closely as possible. They should accurately represent the user population.
2. Variables — These variables are manipulated and measured under controlled conditions in order to test the hypothesis.
3. Hypotheses —A prediction of the outcome of the experiment.

Observational techniques

One of the best ways to gather information is to just observe the user interacting with it. Now by simply observing, we might not get the information we actually need. So it’s better to use some of these Observational techniques.

Firstly we can ask the user to think aloud and describe what they’re thinking throughout the task. And in cooperative evaluation, the user is encouraged to see themself as a collaborator in the evaluation.

The record of an evaluation session of this type is known as a protocol. You can write down using a pen and paper, record the session, or use computer logging. But analyzing these protocols can be time-consuming.

Instead, we can use automatic analysis tools to support the task. For example, EVA.

But sometimes when we use direct observations we can lack interpretations of data. So it’s better to do a post-task walkthrough to fill in the gaps.

Query techniques

We can also ask the user about the interface directly and get their opinion. There are two main types of query techniques: interviews and questionnaires.

Evaluation through monitoring physiological responses

When it comes to evaluation through user participation, we rely heavily on the observations and the users. But by using, what is referred to as, objective usability testing, we get to see exactly what users do when they interact with computers and also measure how they feel. For example, eye tracking and physiological measurement.

Conclusion

In short, evaluation is an important part of the design process and should take place throughout the design life cycle. And as we discussed in this article, there’s a range of techniques available for evaluating an interactive system at all stages in the design process.

Hope you found this article helpful! See you in another article!

Thank you!

Reference: Alan Dix, Janet Finlay, Gregory Abowd, and Russell Beale. 2003. Human–Computer Interaction(3rd Edition). New York: Prentice-Hall