SISU-Usability Evaluation Methods

Heuristic evaluation:

Improving usability interaction by conducting user studies.

SISU is a university’s student information system. The usability of SISU was evaluated part of the Usability Evaluation Methods course. The goal of the evaluation is to provide the developers with a general evaluation of the usability of the system as well as ideas for improving its usability. The usability test will be conducted using pre-formulated test tasks to help focus the test user’s actions to the parts of interest. Think-aloud verbal protocol is used to find out the participant’s thoughts during the test. Other methods used in the test include an interview and questionnaires.

The usability test will focus on evaluating the following parts of the service:
● Study planning
● Course enrollment

Goal of this project was to conduct user research to find out the usability issues, evaluate the severities of problems and create the solutions for them.

Methodology

The aim is to interview users with different backgrounds and experience level with the SISU system. Interviews and observations of system usage are conducted remotely, using the Zoom conferencing. We will have users that have never used SISU before and users that have experience with the system.

Metrics and usability goals:

According to the ISO-9241-11 standard, the criteria for usability are effectiveness, efficiency, and satisfaction. The following measures will be used to evaluate different aspects of usability:

1. Evaluating effectiveness
   ● Completion rate (% of tasks successfully completed)
2. Evaluating efficiency
   ● Task times
   ● Errors per task (and per participant)
3. Evaluating subjective satisfaction
   ● Interview results
   ● Questionnaire results
   ● Observations during the test (think-aloud and the behaviour of the
   participant

It is expected that there will be some errors in performing the tasks but the test users should be able to recover from them. Thus, a completion rate of 90% is the goal for the tasks in this test. Expected task times for each task are listed in the following section on Test Tasks.

The user will be able to give their feedback on the system after the tasks have been run through. This input should be taken into consideration when compiling the final analysis.

Key Findings

Common

● The visual hierarchy was perceived as confusing
● The system does not guide / advise the user within the system.
● The navigation is unclear, during the usability test the test users usually searched for the task in the wrong part of the system at first
● Purpose of system parts unclear, e.g. memory list / control.

Focus area 1: course registration

● The user easily imagines that the course registration has gone through already when clicking on the course
information from the page and ignores that the registration itself is handled through the calendar.
● IF the course registration does not go through but gives an error message, the user may ignore that as well.
● Course registration was felt to require too many clicks

Focus area 2: planning of studies

● The system informs the user that the selections are incorrect, but does not guide how the selections should be corrected
● An experienced user will ignore the error messages, considering them insignificant
● The information hierarchy for adding optional courses directs the user to add a study draft instead of courses.
● Searching for courses is difficult using the search.

Heuristic evaluation

A heuristic evaluation of the SISU system was prepared as the starting point of the entire process.
The focus was on finding problem areas, especially in key areas such as study plan and course registration, which would provide a framework for planning the actual user testing. A total of 50 entries were made during the evaluation, of which 23 were related to the key areas mentioned in the assignment. The observations were discussed, after which the problem areas were ordered according to the severity and usability heuristics they violated.

Usability test participants

• Pilot: State IT employee, comprehensive experience with different systems. No previous
  experience at SISU
• Participant 1: Office worker with no previous experience of university studies or SISU
• Participant 2: Studying at the University of Tampere and using SISU in everyday life. Received
  separate 1on1 training to facilitate the use of SISU

The key problems discovered in the usability test

Pilot

● It took the user 42 minutes to realize that the headings in the course planning tool can be clicked, and pressing it reveals more necessary tools.
● The slowest execution times in test tasks and the smallest success process.
● Operating models learned from other systems clearly affected the use of SISU in a negative way.
● Difficulty using the system’s primary operating models and taking into account pop-up notifications.

Participant 1

● Did not understand the communication system: how and to whom the message is sent is unclear
● The purpose of the memo list feature is unclear.
● An error message may appear when registering for courses without a valid reason.
● Came across several error messages, but got confused because the system didn’t forward.

Participant 2

● Ignore the error message because knowing it doesn’t mean anything.
● Does not react when the course registration is not accepted.