Ross School of Business UX Research and Usability Evaluation

Lessons Learned

Staying in close contact with your users is incredibly important and ideally should be done on a continual basis, not just when first designing or redesigning (for an example of this, read about Meetup.com's dead simple user testing.). And the value you get from simply talking to users can be immense. But interviewing is definitely an art form that takes a lot of practice.

Video Summary

I've written a rather detailed summary of all our reports below. But if you want to cut to the chase or prefer a visual style of learning, you can watch a video summary of our semester-long project here:

The Client

The Ross School of Business is one of the top business schools in the country and in early 2014 they were in the process of redesigning their website. They had contracted with an outside firm to do the redesign. Our group worked with the Office of Marketing and Communications at Ross to do user research on their current site. Our research was conducted over the course of the current semester and included a number of reports:

• an interaction map of the website;
• user interviews, from which we generated personas and scenarios;
• a Qualtrics survey of users;
• a comparative analysis with a comparison matrix;
• a heuristic analysis;
• and finally usability tests conducted with users in the usability lab at the School of Information.

The Team

My teammates for this project were Krista Gray, I-Chun Hsiao, Nehal Molasaria, and Chihiro Jimbo. We all shared equally in the work, though we did have specific roles as well. I served as the client contact. Below is a summary of each of these steps of our project with relevant visuals.

Interaction Map

Because the Ross website fulfills a number of different functions, we quickly realized that creating an interaction map of the entire site would be very time-consuming and possibly not have much of a payoff. We decided to instead focus on the site's most important function: marketing the school to prospective students, primarily full-time Master of Business Administration (MBA) students and Bachelor of Business Administrations (BBA) students. These two student groups account for the majority of students attending Ross. We mapped out the possible interactions a student from either of these two groups would likely take while navigating the website, with a particular focus on the program and admission requirements for the BBA students, and information about Ross and the curriculum for MBA students.

Our conclusions were that navigation elements appeared in both the left-hand and right-hand column, making for an inconsistent layout; and secondly that there were issues with redundant links that could further confuse the user. We did not write up a full report based on the interaction map, but it did set the ground for later reports where we found confirmation of these conclusions.

User Interviews, Personas and Scenarios

We conducted a total of five interviews with two current BBA students, one current full-time MBA student, and two incoming MBA students – one part-time and one full-time. Our findings and recommendations included:

• evidence of strong user interest in student profiles, but dissatisfaction with discoverability and relevance; we recommended updating student stories and placing them more prominently;
• users examine the material on the website thoroughly and are engaged by the content
• navigation of the site was challenging at times for users, and we recommend making paths to sub-sections clearer and navigation options more consistent overall.
• undergraduates felt that the website was geared primarily toward graduate students; undergraduate news and events could be featured more frequently
• a high level of student interest in interdisciplinary studies and other special opportunities that was not well supported on the site. We recommend highlighting these further.

From the interviews we also generated a set of three profiles and scenarios.

Survey

We composed a survey on Qualtrics, again with a focus on the perspective of prospective students. The questionnaire was designed to take five to seven minutes, and also included a rudimentary heat maps feature (pictured above). Our findings and recommendations included:

• The site effectively communicates the values and uniqueness of a Ross education.
• While user satisfaction is high for the quality of information on the site, satisfaction for navigation is lower.
• Internal navigation (e.g. menus) and external search engine queries are the most common ways of finding Ross information. The redesign should continue using current search engine optimization and focus on improving the information architecture.
• Heat maps indicate that highly sought-after information like Class Profile is not in the most intuitive location on the site.
• The information needs of out-of-state students are more demanding than those of in- state students; out-of-state students have more interest in student stories and career outcomes than in-state students do. The upcoming site redesign would benefit from awareness of these differences to improve recruitment of out-of-state students.
• Facebook, Twitter, and LinkedIn are the most popular social networks of BBA students and there is a high level of interest in finding out about events and connecting with Ross alumni and students through social media.
• Laptops and smartphones are the predominant devices used to access the site, and the site experience should be tailored to these two screen sizes (tablets do not yet have enough traction to merit prioritization).

Comparative Analysis

We started out with a wide pool of competitors, both direct and otherwise. After the first round of analysis, this pool was reduced to five business schools (Columbia, NYU Stern, Yale, Stanford, Cornell) and one undergraduate institution (Cooper Union). In this second round of analysis we generated more specific dimensions for analysis that were tailored to educational institutions, and business schools in particular. Our report included a number of findings and recommendations related to the global navigation bar, the header, the footer, the incorporation and use of social media, branding, and the use of images.

Heuristics Analysis

We conducted a heuristic evaluation of the Ross website using Jakob Nielsen’s ten heuristics, an industry standard for evaluating websites and software. By compiling severity ratings individually and collectively, we developed the following findings and recommendations for the redesigned Ross website, organized into four categories:

• Navigation Issues: By not having dropdown menus in the global navbar, users are forced to remember where they have previously found useful pieces of information. Navigation menus on the left-hand side of the page could be more uniform and sparser in order to reduce the cognitive load on users.
• Internal Consistency Issues: The ordering and naming of items in the left-hand navigation menus should be uniform across the website. On major landing pages, images should be placed in the same spot at the top of the page.
• External Consistency Issues: The styling of text and links as well as the use of symbols and icons should conform to user expectations of standard web practices.
• Design Issues: Links embedded in the content section of pages might be overlooked and lead to frustration for the user. Duplicate links add clutter to some landing pages.

Doing a heuristic evaluation with other people was an interesting process, as we could see where we agreed and disagreed on the severity of issues that came up. We did end up catching mostly the same issues and there were five of us, so according to Nielsen that should have caught about 75% of the issues.

Usability Tests

As you can imagine, after all these reports, we were greatly looking forward to getting users in front of a screen and seeing how they fared with the tasks we assigned them. We came up with tasks that sought to confirm previous findings and to explore other areas that hadn't been covered in our previous assignments. These were the research questions that we came up with:

• How do users navigate through the Ross site when looking for specific pieces of information?
• Are navigation paths for these pieces of information clear and easy to discern? Where might things go wrong?
• How does the global navigation help or hinder the user’s process of finding information?
• How do the internal navigation menus located on left or right of the pages assist users in finding the information?
• Are embedded links prominent enough to catch attention of the users while searching?

Participants were selected based on their level of familiarity with the Ross School of Business website (we wanted both novice users with no experience of the site and “expert” users who had used the website previously) and their alignment with key demographics (primarily age and status/year in school). We also sought a balance of in-state, out-of-state, and international students, but due to some cancellations we were only able to test one out-of-state student (the rest being in-state). The chart below provides a brief overview of the characteristics of our participants and a summarization of the data from our data logs.

We used Camtasia to record the user's actions and thoughts and provided them with either a PC or Mac based on what they were familiar with. We also took detailed notes and created a spreadsheet based on all our notes. Below are the tasks we asked users to do, along with our reasoning for choosing these tasks.

Task 1

Ask the participant to discover on the site information about what criteria Ross uses for assessing applications so as to get a better idea if they would be a good fit at Ross.

Purpose: We wanted to learn how quickly and how well a prospective student could learn about the general parameters that Ross uses to evaluate candidates. This might be a prospective student’s first step in figuring out whether or not they should apply to Ross.

Task 2

Ask the participant to find information about classes, and specifically to find out what classes are required to graduate and what order they should be taken in.

Purpose: We had previously recommended combining the “Admissions” and “Academics” sections in the global navbar and wanted to see how users would choose between these two options when seeking out information about classes at Ross.

Task 3

Direct the participant to learn about the current student body so as to see if they will fit in or measure up.

Purpose: From our user interviews we knew that general information about the student body was an important factor for prospective students deciding to apply to Ross, and our survey found that over 90% of respondents reported using the website to access the class profile as prospective students. Here we wanted to know if users could find general information that was nevertheless specifically-located (like a class profile), and in Task 5 we asked to see if they could find specific information about individual students and alumni.

Task 4

Ask the participant to seek out information about costs: specifically, estimated annual expenses including tuition, books, housing, and personal expenses.

Purpose: Once a student decides that Ross is a good fit for them academically and culturally, they might want to consider the financial details of attending Ross. We made this a later task because we did not think students would consider costs as a primary factor since a) BBA students would be paying roughly the same amount as other University of Michigan undergraduates and so this would be known information; and b) prospective MBA students applying to Top 20 business schools across the country can expect to pay roughly the same amount regardless of where they go.

Task 5

Ask the participant to find more specific information (such as profiles) about individual students or alumni.

Purpose: The motivation for this task is explained above in the purpose of Task 3. Discovering individual scenarios is an important factor in being able to understand what life would like at Ross and as an alumnus.

Task 6

Ask the participant to look for information about admissions events and to see what options there are for signing up for an event.

Purpose: Taking a specific action after concluding research on Ross would be an expected task performed by a prospective student. Information about events on the Ross is not consolidated into one spot, so we wanted to learn what path users would take to schedule their attendance at an admissions event.

Here were our key findings and reccomendations from conducting user testing:

• The limited functionality of the global navbar and left navigation menu impedes site navigation. We recommend employing dropdown menus and a navbar pinned to the top of the browser.
• Some labels and categories are not intuitive for users, making site navigation more challenging. We recommend revising these to reflect users’ understanding of the terms.
• The relationship between general and specific information is at times difficult to discern on the site. We recommend creating more subsection overview pages to clarify the context of individual pages.
• The location of key pieces of information, such as tuition and costs, can be difficult to find. We recommend making this information more prominent on the site.
• Users overall report positive impressions of the Ross website.
• In-text links are at times relied on heavily. We recommend that the redesigned website employ in-text links as a supplement to menu-based navigation.

Feedback from the Client

Six months after the project ended, I checked in with our client to see how our report had assisted the redesign effort. Below is her feedback.

"[Your] report was useful and informative. Your research confirmed many of the difficulties that had been reported but put a spotlight on new ones. Specifically,

1. the need to provide more pervasive and consistent brand presence through the site.
2. provide more images of the Ross experience and
3. the need to provide easier access to profile data.

[Our external design agency] originally did not provide drop down secondary menus. I'm not sure if the decision to add them was entirely attributable to your work but I suspect it informed it. It was also interesting to note in your findings that a highly motivated seeker of information could actually find what they were looking for on a site so difficult to navigate. An unexpected data point."

A Look at the Redesigned Website

The site looks good: larger images give a better impression of the school and the Ross brand is much more present across the site. And the functionality is enhanced via the dropdown menus in the navbar; the navigation across the site has also been centralized so that users know to look in one place for any navigation.

Of course, the proof is in the tasting of the pudding: user testing. It would be great to get the team together for another round of testing!