Axos Bank

Usability Testing the Axos Mobile App and Gathering Performance Metrics

Introduction

Axos is a digital bank that aims to provide comprehensive financial tools (deposit/investment accounts, loans, and planning) to its user base. Axos has recently launched their new mobile app and wants to maintain UX best practices by making design decisions based on user data.

My Role

UX Researcher

  • Conducted stakeholder interviews to establish project goals

  • Created a screener survey in order to recruit participants that reflect user population

  • Designed a study plan

  • Generated a session guide including a script with task-based scenarios that reflect core tasks within the app

  • Moderated in person testing sessions with participants

  • Analyzed data gathered from sessions

  • Distilled insights from analyzed data

  • Presented findings to stakeholders

Goal

Establish baseline quantitative usability metrics to enable meaningful testing between future iterations of the Axos mobile app. In addition, overall usability of the Axos app Borrow vertical was measured, and qualitative insights were gathered. This study focused on the Borrow vertical (loans), but similar studies were conducted on Bank and Invest verticals

The Process

Establish Goals

In order to provide meaningful business outcomes, business requirements had to be established early in project planning. Through conducting stakeholder interviews (VPs, SVPs, Heads of Consumer Lending, Project Managers, Product Managers, UX Designers) the following project goals were determined:

  • Identify opportunities for improvement of the mobile app

  • Support future testing

Project Plan

With the overall goals of the project established, the research team proposed a project plan to stakeholders. The project plan outlined:

  • Project goals

    • Identify opportunities to improve UX/UI of the app

    • Establish baseline quantitative usability metrics to support future testing

  • Participant demographic & quantity

    • Mobile finance app users

    • 8 participants

  • Methods

    • In person, 1-on-1, moderated usability test with task-based scenarios

  • Metrics

    • Task success

    • Time on task

    • Lostness

    • Likert scale (difficulty, confidence)

    • SUS (System Usability Score)

    • NPS (Net Promoter Score)

  • Timeline

    • 4 weeks

After proposing the project plan, another meeting was held with stakeholders so they could ask clarifying questions and provide feedback (i.e. incorporating additional task-based scenarios).

Preparing Test Materials

With the project plan finalized, preparation of test materials were underway. These materials included:

Recruitment

The recruitment process consisted of creating a screener survey recruit target population, selecting qualified participants from a pool through Respondent.io, and scheduling sessions. 10 participants were recruited for this study.

Sessions

Test sessions were moderated, in person, and 1-on-1. Participants were directed to complete task-based scenarios and think out loud in order to gather qualitative insights. Participants rated their difficulty and confidence with each task on a Likert scale. Once all tasks were complete, participants completed a questionnaire which was used to generate SUS and NPS.

Quantitative Analysis

From participant sessions, quantitative data was gathered and analyzed.

  • Task success (success, partial success, or fail)

    • Participants completed most tasks successfully except for task 7

  • Time on task (Compared with optimal times)

    • Average times were well above optimal times, potentially resulting from task difficulty, sub-optimal paths, and first time users

  • Lostness (score calculated through formula that intakes optimal, unique, and total page/click counts)

    • Participants were significantly lost on tasks 4, 5, and 7

  • Confidence (Likert scale)

    • Tasks rated with high confidence except for Task 7

  • Difficulty (Likert scale)

    • Task 3 was easier than participants expected

    • Tasks 5 and 7 were more difficult that participants expected

  • System Usability Scale (SUS)

    • Indicated good overall usability - 82

  • Net Promoter Score (NPS)

    • More promoters than detractors in sample, but score lower than industry average of 34

Qualitative Data

An affinity map was created in order to code qualitative data and identify trends. These trends were distilled into insights and actionable design recommendations. The insights synthesized include pain points, disparity between design and mental models, and issues with information hierarchy.

Presentation and Documentation

Research report of findings and actionable insights were presented to stakeholders.

Research process, data, and synthesis was documented and made accessible to relevant business units.

Learnings

Business Learnings

Project goals were achieved. Opportunities to improve UX/UI of the app were identified. Baseline quantitative usability metrics were established that will support future testing. Based on findings, the design team has actionable insights that, when implemented, will improve user experience for core flows within the mobile app. Business units gained user perspectives that will help shape the product and promote customer satisfaction. Findings were placed in a backlog of action items for product/design teams.

Personal Learnings

Regarding the planning phase, through this project I have gained an understanding of the value in communicating with cross-functional stakeholders to gather project goals and requirements, as well as the importance of leveraging those goals to provide impactful findings. Regarding session moderation, I have improved my ability to prompt participants in order understand their perspective, without biasing them. Perhaps my most substantial takeaway from this project relates to data analysis. I have learned to utilize impactful quantitative metrics (i.e. lostness and SUS), use Excel to organize/synthesize/visualize data, and unite quantitative and qualitative data to generate robust findings.