a. Group number and name

Group number: 18

Group name: %eiip

b. Member names

Erica (eportnoy@), Mario (mmcgil@), Bonnie (bmeisenm@), Valya (vbarboy@)

c. Project summary

Our project is a smart bookshelf system that keeps track of which books are in it.

d. Test method

i. Obtaining informed consent

We are obtaining informed consent by having the participants sign a form adapted from the standard consent form template. We feel this procedure is appropriate because it informs them of the ways in which their identifying information will be used. Our consent document is located here:

https://docs.google.com/document/d/1l1k40exeedEf5zZ_EZQEhIP4xqquLzfmiFmbWMpXawA/edit?usp=sharing

ii. Participants

Participant A is an undergraduate student that we selected because she was doing work in a public place with multiple books next to her. Participant B is a graduate student in mathematics that we selected because he is precisely in our target user group. Participant C is a professor of computer science that we selected because he is precisely in our target user group.

iii. Testing environment

We performed a “Wizard-of-Oz” test, using our low-fidelity prototype bookshelf and mockup interface screens. For each test, we placed the bookshelf on a hard, table-height surface in a quiet area. The tests were performed using several (real) paperback books.

iv. Testing Procedure

When testing our prototype, Bonnie ran the paper prototype, Erica greeted the participant and obtained consent, Valya read through the demo and task scripts with the participant, and Mario and Erica were the primary note-takers. Our first task was having the user input an entire new collection, which we rated as difficult. Our second task was having the user add a new book to the system, which we rated a moderate task. Our final task was finding a book on the bookshelf, as well as searching for a non-existent book which we rated as easy. We chose this order because because it captures a more realistic use-scenario/workflow. Our demo and task scripts are located here:

https://docs.google.com/document/d/1GJH9i-mSwuxtb6rsPU5xBE3FU6XHx5Xq0bRYqmR4KCQ/edit?usp=sharing

Usability testing with Participant A

e. Results

We noticed various usability issues that definitely need to be addressed. Most importantly, two out of three users placed the books directly on the bookshelf without going through the software system. Even after realizing that they needed to go through the mobile app, they had trouble figuring out how to begin adding books to the system; for instance, some were unable to easily find the plus button for adding new books. Two participants were annoyed with the length of the book insertion process and expressed a desire to streamline it. They were angered with the length of time it takes, so we will definitely have to address this issue. One user was uncomfortable switching back and forth between using the phone and handling the books. One user mistakenly believed that the ordering of books displayed in the app’s search screen matched the ordering of books on the bookshelf. On the plus side, users experienced a moment of joy seeing the bookshelf light up when they tapped the book on the app. Additionally, after adding their first book to the shelf, two of our testers seemed to adapt quickly to the process and added the remaining books with relative speed, which seems to indicate that this system could be a practical solution.

f. Discussion

Our tests revealed some consistent problems with the interface as it currently stands. One issue is that users were confused about how to add books to the system; also, some users complained that the process of adding books to the system is too cumbersome. At a minimum, we want to remove the “edit/verify book data” step (step 3) and the requirement that users photograph the book cover. We also want to tweak the UI to make the “add books” button more clear. We’ve also discussed alternative ways of identifying books to the system (such as having the user photograph the spine of each book and using character-recognition to get the book’s information) that would allow users to add books to the shelves in a more streamlined way. It seems like users really want to be able to just add books to the shelf without going through a lengthy process; we need to understand how we can minimize the work they have to do. For example, if we tap into an API such as GoodReads, we could remove the “take a photo of the cover” step by matching the ISBN number to a book cover from the database, combining three steps on the user’s end into one. Since users were not sure how to add a new book, we will color the plus button green, and we will add the text “add new book” on the “no books found” empty collection screen, with an arrow pointing to the plus button. In “Subsequent Testing” we present some other ideas for redesigning the book addition process to be more obvious and intuitive to the user. Also, if we present the system to the user as primarily software with a hardware component attached (as opposed to a normal bookshelf with some extra software added on), users may be more inclined to enter the books into the system rather than merely placing them on the shelf.

One limit of our experimental process was that we only tested the bookshelf with a limited number of books. This meant that, for example, some users simply grabbed a book off the shelf instead of testing the book retrieval functionality of the mobile interface. So, it’s possible that, when dealing with many books, problems with our interface might arise that we haven’t yet discovered. We noticed that because all the books could fit on a single screen of the interface, for example, that one user didn’t use the search or scrolling functionality.

g. Subsequent testing

User testing showed us that our prototype is somewhat confusing and tedious to use, though overall users seemed to like it. We want to try to fix these issues and run further testing before making our high-fidelity prototype. In particular, we want to re-design our mobile interface to make the buttons and their functions clearer. For example, when first asked to add books, all the users were confused, and clicked around until they were able to find the correct button. We want to make the first introduction to our application easier for the user, so that they don’t have to play guessing games. To do this, we would need to change the design of our paper prototype, adding more text, labeling the buttons, etc. We plan to make a few different versions of the clearer paper application, and to test it on users within the next week, to figure out what changes will make the screen easiest to use and most understandable. We will then incorporate these findings as we make our high-fidelity prototype.

Our second issue was that adding books to our system is somewhat tedious and annoying. Some users clearly got sick of doing it, even when tested on only a few books. To fix this, we want to see what we can feasibly do to streamline the process of adding books. This would involve testing the RFID sensors for range, and testing different ways to physically attach the RFID tags to books. Moreover, we want to look at image recognition, to see if we can gather book information solely from taking photos of spines, or even just the cover. We plan on doing this during lab next week. Then, once we identify potential improvements, we will adapt our low-fidelity prototype to reflect a few potential simplifications of the system. Moreover, in our user tests, we ordered the tasks in the order in which they would be most likely to perform them in practice. Instead of doing that, we want to run tests that will first allow our users to add a single book (moderate task), to check if they can figure out how to add a book to our system. We would then provide them with many books, to see how tedious they find the process (difficult task). Finally, we would present them with a bookshelf full of books they did not add, to see if they can find a book using the system (easy task). We will conduct these tests within the next week to figure out what is the minimal-effort system that will make our users the happiest.