a. Group number and name
Group number: 18
Group name: %eiip
b. Member names
Erica (eportnoy@), Mario (mmcgil@), Bonnie (bmeisenm@), Valya (vbarboy@)
c. Project summary
Our project is a smart bookshelf system that keeps track of which books are in it. You can see the web application portion of our project here, and our source code here!
d. Introduction
In this user test, we evaluated both the hardware and software components of our system. We had users actually use Biblio-File to add books to their shelf, search for books (both those that were and were not on the actual shelf) and play with the system. In particular, we were testing to see whether or not our system actually made searching for books easier. To test this, we had users search for book both with and without Biblio-File. Moreover, we wanted to see if adding many books was annoying or tedious for the users, and if the delay due to RFID reading was particularly annoying or frustrating.
e. Implementation and Improvement
Our P5 post can be found here. We also made a few changes since P5, which are listed below:
- Completed implementation of the server-client-daemon architecture – we can now user-test using only our system, without needing “Wizard-Of-Oz” simulation
- Modified the RFID library to enable low-latency use of multiple scanners (original code didn’t support multiple scanners)
- Alphabetized entries by title in the web app to let users scroll through books in a sensible order
- Added magic administrative functions to help with user testing: auto-populating bookshelf, clearing database, etc.
- Computer now beeps loudly after a sensor detects an RFID to give users feedback
f. Method
i. Participants
For our user test we chose three undergraduate students with varying levels of technical expertise. Our first tester is an engineer, who often uses technology, and was very interested in how our system worked. Our other testers were less experienced with technology as a whole, and were completely unfamiliar with the Arduino and the other tools that we used in our bookshelf. We chose them to see how intuitive our system was, and how annoying or frustrating the delays might be. People who are less familiar with these tools are also less used to delays, and will therefore have a more natural reaction to them. Similarly, people with less technical expertise are less likely to assume anything about the app, and we can therefore see how they use it, what they try to do but cannot etc.
ii. Apparatus
In order to conduct the test we used a stack of shelves in the TV room of Charter. The location might not have been optimal, because there were other people there playing video games, so there was a lot of noise and distractions. That being said, it was cool that our system was so mobile, and could be applied to any bookshelf anywhere. We attached our system to the shelves, and brought our own books to use for the tasks themselves. We let the users use one of our own phones to test it, to avoid the need to download pic2shop for barcode scanning. That being said, they could have easily used their own mobile device if they chose to.
This is the version of the system that we used for our user tests
iii. Tasks
The easy task is finding a book on the shelf, or searching for a book that is not present. Users can choose to use our mobile app to search for a book, or they may attempt to manually search the shelf. If our system provides added value, we hope that they will opt to consult our mobile app. To do this task we provided the users with a very full bookshelf. Each user saw exactly the same books on exactly the same shelves. We timed how long it would take the users to find a book that was on the shelf, and one that wasn’t, both with Biblio-File and without.
Our medium task is adding a single new book to the system; it consists of adding an RFID tag to the book, adding it to the system using our mobile interface, and then placing the book on the shelf. The purpose of this task was to test how easy our system is to use, and what a user would intuitively want to do given a system like ours.
Our hard task is adding an existing book collection to the system; this consisted of four books, for testing purposes. This is the last task a user would have to complete with our system, and it this is very similar to the previous task. It consists of using the mobile interface and an RFID tag to add books to the bookshelf. The main purpose of this task was to test the tediousness of adding many books to a collection.
A video of one of our user tests can be found here!
iv. Procedure
To conduct the study, we first introduced our team and had the user read and sign the consent form, and fill out the demographic questionnaire. We then explained the concept of a Think-Aloud Study, and practiced the methodology on an unrelated problem (see script in appendices). After this, we demonstrated how our system works (in general, without showing them any of our tasks). We then ran the easy task, and timed it, followed by the medium task and then the hard task. Afterwards, we told the users how our system was meant to work (if they didn’t understand it to begin with), and asked some follow-up questions to check how annoying our system was (if it was at all) and how the user felt using it. The answers to these questions are included in the appendices. The users were encouraged to think aloud and ask questions throughout the study.
g. Test Measures
We measured two within-subjects variables related to book access and retrieval, whether or not a book is on the shelf and whether or not the user was using our system. We chose to measure these to see if our system gave the user any quantitative speedup in common book interaction processes.
- Time taken to retrieve a book on the shelf.
- Time taken to realize a book is not on the shelf.
- Time taken to interact with a book using only the physical bookshelf.
- Time taken to interact with a book using our system.
h. Results and Discussion
Our tests showed that in general, our design is sound, although a repeated measures ANOVA with a sample size of 3 showed no significant difference in using or not using our system (p > .05, see ANOVA output in appendices). Many users were enthusiastic about what we were able to do; in particular, many were delighted that we could gain a lot of information from a single photo of the ISBN barcode. We deliberately did not give users a complete demo of our system because we wanted to judge its intuitiveness. Even without complete instructions, our testers largely understood the system, which we’re very proud of. However, for some tasks, such as removing books from a shelf, it’s clear that more specific instructions would be helpful.
Users relied heavily on receiving some sort of feedback that an RFID tag had been sensed, which we implemented in the form of a loud beeping sound. This worked well. Also, users did not seem impatient when we asked them to add many books to the system at once.
There are some small changes that we’d like to make. For example, when there are no books in the shelf, we shouldn’t display the search bar, since users often attempt to add a book by typing the title into the search bar first. We may also want buttons to display a “not on shelf” message instead of a greyed-out “Light up!” button when they’re not on the shelf, since 1 in 3 users did not recognize the design motif of a disabled button. Also, there are some bugs we still need to fix, such as the barcode scanner redirecting to iTunes occasionally, and the lag with the LEDs on the shelf.
We also need to clarify the tap-in tap-out process to the user. While it is not the most intuitive, it is based on the technical limitations of our hardware, so we will have to compensate in instructions. Since some users attempted to tap a book in before adding it via the software, we should enable our software to allow either ordering. We should also change the instructions to say “place the bookmark inside the front cover,” since some users placed the bookmark too deep inside the book to be read.
i. Appendices
i. All things read or handed to participant
ii. Raw data