Lauren Berdick

The Cereal Killers

Group 24: Andrew, Bereket, Lauren, Ryan

Our Project

A program to create computer shortcuts that can be replayed using hand gestures via a wireless gesture bracelet.

Previous Posts

P1: https://blogs.princeton.edu/humancomputerinterface/2013/02/22/the-cereal-killers/
P2: https://blogs.princeton.edu/humancomputerinterface/2013/03/11/p2-contextual-inquiry-and-task-analysis/
P3: https://blogs.princeton.edu/humancomputerinterface/2013/03/29/p3-brisq-the-cereal-killers/
P4: https://blogs.princeton.edu/humancomputerinterface/2013/04/08/p4-group-24/
P5: https://blogs.princeton.edu/humancomputerinterface/2013/04/22/p5-cereal-killers-team-24/
P6: https://blogs.princeton.edu/humancomputerinterface/2013/05/07/p6-usability-study/

Our Final Project in Action

Gui:

http://www.youtube.com/watch?v=sOxUXQ_aBTc&feature=youtu.be

Gesture Bracelet Design:

Gesture Recognition Demo:

http://youtu.be/7P9_5Gr0QCA

Bracelet Images:

Gesture Bracelet

Side View 1

Side View 2

Bracelet on Wrist

Changes since P6

In P6, since we did not have the gesture recognition or gesture recording part working yet, we made another GUI with temporary buttons. Now we have a GUI with the final layout which reflects the programs ability to record gestures.
One button would “record the gesture,” but when pressed we used a Wizard of Oz technique by simply remembering which gesture the user assigned to an action. Now we decided to go with six predefined gestures.
We had another button to replay the actions. Now, the actions will be replayed when a certain gesture (which was mapped to that action) is performed.
We have gesture recognition.
We created the gesture bracelet using wireless xbee devices, an arduino uno, and accelerometer
We wrote code in python to receive data from the bracelet wirelessly, compare it with a library of data, and find the closest match using a nearest neighbor DTW algorithm
We were aslo able to make the use of the bracelet easier. We replaced the idea of shaking before and after a gesture to signal its start and stop by instead just jerking the bracelet in any direction fast enough to trigger the start of transmission form the bracelet to the laptop, and sending roughly 3/4 a second of data to the computer. Thus, we no longer needed the user to do an action to end the gesture
The keylogger handles multiple keys. In P6, it would not record when multiple keys were pressed simultaneously.
Now that we have predefined actions, our GUI maps gestures to action by writing the mapping to a file.
And therefore we have a page in the gui that lists the available gestures.

Evolution of Our Goals and Design

Our goal to create an easy to use system that would make users more connected to their laptops has not changed. However, due to feasibility, module functionality and user feedback, we made some changes to the implementation of this goal.

Six Predefined gestures vs User Created gestures

We changed some design decisions based on feedback from users. We went with six predefined gestures, as opposed to users having to create their own gestures, based on our interviews. Our users found that it was hard to think up gestures on the spot. This way, a user does not have to worry about creating a gesture, and all the gestures are distinct. This limits the variety for the user. However, in our usability tests, users spent quite a few minutes simply trying to think of a few gestures, and they often needed suggestions, or were hesitant about their choice of movement.

Bracelet size

Our initial plan was to have the bracelet small and light for comfort and ease of use. In order to do this, we had planned to use bluetooth to transmit the accelerometer data to the computer. However, we had a problem with our bluetooth module not working properly, so we had to go for the bulkier radio frequency option with xbee devices. Additionally, after testing a femtoduino and arduino micro, neither of these would work with the bluetooth or xbee devices, so we had to use the larger arduino uno. Therefore, we had to modify and increase the size the of the bracelet.

Evaluation

We were happy with the gui we were able to make and the ability to log both keyboard and mouse input and replay them. We felt the gui was intuitive and provided everything the user needed to understand and interact with the program. We were also very happy with our gesture recognition. We could do six gestures and recognize each: left and right swipes, up and down swipes, and clockwise and counterclockwise circles. We felt that these covered a nice range of intuitive gestures, and were different enough to be recognized effectively.The unfortunate part of our project was the inability to link the gesture recognition with the gui. We had problems installing the necessary libraries for gesture recognition in windows, in which the gui and shortcut processing was written, so we were left with a mac that could recognize gestures and a windows laptop that contained the guy.

We definitely felt that this was applicable in the real world, and our users who tested the product agreed. It has a place in the market for expanding computer use, and we think that the three scenarios we came up with are still applicable. It would be exciting to see how the device would do in the real world, and since we mostly use low cost materials that could be further reduced in cost if buying in bulk, we think the device could be priced in an attractive range. Of course, the program would need to be linked to the gesture recognition so we could run both on the same laptop, but we feel that this project is very applicable and even had some users ask if we could update them when we finished because of their interest in the product.

Moving Forward

We would of course like to link the gestures and the gui on the same computer. We think that with some more time, we could either figure out how to install the problematic library (pygsl on windows), or change the machine learning library we are using to make it compatible with windows. We would also like to investigate more wireless and microcontroller options so we could reduce the size of the bracelet. We were happy with how compact the final product was, but we feel that it could even be further reduced for a sleeker design. We would also like to replace the xbees, which require a receiving end hooked up to the computer, with just a bluetooth transmitter on the bracelet that could pair with any bluetooth compatible laptops.

Future testing would include observing users using the bracelet with the gui and seeing how easily they are able to pair the gestures with actions. Additionally, we would like to see users using the bracelet. We were happy that we were able to put code on the bracelet that activated it with just a simple jerk, and felt that this made it easier to use than the initial shake actions. We would like to see if users agreed with this, and if they felt it had an intuitive feel and ease of use.

Code

Zip file:

https://www.dropbox.com/s/cohxt7l9eo6h378/brisq.zip

The libraries we used with our code were numpy, scipy, pyserial, pygsl, and mlpy, all python libraries. Numpy was used to store and load data, mlpy was used to compare accelerometer vectors (x,y,and z), pyserial was used to read data from the usb port, and scipy and pygsl were required for the mlpy library.

Poster/Presentation Documents:

https://docs.google.com/presentation/d/1GAqPCLOdXqt-w-z_E4qPgnXzB4sn4m1c-HoclzbxlmE/edit?usp=sharing

Lauren Berdick I paper prototyped an app that uses the freefood listerv in order to display and give information about where to find free food on campus.

Observations:

I observed people in-between lectures. People entertain themselves with an eclectic mixture of habits. Some people munch on food, either snacks or two course meals they got from late meal. Others, most likely those who have not been tired out for the week already, start to flip through a textbook, or lecture slides of the class to come. Many immediately take out their computer and start facebooking, playing random games, checking their Princeton gmail. The teachers, if they get there early, are generally setting up their computer for use with the projector, making sure their slides are organised. Some who arrive quite early may strike up a chat with a nearby student, or perhaps one of their TAs. I observed and then interviewed the following people directly to find out what they generally do, and some things they might feel would broaden their experience in these few minutes.

Nicole Loncke

She has one class before which she has a 10 minute break. They are both in McCosh, so she isn’t running frantically. She usually chats with some people if she sees anybody she knows. She’ll walk over slowly, get a good spot and then sit around. It is a music class, so the professor has music playing before class. She may talk to the person next to her if she knows them. Or, she may grab a snack. If she has a little more time, she will go to late meal. Alternatively, she will call her parents or a friend she hasn’t talked to in a while. If there are comfy chairs, she’ll take a nap, but at least 20 minutes are needed.

Daniel Brooker and Kaleb Bradford

Dan usually checks his email, because he is perpetually forgetting about things, events, etc. Kaleb calls relatives, friends to chat. He also checks facebook, emails. He may go over material for class that he is about to go to, but generally only if there is a quiz. Both also wanted to take naps. Dan likes music, so he would go for something in entertainment, music related, maybe play some game. Kaleb and Dan are sports fans, so they would check sports news, team scores. Kaleb would prefer to be able to follow specific teams. Dan feels he would want to check news, current events, etc, because he thinks Princeton students seem isolated from the daily goings on. They would prefer not to work during that small break, because they already spend most of their time outside class working, and then of course the time for the class lectures, precepts, labs. They don’t want to do more work during those 10 minutes. They need a break.

Brainstorm:

FreeFoodFinder: app to find food using the freefood listserv. Find food near you!
CallMe! : app that gives you an alert of who you haven’t called recently (but you have the ability to ignore that alert if you want)
OrganiseMe! : integrated with google calendar. Tells you what assignments you should start working on
Reminderapp: tells you what assignments you haven’t worked on for a while. “Why don’t you do a questions from this problem set?”
Sleepify: app to show you where comfy places are to sleep/sit. (possible name) “Nooks and Crannies”
app integrated with blackboard, so has practice quizzes. Gives you a practice quiz question
Or practice LSAT, MCAT question a day, or when you open app
FunnyBone: app that you can enter funny moments with your friends and so when you are bored, open the app and say, “Show me a funny moment in my past”, and then it will pull up a random memory
Scan emails for keywords, make a calendar out of it, and then you can check it what’s coming up
NapTaker, say I want to nap now, wakes you up 10 mins later.
Nap schedule, shows graphic of when can you take a nap
Filters for news, so you can be alerted of those kind of things in that area of news
Name game. Get to know your neighbours
A collaborative art project, maybe facebook photos that people contribute to
One tough problem that everyone contributes to

I chose free food because I had been wanting to do something like this for a while. Also, it may be a stereotype, but I hear Princeton students love free anything, especially food. This is what I got user feedback on. I also chose the CallMe app, because it seems like a lot of students like to call people they have not spoken to in a while during that break time.

Prototype:

FreeFoodFinder – app that finds free food on campus for you, using the freefood listserv

In the field:

Ajibike looking at loading page

Nicole looking at the Food Neat You! screen. Screen showing your current location as a red dot and food locations as blue dots. Food near you shows food locations in a certain radius surrounding your location.

Screen which shows all food choices on campus, regardless of your location.

Screen showing menu options for users. The buttons lead to the following screens.

Kwaku, another user, deciding to look for free food

Kwaku selecting Food for You! This will bring him to the map page showing his current location and free food spots within a certain radius

Screen giving specific details on food: where, when, what. One way to get here is to select one of the blue food dots on the map.

Greg looking at the AlertMe! screen. This alerts you when there is food in certain locations (locations that the user previously set as a Favourite).

Greg selecting a location to find food in.

April checking out what her free food will be

Settings page — where the user can set preferences, e.g. “Favourite” locations that the user wants alerts from

Feedback

Ajibike Lapite

–wants background music
–“I like it”
–would definitely use it

Kwaku Ohemeng

–is there a way of making availability of food in real time?
–How much food is left?
–in settings, be able to change colour. Personalise app
–Main concern: Can we reserve food? If we ran and not there anymore would be a waste of time
–but it is cool
–would use it
–wants extra features available to independent students
–because they might be most likely to use this

April Liang

–What is the time? Because most free food emails say “food right now!” So would have to use email timestamp instead of finding time in email
–Select location should include a radius set by user
–Yes would use it

Greg Owen

–Might use, but probably not every day
–Would be great app if I had more time, like 30 mins in between class, because might have to go out of my way
–Definitely use Alert Me at a given time feature, because I know regularly when I am craving some food

Human-Computer Interface Technology

Princeton University

Author Archives: Lauren Berdick

Final Project Documentation – The Cereal Killers Group 24

A2 Free Food Finder