Group Num­ber: 4
Group Name: Team TFCS
Group Mem­bers: Collin, Dale, Farhan, Ray­mond

Sum­mary: We are mak­ing a hard­ware plat­form which receives and tracks data from sen­sors that users attach to objects around them, and sends them noti­fi­ca­tions e.g. to build and rein­force habits and track activity.

Previous Posts:

P6: https://blogs.princeton.edu/humancomputerinterface/2013/05/06/p6-team-tfcs/
P5: https://blogs.princeton.edu/humancomputerinterface/2013/04/22/5642/
P4: https://blogs.princeton.edu/humancomputerinterface/2013/04/08/p4-team-tfcs/
P3: https://blogs.princeton.edu/humancomputerinterface/2013/03/29/p3-2/
P2: https://blogs.princeton.edu/humancomputerinterface/2013/03/11/p2-tfcs/
P1: http://blogs.princeton.edu/humancomputerinterface/2013/02/22/964/

Final Video

Changes from P6

– Added a “delete” function and prompt to remove tracked tasks
This was a usability issue that we discovered while testing the app.

– Improved the algorithm that decided when a user performed a task
The previous version had a very sensitive threshold for detecting tasks. We improved the threshold and also used a vector of multiple sensor values to decide what to use asa cutoff instead of only one sensor.

– Simplified choice of sensors to include only accelerometer and magnetometer
This was a result of the user testing which indicated that the multiple sensor choices vastly confused people. We simplified it to two straightforward choices.

– Updated text, descriptions and tutorial within the app to be more clear, based on user input from P6
– Updated each individual sensortag page to display an icon representative of the senor type, simplified the information received from the sensortag in realtime, and added a view of user’s progress in completing tasks

Goal/Design Evolution

At the beginning of the semester, our goal was to make an iPhone application that allowed users to track tasks with TI sensortags, but in a much more general way than we actually implemented. For example, we wanted users to decide which sensors on our sensortag–gyroscope, magnetometer, barometer, thermometer, etc–they would use and how, and we would simply assume that users would be able to figure out how best to use these readings to fit their needs.  This proved to be a poor assumption, because it was not obvious to nontechnical users how these sensors would be used to track tasks they cared about.

We quickly reoriented ourselves to provide not a sensortag tracking app but a *task* tracking app, where the focus of the app was in registering when users took certain actions–opening a book, taking a pill from a pillbox, going to the gym with a gym bag–rather than activated the sensors on certain sensortags. Within this framework, however, we made the model for our application more general, exposing more of how the system functions by allowing them to set up sensors for each task, rather than choose from a menu of tasks within each application. This made our system’s function easier to understand for the end user, which was reflected in our second set of interviews.

Critical Evaluation
Our work over the semester provided strong evidence that this type of HCI device is quite feasible and useful. Most of our tested users expressed an interest in an automated task-tracking application and said that they would use Taskly personally. Still, one of the biggest problems of our implementation of a sensor-based habit tracking system was the size and shape of the sensors themselves. We used a sensortag designed by TI which was large and clunky, and although we built custom enclosures to make the devices less intrusive and easier to use, they were still not “ready for production.” However, as mentioned above, this is something that could easily be fixed in more mature iterations of Taskly. One reason to believe that our system might function well in the real world is that the biggest problems we encountered–the sensortag enclosures and the lack of a fully-featured iPhone app–are things we would naturally fix if we were to continue to develop Taskly. We learned a significant amount about the Bluetooth APIs through implementing this project, as well as about the specific microcontroller we used; we expect BLE devices, now supported only by the iPhone 4S and later phones, will gain significant adoption.

The project ended up being short on time; our lack of iOS experience (initially) made it difficult to build a substantively complex system. The iPhone application, for example, does not have all of the features we showed in our early paper-prototypes. This was partly because those interfaces revealed themselves to be excessively complicated for a system that was simple on the hardware side; however, we lost configurability and certain features in the process. On the other hand, we found learning new hardware platforms (for both iOS and the SensorTag) to be something that could definitely be accomplished over the course of weeks, especially making use of previous computer science knowledge.

One final observation that was reinforced as a result of our user testing was that the market for habit-forming apps is very narrow. People were very satified with the use cases that we presented to them and their recommendations for future applications for the system very closely aligned to the tasks we believed to be applicable for Taskly. Working on this project helped us recognize the diversity of people and needs that exist for assistive HCI-type technologies like this one, and helped us gain a better idea of what kind of people would be most receptive towards systems where they interact with embedded computers.

Moving Forward

One of the things we’d most like to improve upon in later iterations of Taskly are custom sensortags. The current sensortags we use are made by Texas Instruments as a prototyping platform, but they’re rather clunky. Even though we’ve made custom enclosures for attaching these sensors to textbooks, bags, and pillboxes, they are likely still too intrusive to be useful. In a late iteration, we could create a custom sensor that uses just the bluetooth microcontroller core of the current sensortag we’re using (called the CC2541) and the relevant onboard sensors like the gyroscope, accelerometer, and magnetometer. We could fabricate our own PCB and make the entire tag slightly larger than the coin-cell battery that powers the tag. We could then 3D print a custom case that’s tiny and streamlined, so that it would be truly nonintrusive.

Beyond the sensortags, we can move forward by continue to build the Taskly iPhone application using the best APIs that Apple provides. For example, we currently notify users of overdue tasks by texting them with Twilio. We would like to eventually send them push notifications using Apple Push Notifications Services, since text messages are typically used for communication. We could also expand what information the app makes available, increasing the depth and sophistication of historical data we expose. Finally, we could make the sensortag more sophisticated in its recognition of movements like the opening of a book or pillbox by implementing Machine Learning data to interpret these motions (perhaps, for example, using Weka). This would involve a learning section where the user performs the task with the sensortag attached to the object and the system would learn what sensor information corresponds to the task being performed.

Another thing we need to implement before the system can go public is offline storage. Currently the sensor only logs data when the phone is in range of the sensortag. By accessing the firmware on the sensortag, it is possible to make it store data even when the phone is not in range and then transmit it when a device becomes available. We focused on the iOS application and interfacing to Bluetooth, because the demonstration firmware already supported sending all the data we needed and none of us knew iOS programming at the start of the project. Now that we have developed a basic application, we can start looking into optimizing microcontroller firmware specifically for our system, and implementing as much as we can on the SensorTag rather than sending all data upstream (which is more power-hungry). A final change to make would be to reverse the way Bluetooth connects the phone and sensor: currently, the phone initiates connections to the Bluetooth tag; reversing this relationship (which is possible using the Bluetooth Low Energy host API) would make the platform far more interesting, since tags would now be able to push information to the phone all the time, and not just when a phone initiates a connection.

iOS and Server Code

https://www.dropbox.com/s/3xu3q92bekn3hf5/taskly.tar.gz

Third Party Code

1. TI offers a basic iOS application for connecting to SensorTags. We used it as a launching point for our app. http://www.ti.com/tool/sensortag-ios

2. We used a jquery graphing library for visualization. http://www.highcharts.com

Demo Poster

https://www.dropbox.com/s/qspkoob9tggp6yd/Taskly_final.pdf

Richter Threads

Group Name and Number: VARPEX, Group 9

Group Members: Abbi, Dillon, Prerna and Sam

Project Description

Richter Threads is a jacket that creates a new private music listening experience through motors which vibrate with bass.

Previous Blog Posts:

• P1: https://blogs.princeton.edu/humancomputerinterface/2013/02/22/team-varpex-group-brainstorming/
• P2: http://blogs.princeton.edu/humancomputerinterface/2013/03/10/p2-feel-the-music/
• P3: http://blogs.princeton.edu/humancomputerinterface/2013/03/31/p3-team-varpex/
• P4: http://blogs.princeton.edu/humancomputerinterface/2013/04/07/p4-varpex/
• P5: http://blogs.princeton.edu/humancomputerinterface/2013/04/22/p5-team-varpex/
• P6: http://blogs.princeton.edu/humancomputerinterface/2013/05/06/group-9-varpex-p6/

Video

Changes Since P6

• New protoboard with components soldered on: In our feedback from P5, many of our subjects complained that the box they had to carry around containing the system was cumbersome. To create a sleeker and more integrated experience, we soldered our system onto a small protoboard that could be affixed with velcro inside the jacket. The prototype box is no longer necessary.
• Velcro fasteners for motors: The cloth pockets were not effective at holding the motors; we needed to sew them in our last prototype. For this iteration we put velcro inside the jacket and velcro on the motors to make them easier to attach and detach. This also enhances sensation from the motors since there is not an additional layer of cloth over them
• Threshold potentiometer: We discovered in P6 that users liked being able to use their own music, but different music pumps out different bass frequencies. To allow users to change the threshold of bass volume needed for the jackets to vibrate, we included a knob users could use to adjust the amount of bass required to vibrate the motors when listening to their music.
• Volume knob: We reversed the direction of the volume knob to make it more intuitive (turning clockwise increases the volume now).
• New sweater: We chose a much more stylish and, more importantly, a better-fitting sweater for our system, since no one found our original sweater particularly attractive or comfortable.
• Power switch: We added a power switch for just the batteries, to allow people to more easily listen to their music without the jacket on.
• Length of motor wires: We lengthened some of the wires driving the motors, so that the jacket would be easier to put on despite the presence of a lot of wires.
• “Internalizing” components: The board is affixed to the inside of the jacket while the controls (volume/threshold knob, power switch) are embedded in the pocket of the jacket to make the entire system more conspicuously hidden.
• Batteries consolidated: Our battery pack has been consolidated and secured in a different part of the jacket. The batteries impose a large space bottleneck on our design (hard to miniaturize power needs like we did the board), so they require special placement within the jacket.

Evolution of Goals and Design

Our goals over the course of prototyping definitely broadened as we received feedback from users. The original inspiration for the jacket was the dubstep and electronica genres, which are known for being bass-heavy. In testing however, users responded positively to listening to other genres of music through the headphones; by limiting our goals to representing the concert going experience for dubstep/electronica, we were missing out on exploring the potential of the jacket in other musical genres. This development actually lead to a concrete design change when we added a threshold knob to allow users to adjust the sensitivity of the jacket. This actually leads into a different change in goals. Though we set out to replicate the concert-going experience, we succeeded in creating a unique experience in its own right. Our jacket offers users a tactile experience with music beyond mimicking large subwoofers.

As for our design, our early design goals sought to use an Arduino to perform the audio processing needed to filter low bass frequencies to actuate the motors. We quickly learned that this introduced a delay, and unfortunately in music-related applications, any delay can make the application useless. We found an excellent solution however, by implementing a low-pass filter with a threshold to actuate the motors in an analog circuit. This design change also allowed us to work on making the circuit more compact and portable with every iteration of our design. It also eliminates the need for a lot of complicated set-up of a microcontroller; it is hard to imagine that the set-up with an Arduino would have been as easy as plugging in your mp3 player and headphones and turning the jacket on, as it is in our current design.

 Project Evaluation

We will continue to create improved iterations of this jacket in the future. The users who have tested our project have expressed interest in the experience our jacket had to offer, and through the prototyping process we have worked on making the system as portable and inconspicuous as possible. In our last two iterations alone, we moved our system from a bulky box containing our hardware to a much smaller protoboard that we could embed in the inside of the jacket. If we ever wanted to produce this jacket for sale, surface mounted circuits could be made to decrease the electronic size even further. As our jacket stands right now, the jacket does not outwardly appear to be anything but a normal hoodie jacket. It is this sort of form factor that we think people could be interested in owning.

We worked in an applications space we would label as “entertainment electronics.” The challenge this space poses to us is that user ratings are, at their core, almost entirely subjective. We had to evaluate areas like user comfort and the quality of a “sensation,” none of which can really be independently tested or verified. In a way, our design goals boil down to going after a sort of “coolness factor,” and our real challenge was making the jacket as interesting of an experience as possible while minimizing the obstacles to usage. For instance, in P6, people reported difficulty in having to carry around a box that contained the system’s hardware, and while they liked the sensation of the jacket they found the form factor cumbersome. We sought to fix this in our final iteration of the jacket’s design- it is now as comfortable and inconspicuous as a normal jacket, which we hope would allow people to see the jacket as something purely fun at no cost of convenience.

We set out with the goal of replicating a concert-going experience, but in the process created something new entirely. We believe we were successful in the regard of giving something “cool” to people that brought them joy. It is not clear to us what sort of objective measures alone could have achieved this. While we could have strictly timed how long it took people to figure out how our jacket worked, for example, this wouldn’t have informed us about what people really would want out of our system. While we had tasks in mind when we set out to build and test our jacket, the users themselves came up with their own tasks (the jacket’s benefits in silent discos or at sculpture gardens, its uses in a variety of musical genres, etc) that forced us to consider our jacket beyond the narrow goals we had defined for ourselves. It is this sort of feedback that makes designing for entertainment an exciting and dynamic application space.

 Future Plans

If we had more time, we would form a product as sleek, comfortable, and safe as a normal jacket. We would address problems introduced by moisture (sweating or rain) to ensure user safety and proper functioning. In order for this to become a product used in the long term, these concerns must be addressed. Next, we would seek to minimize the impact of the hardware within the jacket. We would use smaller, more flexible wires to attach to the motors. Printed circuit boards would decrease the size of the circuitry and we would optimize the layout of the components. In even further iterations, we would use surface mount electronics to decrease the size of the electronics. We would also like to find a better power solution. For instance, a 9.6V NiCd rechargeable battery would be capable of supplying the necessary current for each of the voltage regulators and can fit in a better form factor. It would also prevent users from having to periodically buy new batteries. We would also test the battery-life of the device to determine acceptable trade-offs between physical battery size and total battery energy. The motors would also be integrated into a removable lining which would hold down the wires and improve the washability and durability of the jacket. We would like to add additional features, including automatic threshold adjustment and a pause button. These new features will require additional understanding of analog electronics and the standards for audio signals on different mobile devices.

The next stages of user testing would inform our decisions about battery type and the long-term usability of the jacket. We would have different sizes of the jacket so we can accommodate users of many body sizes, and we would ideally give the jacket to users for several hours or over the course of days to get quality feedback about practicality and durability. These experiments would also help us understand how users react to the jacket’s sensations after their novelty has worn off. This user testing could further shed light on additional features we might add to the jacket.

 README

How it works:

A male auxiliary cable plugs into a user’s music player and receives the audio signal. The signals from each of the two channels are fed to a dual potentiometer to reduce the volume for the user. The ground of this potentiometer is at VREF (1.56V) because a DC offset is needed in order to keep all the audio information with single supply op-amps. The output of this potentiometer is fed to a female auxiliary cable where the user plugs in headphones. One channel of the audio is also fed to a voltage follower which helps protect the signal from the rest of our circuit. This signal then goes to a low-pass filter which filters out frequencies above 160 Hz. The filtered signal is compared to a threshold value to determine the gate signal for the transistors. When each transistor turns on, current flows through its corresponding motor pair. The circuit is powered through three 9V batteries, each attached to a 5V voltage regulator (L705CVs). Two of the batteries power the motors, which draw significant current, and the third battery powers the operational amplifiers used for the comparator and voltage follower.

Schematics

 Budget

Since our project is mostly hardware based, we’ve included a parts list and final budget below.

 Third Party Code

In earlier iterations, we attempted to use the Arduino to do the audio processing.

http://wiki.openmusiclabs.com/wiki/ArduinoFFT

Demo Materials

https://www.dropbox.com/sh/qxxsu30yclbh7l1/wv1qeXYeM-

Group 24 – the Cereal Killers
Andrew, Bereket, Ryan, and Lauren

The brisq bracelet is an easily-programmable, lightweight, wearable system that allows users to control their computers with gestures picked up by a motion-sensitive bracelet.

Introduction
Brisq is designed to be a simple, wearable interface that allows users to have customizable control over their computers without making use of a mouse or keyboard.

Implementation and Improvements
P5: P5 Blog Post Link

In addition to what we had in our previous prototype, we have finally integrated out GUI with the backend for the program, enabling users to actually record gestures without a “wizard of oz” approach to testing. However, since we do not yet have the accelerometer-based bracelet working, w will still be using one of our team members to simulate gesture execution.

Method
We chose 3 volunteers who had a diverse background of majors, in order to attempt to spread our test subjects across a wide spread of computer expertise. One of our test subjects Is a Woody Woo major, one is a Civil engineer, and one is a COS major. We chose this spread in order to give us a good spread of computer expertise in our test subjects.

For this test, we used our current, semi-functional brisq prototype. It consists of the program backend which can record and execute a series of mouse and keyboard shortcuts, our GUI which is now functionally connected to the program backend, and an arbitrary bracelet + “Wizard of Oz” acting in place of the accelerometer-based bracelet which is still under construction. The tests were all conducted in Quadrangle Club on members who volunteered for our experiment.

Task 1, Cooking: The test subject is asked to look up the recipe for a new dish they do not know how to prepare, and then prepare it while using brisq to navigate through the online recipe while cooking.

Task 2, Watching a movie: The test subject is sitting on a couch and is asked to map gestures to various controls for the movie, e.g. volume changes, pausing, fast forward, rewind, etc, and then use them to control the movie during playback.

Task 3, Disability test: The test subject is asked to perform a specific test while only using one arm to simulate a disability. The subject is given a cast to wear on their dominant arm, and then asked to type, and to open and close various programs.

Procedure
The user was able to record the computer action using the gui.

We used a remote desktop in order to replay the gestures to simulate the bracelet gesture recognition.

When the user did a gesture, the action was replayed using the remote desktop so the user did not have to press the replay button.

For each user, we changed up the order of the tasks

Test Measures
Qualitative:
Do users pick commands that are appropriate for the task.
– If their choice of commands if inefficient, do they need suggestions on tasks?
Do people easily remember which gestures they assigned to which actions?
– If they get confused, what confuses them? Do they mistake one gesture for another, or forget them entirely?
– Do they get flustered, confused, agitated and mess things up when they forget?
Do people actually use the gesture they create, or do they just forget after a while that they have the bracelet at their disposal?

Quantitative:
Tasks 1 and 2:
record the number of gestures performed during set cooking/movie time
Task 3:
record the time taken to complete the given task

Results and Discussion

User 1
Background: 21, male, sophomore, computer science, Dell Windows 8, computer level 5, no other input devices, no bracelets or watches, >10 hrs of active daily use, > 20 of passive daily use (confirmed), most common activity: running big stuff while I sleep

Cooking
-He was confused by the labels: record gesture or record computer action?
-maybe an explanatory paragraph would help to clarify
-we needed to explain that Esc ends the recording (the user didn’t want to minimise the new window to go back to the GUI to end the recording)
-he didn’t want to have the GUI window up after when he wanted to use the gesture
-page up/down doesn’t work

Gesture1 – swoosh up → open window with his favourite recipe site
Gesture2 – right/clockwise twist → page up
Gesture3 – left/counterclockwise twist → page down

TV
-he had trouble remembering functions and their associated gestures.
-he discovered he needed to record more gestures as he went

Gesture1 – big counterclockwise circle → open TV guide
Gesture2 – big clockwise circle → change the input activity
Gesture3 – twist clockwise → channel up
Gesture4 – twist counterclockwise → channel down
Gesture5 – hand palm up → volume up
Gesture6 – hand palm down → volume down
Gesture7 – punch → ok

Injured
-he had trouble thinking of good actions for computers, but using a computer involves complex tasks

Gesture1 – 90 degree right turn of the wrist → make a key work as its mirror key (to allow typing with one hand)
Gesture2 – a tap → backspace
Gesture3 – swipe to the left → close window
Gesture4 – swipe to right → open chrome
Gesture5 – lift hand up quickly → enter key

User 2
Background: 21, female, junior, civil and environmental engineering, laptop: OSX, level 4 computer user, no other input device, 9 hrs of active daily use, 11 hrs of passive daily use, most common task: doing homework, playing music, facebook

Injured
– she was confused about the order of pressing buttons
-she wanted multitouch gestures form IOS
-she used gestures in a limited way → need suggestions for gestures?
-used Esc button to not open window aagain
-play/pause doesn’t work
-page up/down and window key doesn’t get recorded
– she had trouble remembering actions
-she wanted to be able to hold and move icons, files and windows

TV Room
-wanted 2 handed gestures
-she wanted to have volume as level of arm height

Gesture1/2 – hand up/down → channel up/down
Gesture3/4 – hand side/side → volume up and down
Gesture5 – big swipe → mute
Gesture6 – circle hand clockwise/ counterclockwise→ forward / rewind
Gesture7 – punch → play/pause
Gesture8 – diagonal swipe → on/off

Cooking
-she thought that we could have a recipe sponsorship, so when the user does a gesture, it goes to the sponsored recipe site

Gesture1 – swipe hand down → scrolling
Gesture2 – hand to the right/ left → next page/ previous page
Gesture3 – hand in → magnify

User 3
Background: 20, female, sophomore, woodrow wilson school, computer level 2, input device: touchscreen, both bracelets and watches, 5-6 hrs of active daily use, 7-8 hrs of passive daily use, most common task: internet, playing music, writing papers

TV Room
– wanted to know how subtle the gestures are, i.e. how fine can the arm motions be?
– she thought arm motions are difficult, would rather do finger gestures. She wanted to replace the bracelet with a ring of glove.
– wanted more instructions. was confused about what gestures to pick and how to use the device.
– Some of the gestures she thought of were similar to each, and would be difficult for the classifying algorithm

Gesture1 – up/down → volume
Gesture2 – side/side → channel

Injured
Gesture1 – move arm in xy plane to control
Gesture 2 – punch in the z axis to click and double click

Cooking
Gesture1 – up/down → scroll
Gesture2 – diagonal arm swings → zoom in and out

User 4
Background: 21, male, junior, economics, lenovo laptop Windows 7, computer level 3, imput device: fingerprint scanner, neither bracelets nor watches, 6-7 hrs of active daily use, 6-7 hrs of passive daily use, most common task: facebook, buzzfeed, cracked articles, youtube videos

Injured
Gesture1 – hand side/side → switch between window
Gesture2 – diagonal hand swipe → close window
Gesture3 – hand up/down → minimise/maximise window

Tv Room
Gesture1 – wavy hand motion → play
Gesture2 – stop hand motion → pause

Test subject 1, cooking

Test subject 1, watching TV

Test subject 2, watching TV

Appendices

Consent form – Consent Form Link

Demographic Questionnaire –
Do you own a computer? If so, what kind (desktop or laptop? OS?)
Please describe your level of computer savviness, on a scale of 1 to 5 (1 – never used a computer/only uses email and facebook; 5 – can effectively navigate/control your computer through the terminal)
How many hours a day do you actively use your computer?
How many hours a day is your computer on performing a task for you (i.e. playing music, streaming a video)?
What activity do you perform the most on or with your computer?
Have you ever used something other than a keyboard or mouse to control your computer?
Do you wear bracelets or watches?
Have you ever needed crutches, a cast, a sling, or some other type of device to assist with a disability?

Demo Script –
Our project intends to simplify everyday computer tasks, and help make computer users of all levels more connected to their laptops. We aim to provide a simple way for users to add motion control to any of their computer applications. We think there are many applications that could benefit from the addition of gestures, such as pausing videos from a distance, scrolling through online cookbooks when the chef’s hands are dirty, and helping amputees use computers more effectively. The following tests will be aimed at seeing how users such as yourself interfaces with brisq. Brisq is meant to make tasks simpler, more intuitive, and most of all, more convenient; our tests will be aimed at learning how to better engineer brisq to accomplish these goals.

Our GUI has a simple interface with which users can record a series of computer actions to be replicated later with a motion-based command of the brisq bracelet.
[on laptop, demonstrate how users can record a mouse/keyboard command]

Now that a gesture has been recorded, you can map it to one of our predetermined gestures. shake brisq to turn it on or off, performing the gesture at any time when on to activate the series of actions on your computer.
[show how to set gesture to a command]

We think life should be simple. So simplify your life. Be brisq.

Interview script –
We have just finished showing you a quick demo of how to use brisq. Please use our program to enter the computer shortcuts you would like to use and the gestures to trigger them for the task of cooking with a new online recipe using your laptop. The bracelet we will give you is a replica of the real form, and one of our lab members will assist us in simulating the program.
We ask that you then carry on with the activity as you normally would. Do you have any questions?

[user performs task 1]

Now, please use the program to enter the computer shortcuts you would like to use and the gestures to trigger them for the task of watching a movie with your laptop hooked up to a tv, and you are lying on the couch. The bracelet we will give you is a replica of the real form, and one of our lab members will assist us in simulating the program.
We ask that you then carry on with the activity as you normally would. Do you have any questions?

[user performs task 2]

Lastly, please use the program to enter the computer shortcuts you would like to use and the gestures to trigger them for the task of using a computer with an injured arm/cast/etc. For this, we will ask that during the test, you refrain from using one of your arms for simulation purposes. The bracelet we will give you is a replica of the real form, and one of our lab members will assist us in simulating the program.
We ask that you then carry on with the activity as you normally would. Do you have any questions?

[user performs task 3]

Thank you! We will now ask you a short survey based on these tests.

Post-Task Questionnaire –
Did you find the device useful?
How well did the gestures help you perform the task? 1 to 5 where 1 is it hindered you, 3 is neutral/no effect, and 5 is it helped you a lot
How much would you be willing to pay for such a device?
What was the best thing about the product? What was the worst thing?
What other times do you think you would use this?
Any ideas for improvement/other comments?

Group 10 — Team X

-Junjun Chen (junjunc),

-Osman Khwaja (okhwaja),

-Igor Zabukovec (iz),

-(av)

Summary:

A “Kinect Jukebox” that lets you control music using gestures.

Introduction:

Our project aims to pro­vide a way for dancers to inter­act with recorded music through ges­ture recog­ni­tion. By using a Kinect, we can elim­i­nate any need for the dancers to press but­tons, speak com­mands, or gen­er­ally inter­rupt their move­ment when they want to mod­ify the music’s play­back in some way. Our moti­va­tion for devel­op­ing this sys­tem is twofold: first of all, it can be used to make prac­tice rou­tines for dancers more effi­cient; sec­ond of all, it will have the poten­tial to be inte­grated into impro­visatory dance per­for­mances, as the ges­tural con­trol can be seam­lessly included as part of the dancer’s movement. In this experiment, we want to use three tasks to determine how well our system accomplishes those goals, by measuring the general frustration level of users as they use our system, the level of difficulty they may have with picking up our system, and how well our system does in recognizing gestures and responding to them.

Implementation and Improvements

P5: https://blogs.princeton.edu/humancomputerinterface/2013/04/22/p5-group-10-team-x/

Changes:

• Implemented the connection between the Kinect gesture recognition and music processing components using OSC messages which we were not able to do for P5. This means that we did not need to use any wizard of oz techniques for P6 testing.

• Implemented the GUI for selecting music (this was just a mock up in P5).

Method:

Participants:

The participants were selected at random (from Brown Hall). We pulled people aside, and asked if they had any dancing experience. We tried to select users who had experience, but since we want our system to be intuitive and useful for dancers of all levels, we did not require them to have too much experience.

Apparatus:

The equipment used included a Kinect, a laptop, and speakers. The test was conducted in a room in Brown Hall, where there was privacy, as well as a large clear area for the dancers.

Tasks:

1 (Easy). The first task we have chosen to support is the ability to play and pause music with specific gestures. This is our easy task, and we’ve found through testing that it is a good way to introduce users to our system.

2 (Medium). We changed the second task from “setting breakpoints” to “choosing music” as we wanted to make sure to include our GUI in the testing. We thought that the first and third task was adequate for testing the gesture control (especially since the gesture control components are still being modified). We had participants go through the process of selecting a song using our graphic interface.

3 (Hard). The third task is to be able to change the speed of the music on the fly. We had originally wanted to have the music just follow the user’s moves, but we found that the gesture recognition for this would have to be incredibly accurate for that to be useful (and not just frustrating). So, instead, the speed of the music will just be controlled with gestures (for speeding up, slowing down, and returning to normal).

Procedure:

For test­ing, we had each user come into the room and do the tasks. We first read the general description script to the user, then gave them the informed consent form, and asked for verbal consent. We then showed them a short demo. We then asked if they had any ques­tions and clar­i­fied any issues. After that, we fol­lowed the task scripts in increas­ing order of dif­fi­culty. We used the eas­i­est task, set­ting and using pause and play ges­tures, to help the user get used to the idea. This way, the users were more com­fort­able with the harder tasks (which we wanted the most feed­back on). While the users were completing the tasks, we took notes on general observations, but also measured the statistics described below. At the end, we had them fill out the brief post-test survey.

Test Measures:

Besides making general observations as the users performed our tasks, we measured the following:

• Frustration Level (1 to 5, judged by body language, facial expressions, etc.), as the main purpose of our project is to make the processes of controlling music while practicing as easy as possible.

• Initial pickup – how many questions were asked at the beginning? Did the user figure out how to use the system? We wanted to see how intuitive and easy to use our system was, and whether we explained our tasks well.

• Number of times reviewing gestures – (how many times did they have to look back at screen?) We wanted to see how well preset gestures would work (is it easy to convey a gesture to a user?)

• Attempts per gesture  (APG) – The number of tries it took the user to get the gesture to work. We wanted to see how difficult it would be for users to copy gestures and for the system to recognize those gestures, since getting gestures to be recognized is the integral part of the system.

• Success – (0 or 1) Did the user complete the task?

Results and Discussion

Here are statistical averages for the parameters measured for each task.

From these statistics, we found that APG for each tasks was relatively high, and high APGs correlated with higher frustration levels. Also, unfortunately, the APG didn’t seem to go down with practice (from task 1 to task 3, and with the 3 different gestures in tasks 3). The tasks with the highest APG (10+ on normalize speed, and 8 on speed up) both happened towards the end of the sessions.

These issues seem to stem from misinterpretation of the details of the gesture on the screen (users tried to mirror the gestures, when in fact the gestures should have been performed in the opposite direction), as well as general issues in the consistency of the gesture recognition. All three users found the idea interesting, but as users 1 and 3 commented on, gesture recognition needs to be robust for the system to be useful.

Changes we plan to make include a better visualization of the gestures, so that users have less trouble following them. Based on the trouble some of our users had with certain gestures, we’ve decided that custom gestures are an important part of the system (users would be able to set gestures that they are more comfortable with, and will be able to reproduce more readily). This should alleviate some of the user errors we saw in testing. On the technical side, we also have to make our gesture recognition system more robust.

Appendices

i. Items Provided / Read to Participants:

• Consent Form

• Introductory Demo Script

• Task 1 Demo Script

• Task 2 Demo Script

• Task 3 Demo Script

• Post-Task Questionnaire

ii. Raw Data

Links to Participant Test Results:

• Participant 1 Results

• Participant 2 Results

• Participant 3 Results

Individual Statistics:

Participant 1:

Participant 2:

Participant 3:

Group Number: 1

Group Members:

Avneesh Sarwate, John Subosits, Yaared Al-Mehairi, Joe Turchiano, Kuni Nagakura

Project Summary:

Our project is a serving system for restaurants, called Ser­vice­Center, that allows wait­ers to effi­ciently man­age orders and tasks by displaying information about their tables.

Description of tasks we chose to support:

In our working prototype, we have chosen to support an easy, medium, and hard task. For the easy task, users are simply asked to request help if they ever feel overwhelmed or need any assistance. This task is important to our system, because one of our functionalities is the ability to request assistance from waiters who may have a minute to spare and are watching the motherboard.

For our medium task, we ask users to input order information into the system for their table. Once users input orders, the motherboard will be populated with orders and the kitchen will also be notified of the orders. This is a medium task since this is simply inputting information.

Finally, for our hard task, we ask our users to utilize customer statuses (ie. cup statuses, food orders) to determining task orders. This task covers both ensuring that cups are always full and that the prepared food is not left out for too long. The motherboard’s main functionality is to gather all this information in an easy to read manner, allowing waiters to simply glance at the customer status to determine what order they should do things.

How our tasks have changed since P3 and P4:

We didn’t end up changing our tasks too much, since we thought it covered the functionality of the system pretty well. Based on user testing, testing, our easy task was performed as needed, and our other two tasks were performed regularly throughout the testing process. We decided to combine the medium and hard task into a hard task and add a new task that requires users to input customer orders into the system. This last task was added so that we could test the full functionality of our system. We also didn’t test the input interface in P4, so we really wanted to add this as a new task.

Discussion of revised Interface Design:

Initially, we had designated a dedicated person to take the order-slips of the waiters and then input them into the system. However, during our paper-prototype test, we saw that this designated inputter was by far the slowest element of the system. We decided, then, that it may be easier to have the waiters enter their own orders. Even though there is a lag involved in typing out items twice, there could be more of a lag if a single person manning the input screens becomes overwhelmed, as then that worker could make mistakes that lead to errors that cause problems for other waiters. By having each waiter responsible for their own input, we could reduce the volatility of our system.

Our initial design for the order-input interface was fairly involved, and displayed the restaurant menu as a clickable menu with a list and checkboxes, but after seeing how long basic tasks took on the paper prototype, we thought we should strip down to make it more efficient. We decided, for the purposes of this prototype, to have a simple text entry box and button for each table. Having a physical menu next to the entry station could also save screen space, and allow the input interface to be displayable on much smaller screens (cheap tablets, old portable TVs, smartphones). We decided to move to a typing interface because we had a very minimal amount of information that needed entering, and typing seemed faster for this limited interaction.

Easy Task 

Medium Task

Hard Task

Overview and Discussion of New Prototype 

The table-end side of the system has been fairly extensively prototyped although the interface between it and the rest of the system has not been implemented.  The current system for each restaurant table consists of three coasters for the patrons’ drinks which incorporate FSR’s to estimate the fullness of the glass based on its weight.  The coasters were manufactured from clear Plexiglas.  An Arduino monitors the sensors on the table and sends their states to a laptop computer via a serial connection.  The status of each of the glasses then prints to a serial monitor window on the laptop.  For the purpose of this prototype, a wizard-of-oz technique used a person at this computer communicating via a chat application with another person at the computer controlling the motherboard which displays the status of the various tables.  The wireless communication between the Arduino and the motherboard via a laptop was not implemented for this prototype because of the ease with which wizard-of-oz techniques could be used to simulate it and the relative difficulty of this part of the project.  The code for monitoring the FSR’s with the Arduino was loosely based on the Adafruit FSR tutorial code, available at http://learn.adafruit.com/force-sensitive-resistor-fsr/using-an-fsr, although the code was used primarily as inspiration.  Little was retained from the code expect the variable names and setup routine.

We implemented the changes to the interface design for the order input system. The minimal language that the users would enter into the box of each table would be “DishName Command.”  DishName could be any string (so users could use their own shorthand), while Command would be either “add,” “done,” or “del.” Add would add the item to the list for that table, done would mark that item as delivered, and delete would delete them from the order list. Pressing the button would “send” the information to the motherboard.

We chose to implement the Motherboard (the actual display screen for the app) using Python, with the Pygame module and a “Standard Draw”-like module included for ease of creating the actual screen. We chose Python since it is a simple, easy-to-use and debug language which fully supports object-oriented programming. Each “Table” on the screen is implemented as an object, which contains a variable number of “Order” objects and a single “Cup Display” object (which displays the current number of empty, medium, and full cups). We deemed it important to implement these objects to allow for modularity in the display screen to adjust to different restaurant floor plans. With the current implementation, it is possible (and fairly simple) to draw a rectangular table of any size at any position on the screen along with all of the table status features. In the final implementation, we plan to include support for circular tables as well. The screen updates itself whenever it receives new information to display it in real time.

We decided to leave out implementing the communication between different parts in our working prototype. Our system currently is made up of independent components with a set API. We have defined an API for each system so that cup statuses and order inputs/statuses can be easily changed through these functions once the communication channel has been implemented. In testing our prototype, we will use a Wizard of Oz technique for the communication between devices, and have our group members relay the information from each sytem. We chose this functionality to leave out of the working prototype, because we wanted to focus primarily on the layout of the motherboard and input system, and the functionality of our coasters. The help button is still a paper prototype, since in our ultimate system, this will simply be a button that alerts the motherboard. We decided that having a paper prototype and an observer to keep track of when the button is pressed (an easy task as P4 showed) were enough for user testing.

The Motherboard section used the Pygame module for Python. Apart from this, all code was written by Kuni Nagakura and Joe Turchiano.

A Plexiglas coaster with protective covering still in place showing temporary FSR attachment.

Circuitry showing three inputs and 10K resistors.

Simple Order Input Interface

Screenshot of Motherboard

Your group number and name
Group 8 – Backend Cleaning Inspectors

First names of everyone in your group
Dylan
Green
Tae Jun
Keunwoo Peter

1-sentence project summary. 1 paragraph describing the tasks you have chosen to support in this working prototype (3 short descriptions, 2–3 sentences each; should be one easy, one medium, one hard).
Our project is to make a device that could help with laun­dry room security.

Tasks supported in this prototype
1. Cur­rent User: Lock­ing the machine:

– The Current User inputs ID number into lock­ing unit key­pad. The product will then try and match his number to a netid in our system. If it finds a match, it will ask the user if this is their id. They can then answer yes or no and if yes, the machine locks. This netid is also the netid that will be sent emails if an alert is sent. This task would be medium in difficulty as the user has to ensure that he/she enters the right 9-digit number.

2. Next User: Send­ing mes­sage to cur­rent user that laun­dry is done and some­one is wait­ing to use the machine:

– When there are no machines open, the next user can press a but­ton to send an alert at any time dur­ing the cycle. When the but­ton is pressed, an alert will be sent to the current user saying someone is waiting to use the machine. The difficulty of this task would be easy. It is lit­er­ally as easy as press­ing a button.

3. Cur­rent User: Unlock the machine:

– If the machine is cur­rently locked and the current user wishes to unlock the machine, the cur­rent user must input his princeton ID number. Once he has done this, the system then checks this id and tries to find a potential match to a net id. If it does, it will ask the user if this is his net id and on yes, it will unlock. This is a medium/hard task, as the user must input his number and confirm to unlock.

1 short paragraph discussing how your choice of tasks has changed from P3 and P4, if at all, as well as the rationale for changing or not changing the set of tasks from your earlier work.
The only change to our task was that instead of unlocking automatically when the cycle and following grace period ended, our system will unlock only when a new user alerts the owner of the locked machine (followed by a subsequent, shorter grace period).

2–3 paragraphs discussing your revised interface design
Describe changes you made to the design as a result of P4, and your rationale
behind these changes (refer to photos, your P4 blog post, etc. where appropriate)
We decided to only make one significant change to our system. Before, we planned to have our system automatically unlock after the grace period (the 10 minute-ish period after the laundry finished) was finished. However, we decided that there is no reason to unlock the machine if no one is waiting to use it. That would be unnecessarily risky. Thus, we decided to start the grace period only after an alert has been sent to the current user.

Provide updated storyboards of your 3 tasks
first task

second task

third task

Provide sketches for still-unimplemented portions of the system, and any other
sketches that are helpful in communicating changes to your design and/or
scenarios

3–4 paragraphs providing an overview and discussion of your new prototype

Discuss the implemented functionality, with references to images and/or video in the next section
Our product implements each of the three tasks described above. In addition to the procedures mentioned above for taks one and two, our product also gracefully handles backspacing, clearing, and various other errors, such as entering the wrong id or an id not yet in our database system. As for the matching of the numbers to ids, currently we have just hardcoded a few pairs into our system. Were we to make our product for real, we would have to get a list from the school and integrate it with our product. This is unnecessary for our product at this stage however.

Describe what functionality from the proposed complete system you decided to leave out of this prototype, and why
As of right now, we have not implemented the grace period system yet. It is intended to be a system where an alert can be sent, once the machine is finished running. When someone presses the alert button, our product sends an email to the current user of the machine telling them that someone is waiting to use the machine, and that the machine will be unlocked after a certain period of grace time has passed. We haven’t decided how long this grace period will be yet, but we think something like 5 or 10 minutes should suffice. The alert button can also be pressed before the laundry is done (before 35 minutes have passed), but the alert is still only sent once the laundry is done. The reason we have not fully implemented this grace period functionality yet, is because we had trouble with the code for implementing a third party timer class. We plan to have this functionality up and running in a week or so. We do however have the “send alert” functionality already running. When the button is pushed an email will be sent to the current user alerting them that someone wishes to use their machine. We just haven’t implemented the grace period timing yet.

iii. Describe any wizard-of-oz techniques that are required to make your prototype work

No wizard of oz techniques are required to make our prototype work.

In the above sections, be sure to provide your rationale for choosing what
functionality to implement, what to wizard-of-oz, and what to ignore.
The only thing we havent implemented yet is the timing mechanism for the grace period interval. This is due to unfamiliarity using third party library to implement the timing code.

Document any code you used that was not written by your team (e.g., obtained
from an online tutorial, third-party library, etc.). If you did not use code from other
sources, say so.
The additional libraries we use are: Keypad, LiquidCrystal, Servo, SimpleTimer, and WiFly Shield.

Video and/or images documenting your current prototype

keypad

wifly shield and lcd

inside of lock mechanism

whole set up

lcd screen

lock mechanism

Videos: