Group 24 – the Cereal Killers
Andrew, Bereket, Ryan, and Lauren

The brisq bracelet is an easily-programmable, lightweight, wearable system that allows users to control their computers with gestures picked up by a motion-sensitive bracelet.

Introduction
Brisq is designed to be a simple, wearable interface that allows users to have customizable control over their computers without making use of a mouse or keyboard.

Implementation and Improvements
P5: P5 Blog Post Link

In addition to what we had in our previous prototype, we have finally integrated out GUI with the backend for the program, enabling users to actually record gestures without a “wizard of oz” approach to testing. However, since we do not yet have the accelerometer-based bracelet working, w will still be using one of our team members to simulate gesture execution.

Method
We chose 3 volunteers who had a diverse background of majors, in order to attempt to spread our test subjects across a wide spread of computer expertise. One of our test subjects Is a Woody Woo major, one is a Civil engineer, and one is a COS major. We chose this spread in order to give us a good spread of computer expertise in our test subjects.

For this test, we used our current, semi-functional brisq prototype. It consists of the program backend which can record and execute a series of mouse and keyboard shortcuts, our GUI which is now functionally connected to the program backend, and an arbitrary bracelet + “Wizard of Oz” acting in place of the accelerometer-based bracelet which is still under construction. The tests were all conducted in Quadrangle Club on members who volunteered for our experiment.

Task 1, Cooking: The test subject is asked to look up the recipe for a new dish they do not know how to prepare, and then prepare it while using brisq to navigate through the online recipe while cooking.

Task 2, Watching a movie: The test subject is sitting on a couch and is asked to map gestures to various controls for the movie, e.g. volume changes, pausing, fast forward, rewind, etc, and then use them to control the movie during playback.

Task 3, Disability test: The test subject is asked to perform a specific test while only using one arm to simulate a disability. The subject is given a cast to wear on their dominant arm, and then asked to type, and to open and close various programs.

Procedure
The user was able to record the computer action using the gui.

We used a remote desktop in order to replay the gestures to simulate the bracelet gesture recognition.

When the user did a gesture, the action was replayed using the remote desktop so the user did not have to press the replay button.

For each user, we changed up the order of the tasks

Test Measures
Qualitative:
Do users pick commands that are appropriate for the task.
– If their choice of commands if inefficient, do they need suggestions on tasks?
Do people easily remember which gestures they assigned to which actions?
– If they get confused, what confuses them? Do they mistake one gesture for another, or forget them entirely?
– Do they get flustered, confused, agitated and mess things up when they forget?
Do people actually use the gesture they create, or do they just forget after a while that they have the bracelet at their disposal?

Quantitative:
Tasks 1 and 2:
record the number of gestures performed during set cooking/movie time
Task 3:
record the time taken to complete the given task

Results and Discussion

User 1
Background: 21, male, sophomore, computer science, Dell Windows 8, computer level 5, no other input devices, no bracelets or watches, >10 hrs of active daily use, > 20 of passive daily use (confirmed), most common activity: running big stuff while I sleep

Cooking
-He was confused by the labels: record gesture or record computer action?
-maybe an explanatory paragraph would help to clarify
-we needed to explain that Esc ends the recording (the user didn’t want to minimise the new window to go back to the GUI to end the recording)
-he didn’t want to have the GUI window up after when he wanted to use the gesture
-page up/down doesn’t work

Gesture1 – swoosh up → open window with his favourite recipe site
Gesture2 – right/clockwise twist → page up
Gesture3 – left/counterclockwise twist → page down

TV
-he had trouble remembering functions and their associated gestures.
-he discovered he needed to record more gestures as he went

Gesture1 – big counterclockwise circle → open TV guide
Gesture2 – big clockwise circle → change the input activity
Gesture3 – twist clockwise → channel up
Gesture4 – twist counterclockwise → channel down
Gesture5 – hand palm up → volume up
Gesture6 – hand palm down → volume down
Gesture7 – punch → ok

Injured
-he had trouble thinking of good actions for computers, but using a computer involves complex tasks

Gesture1 – 90 degree right turn of the wrist → make a key work as its mirror key (to allow typing with one hand)
Gesture2 – a tap → backspace
Gesture3 – swipe to the left → close window
Gesture4 – swipe to right → open chrome
Gesture5 – lift hand up quickly → enter key

User 2
Background: 21, female, junior, civil and environmental engineering, laptop: OSX, level 4 computer user, no other input device, 9 hrs of active daily use, 11 hrs of passive daily use, most common task: doing homework, playing music, facebook

Injured
– she was confused about the order of pressing buttons
-she wanted multitouch gestures form IOS
-she used gestures in a limited way → need suggestions for gestures?
-used Esc button to not open window aagain
-play/pause doesn’t work
-page up/down and window key doesn’t get recorded
– she had trouble remembering actions
-she wanted to be able to hold and move icons, files and windows

TV Room
-wanted 2 handed gestures
-she wanted to have volume as level of arm height

Gesture1/2 – hand up/down → channel up/down
Gesture3/4 – hand side/side → volume up and down
Gesture5 – big swipe → mute
Gesture6 – circle hand clockwise/ counterclockwise→ forward / rewind
Gesture7 – punch → play/pause
Gesture8 – diagonal swipe → on/off

Cooking
-she thought that we could have a recipe sponsorship, so when the user does a gesture, it goes to the sponsored recipe site

Gesture1 – swipe hand down → scrolling
Gesture2 – hand to the right/ left → next page/ previous page
Gesture3 – hand in → magnify

User 3
Background: 20, female, sophomore, woodrow wilson school, computer level 2, input device: touchscreen, both bracelets and watches, 5-6 hrs of active daily use, 7-8 hrs of passive daily use, most common task: internet, playing music, writing papers

TV Room
– wanted to know how subtle the gestures are, i.e. how fine can the arm motions be?
– she thought arm motions are difficult, would rather do finger gestures. She wanted to replace the bracelet with a ring of glove.
– wanted more instructions. was confused about what gestures to pick and how to use the device.
– Some of the gestures she thought of were similar to each, and would be difficult for the classifying algorithm

Gesture1 – up/down → volume
Gesture2 – side/side → channel

Injured
Gesture1 – move arm in xy plane to control
Gesture 2 – punch in the z axis to click and double click

Cooking
Gesture1 – up/down → scroll
Gesture2 – diagonal arm swings → zoom in and out

User 4
Background: 21, male, junior, economics, lenovo laptop Windows 7, computer level 3, imput device: fingerprint scanner, neither bracelets nor watches, 6-7 hrs of active daily use, 6-7 hrs of passive daily use, most common task: facebook, buzzfeed, cracked articles, youtube videos

Injured
Gesture1 – hand side/side → switch between window
Gesture2 – diagonal hand swipe → close window
Gesture3 – hand up/down → minimise/maximise window

Tv Room
Gesture1 – wavy hand motion → play
Gesture2 – stop hand motion → pause

Test subject 1, cooking

Test subject 1, watching TV

Test subject 2, watching TV

Appendices

Consent form – Consent Form Link

Demographic Questionnaire –
Do you own a computer? If so, what kind (desktop or laptop? OS?)
Please describe your level of computer savviness, on a scale of 1 to 5 (1 – never used a computer/only uses email and facebook; 5 – can effectively navigate/control your computer through the terminal)
How many hours a day do you actively use your computer?
How many hours a day is your computer on performing a task for you (i.e. playing music, streaming a video)?
What activity do you perform the most on or with your computer?
Have you ever used something other than a keyboard or mouse to control your computer?
Do you wear bracelets or watches?
Have you ever needed crutches, a cast, a sling, or some other type of device to assist with a disability?

Demo Script –
Our project intends to simplify everyday computer tasks, and help make computer users of all levels more connected to their laptops. We aim to provide a simple way for users to add motion control to any of their computer applications. We think there are many applications that could benefit from the addition of gestures, such as pausing videos from a distance, scrolling through online cookbooks when the chef’s hands are dirty, and helping amputees use computers more effectively. The following tests will be aimed at seeing how users such as yourself interfaces with brisq. Brisq is meant to make tasks simpler, more intuitive, and most of all, more convenient; our tests will be aimed at learning how to better engineer brisq to accomplish these goals.

Our GUI has a simple interface with which users can record a series of computer actions to be replicated later with a motion-based command of the brisq bracelet.
[on laptop, demonstrate how users can record a mouse/keyboard command]

Now that a gesture has been recorded, you can map it to one of our predetermined gestures. shake brisq to turn it on or off, performing the gesture at any time when on to activate the series of actions on your computer.
[show how to set gesture to a command]

We think life should be simple. So simplify your life. Be brisq.

Interview script –
We have just finished showing you a quick demo of how to use brisq. Please use our program to enter the computer shortcuts you would like to use and the gestures to trigger them for the task of cooking with a new online recipe using your laptop. The bracelet we will give you is a replica of the real form, and one of our lab members will assist us in simulating the program.
We ask that you then carry on with the activity as you normally would. Do you have any questions?

[user performs task 1]

Now, please use the program to enter the computer shortcuts you would like to use and the gestures to trigger them for the task of watching a movie with your laptop hooked up to a tv, and you are lying on the couch. The bracelet we will give you is a replica of the real form, and one of our lab members will assist us in simulating the program.
We ask that you then carry on with the activity as you normally would. Do you have any questions?

[user performs task 2]

Lastly, please use the program to enter the computer shortcuts you would like to use and the gestures to trigger them for the task of using a computer with an injured arm/cast/etc. For this, we will ask that during the test, you refrain from using one of your arms for simulation purposes. The bracelet we will give you is a replica of the real form, and one of our lab members will assist us in simulating the program.
We ask that you then carry on with the activity as you normally would. Do you have any questions?

[user performs task 3]

Thank you! We will now ask you a short survey based on these tests.

Post-Task Questionnaire –
Did you find the device useful?
How well did the gestures help you perform the task? 1 to 5 where 1 is it hindered you, 3 is neutral/no effect, and 5 is it helped you a lot
How much would you be willing to pay for such a device?
What was the best thing about the product? What was the worst thing?
What other times do you think you would use this?
Any ideas for improvement/other comments?

Lab 3: Motor Control

Bereket Abraham
Lauren Berdick
Ryan Soussan
Andrew Ferg

Group #24: The Cereal Killers

Z Flipper

We built the Z-Flipper, a robot designed to flip over itself multiple times. It was made out of 3 panels connected by servo motors at the joints. The construction was good, but our servo motors were too weak to lift the entire weight of the robot. The second robot we constructed, the Wormotrom, moved like an inchworm. The robot moved at a slow but steady pace, and put less strain on the motors. However, we were unable to control the direction the robot moved in. In the future, we would remember to consider the strength of our motors during the brainstorming and idea selection process.

12 Robot Ideas

1. 2 servos on either side of a piece of styrofoam and have it crawl along.

2. 2 DC motors attached to wheels on either side of a piece of styrofoam with a passive trailing wheel. A 3 wheel cart. You could turn by spinning one side faster than the other.

3. Put a piece of styrofoam on a 3-wheeled tripod. Attach a DC motor on the back as a fan. Blows across the table.

4. 2 servos attached to a ribbon. The servos turn in opposite directions, causing the ribbon to contract and expand. Much like a worm.

5. One servo with a gripper claw attached. The motor sweeps from 0 to 30 deg slows, and then back quickly. The robot claws it way across the table. Perhaps you would need two trailing legs for stability.

6. The same thing as above except now we have 2 gripper claws on either side. This configuration could also be used to climb as well as crawl.

7. Attach tiny helicopter blades to the DC motor. The robot flies into the air with a burst of speed, then glides over to the other side of the table.

8. Two DC motors attached to wheels and connected by some kind of thick rubber band or track. The motors are fixed together and the track rotates around them, like a really basic tank.

9. Create a hollow cylinder with a DC motor at the central axis. Attached to the motor is a weighted rod, or a pendulum. As the motor turns, it shifts the cylinder’s center of mass, causing it to turn.

10. A four legged robot with a rotating hip. One servo would control the hip and one DC motor would control the front two legs. The back legs would be passive / wheeled. The front legs would be 180 degrees out of phase, so that as the hip turned one leg would be in the air and the other would be pulling the robot forward.

11. The entire path has a track or ladder. The robot simply has a DC motor that pulls itself along the track.

12. Cover the table in a few inches of water. Make a small boat and with 2 paddles connected to servo motors. Or a DC motor connected to a propellor.

13. 3 flat plates in the shape of a “Z”. At each joint is a servo motor. The plates open up, causing the robot to flip over. The robot moves forward by flip over and over in the right direction.

Final Idea: #13

At first, we thought about doing the crawler, #6.

But then we decided on the “Z Flipper”, #13.

Here is a diagram on how the Z Flipper would move.

Parts List

3 Cardboard plates made from gum packaging.
2 stepper motors
1 arduino microcontroller
2 long, stiff wires for attaching the motors
tape
jumper wires

Instructions

First construct the flipper. Attach the motors to the plates using tape and wires. At a joint, the motor body connects to one of the plates using tape. The motor arms connect to the other plate. This connection is much weaker and needs the wire to reinforce it. When that’s done, use the jumper cables to connect the motors to the arduino. Finally, upload the code and let it go. The code is designed to turn the motor, wait a set amount of time, and then turn the other motor. And so on.

Testing

Video of the Z-Flipper. We were able to construct the flipper mechanism, but the motors were too weak to move unaided.

In order to salvage a workable robot, we stripped off the second joint, leaving only 2 panels connected by a servo motor. Now the robot was basically a hinge that could open and close. Instead of flipping over itself, the robot would now move using an inch-worm type motion. To create forward motion, I covered the back panel in smooth electrical tape and put a flap of scotch tape on the front panel. When bending open, the flap would curl under itself and slide forward. When bending close, the sticky side would catch on the table and pull the rest of the robot forward. I call it the Wormotron. Check out our new diagrams and video.

Diagram

Wormotron in action.

Arduino Code

/*
Lab 3, COS 436
Z Flipper Robot
Group 24: The Cereal Killers
*/

#include  

int jointPinA = 9;
Servo servoA;
int angleA = 0;
int steps = 10;
int on = 1;
 
void setup() {
  servoA.attach(jointPinA);
  
  Serial.begin(9600);
  Serial.println("Ready");
}
 
 
void loop() 
{   
  if(on == 1) {
    for(int i = 0; i < steps; i++) 
    {
      openJointA();
      delay(15);
      closeJointA();
      delay(15);
    }
    on = 0;    
    servoA.detach();
  }
}

void openJointA() {
  // scan from current position to 180 degrees
  for(; angleA  0; angleA--)  
  {
    servoA.write(angleA);
    delay(15);
  }
}

Group 24:
Bereket Abraham
Andrew Ferg
Lauren Berdick
Ryan Soussan

P2: Contextual Inquiry and Task Analysis
The Cereal Killers

Brisq: A Gesture Control Device

Problem: Current WIMP and touchscreen interfaces force you to constantly use your hands in order to interact with a computer.
Solution: We want to create a wristband that allows you to map hand and arm motions to computer commands. We also want to create software that allows the user to create the commands/routines and map gestures on their own.

1. Interviews
We have a couple of target groups for the app, each that can benefit in a different way. The main ones we identified were housewives who would need to interact with their computer when their hands were dirty cooking, amputees who have trouble interacting with computers, heavy computer users who would like to make their computing experience more efficient, and DJ’s that would like to add a more fluid control of their music. Housewives are relevant since cooking can be messy, and often times they enjoy trying new recipes which nowadays are being accessed with digital devices. If they could scroll through their recipes without actually touching the device, it would keep them clean, out of danger, and would save paper. For amputees, we felt that gestures could enhance their computing experience, such as someone missing a hand could put on the bracelet and no longer have to rely solely on their one available hand. For heavy computer users, we felt there were many programs, games, and shortcuts that could be programmed with gestures that would save heavy computer users time and make them quicker with their computers. Repetitive tasks such as going to the desktop, opening a web browser, and change a song could be done with a gesture. Finally, we thought DJ’s could use the device since gestures would give them a new way to control their devices which could be more fluid and fun. We interviewed four people from 3 different target groups.

The first person we interviewed was a housewife. We interviewed her over the phone around 8:00 pm. She is 52 years old and lives with her husband and two children, aged 11 and 15. She enjoys being with her family. We asked her some questions about her daily routine before introducing our device to her. She is a stay at home mom, and runs errands and does chores until she has to pick up her kids from school. She starts cooking dinner around 5:30 so that they can eat as a family when her husband arrives home at 6:30. When asked about cooking, she said she has a set of meals she switches between for dinner. We asked if she ever tries new meals, and she said she does occasionally, maybe once every week or two. When asked where she gets her new recipes, she said the internet. She normally prints out the recipes and follows them then. She does have a laptop though, so when introduced of the idea of being able to scroll down with a gesture bracelet while cooking, she immediately thought it would be nice to save paper and eliminate the hassle of running out/low on paper and ink. She said she would consider using her laptop as a cooking aid if she had the bracelet, but wasn’t sure if she would like wearing a bracelet while cooking. She was also worried it would get damaged while she cooked. Overall, she had mixed feelings towards the idea, and wasn’t sure if she would use it.

The next person we interviewed was a DJ. We interviewed him while he was setting up for a party at an eating club. He works for various clubs, parties, and bars and we met him when he came to DJ for an eating club. He was concerned with getting more gigs, and having a very responsive turntable system. when we asked if he felt his controls could be helped with a gesture option, he was excited by the idea but was worried about the delay. He thought it wouldn’t be that helpful if he had to spend time initiating a gesture, and wasn’t sure what specifically it would be used for since his DJ equipment was analog so that it was respond quickly to his changes. He thought it might be useful as a novelty item, or in slower sections or when he was talking on the microphone.

The next target group we interviewed was heavy computer users that were more tech savy. We will call the first interviewee “Steve”.

We interview Steve in the Frist Gallery. Steve is a heavy computer user and multi-tasker. He really enjoys video games. He would definitely consider using our product. Steve once had carpal tunnel after using his computer way too often, and he had to stop using the mouse. He was left with no alternative which made it very difficult for him to use his computer and he had to limit his use of the machine. He feels he would have probably used a gesture controller like ours had it been available and it would have made his computer use much more painless and comfortable.

He says gesture control would make simple shortcuts for him – he would love to see anything with multiple steps reduced into one thing. When he first gets up, he has a routine with his computer, opening his favourite tabs and websites like Facebook, mail, YouTube. It would be great for him if with one swipe of his hand from his bed he could start those up. If he could just open the browser and then swipe his hand to get his favourites that would make his life easier. Conversely, it would also work for a shutdown procedure: if with one hand gesture he could save all his documents instead of having to go through all the windows and then turn off the computer.

Steve would like to see in the future the hand gesture extended into finger motion as well, to make more precise movements. If sensitivity could spread to the fingers without making the device bulky, it would definitely make the product even more useful.
Most importantly, he states that you need a way to activate the gesture control device. He would like to see us put some kind of switch or button on the gesture bracelet so that normal actions are not confused with gestures.

Last, we interviewed “Michael”.

We interviewed Michael in Frist at a first floor table. Michael is also a heavy computer user. He considers himself a multi-tasker. He thinks this would be a product that he would use, because he thinks the idea of being able to control the computer even when he is not right there would be useful. His pet peeve with the mouse is clicking, dragging and resizing windows. He really gets annoyed with the little pointer. He feels that gesture control would just make that part of his computer use a lot easier.

Michael has also had to use Smart Boards from some of his many presentations. But he has had to be right next to these boards. Even though the board is quite advanced, he still feels being able to be away from the board would make presentations more effortless and would also add grandeur to it, if he could swipe along the slides. He says this idea reminds him of Iron Man movies.

There are also more common tasks for which he would want to use gesture control. For example, skipping songs when you are not on the computer. Perhaps you are playing music with your computer on your desk, but you are all set up in your bed studying. Being able to manipulate the computer from far away is the main point that he likes about this idea.

He thinks it would be interesting if this could be used for art projects. If gestures could control a design program, and the hand could be used to trace or paint something. He would also find it useful if there could be a gesture for each website that he generally visits.

To conduct the interviews we asked interviewees first to describe themselves, their daily routines, and their interactions with computers. Based on those questions or how they were sought out, we then asked more specifically about a task that was specific to their target group. We asked them about the task, and if they felt gestures would improve the task. We observed people in various environments: over the phone, before an eating club party, and in Frist.

Common themes we noted we people excited by the idea of a gesture device and immediately had an idea or two of where it could be useful. There was a lot of talk about using it to aid home media, for computer actions that get tedious, and for video games. People often did not understand the limitations of the device, and sometimes proposed suggestions such as moving workout equipment with a gesture or controlling sports balls that were not feasible with a simple computer program. We noticed that heavy computer users seemed to have more ideas for its application, and the housewife and DJ were more concerned with it for their main objective. This is probably since computers are a significant part of heavy computer user’s lives, so they had already been annoyed with some of their shortcomings. We really liked some of the ideas Steve had, including an aid for carpal tunnel syndrome and gestures for opening and switching between common programs.

2. Task Analysis

1. The main users for our gesture bracelet fall into specific groups for specific tasks and a more general range for people seeking to add gesture functionality to computer programs by their choice. Our target groups are housewives, heavy computer users, DJ’s, amputees, and computer users that rely on their computers for home media.

2. Housewives currently cook while following online recipes. Heavy computer users are always interacting with their computers. DJ’s use turntables to play music at parties and amputees use computers for everyday functions.

3. Housewives would like to flip through their online recipes without getting their computer dirty, For general use, computer users would like to add gestures to programs where they make the action more natural, or enable them to interact with a computer at a distance. These interactions include giving a slideshow presentation or watching a movie with your computer hooked up to the TV. TV controls could include changing the volume, pausing, playing etc. without getting up from the couch. Heavy computer users would like to reduce repetitive mouse clicks to relieve carpal tunnel syndrome. Finally, amputees would like a better computing experience and the ability to type emails, browse the web, chat, and more faster.

4. Users set the tasks themselves, and use buttons and a computer program to pair them. The user turns the bracelet to a “make gesture” setting and performs the gesture with the bracelet on their wrist. To exit the “make gesture” mode, the user will either perform another gesture or wait a few seconds. The user then uses the computer program to listen to the following keyboard inputs and perform the input sequence. Both of these are repeated, and the user can then pair gestures with computer shortcuts. For example, the housewife could move her wrist up and down, record the gesture, then with her web browser and gesture program open she would press listen and then scroll down in the browser. Last, she would go to a gesture list and drag the up and down movement to the scroll command. Now when she wants to scroll while she is cooking, all she has to do is shake the bracelet, moves her hand up and down, and shake the bracelet again. Without having to move or touch anything, the page will have scrolled down.

5. Wherever the person happens to be – in the kitchen, bedroom, living room, etc. The bracelet frees the user from the need to be physically close to the computer. All they have to do is stay within range of Bluetooth / WiFi.

6. Because users set the gestures to fit their specific needs, the only data we need is the shape and form of their hand and arm gestures. An important problem we need to solve is how to recognize these gestures based on sensor data from the bracelet. Thus, we will need to monitor their arm / hand movements and compare them to a history or previous such movements in some sort of algorithm.
The only time users will have to interact directly with their data is when they have to initiate set up and record the hand gestures. After that, the system should operate smoothly, with minimal input from the user.

7. Within our system, users have the bracelet and attached computer program to work with. Outside of our system, people can use traditional methods to interact with their computers, such as a mouse and keyboard (like with pc gamers). Media center enthusiasts often have a complicated remote or a set of remotes to control their system. Office workers giving presentations can use special clickers or simply the arrow keys on their computer. Amputees, especially those with part or all of their arm, could probably still navigate a computer. However, their experience will be slow and difficult, because almost all input devices (mouse, keyboard, touchpad, joystick) are designed for skilled and continual hand use. Finally, people trying to look at recipes while cooking don’t really have a good alternative tool. The next best thing would probably be voice activated commands, which are notoriously unreliable.

8. Users could chat in an online forum. There, they could help each other or swap ideas for cool applications.

9. For people within a specific usage group, Brisq will be used almost continuously for a short period of time. For example, while cooking a meal you will be flipping to the next instruction fairly often. Amputees and computer gamers will probably be the highest frequency users, continually clicking on a link or shooting the next alien. Casual users in engage the system a lot less often but over a much wider time scale. While watching a movie, you may change the volume only once or twice. But, being able to do that with the wave or your hand will save a lot of effort.

10. For any task, the bracelet will have to respond accurately and with no noticeable delay in order to avoid annoying the customer. While cooking, you usually have some leeway but there are times when 20 seconds is the difference between browned and burnt. Slideshow presentations are a prime example of need for accurate, reliable technology. Beforehand, you don’t want to keep your audience waiting while you set up. During the presentation, a few misinterpreted clicks can completely disrupt your flow and distract audience members. For amputees and pc gamers, speed is absolutely essential. A lot of games are reaction based, where you have to outdraw your opponents. In our software’s gesture recognition algorithm, we will have to make tradeoffs between accuracy and speed. Because of the different needs, it might make sense to allow users to choose how much of one they want, at the expense of the other.

11. When things go wrong, either a false positive or a false negative is triggered. A false positive is when the user engages in an arbitrary hand-arm motion that is incorrectly interpreted as a gesture. Thus, things would happen for seemingly no reason. We try to counteract this by creating a “make gesture” mode that you would have to enter by shaking or twisting the bracelet. This is hopefully a unique gesture that would not happen normally. A false negative is when you attempt to perform a gesture but it is not recognized a part of your library of gestures. Or the wrong gesture would be recognized and trigger the wrong command. This also needs to be avoided, because it will lead to user frustration and eventually abandonment of our product. We will fight this threat by improving our gesture recognition algorithm and by continuing to train it with every new gesture formed.

3. Three Tasks
1. People in the kitchen
People listen to music and use online recipes on their computer while cooking. They will wear the bracelet on their wrist, engage it for a gesture and gesture to scroll down or change the song, all without having to clean their hands. This is an easier task since there is only one gesture to recognize, so there will be a practically zero chance of misinterpreting the gesture.

2. Couch Control
User will change volume, play, pause, and change the volume on a computer movie which is hooked up to a tv, all without leaving the couch. We will need around 4-6 gestures to recognize here, for increasing and decreasing volume, playing and pausing, etc. This will make the task moderately difficulty, but 4-6 gestures should have a high success rate (i.e. over 95% based on research papers).

3. Giving Presentations
Users can use the bracelet during presentation for a variety of tasks. Swipe left and right to change slides, twist to switch programs and start playing a short video that you wanted to include. twist the other way to pause it and return to your slideshow, and more. This task could range from a simple 1 or 2 gesture interaction (changing slides) to potentially something much more diverse and complex (maybe even 8 gestures) for a long and interactive presentation.

Interface Design:

Brisq aims to make common secondary computer tasks simple and streamlined. People listen to music while cleaning the house, look up recipes on an on-screen cookbook while cooking, stream movies from laptops to their TVs (because who still uses DVDs?), and integrate their computers into their lives in all sorts of ways. But no one wants to have to dry off their hands just to change the volume on their computer, no one wants to wash their hands just to flip the page of their e-cookbook. Our users will be anyone and everyone who regularly uses their computers to complement their day to day lives. Enable Bluetooth on your computer, and use our program to easily map a sequence of keyboard presses or mouse clicks to one of 8 pre-programmed gestures that brisq can recognize; then put the brisq bracelet on your wrist and go! The small, stylish bracelet will communicate with your computer whenever it is in Bluetooth range. Shake brisq to turn it on, then swipe right to skip a song while you’re barbecuing; twist to crank up the bass at your party when theres a dancing crowd around your speakers; change the volume up on your movie without leaving the couch, or even looking for the remote. Simplify your life. Be brisq.

Users can program gestures and map them to computer inputs. They will do a predefined gesture, and brisq will send the signal captured from an accelerometer to a computer via bluetooth. On the accompanying GUI, users can enter and save keyboard and mouse inputs for specific computer programs, and match them up with gestures. Later, they just shake brisq (or press the small on/off button), then do a gesture to have your computer execute the command saved for the open program. Users can thus substitute simple and intuitive movements for keyboard and mouse inputs that may be long, repetitive, inconvenient, and more. Existing “smart” bracelets monitor heart rate, act as pedometers, and perform other simple functions but do not serve as a universal, customizable interface you your computer. Additionally, we are not aware of programs to set shortcuts by “listening” to keyboard and mouse inputs and saving them in a bank; we believe this we make the act of setting shortcuts very simple and intuitive for everyone.

Andrew Ferg
Bereket Abraham
Lauren Berdick
Ryan Soussan

Group 24

Before we tell you about our amazing resistor piano, we just wanted to show you this awesome picture of the acceleration picked up by the accelerometer in Part 5.

Data picked up from the accelerometer

Three ideas:
– Make a drum out of the piezo. The piezo readings can be run through processing to play a tune/melody each time the piezo is hit.

IMG_0433

– a therimen made from an accelerometer. As you rotate the accelerometer through an angle theta, the pitch changes. As you rotate through a different angle, the tempo changes.

IMG_0221

– A piano made out of a slimpot. As you press different areas of the slimpot, the buzzer plays different notes.

Piano Slider

Final result:

In the end, we chose the piano. However, for the final design we ditched the buzzer and played the notes through the computer. Using the serial port, we sent the notes to a processing script. Using the SoundCipher library, we were able to play computer generated notes and simulate the piano keys visually. We thought it was a success. The graphics synced well with the hardware and it sounded realistic. We did have some issues with the “debouncing” of the keys — if you held a key down it played the note twice. But it was not that noticeable.

IMG_2054

Parts used in final system

Arduino
Wires
Slimpot
Resistors

Instructions to recreate

A slimpot has variable resistance when the user presses different areas of the slimpot. Seeing this, we decided to segment the resistances, so each resistance was assigned an interval. Each interval corresponded to a different piano key. We sent this piano key number to Processing over the serial port. We then downloaded Soundcipher library off the internet which plays notes given a frequency. This allowed us to play the keys. We used rectangles to visualize the keys of the keyboard on the screen (using Processing for the graphics). The keys dimmed in colour when they were pressed. The 32bit Processing had to be downloaded in order to use the serial port.

Source code

Arduino Code:

// these constants won't change:
const int pin1 = A0; // the piezo is connected to analog pin 0

// Voltage thresholds of the piano keys
// 15,20,33,50, 130, 200,
// these variables will change:
int sensorReading = 0;      // variable to store the value read from the sensor pin

void setup() {
 Serial.begin(9600);       // use the serial port
 
}

void loop() {
  sensorReading = analogRead(pin1);    
  
  if (sensorReading > 200) Serial.write((uint8_t)0);
  else if (sensorReading > 130) Serial.write((uint8_t)1);
  else if (sensorReading > 50) Serial.write((uint8_t)2);
  else if (sensorReading > 33) Serial.write((uint8_t)3);
  else if (sensorReading > 20) Serial.write((uint8_t)4);
  else if (sensorReading > 15) Serial.write((uint8_t)5);
  else if (sensorReading > 2) Serial.write((uint8_t)6);
  else;
  
  delay(100);  // delay to avoid overloading the serial port buffer
}

Processing Code:

import arb.soundcipher.*;

SoundCipher sc;
int keysNoteMap[] = {59, 60, 63, 64, 65, 67, 69};
import processing.serial.*;
Serial myPort;  // The serial port

void setup(){
  sc = new SoundCipher(this);
  //keysNoteMap = new int[7];
  
  //keysNoteMap[7] = {59, 60, 63, 64, 65, 67, 69};
  size (250,150);
  println(Serial.list());
  myPort = new Serial(this, Serial.list()[0], 9600);
  println("starting....");
}

// keep processing 'alive'
void draw() {
  boolean pressed = false;
  int note = 9;
  while (myPort.available() > 0) {
    note = myPort.read();
    pressed = true;
    println(note);
    sc.playNote(keysNoteMap[note], 100, 1);
  }
  
//white tut 
  fill(255);
  if( pressed && note==6){
    fill(204);
  }
  rect (10, 10, 30, 100);
    
  fill(255);
    if( pressed && note==5){
    fill(204);
  }
  rect (40, 10, 30, 100);
    fill(255);
    if( pressed && note==4){
    fill(204);
  }
  rect (70, 10, 30, 100);
    fill(255);
    if( pressed && note==3){
    fill(204);
  }
  rect (100, 10, 30, 100);
    fill(255);
    if( pressed && note==2){
    fill(204);
  }
  
  rect (130, 10, 30, 100);
    fill(255);
    if( pressed && note==1){
    fill(204);
  }
  
  rect (160, 10, 30, 100);
    fill(255);
    if( pressed && note==0){
    fill(204);
  }
  
  rect (190, 10, 30, 100);
  
  //black tut
  fill(0);
  rect (32,10,15,60);
  fill(0);
  rect (62,10,15,60);
  fill(0);
  rect (122,10,15,60);
  fill(0);
  rect (152,10,15,60);
  fill(0);
  rect (182,10,15,60);
  
  if(pressed){
    sc.playNote(keysNoteMap[note], 100, 4.0);
    delay(100); 
  }
  pressed = false;
}

Final Project Ideas

Bereket Abraham
Andrew Ferg
Ryan Soussan
Lauren Berdick

Part I
Final Project Ideas

1. Smart food bin:
Problem: forgetting / not knowing how much food is left in your fridge
Possible solution: special bins for certain common foods that get used a lot. Use a (pressure?) sensor to detect how much is left. A computer will then send you alerts if you are running low. Or you can look up recipes based on what you have left.

2. Automatic bartender:
Problem: the amount of liquid poured in each cup can vary a lot. We desire a system to pour standard, repeatable amounts into each cup.
Possible solution: A small reservoir tank and an arduino controlling a valve. Similar to a water cooler.
Possible solution: Same as above but now the cups travel on a convenient conveyer belt.

3. E-composer:
Problem: Writing music down on a score is annoying / difficult.
Possible Solution: create a computer program where you play an instrument and the computer transcribes the music.

4. E-composer II:
Problem: We want something to transcribe a music file, and break it into individual instrument parts.
Possible Solution: Some computer programs take in a music file and convert it into a score with notes. We just need to separate the individual instruments.

5. Smart light switch:
Problem: Hard to remember to turn off the light and leaving them on wastes electricity. It’s also annoying to turn off the lights from bed.
Possible Solution: Programmable light switch.
Possible solution: monitor a person’s light switch and then use machine learning to look for patterns. Similar to an implementation used for eco-friendly thermostats.

6. Indoor Directions:
Problem: It’s hard to make good directions for buildings and wall maps are annoying to use.
Possible Solution: Have a central screen where you input your destination. The lights in the floor will guide you to your destination. Your movements are tracked with (pressure/vibration?) sensors. Or just light up the entire path.

7. Shower Notes:
Problem: Can’t take notes during a shower.
Possible Solution: Create a waterproof iPad case. Create a way to use the touchscreen across the plastic casing. Indirect contact.

8. Networked Alarm Clock:
Problem: Several time keeping devices, including your online calendar, alarm clock, and cell phone.
Possible Solution: Program that interfaces mostly with google calendar. You can sleep until a set time or until your next class. Sets alarms on the clock and text/calls you on your phone.

9. Printed To Do List:
Problem: There are many good productivity tools online. How do they convert to the real world?
Possible Solution: Create printed post-it notes for each task or set of tasks.

10. Laundry folder
Problem: folding laundry is a pain.
Possible Solution: robot.

11. Breadboard Simulation / Avatar
Problem: Making circuits is hard.
Possible Solution: When using a breadboard, your parts are simulated on a computer. Computer can tell you stats on every wire/part.

12. Better oscilloscope
Problem: hard to use. Needs better interface.

13. Traveling Salesman
Problem: When you have a lot of destinations/errands, it is hard to plan the optimal route / order of events.
Possible Solution: event planner that takes into account opening/closing times, map directions, relative proximity, etc.

14. Digitizing notes
Problem: Lots of old notebooks are thrown away to save space.
Possible Solution: save your notes with some kind of automatic scanner. Enable searching.

15. Save energy
Problem: Several types of exercise can also produce energy.
Possible Solution: Attach a generator to a exercise bike, treadmill,

16. Dry shoes / socks
Problem: Canvas shoes and loafers get wet very easily. Uncomfortable.
Possible Solution: chemical treatment.
Possible Solution: Some kind of quick drying system.

17. Automatic tailor
Problem: Hard to find right size for online clothes shopping.
Possible Solution: special shirt / pants that finds your size.

18. Outlet finder
Problem: Hard to find a plug.
Possible Solution: detect radiation from AC current.

19. Bathroom finder.
Problem: Can never find the bathroom when you need it.
Possible Solution: App looks up the building plans and finds the nearest bathroom.
Possible Solution: tag bathroom with homing signal.

20. Lost Keys
Problem: Always losing your keys and other small objects.
Possible Solution: Close range tracker

21. Hologram
Problem: We don’t have holograms.
Possible Solution: use sound to set up standing waves. Trap gas in the nodes and shine a laser through it. Thus you get pixels.

22. Singing trainer
Problem: Hard to sing / embarrassed to practice in front of other people.
Possible Solution: program with feedback, scales, and other tools.

23. Music Updater
Problem: Hard to keep up to date with all of the musicians you like.
Possible Solution: Make an app that tracks whenever one of you artists releases a new song or album on Spotify or iTunes.

24. Outdoor refrigerator
Problem: During the winter, people heat their houses then cool them again to refrigerate food. Wastes energy.
Possible Solution: Pipe in outdoor air to cool the refrigerator. Somehow connect it to the outside world without cooling the rest of the house.

25. Robot Puppy
Problem: Usual robotic pets are stupid / unfeeling.
Possible Solution: Mod a robotic dog so that it recognizes faces and/or gestures.

26. Personalized ads
Problem: A lot of ads are not relevant.
Possible Solution: look at the clothes / other of a person to determine what types of ads to display.

27. Input automation
Problem: It’s annoying to do long sequences of clicks or enters on a computer, especially if you have to do it a lot of times.
Possible Solution: Have a program that monitors a sequences of clicks and turns them into an exe. Thus, you can simply a difficult or annoying interfaces into a series of chunks.

28. Commercial muter
Problem: TV commercials are annoying.
Possible Solution: Make a device/app that mutes ads and/or plays music. Maybe also include a short game during the commercial break.

29. Read while moving
Problem: It’s hard to read or watch movies while on the treadmill, bike, erg, etc.
Possible Solution: create a program that tracks your head and moves the display to match you.
Possible Solution: create a simple version of Google Glass to read books. Miniprojector / Mini screen?

30. Color Blind
Problem: People who are color blind might accidently mismatch clothes.
Possible Solution: Create an app that checks colors to make sure they are what you think they are. Or converts them into RGB numbers or sounds.

31. Artificial Synesthesia
Problem: It would be cool to experience colors / sounds in a new way.
Possible Solution: create a program that converts paintings into sounds or music into colors in an interesting way. Also leads to a new way to interface with music / painting.

32. Regular laptop into touchscreen
Problem: Regular laptop is not a touchscreen.
Possible Solution: use an attachable sensor to detect either finger touches or a special pen on the screen, turning any normal screen into a touchscreen.

33. Auto Tie Tyer
Problem: When tying a tie, it’s hard to judge the proper length of the end ahead of time.
Possible Solution: program measures the tie and determines how much slack you should give to get the right length.

34. Noise cancelling
Problem: Sometimes you want to play an instrument late at night.
Possible Solution: Use the same tech as noise cancelling headphones. A device at the end of your instrument will exactly cancel out its noise. Only someone in its immediate vicinity can hear it.

35. LateX Math Handwriting Recognition
Problem: It’s annoying to use LaTeX – why can’t we just write math and digitize it? but computer-typeset math is very readable and modifiable. let’s build a system that lets you interact with a LaTeX document with just a hand-edited document.

36. Gesture control TV remote
Problem: this does not exist.
Solution: Wear gesture detecting device, i.e. a camera or kinect thing. Then hack a TV remote and use gestures to control a TV.

36. Gesture control bluetooth
Problem: It’s hard to easily move files.
Solution: Wear gesture detecting device, i.e. a camera or kinect thing. Then use gestures to move files between a computer and your wearable device (possibly with cloud storage).

36. Gesture control magnets
Problem: I do not have superpowers.
Solution: Wear gesture detecting device, i.e. a camera or flex sensors. Then use gestures to control electromagnets in your gloves.

37. Surveillance Orb
Problem: It is difficult to monitor an entire room by looking at multiple camera feeds.
Solution: Stitch several cameras together and display them onto an orb (panoramic view). Then use control stick or gestures to move around in the room.

38. Surveillance Drone
Problem: Need to move to a different city, and need to apartment hunt.
Solution: Attach a camera to a robot (drone). Drive around the apartment.

39. Latex Checker
Problem: There’s no way to check the math in a latex document
Solution: Send the latex math to a math solver (like Mathematica) to make sure results are correct

40. Shortcut Recommender
Problem: Do lengthy or repetitive tasks inefficiently
Solution: Monitor computer use and automatically recommend shortcuts (keyboard, external buttons) for actions

41. Software/Hardware Recommender
Problem: We don’t know what efficient technology is available
Solution: Program to monitor computer use, recommend software and hardware devices that might help the user improve efficiency or generally improve their computing experience

42. Facial Hair Planner
Problem: Don’t know what facial hair would look awesome on face
Solution: Program to display different facial hair styles on image of user

43. Foot Pedal Interactor
Problem: We don’t use our feet when we use computers
Solution: Foot Pad/ Buttons to control actions on computer and games

44. Projectable Interactor
Problem: Some actions in software, games don’t map well to the keyboard
Solution: Have image projected on desktop, sense touches on the image and respond the the actions, change image/layout for each program

45. Guitar Ear Trainer
Problem: Hard to learn to play by ear, inefficient feedback or need a second person
Solution: Program to play notes on a guitar, listen to guitarist’s response and note mistakes and accuracy

46. Resistive Electronic Pen
Problem: Electronic pens don’t have any resistance / feedback.
Solution: Make a pen with a magnet in the tip. Underneath the surface will be an electromagnet. More strength = more attraction = more surface friction. Maybe different brushes are more resistive.

47. Repelling Electronic Pen
Problem: Electronic pens don’t have any resistance / feedback.
Solution: Same thing as above, but now the electromagnet is repelling. You would also have a way of forcing the electromagnet to be directly under the pen. Now, the pen will have a buffer directly over the surface that it cannot penetrate. But, you could push down with different strengths, which controls about an inch of z direction. Would make for a really interesting click or button.

48. Rotation Cubes
Problem: Its hard to rotate in 3D modeling or AutoCAD software.
Solution: A series of networked cubes. Each cube rotates and translates selected objects. A special cube can be assigned to rotate globally.

49. Thermal Beanie/other piece of clothing
Problem: Clothes don’t always keep you at the temperature you want
Solution: Have heat pads in clothing item that adjusts heat to keep you at desired temperature.

50. Magnetic Adjuster knob
Problem: Traditional slider knobs are boring / uninteresting.
Solution: Create a floating magnet suspended between two electromagnets. You can move the magnet up and down within the field, and the electromagnets will detect this and adjust accordingly.

51. Child Tracker
Problem: Nerve racking leaving a child alone in a room
Solution: Have the child wear a specific color and track them with cameras. Make sure they don’t go to dangerous areas.

Part II

Winner: Smart Food Bins
Why This Project: Out of our 51 ideas, this one had the most direct plan to carry it out, a very obvious utility, and easily fits within the budget. We liked the magnetic gloves and synesthesia idea as well, however, they gave difficult mechanical and creative concerns, respectively. The food pads had a clear cut implementation and utility that we felt we could create during the semester.

Part III

Smart Food Bins
Target user group: Very busy adults who have to cook for a household. This would most likely benefit families, where the parents often cook for three or more people, or roommates who cook for themselves, or co-ops who cook for a group. Their hectic lives mean that keeping track of food amounts is a very burdensome chore, and the fact that there are multiple people in the household means that it’s harder to keep track of what’s available since everyone is constantly eating and changing the amounts left. Their needs mainly center around adequately feeding their families. Their wants include saving time by reducing the number of trips to the grocery store and saving money by reducing waste.

The closest group we could find to model our target group is college students eating independently in large groups. This would include members of food cooperatives and residents of Spellman that cook together.

Problem Description: People are not always aware of the amount of groceries they have in their refrigerator or food pantry, especially when they are away from home. They might forget to buy certain items at the store and not be able to make certain recipes. Also, refrigerators and pantries can get very cluttered and disorganized. Thus, people may not be able to easily figure out they have left and what they are able to cook.

Possible Solution: We would make pads for food that would stream to a website the amount of each food that is available. We would make pads out of a durable, waterproof fabric with pressure sensors inside them to detect the food’s weight. The pads would communicate wirelessly over Bluetooth to an arduino, which would then stream data over wifi to a website. Users can login to the website to check the percent of each food that they still have. Users can label each pad on the website (i.e. eggs, milk, cereal) to indicate what it stores. The pads would have buttons to set (by measuring or by manual entry) their minimum and maximum weight, so that for example if a box of cereal is being used, the minimum weight would be the empty box, and the maximum weight would be the full box. The pads could come in different shapes and sizes.

An extension on the website could be a recipe checker, that allows the user to check what they still need in order to make a certain amount of servings for a dish. Additionally, the website could recommend dishes to make based on the ingredients that the user has.