Team Colonial – L1

Team Colonial

Group Members: David Lackey, John O’Neill, Horia Radoi

Group Number: 7

Short Description

We built a physical interface (with potentiometers) to change the RGB values of a window filled with a single solid color.  RGB values can be tricky to understand.  Providing physical knobs to adjust them with immediate visual feedback allows the user to better grasp the concept of RGB values.  We feel that our project was successful and we really liked how Processing allowed us to give such quick feedback.  One of the main drawbacks of our project is that a user can only accurately control 2 potentiometers at a time, making it really difficult to affect all three RGB values simultaneously.

Sketches

RGB Interface

photo 1

This sketch involves our physical interface for adjusting the RGB values of a solid color, displayed via laptop.

Cup Temperature Indicator

photo 2

This sketch involves an LED setup mounted on the side of a cup.  A thermistor hangs just barely into the cup through a small hole in the lid.  The information from the temperature sensor is passed to the Arduino, which then powers certain LEDs based on the temperature of the inside of the cup.

Rooster

photo 3

This sketch involves a Photo Sensor attached to a window to get information about the lighting outside of the user’s room.  If the light outside reaches a certain threshold (daylight), a buzzer connected to the Arduino will go off, waking the user.

Storyboard

photo

System in Action

THE RGB PANEL (video)

Parts Used

  • Arduino, wires, USB cord, breadboard
  • 3 Rotary Potentiometers

Creation Instructions

  1. Add three rotary potentiometers to a breadboard
  2. Add power to the rotary potentiometers, connect them to ground, and connect each one to a separate analog input
  3. Read potentiometer values and convert them to RGB values
  4. Use Processing to display a window filled with a solid color

Source Code

Arduino

/* RGBKnobs */

// pins for knobs
int rPin = 0;     
int gPin = 1;
int bPin = 2;

// the analog reading from the knobs
int rReading; 
int gReading;
int bReading;

double knobMax = 1023.0;

void setup(void) {

  // We'll send debugging information via the Serial monitor
  Serial.begin(9600);   

}

void loop(void) {

  rReading = 255 - (255 * (analogRead(rPin) / knobMax));  
  gReading = 255 - (255 * (analogRead(gPin) / knobMax));
  bReading = 255 - (255 * (analogRead(bPin) / knobMax));

  Serial.print(rReading);
  Serial.print(","); 
  Serial.print(gReading);
  Serial.print(",");
  Serial.print(bReading);
  Serial.println();
}

Processing

/* RGB Knobs, created by Lab Group 7 (jconeill, dlackey, hradoi) */

 import processing.serial.*;
 Serial port;
 // rgb values
 int[] vals = {0,0,0};

 void setup() {
   size(555, 555);

   println("Available serial ports:");
   println(Serial.list());

   port = new Serial(this, Serial.list()[4], 9600);  
   port.bufferUntil('\n');
 }

 void draw() {
   background(vals[0], vals[1], vals[2]);
 }

 void serialEvent(Serial port) {
   String in = port.readStringUntil('\n');
   in = trim(in);

   if (in != null)
     vals = int(split(in, ","));
 }

Lab 1 – Intuitive Computer Navigation

Group Members: Abbi Ward, Dillon Reisman, Prerna Ramachandra, Sam Payne

Group Number: 9

Ideas and Sketches:

1. Etch-A-Sketch – a version of the famous Etch-A-Sketch game using 2 rotational potentiometers

EtchASketch

2. Intruder Alert – An alert if someone opens the door to your room while you’re asleep using photocells and a potentiometer

CreeperInTheRoom

3. Intuitive Control for the Computer – Use of finger flicking movement to quickly minimize windows on the computer.

FlexFinger

We decided to select Idea number 3, since it seemed the most intuitive and feasibly implementable for the lab.

Project Description:

For L1, we built a glove that allows the user to navigate the Internet more naturally. For this implementation we sought to enable a user to do a simple “flick” gesture to minimize the browser quickly for privacy. Currently, the process of minimizing all windows at best requires the user to enter a key macro on their keyboard. The problem with this is that when surfing the Internet the user is not always engaged with the keyboard. This glove uses a flex sensor on the user’s non-dominant index finger so the user does not have to inefficiently engage the keyboard in the event that they have to close their browser quickly. The difficulty with this project was that the Arduino Uno does not have a native API to override the keyboard. To accomplish the task of associating a flex sensor with the keyboard macro to minimize all windows, we used Processing and had the signal indicating a flick written to a file that would be simultaneously read from by a Java module. Java’s “Robot” class allowed us to trigger the keyboard macro we needed to minimize all windows. We are very pleased with the final result- our Java module could be adapted to implement even more functionality by simply triggering different keyboard macros given different signals from Processing. In the future, however, we would like to be able to find a more efficient system to get the Arduino to interact with the computer’s hardware than this write/read process that is happening between Processing/Java.

Project Storyboard

Frame 1

storyboardf1

Frame 2

storyboardf2

Frame 3

photo

Frame 4

photo (1)

Project Schematic

Photo Feb 23, 2 48 58 PM

Demo Video

List of Parts Used

  • Software
    • Arduino
    • Processing + our keyeventsprogram program
    • Java + our KeyboardInteractor.java
  • Hardware
    • flex sensor
    • Arduino
    • 10 kOhm pull up resistor
  • Other
    • glove (to hold the sensor)

Instructions to Recreate Design

  1. Follow the given schematic.
  2. On the Arduino, upload FirmataStandard (in the Example Software, under Firmata) to enable the Arduino to work with Processing.
  3. Go to https://github.com/pardo-bsso/processing-arduino to get an arduino-processing interface library with fixed bugs.
  4. Open our Processing code(keyeventsprogram), Java code (KeyboardInteractor.java) and a command line. You will need to adjust the pathnames stored in the files for your own computer.
  5. With the circuit all hooked up, start the Processing program. A small gray box should pop up.
  6. At the command line, type “tail -f desktop.txt | java KeyboardInteractor”. This takes the output of the Processing program (that is stored in desktop.txt) and pipes it to our Java program. Now, when you interact with the sensor (i.e. by flicking your finger), the shortcut should happen (i.e. windows+d takes you to your desktop)

Source Code

Code for Arduino

/**
* Names: Sam Payne, Abbi Ward, Dillon Reisman, Prerna Ramachandra
* Dates: 2/22/13
* COS 436, L1
*
* NOTE: since Robot objects are not allowed due to graphical
* interference, this program interfaces with an external application
* to trigger desktop results
*/
import processing.serial.*;
import cc.arduino.*;
Arduino arduino;
int desktopPin = 0; // pin reads from flex sensor
int desktopReading; // reading from flex sensor pins
PrintWriter output; // write to file when triggered
//Paramters for sensor reading
int FLICKTOP = 280; //threshold to trigger flick motion
int desktopCounter = 0; //number of times which flick has triggered
//setup
void setup()
{ 
 arduino = new Arduino(this, Arduino.list()[0], 57600);
 arduino.pinMode(ledPin, Arduino.OUTPUT);
 //create pipe file for Robot helper to read
 output = createWriter("C:/Users/Abbi/Programming/hcilab/src/desktop.txt"); 
}
void draw()
{
 // for this prototype only perform the shortcut once
 if (desktopCounter == 0) {
 desktopReading = arduino.analogRead(desktopPin);
 //output.println(desktopReading);
 }
// if above the threshold, then go do desktop
 if (desktopReading > FLICKTOP) {
 desktopCounter++;
 desktopReading = 200;
 //Send command to buffer file for robot object to interpret
 output.println("D"); 
 output.flush();
 }

 //exit the program after 5 triggers, this is a prototype function
 if (desktopCounter > 5) {
 output.flush();
 output.close();
 exit(); 
 }
 //arduino.digitalWrite(ledPin, Arduino.HIGH);

 delay(20); // delay 20 ms
}

Java Code with Robot class as workaround for using a different Arduino

/**
* Names: Sam Payne, Abbi Ward, Dillon Reisman, Prerna Ramachandra
* Dates: 2/22/13
* COS 436, L1
* Keyboard interacting program to interface with Arduino
* since robot class is not allowed due to graphics interference
*/
import java.lang.Object;
import java.awt.Robot;
import java.awt.AWTException;
public class KeyboardInteractor {
 public static void main(String[] args) {
 Robot rob; 
 //In in = new In("C:/Users/Abbi/Programming/hcilab/src/desktop.txt");
 try {
 rob = new Robot();

 //character mappings in java
 int VK_D = 68;
 int VK_WINDOWS = 524;
 int VK_BACK_SPACE = 8;

 char c;
 String line;
 while(true) {
 //c = StdIn.readChar();
 line = StdIn.readLine();
 if (line != null) StdOut.println(line);
 if((line != null) && (line.equals("D"))) {
 //StdOut.println("Desktop Command Detected");
 //trigger keypresses
 rob.keyPress(VK_WINDOWS);
 rob.keyPress(VK_D);
 rob.keyRelease(VK_D);
 rob.keyRelease(VK_WINDOWS);
 }

 //example for additional functionality
 //with additional sensors and triggers
 if((line != null) && (line.equals("B"))) {
 rob.keyPress(VK_BACK_SPACE);
 rob.keyRelease(VK_BACK_SPACE);
 }

 }
 } catch (AWTException e) {
 e.printStackTrace();
 }

 }

}

 

Team Colonial – P1

Team Colonial:

Dave Lackey (dlackey@)

John O’Neill (jconeill@)

Horia Radoi (hradoi@)

Brainstorming ideas:

  1. (Concentration – sketched) Device that looks at what you’re browsing and shocks you / punches you when you procrastinate. Sketch.
  2. (Education) Interactive game that allows you to control things virtually that would otherwise be too dangerous (nuclear waste or dangerous chemicals).
  3. (Exercise) A game involving lasers that force you to exercise by performing quick foot movements in order to win.
  4. (Health) Laptop stand that raises up after a certain amount of time so that you have to stand (relieves back pain). Sketch.
  5. (Leisure) Beverage launcher controlled by voice.
  6. (Music) Virtual DJ board controlled by hand gestures.
  7. (Navigation – sketched) A device that taps your shoulder when you should turn right or left (vehicle or on-foot navigation).  Could possibly integrate with voice navigation. Sketch.
  8. (Other) Coding through gestures and voice (in addition to keyboard).
  9. (Other) Device that helps you wake up by giving you a challenging set of physical and virtual tasks to turn off its alarm mechanism.
  10. (Other) Pads under your sheets that slowly vibrate to wake you up. Sketch.
  11. (Robotics – sketched) A robot who crawls across a chalkboard to clean it when activated. Sketch.
  12. (Accessibility) Heads Up Display – Attached to your glasses and connected to your phone through bluetooth, displays information about callers, number of unread texts and/or emails. Sketch.
  13. (Health) Barcode reader for fridge – lets you know when your food is going bad.
  14. (Accessibility) Round touchscreen phone, that can be worn on your forearm.
  15. (Entertainment) A walking FURBY® toy, that can follow you around when it is hungry.
  16. (Accessibility) Local GPS/Wifi signal generators that can be attached to important devices that are easily misplaced(Phone, Keys, Prox, Backpack) – in order to find them easier.
  17. (Accessibility) Turning any display into a touchscreen by using a glove with an emitter attached to the fingers, which transmits a signal whenever it sense pressure (touching a surface), and 4 receivers placed on the corners of the display, which pinpoint the positon of the fingers.
  18. (Accessibility) Number 16 for generating 3D points by pressing a button and drawing in mid air (would require computer to see generated image/series of points)
  19. (Lifestyle) Travel mug with a heater incorporated, that heats your beverage whenever it gets too cold. Sketch.
  20. (Lifestyle) Backpack with a solar panel incorporated, which can recharge you phone or ipod.
  21. (Autonomous Vehicle) Cars that park themselves in a designated parking lot.
  22. (Autonomous Vehicle) On-campus golf cart taxi service for injured athletes – fleet of self driven golf carts that pick up and drop off injured students in an efficient way.
  23. (Lifestyle) Breathalyzer car start – you can’t start your car unless your BAC is in normal limits.
  24. (Lifestyle) Roomba/Helicopter that brings you a glass of water when you are in bed (autonomous or pre-programmed).
  25. (Lifestyle) Polarized TV programmes – two different programs running on the same device, but on different polarizing streams – you need a pair of glasses to watch your own show.
  26. (Lifestyle) Equipment that prepares a set breakfast when you wake up (start when you hit the alarm button on the alarm clock)
  27. (Lifestyle) Shampoo estimator – dispense enough shampoo based on the length of your hair
  28. (Lifestyle) Automated TShirt folder – give it crumpled Tshirts and it outputs them neatly folded.
  29. (Education) Piano assistant – load a music file and it teaches you how to play it on the piano, by lighting up the key you need to press.
  30. (Education) Number 29 for guitar – because guitars are cool.
  31. (Lifestyle) Device that changes the song on your ipod based on the intensity of your workout (mostly for running) or based on blood pressure. Sketch.
  32. (Lifestyle) Device that detects a song you dislike on the radio and replaces it with a song from your ipod (also serves as ipod charger)
  33. (Lifestyle) Glasses that light up if alcohol is detected in a drink
  34. (Entertainment) Racing game controllable with your steering wheel and car levers, for when you have to wait in the car for long times.
  35. (Lifestyle) Radar that detects how far you are from the car in front of you/ Alerts you when it has moved far enough – for stop-and-go traffic
  36. (Lifestyle) Camera incorporated in your eyeglasses, that can take an exact picture of what you can see.
  37. (Accessibility) Automatic page turner when eyes reach last element on page
  38. (Heath) Sensors improper posture, delivers some feedback – vibrations where incorrect.
  39. (Health) Use kinect to train user into proper lifting / workout form.
  40. (Health) Use kinect to detect if people are stalking you / behind you.
  41. (Health) Toothbrush that tells you if you’ve missed any spots
  42. (Lifestyle) Helping blind people shop for clothes, or just choose clothes in the morning
  43. (Lifestyle) Glasses with facial recognition so you never forget someone’s name, or perhaps those who have trouble recognizing faces, such as those with helps those with Asperger’s
  44. (Entertainment) Glasses that read the depth of objects in front of you, and recreates relative depth on a surface that can be felt (like a pinpression) — this way, a blind person could “feel” the objects and space in front of them (e.g. the way performers move left/right and forward/back on a stage)
  45. (Music) Using nerf guns as musical inputs (place in space alters note)
  46. (Lifestyle) Using myoelectric sensors to help amputees control machines (e.g. something that controls a paintbrush or prepares dinner)
  47. (Music) Rockband using flex sensors – someone plays air guitar or air drums, and the sensors on their limbs translate motion into a sound
  48. (Lifestyle) Travel mug with an LED that displays how hot a beverage is.
  49. (Lifestyle) Stereo cameras for glasses that can approximate distances.
  50. (Kinect) TV remote control using gestures.
  51. (Kinect) Martial arts/Krav Maga form/posture corrector in real time
  52. (Kinect) Light art – Use a Kinect controller to detect the position of a digital spray can, and based on its position, back project a grafitti drawing generated by the user. Sketch.

Idea Choice Explanation

For our project we are going with sensors that detect improper posture.  People who monitor their health and have seating posture issues could benefit from this.  A possible interface could be a simple set mechanism to set a desired posture, and then an alert mechanism to notify the user when they stray from the posture.  To reach this idea, the group members each picked his favorite idea and polled third-parties (students outside of the class) about their favorites.s

Check out our sketch here.

Project Description

Our target group of users are people who sit in chairs, are interested in monitoring their health through the use of technology and who have issues with correct posture and/or back pain.

It’s not too difficult to find out what good seating posture is.  Most people are probably already aware of it.  However, people almost inevitably end up twisting their body frames into unhealthy positions.  If people are aware that they’re straying from good posture, we believe that it will help them to stop straying.  For example, if they are slumping in their chair the product may alert the user about their mistake or penalize them in a virtual way.  This is a product that could be positioned during long periods of sitting.  Another important thing to realize is that they’re often sitting in a chair and in front of a screen, which makes the screen a perfect medium for relaying information about their posture.

One benefit of using flex sensors is that they can easily bend to any bodily contours. A preliminary idea involved creating a chair with properly-placed force sensors, but the arrangement of sensors would have to be reconfigured for each individual using it, unlike the flex sensors, which adapt to the shape of the user.  Additionally, flex sensors can be portable, allowing us to make the product accessible during other areas of life, such as with good weight-lifting posture.

P1: Team Chewbacca

Brainstormed Ideas:

1. Posture helper—Detects and notifies you when you do not have the right posture.

2. Push-up monitor—detects motion along your body as you do push-up and ensures that your push-up form is at tip-top shape; can also add some sort of motivational game that helps improve your form

3. Built-in shoe sensor—help improve the monitoring of a runner; detects how many steps you have taken, how fast you have taken them, distance, and impact, and compares to previous runs/makes suggestions about how to avoid injury

4. Alarm clock stimulator—Uses a combination of features: tactical, sound, touch, and smell to guarantee that user wakes up

5. Sleep-movement regulator— detects lapses in the REM cycle; plays soothing sounds when movement is detected to help user sleep better

6. Classroom alarm clock— detects when a student is beginning to fall asleep and uses sensory stimulation such as vibration to keep the student awake

7. Pen that does not write, but senses pressure and shapes and identifies writing so you can write on any surface and it will compile it into a digital document.

8. Proximity reminder—signals to you what types or errands or tasks you should do when you’re within a certain area (or within proximity of something like the grocery store)

9. Office-hour scheduler—helps coordinate meetings with other people (allow them to see your location)

10. Parking-locator— Indicates where you have parked and sends you the location; also can issue an alarm that emits from the car if button is pressed (alarm button attached to key only works within small distance)

11. Item-locator—helps locate most valuable misplaced items simultaneously (cell-phone, wallet, keys, etc.)

12. Smart shower: before shower, collects data from skin contact and determines water
temperature and duration needed for skin’s dryness and dirtiness

13. Expanding on #12, during the shower, skin sensor that prevents skin from becoming too hot (proven to be unhealthy), gives you/shower feedback)

14. Small wearable sensor worn on skin that detects skin’s dryness and suggests when lotion should be applied.

15.  Better sensing techniques for a blind person—modify a cane with additional detection of distance and sound/touch feedback

16.  Breath detection—resolves the problem of bad breath; detects the chemical composition of breath and warns user whether or not they need mints or breath-refreshers

17.  Hydration detection— building off of #16, resolves the problem of bad dehydration; detects the chemical composition of breath and warns user whether or not they need to drink water

18.  Bacteria detecting utensils—prevent user from ever getting stomach-viruses by warning user whether the food they are picking up with their utensil has potentially harmful bacteria.

19. Interactive learning game (e.g. for multiplication tables) with sensory feedback such as vibration, sounds, or smells that correspond to the associations (helps user remember by using multiple senses)

20. Visual note-taking: Records what you write, detects circled terms, and creates a digital version of your notes with images of the terms you circled. Could also automatically detect keywords in your notes and create a “collage” for visual learners/quick refreshing.

21. Glasses that display notifications on the lenses, could be used for emails, schedule notifications.

22. Discreet texting device: personal projector/video camera that projects a keyboard onto the surface in front of you, detects which letters you press, and allows you to send texts discreetly during classes, meetings, etc.

23. Discreet device attached to arm or clothing that warns you when your stocks are going down with vibrations, includes a quick sell button or a more complex interface based on personal projector (see texting device).

24. An accelerometer or flex sensor for hand and arm motions that senses and remembers if you’ve done certain actions (locking doors, closing door, etc)

25. Commute to-do list — user uploads to-do list (quick calls, small articles/memos to read), and device detects traffic/stop-and-go motion of car and suggests appropriate to-do list for commute (may give warning/turn off when car is in motion)

26. Shoe/step sensor — worn inside shoe, gives information about step style, detects pronation and supination and recommends appropriate shoes/insoles

27. Smart closet — takes data about weather and suggests appropriate clothing (if complete integrated system, might move those pieces of clothing to front/middle of closet)

28. Habit-fixer — a small accelerometer device that is affixed to appropriate part of body and detects habits that you want to fix (biting nails, jiggling legs)

29. Cell phone as Mouse – attach your cell phone to your computer, and then use that as a mouse / menu options to reduce clutter on the screen, use ultrasonic information too.

30. Calorie detector – probes food items and determines the nutrition information.

31. Smart Room – changes music and TV upon entering a room according to predetermined tastes of the users who enter (using device carried by user or mobile data). If more than one user enters, it will figure out the common interests among them all.

32. Athletic-form helper – for any type of athletic activity, a certain technique is necessary; use some sensors that detect the motion of the user, and the movement can be shown through a video feed, and thus, the form of the athlete can be improved (ice skating, pitching a baseball, etc.)

33. Hand sensor for playing violin, guitar, etc, that detects hand posture and gives advice on improvement, also can download specific pieces and it can tell you which finger intervals were wrong

34. Orchestra section communication device: attaches to music stands, detects when changes are made to music of principal stand, and notifies rest of stands about type of change (bowing, etc) and location (measure number)

35. Building off of #34, Kinect-type device attached to music stand that detects when bowings within section are not coordinated.

36. Contextual music page-turner: detects the notes that are being played or sung and lights up the corresponding notes on the sheet music; mechanically flips the pages when the end of the sheet music page is detected

37. Instrument upkeep device — small device that can be run over stringed instruments to detect moisture of wood, rosin/oil buildup, and state of varnish to detect when humidifier should be used or cleaning/varnishing should be done (already analog devices fro humidity, but need more exact/holistic sensor)

38.  Urine characterization – takes input of color, chemicals, etc. and determines certain characteristics about your healthiness, and suggests improvements.

39.  Virtual mouse – acts like a mouse, you can move around empty space to control the mouse on a computer.

40.  Smart baking pan — detects which areas have been greased, so you can cover whole surface and grease enough

41. Smart beat – Creates music based on your interactions with the environment (if you are walking, takes up the beat of your steps)

42. Remote button pusher – you can have a button that will adapt to whatever device you want to click (camera so you don’t need a self-timer, turn off a light from far away, etc.)

43. Handheld scanner — scan a physical page (notes, etc), and uploads it with good design (checks alignment, spacing, etc)

44. Interactive scanner for physical books — looks like (and may act as ) clip-on reading lamp, but detects when words are underlined or notes are taken, and scans those pages for easy recap of significant passages

45. Individual temperature-sensing thermostat — has infrared sensor that detects the temperature of individuals inside the room and adjusts (if someone is exercising within the room, the heater would turn off/air conditioner would come on).  Right now thermostats only detect the overall temperature of the whole room, which adjusts very slowly to individuals’ movement or activities.

46.  Instant feedback for lecturers — projector-like device that can be mounted in front of blackboard, is given information about dimensions of lecture hall, and immediately gives feedback to instructor about whether they are writing too small for those in farthest seats.  Could also be implemented with device that measures microphone input/speaker output to tell if lecturer is speaking loud enough to be heard

47.  Light-detecting window blinds — detects brightness of sunlight and opens/closes automatically in order to create appropriate amount of light in room and prevent overheating in room (saves energy)

48.  Kinect-like device in doorway of retail stores, connects to mobile device to tell customers as they walk in what size they should look at for different articles of clothing (shirt, pants, etc), specific garments or styles that would be flattering to their body type (sizing for different stores vary greatly, would reduce time trying on different sizes, helps marketing for each store)

49.  Accompaniment-producer: Sound-detecting device that listens to the piece you are playing, identifies it, detects the speed at which you are playing it, and prepares/plays accompaniment at appropriate speed.  Works for singing (karaoke soundtrack) or instrumental pieces/ concertos (orchestral or piano accompaniment recording), and can speed up/slow down recording to match your speed.

50.  Plates/cups that detect content volume and give information to central device (can be used by restaurants to detect which diners need drink refills/are done with their courses)

51. Alternative to #50: Proximity sensors in restaurant tables that detect when devices worn/carried by waiters stop near the table for an extended period of time, gives data about which tables are being neglected/allows waiters to see which tables have already been visited by waiters who were not assigned that specific table

52. A wearable device that creates your own personal soundtrack – all of your movements are converted to sounds. For example, as you’re walking, your rate may be converted to the tempo of the song.


Selected Project:

Target Users:
Our target users are musicians, who often sheet music but have difficulties turning the pages at the appropriate times. This is particularly true for pianists, who often have people turn their pages for them during performances.  In addition, page turns often do not occur at appropriate times in the music, and are often during complex or fast passages that require a player’s full attention and do not lend themselves well to pauses.  Turning pages by yourself or using a human page-turner can introduce error into musical performances. Musicians would therefore benefit from an automated page-turning device, to eliminate the need for extra people during performances and make the page-turns more accurate and dependable.
Problem Description:
The problem is the lack of an automated way to reliably and robustly turn pages of sheet music.  This problem occurs most often for solo performers, who are currently forced to break the music’s rhythm and their concentration in order to flip these pages. Because this problem is most important for live performances, the solution must be discreet, and not disrupt the performance (quiet, be able to detect a single user’s instrument even when playing with accompaniment, and function even if performer does not follow music exactly).  Currently, there are some mechanical page-turners implemented primarily for organ players that respond to mechanical input (pushing pedal), but the process has not been completely automated.
Why the chosen platform/technology?
An Arduino-based solution would be ideal for this type of problem. Beforehand, the Arduino must have access to an online version of the sheet music, where it can read in the notes and know where the page breaks are for the given sheet music. The sound played by the musician would then be detected by an Arduino’s microphone, so that the Arduino could detect different pitches and speeds of notes played. It would then compare the given note to the sheet music stored in its online database, and identify where the user is currently playing the music. When the user reaches a note that is near the end of the current page, it will automatically turn the page for the musician.The Arduino is a good platform for these functionalities–specifically for powering the motor to mechanically flip pages, and also detecting analog inputs from sound changes.

 

 

P1

P 1

TFCS

Collin Stedman, Raymond Zhong, Farhan Abrol, Dale Markowitz

Ideas

  • NFC device that allows user to “transport” documents and websites from one computer to another.
  • Habit-acquisition system.  Tag physical objects like dumbells, medicine bottles, etc with RF tags or microcontrollers, and use them to detect and log users’ interactions with these objects.

  • Forget-me-not backpack. If it is zipped without an NFC-tagged laptop, it beeps to alert its user he/she has left something behind.

backpack

  • Portable 3D pointer/mouse which consist of a string whose tension is measured to determine its position in 3D space.

  • Interface for navigation through multiple degrees of graphs (such as WordNet) with gestures using Kinect/Leap Motion.
  • Real-time tracking of facial microexpressions using a wearable camera, to provide instant feedback on social interactions and crowds.
  • Real-time tracking of facial microexpressions using a stationary webcam, to track energy and dynamics in a room during meetings, lectures, etc.
  • Intelligent conference tables that transfers powerpoints/handouts to and from users who place their devices on it.
  • Augmented-reality phone apps that overlay location data from Facebook, Linkedin, etc on the real world, like Yelp Monocle but with open data integrations.
  • Panoramic photography application for iPhone, which allows tagging things around you and immersively recreating the place where you took the photo.
  • Pen peripheral device for iPhone, which has a pencil tip and can write on paper, but also saves digital copies of what users have written.
  • Add electrical/magnetic field sensing to an iPhone or Pebble, and use it to provide an additional sense through vibration, tension, etc.
  • iPhone peripheral multimeter attachment

  • Geo-aware reminder app: App that reminds you to do tasks when you are in the appropriate location, sensed using GPS.
  • Public computer setup that senses’ users presence using face recognition and/or NFC, and automatically loads their personal desktop from the cloud.
  • An audio looper that uses Leap/Kinect to control dynamics, pitch, etc.
  • Virtual plants and creatures that simulate the health of real things, like work, schedules, or relationships (as measured by online communication). Best if a clear interface for tracking many different things.
  • Interactive canvas that can be touched or walked on to draw stories or animated art/videos/slideshows.
  • SMS enabled door lock – users text their friends keys which enable the friends to open the lock.
  • A tangible interface for tangibly representing and manipulating object-oriented programming constructs.
  • Input/output integrations between two computers that allow them to work together seamlessly.
  • Keyboard gloves. Design a different keyboard that allows typing from memory.
  • Digital business cards that are exchanged by phone via NFC.
  • A game played on a digital surface where the physical location and orientation of game pieces upon the board causes different game behaviors, events, and outcomes.
  • Photo frames whose photos are static until users stop to look at them, at which time the photos become videos and possibly react to the observer(s).
  • 3D scanner which reads and interprets braille.

  • A petlike robot which is meant to be a dynamic presence in the homes of people living alone.
  • Wifi-detector necklace/keychain, which glows when open wi-fi is available, or for important calls.

  • Kinect program which allows users to “walk through” panoramic google maps.
  • NFC-key bracelet, which stores NFC keys like a virtual keyring would.
  • Multiuser NFC laptop lock screen which tracks recent users of the computer, retaining traces of their presence and usage of the computer. (Useful for making computer applications more discoverable, if people can see what software is used at a certain workstation.)
  • Flash drives that plug into a computer and download content from Internet livestreams, and replay them until they are exhausted and must be recharged.
  • Lights that automatically turn off when user gets into bed (detects this with pressure sensor)
  • Chandelier that displays when user receives an email, Tweet, etc with different color LEDs.

  • Persistence of Vision machine that sits on users desks, displays tweets, email titles, etc.
  • Lecture attendance taker using NFC login.
  • Headband that measures user’s concentration and vibrates to alert them when they lose concentration (for use in lecture, studying, etc).
  • Posture monitor using a stretch sensor, which vibrates to alert user when he/she is slouching.
  • Musical device that consists of streams of water (water harp, for example).  Perhaps for hearing-impaired users to visualize and create music.
  • Controllers for complex theater lighting configurations using gestures/tactile interfaces.
  • Redesigned controls for mixers and synthesizers, perhaps gesture-based using Kinect.
  • Interactive table that can be drawn on and that makes noise when users touch it, also responding to pressure of touch.
  • A set of triggers for safely receiving a delivery carried by a small quadrotor/helicopter.
  • A gestural user interface used to provide navigation instructions for a robotic reconnaissance drone before it is deployed.
  • Intelligent teapot that brews tea when it reaches the right temperature.
  • Canvases that interactively display text/poems/etc through gestural input. Think interactive museum exhibit but with text.
  • Journal that automatically logs daily temperature, pulse, location based on data taken by phone, fitbit, etc.
  • Directional sensing of wifi or cell signals – overlay where wifi or cell signals are coming from on a webcam view or Google Glass.
  • Virtual spatial interface across multiple computer screens: ability to throw virtual objects from one screen or computer to another in physical space using gestures.
  • Object oriented programming language which is embodied by physical lego-style blocks. Each block corresponds to an object with a series of functions. Plugging blocks together in different ways creates complete programs.
  • Shoes that sense and warn athletes of poor posture in real-time and historically.
  • Using a sheet as a screen. Video camera detects indents or ripples in sheet, interprets command and projects new image on sheet.
  • 3D scanner, and interface for virtually rearranging scanned objects in e.g. a room.

Our Choice

Habit Tracking for Object Interactions

We will create a variety of sensors to be attached to everyday objects like medicine boxes and exercise equipment, which detect when they are disturbed and update a habit-tracking application on your phone or computer.

We thought of many physical computing ideas, and chose the one that enabled the most interesting interaction cycle between the user and computer. Dale and Collin were interested in using NFC tags to remind themselves when they forgot to bring things with them. Raymond noticed he left things untouched that he wanted to use, like vitamins, whey, and dumbbells. We realized that it’s hard to profile what objects should be on someone at all times, but it is much easier to couple everyday interactions with objects like medicine bottles and dumbbells to a computer. What if those objects could remind us when they were left unused? This is useful; it would solve problems for anyone on a medical regimes or trying to build a new habit. It also enables some of the most interesting human-computer interactions among all the ideas we considered, since it it imbues common objects with intelligence which can go far beyond sensing disturbances, enabling a whole class of interactions between people and things around them.

 

Target Users and Context

Habit tracking serves strong needs for a few user groups. The elderly and those suffering from Alzheimer’s would benefit massively from reminders for their daily habits. For both the elderly and the infirm, daily medication schedules are vital; both overdosage and underdosage are unacceptable. Elderly individuals frequently use pill boxes labeled by day in order to remember which medicines to take on a particular day of the week. While this humble solution is simple and reliable, it does not provide active reminders and is only effective when used regularly. Even more importantly, the elderly population is provided with very few tools for remembering non-medical habits or behaviors, such as calling a family member, attending a scheduled appointment, feeding a pet, or checking the mail. This demographic is likely to have few but more significant uses for mobile or desktop computers, like calling and messaging relatives or reading news.

Another group of people who would benefit from habit tracking is athletes. In order to prepare for a marathon, be in peak shape for a championship, or simply maintain physique, athletes must be diligent in their workout routines. They need to remember to workout with a certain frequency, or want to make sure they they work out at a certain time of the day and for a certain amount of time. Most importantly of all, they need to make sure that they manage to maintain whatever changes they make to their established routines. The category of athletes extends to larger groups like students or the general population, as we are all too familiar with making resolutions to go to the gym or floss every day. Many people are interested in either acquiring new habits or reinforcing existing ones.

Problem Description

Our product solves two general problems. The first is struggling to acquire habits, which requires a solution that is reliable and easy to use. A reminder system must not fail upon losing Internet connectivity, although it could expect a phone to stay powered on, and it should not disrupt existing non-technological solutions like pill boxes. The habit tracker will be used mostly at home, or perhaps at work, by people with limited experience with technology. The system should require little maintenance, although users are not likely to be busy.

Another user segment is those who would like to acquire new habits, like flossing, taking vitamins, or exercising. These users currently use Seinfeld-style calendars where they cross out days when they have accomplished their goals, or habit tracking apps that are general-purpose (like Lift or Beeminder) or specific (like RunKeeper). Calendars are limited in how many habits they can track and do not provide reminders. Habit tracking apps require a superfluous interaction beyond the actual activity of the habit. Since these users are likely to have limited motivation during most times (but high motivation occasionally), a lower-friction interaction is beneficial for them.

Choice of Technology

Our system will use a combination of platforms to sense and track object interactions. The frontend of the app will be a smartphone or web app which lets you identify and track the habits and objects that are being tracked. It generates reminders and alerts for objects that haven’t been disturbed by the user in a set time period, and provide reminders to enforce important actions/habits the user is trying to follow.

The sensors must detect usage of the object; since the interaction varies between object, we would use a number of sensor integrations.

  • Weight-sensing pads for: pill boxes, dumbbells, laundry bags, sports equipment
  • Tilt/shake sensors for: dental Floss, hygiene items, tilt-sensors for pill boxes
  • Light sensors for: unused rooms
  • Electrical sensors for: windows, doors, lights

More than just force-sensors, these should sense time, so we can also track how long the object in question was used for. These sensors could be connected to microcontrollers which broadcast a signal. For a more polished system, we could use RF (radio frequency) tags and transmitters to reduce cost and size. The information could then be relayed to a hub or an Internet connection.

NFC-based sensors and NFC-reader phones are one other feasible implementation for the system. NFC tags are cheap and easy to tag an object with, and the data error rates are close to zero. They are also passive, requiring no power supply, which is ideal for sensors that you would attach to objects of daily use. The caveat of using an NFC based system is that the reader (phone) needs to be physically close (3-4 cm) to the tag in order to detect it and get data. This limits the kinds of information that we can get about patterns of usage of objects to only situations where you would physically touch the object with your phone to register a reading. Another implementation choice following in the same vein is using RFID-tags, active or passive as the sensors, and an RFID reader for receiving data.

P1: Runway

Group Name: CAKE

We are not a lie.

Team members:

Connie Wan (cwan), Angela Dai (adai), Kiran Vodrahalli (knv), Edward Zhang (edwardz)

Brainstorming List:

  1. 3D modelling with stereoscopic output overlapping with gestural input space for natural user interaction (perhaps a Maze game where you’re inside a cube)
  2. Enhanced music stand for rehearsal and concerts, with gestural/automatic page turning, automatically jumping to sections based on conductor’s decisions, and easy part markup
  3. Natural navigation in virtual worlds/virtual reality using gestural input.
  4. Solve the problem of limited screen real-estate by projecting a virtual desktop on every surface in the room that you can interact with using pen, hands, and gesture.
  5. 3D data exploration with visor – you put the visor on your head and you see streams of data everywhere like you’re in the matrix. just kidding, you should be able to interact with graphs with your hands, move them around, set axes differently, see different kinds of graphs for the same data, see Google maps with 3D cities, manipulate molecular models etc.
  6. Hands-free transportation for the lazy – Segway chair
  7. Make a enhanced web browser with a head-mounted display: instead of hidden tabs with no relationships, you can drag different pages around in the space surrounding you and organize them into your own mini-web (using gesture, of course).
  8. Modern laser tag – have computers know where “portals” are, have gun-like thingies that have cameras, and you “fire” into portals which come out an associated portal to hit someone
  9. Screenless navigation assistance in new environments using an audio and tactile interface – e.g. vibrating elements in clothing or gloves that warn of obstacles, audio signals that get louder or softer as you face places of interest, etc. Useful for the blind
  10. Piano Tutor – projection onto actual piano keys that can light up keys or recognize when keys are played, synchronized with page turning.
  11. living room aware computing — pointing at “TV”, turns on, knows where you are: we would use a model room and put hardware in the walls, chairs, etc. Simulate “The Living Room of 2040” or something like that
  12. Smart food storage: refrigerator: texts you if it shuts off, keeps updated shopping list, keeps track of when food goes bad, organizes meals and moves ingredients to front of fridge if you need to use them, talks to your computer to get search history– what food have you been searching/ looking at recently, it compiles a list of ingredients and sends them for you to get
  13. Smart food cooking: toaster, stove, microwave: control from your phone or computer, or via speech–it cooks it the way you want it (updated over time); alerts you if something burns, can be remotely shut off (like say you leave the house and realize you left the stove on, you could shut it off with an app, maybe with phone)
  14. Smart dishwasher: control remotely, automatically knows when to wash dishes, rotating water jets that sense dish shapes, make sure everything is hit. Could integrate it with your calendar
  15. Kitchen sink that automatically or hands-free turns on and off depending on when it’s needed, precisely filling up containers.
  16. Smart closet and smart clothes hangers: keeps everything sorted for you, automatically sets out clothing for the day based on weather (thicker clothing for cold days) and calendar (bathing suit for beach days), keeps your clothing updated online so friends can like clothes, and maybe even suggest fashionable combos for you to impress everyone! (this may sort of exist?) also could be helpful as a shopping app–that way, you can automatically know if you have a certain type of clothing so you don’t buy too much of the same type
  17. Smart mirror that allows you to virtually try on clothes (already exists?) – large screen w/ camera, can zoom in on parts, bring up clothing images, etc. update to fb/twitter if you think you’re looking pretty good today wink wink
  18. Enhanced clothing, e.g. integrated music player and cellphone with speakers and microphone in hood, control system in the sleeves (combine with those t-shirts that have displays in the front).
  19. Smart desk in schools – automatically sense who you are, directly integrated with your student account at a university. Personalized instructions in class/lab
  20. Augmented window with touch and display capabilities – use to display widgets like weather, news, etc. Automatically raises and lowers blinds
  21. Smart lamp (could double as projector — display internet pages on your desk, could be part of a desk UI (tabletop),  doubles as a disco ball for room parties), automatically turns off when you get in bed
  22. Smart gym: dumbells that keep track of your weight-lifting abilities, prompts you sarcastically (“do u evn lft”) (“friends don’t let friends skip leg day”), all the devices talk to each other, give you suggestions as to what you should do next for an ideal workout
  23. Smart bike: knows directions and your schedule, “automatic”, i.e. changes gears
  24. Smart backpack: knows what’s inside, is connected to calendar so knows your schedule, so it can alert you if you need a book/notebook that you need for a class, reminders to pick up packages, lets you know if your water bottle is running low, interact with it there’s an interface you can strap on, or on strap, plug in Android or iPhone to give backpack computing power and access to calendar stuff, pop out stuff you need immediately, organizational system that lets you put stuff in only one compartment of the backpack
  25. E-readers that track your gaze and turn pages automatically
  26. Smart bookshelf (like the one in Google NY, but better) that links to a site like shelfari or goodreads (maybe tie to online living room) and allows you to browse books that you might like; you can insert an e-reader or tablet where a book is “displayed” and it downloads that book for you to read
  27. Voice and camera on computer (like Jarvis). Framework that connects with your whole house and provides an interface to turn off the lights, make toast, open your garage, anything you might have to walk to do, you can do it from your computer/voice
  28. smart toilet cubicle, with a projected display and hands free navigation — no more magazines/books, no more phones dropping in toilet.
  29. sassy oven– tell it what to bake, it talks back (maybe it tells you that you’re getting a bit chubby, hold off on the cookies) — the point is to have even more fun with baking and cooking in general. Good for lonely grandparents who don’t have anything else to do
  30. Go-go-gadget arm – Quadrocopter that fetches faraway things for you.
  31. Sunburn warning (hat that has a timer and knows what the weather is like in your area (temp sensor, maybe some sort of UV sensor based on smart glass, maybe knows your skin color/type), and calculates when you’d be exposed to too much sun, and notifies you 10 minutes before by beeping)
  32. Device that senses if you are falling asleep in class and subtly wakes you up – in clothing, vibrates? or a device in ear that makes a small beep straight into your ear
  33. Gestural Video Editor – view video as rectangular prism, make operations such as cuts/transitions/timing control more natural than with mouse. You can stretch out a square prism to slow down the video, compress it to speed it up, color different parts with different music, slice the prism and rearrange it for scene editing, etc.
  34. Gestural Music Editor – can be either for notation/composition or for mixing/mastering. Use gesture contours to input notation with pitch, time and dynamics. Or make cuts/EQ/effects more natural with gestures instead of mouse+dialogs.
  35. Ultimate student: always helps you be working on something (uber productivity) based on how long tasks will take – brings up appropriate tasks on computer, pop out homework/problem sets from your binder, etc. (might be better as a desktop application?) — have a printer built in, new form factor for “laptop” — portable desk instead of tablet. (maybe it folds up)
  36. Advanced video conferencing – 3D images, room-sized conference room space, locational audio.
  37. Whisper/subvocalization translator (like in Ender’s Game sequels) – can understand sound without you having to vocalize, for private conversations. Have a sensor on throat or tongue that detects vibrations and translates to speech
  38. Educational AI : speech, personality: teaches math to elementary school kids through interactive games, learns what sort of games work well for specific individuals on the fly, encourages people — could be through voice, games could be gesture/voice based
  39. medical auto diagnosis: can tell what your heart rate is, cholesterol level, how “sick” you look–maybe you have the flu. tells you in advance if you’re catching a cold and tells you what to do to stop it, contacts doctor if detects something serious (would use computer vision to detect stuff, maybe other sensors as well)
  40. Device that finds and removes split ends from hair, either using visual or physical processing of hair strands
  41. Invisible mouse – instead of trackpad for laptops, just use your hand as a mouse (activated when you put your hand on the table).
  42. make specialized pen better (write on any old paper, updates stuff in the cloud (write subject on top, it knows what document, edits that for you) (Already exists)
  43. Artist’s pen that can change ink color / thickness / type / etc. with small gestures/indications
  44. 2D character rigging with a drawing-based interface or an action figure.
  45. Tactile interface – deeply integrate vibrating actuators into laptop/mouse/clothing for another output modality (like in game controllers or in phone games). E.g. use it in photo editors (indicating alpha value of pixels) or in web browsing.
  46. Forcing you to do exercise – treadmill-activated computer that only runs while you run
  47. 3D game that requires actual running to move
  48. desk that categorizes any object you put on it and lights it up in a different color and/or actually moves it to gather with other similar objects
  49. mood lamps/screens/etc. that change color/brightness/temperature/etc. with the environment’s temperature/sound/brightness/general color/etc.
  50. touchpad that can sense when finger hovers above, gives 3rd dimension special functionality (e.g. switch between stack of applications/windows/tabs)
  51. origami tutor: camera finds where the paper is on the table and projects dotted lines where folds should go directly onto the paper, plus a ghost of how the paper should be folded next
  52. actual (new or invisible) instrument embedded in clothing (e.g. invisible violin, or instrument that depends on how your limbs/body are positioned)
  53. virtual interior decorator that sees what is in a room and projects things/colors from a store’s inventory that would go with them onto walls/furniture/etc
  54. Projector that can project onto any surface and still look flat
  55. Virtual tape measure based on hands for distance or start/end points
  56. Camera/projector combo that lets you highlight portions of a book with a finger, then can display your highlights by figuring out what page you’re on
  57. Building blocks for small children with letters on them that say any word that they are stacked into
  58. Taste-tester (can be used in cooking) – tells you how much seasoning to add, or whether or not you might enjoy something based on how similar it is to known foods. Does it really taste like chicken? http://www.digikey.com/us/en/techzone/sensors/resources/articles/The-Five-Senses-of-SensorsTaste.html
  59. Roomba++ – knows when to do vacuuming, as well as mopping/sweeping. Also cleans up counters, and intelligently puts objects on the floor into appropriate places.
  60. actual shadow boxing game: your shadows fight with each other (with tactile feedback)
  61. smart socks: text with people in class without anyone noticing (use flex sensors and do a morse code style communication — could be marketed for elementary school students, teach students morse code!)
  62. completely virtual board game playing and tutoring like chess (gesture/ 3d display based)
  63. object recognition chess (camera identifies when you move a physical piece, updates internet board in real time)

Sketches

Segway Chair, Auto-flipping Music Stand, Exercise-powered Computer, Origami Teacher, Wearable electronics, Building Blocks that spell out words to toddlers, Gestural Internet Browser (see windows in 3D and move them around!)

Segway Chair, Auto-flipping Music Stand, Exercise-powered Computer, Origami Teacher, Wearable electronics, Building Blocks that spell out words to toddlers, Gestural Internet Browser (see windows in 3D and move them around!), Invisible Instrument, Go-go-gadget Arm [with Quadrocopter], Eye-tracking e-reader, Toilet Cubicle Entertainment System

Our Chosen Idea!: 

Project Description:  

The manipulation environment consists of a 3D stereoscopic monitor that makes objects “pop-out” and look like they are floating in mid air. A gesture camera (the Leap) has its input space aligned with the output space. We also use a webcam to track the primary user’s viewpoint so that users can move their heads and see different views of a scene. All in all, users are able to translate objects by pinching them and moving them directly to where they want to go, to scale objects by stretching our hands apart, to select objects by touching them, and so on.

Why we chose our idea:

For starters, the concept is super cool! We’ve all been inspired by this picture:

Tony Stark using gestures!

and since we have some pretty awesome hardware already (Leap, Stereoscopic Monitor, and a Kinect), we think it would be feasible to implement a neat prototype of a 3D gesture interface which interacts with virtual, floating 3D objects.

Also, our idea is very modular: it’s possible to create more specific subsets of this project by building more functionality on top of the core structure. Specifically, it might make sense to add speech recognition and processing later on (if we have time) to improve that naturalness of the user interface.

Furthermore, it’s just a really cool user interface that hasn’t been properly built yet! We envision that in the future, computers will move away from the screen paradigm and become more immersive.

Target User Group(s):  Artists, architects, designers, and animators who do 3D modelling, as well as scientists who manipulate 3D data, such as researchers in 3D capture who have to deal with point clouds or geneticists who manipulate and fold protein models.

Problem Description and Context: Current 3D modelling applications, such as Blender and Maya, though powerful, are unintuitive and awkward for navigation, relying on the 2D monitor and 2 degree-of-freedom mouse to represent and manipulate 3D objects. Because of these restrictions, it is very difficult to select arbitrary points in 3D space and perform rotations. By displaying objects stereoscopically, we can make them look like they are holograms, floating in the real world. Users could then interact with these objects as if they were real (as with Iron Man’s Jarvis), to make full use of our physical intuition. Although experimental gestural interfaces for 3D modelling exist, they have been shown in academic studies to be less efficient than mouse interfaces. By aligning the gestural input space with the stereoscopic output space, we make a fully natural interface for 3D modeling.

Justification of Technology Platform: The fundamental parts of our system will involve a stereoscopic display and a gesture camera. We will most likely use a standard stereoscopic monitor for output, although we are considering using a head-mounted display as well. For a gesture camera, we intend to use the Leap because of its low data latency. We also need an additional webcam to perform viewpoint tracking. These input and output technologies enable the fundamentally novel part of our system, which is colocated input and output spaces. We would also like to incorporate tactile feedback, namely a simple glove with vibrating actuators, to enhance the visual feedback provided by the 3D display (which for example vibrate when your hands intersect an object). These all add to the illusion that 3D objects are in fact “real” and thus are uniquely suited to our application of manipulating 3D objects.

Design Sketch:

A User manipulates a virtual cube in front of a screen! A Leap motion tracks the users hands, and the depth camera helps out.

A User manipulates a virtual cube in front of a screen! A Leap motion tracks the users hands, and the depth camera helps out.

 

 

Life Hackers P1

Group Members

Colleen Carroll (cecarrol@)
Gabriel Chen (gcthree@)
Prakhar Agarwal (pagarwal@)

I. Brainstorm

  1. Cell phone alarm clock extension, through maybe the speaker port or MicroUSB port, that uses muscle pulses to wake you up.
  2. Kinect based alarm clock that will ring if it doesn’t recognize that a body has moved out of the bed
  3. Alarm clock that vibrates the entire bed when it goes off (vibrations can be achieved by putting something like an eraser off center on a motor).
  4. Proximity alarm that tests for human presence using a combination of different sensors (temperature, photo, visual shape?). This can be used as a homemade security system.
  1. An AI “RCA” that complains if the noise levels in a room are too high by using volume sensors.
  2. “RCA” system that uses a combination of temperature, noise, and light sensors (rave lighting?) to sense whether a party is going on in the room and complains.
7. FootMouse

7. FootMouse

  1. *** Foot pedal based mouse so that one can use the keyboard and mouse simultaneously
  2. Facial movement and gesture recognition system that lets you control different operations in a computer interface by actions such as winking or moving your mouth in different ways.
  1. Automatically adjusting backpack straps with a force sensor to determine whether the backpack is being used and adjust to allow for best back support.
  2. Backpack straps that open up to allow a user to get arms in easily when picked up and then go back to the original setting after the backpack is on.
  1. There’s already hand-based controls for lights (snap your fingers to turn on/off the lights); what about blowing out lights in the same way you blow out candles?
  2. Control the lights in a room by covering or uncovering a photo sensor to a certain extent. This would be intuitive and easy to do!
13. MatchMirror

13. MatchMirror

  1. A mirror that checks the coordination of your outfit to your skin color, body type and other parts of the outfit.
  2. Use the kinect to help you cut your own hair. It shows the user how to cut their hair to a particular style and uses gesture recognition to correct the user as they go.
  3. Use the Kinect to recognize your face shape and recommend hairstyles that are the most flattering.
17. SignGlove

17. SignGlove

18. RepGlove

18. RepGlove

  1. *** “Secret handshake” glove that unlocks bikes, doors, laptops that can be programmed with certain gestures
  2. *** Sign language glove that recognizes what is being signed by a person wearing it in order to transcribe it onto the computer.
  3. *** Fingerless glove for lifting weights that records how many reps you do to an interface that records your workout progress and whether or not you are meeting your goals.
  4. *** A sensor on your clothes (flex, etc.) that recognizes good form in exercise moves and stretches (push ups, planks, sit ups, running, swimming)
  5. A glove that tells you which notes to play on the piano
  6. A gesture-controllable glove to control household tasks (is the stove off? find your keys?).
  7. Glove that recognizes gestures for different functions that may be useful while driving such as picking up the phone (move thumb to ear and pinky to mouth like a telephone).
  1. Webcam attached to your face that, when you point or gesture to certain parts of a book, takes notes or snapshots of active areas.
24. SafeStove

24. SafeStove

  1. A sensor on a dangerous object (stove, table saw) that detects the proximity of your finger and turns the object off when you’re too close.
  1. An application that tells you which notes you’ve played and records them to a computer program that transcribes the music
26. HydrationBottle

26. HydrationBottle

27. DetoxFlask

27. DetoxFlask

  1. Water bottle that keeps a log of how much water you intake during a day by sampling the weight of the water periodically. Lights up if at a given time, you haven’t had enough of the water.
  2. Alcohol flask that keeps a log of how much alcohol you’ve had during a night by sampling the weight of the alcohol periodically. Estimates BAC and displays on flask.
  3. A flex sensor on the ketchup bottle that lights up different amounts based on how much ketchup is in the bottle, the orientation of the bottle, and how hard you’re squeezing.
  1. Pressure sensors on your bike brakes and handlebars that light up brake and turning signals (useful for drivers trying to avoid bikers).
  2. Pressure sensors on your shoes that light up turning signals when you turn on your feet (useful for bikers trying to avoid pedestrians).
31. VolleyHelper

31. VolleyHelper

  1. Accelerometer and pressure sensor on a volleyball that detects and lights up when you should hit a volleyball when it’s falling (useful for people trying to learn when to begin an approach to serve).
  2. A device that detects your posture and helps you improve it using classical conditioning
  3. A Kinect based application that helps you improve your presentation skills by looking at both physical cues (such as posture, nervous habits) and audio cues (number of “um”, “like”, etc)
  4. Presentation tool to write or draw in the air and transfer to screen.
  1. A system that can listen to users singing (often imperfectly) and use machine learning to determine and transcribe the notes they are actually trying to sing (or at least those that would musically make sense)
  1. Using kinect, a waste bin that senses the waste item, classifies it as a particular type of trash and opens the appropriate hole for deposit
  1. Pressure-sensing belt that tells you when to stop eating.
  1. Application that recognizes human presence in a lecture hall and control the blinds to direct light towards locations where it won’t get in people’s eyes
  2. Remote controlled vehicle for which the remote is based on one’s hand movements (point forward to move forward, rotate wrist to turn, etc)
  3. A sensor that can detect the need for handicap facilities such as an automatic door by recognizing the image of a person on a wheelchair or crutches.
  1. A better tape measure – use a fixed armspan as the base, put a sensor on each hand and use specific gestures.
  2. 3d drawing like on a 2D tablet. Use a simple stylus that is easily detected and draw in the air to add the model to CAD.
  3. Kinect in the hallway to direct and measure traffic during an emergency, and safely help people exit the building.
  4. Vending machine where you point to the item that you want (using kinect).
  5. Solve a puzzle on a gift box to make the gift opening experience more interactive! For example, a Christmas gift box that has a countdown on the box, with a new puzzle every day.
  6. Stethoscope that lights up when there is a heartbeat and records and analyzes sound for listening to breathing.
  7. A multi-head screwdriver that can look at the screw you are going to use and the correct size head lights up.
  8. A tray that can measure the amount of food that you are getting to go and print a price tag. The tag can then be easily brought up to the cashier and paid for quickly.
  9. A coffee mug that tells you when your drink is cool enough to drink.
  10. Make passwords more secure by moving them from the keyboard to any chosen object where pressure sensor can sense a specific pattern to unlock the computer.
  11. Attach an arduino to a dog’s collar that the dog can be trained to interact with (barks, licks, breath, light) and open the door or turn on a water dispenser.
  12. A smart crib that can rock the crib when it senses too much noise, that alerts you when the baby does certain things, etc
  13. Mount to skateboard that gives you points based on spins, acceleration, jumps (real life Tony Hawk).
  14. Motivation for running – virtual reality chase game to keep you running while on treadmill and meet workout goal (think real life temple run).
  15. Hands-free way of interacting with your phone using kinect to detect gestures that correspond to mechanical motion on the phone’s touch screen and accelerometer.

II. Idea Choices:

Kinect-based application:
Through the brainstorming process, we tried to come up with ideas by looking at inefficiencies we saw around us and by considering problems those that we or our friends recently had. The idea we have chosen is to develop a mechanical, hands-free means to control one’s phone and we actually came up with it after discussing how one of Gabe’s friends recently dropped and broke his phone as he tried to pick up a call while on a treadmill. With our incessant dependence on mobile communication, we thought that there was definitely room for a hands free means to control a phone (outside of Siri, which is pretty terrible at recognizing a lot of commands). Our proposed solution uses a Kinect device set on the dashboard of the treadmill. Motions of the user would be used to control a mechanical mount in which the phone could be set. This mount will consist of a mechanical “finger” that can be used to touch different parts of the screen, and ideally, we will be able to move the entire mount in order to control accelerometer related functions of the phone. Through this hybrid, mechanical solution, we can translate our own full body actions to actions on the phone without actually touching the device. One could imagine waving to pick up a phone call or pretending to bob one’s head to music to start iTunes.

55. OnTheRun

55. OnTheRun

Our second best idea:
Our second idea came from realizing the limitations in existing password systems. Most are based on a string, comprised of basic, finite character sets. This puts users in the position of either memorizing a complicated password or risk being vulnerable to simple attacks. Users tend to opt for convenience and choose passwords that are susceptible even to simple, brute-force attacks. We imagine a system for “passwords” that is virtually infinite, by taking gestures as input. A glove, when hooked up to a computer, can sense the gesture of a user’s hand and a software program can memorize the sequence of gestures that the user creates and then check future inputs against it. Each gesture, or “character”, is a product of the user’s imagination and thus is much harder for a hacker to guess.

III. Project Description

Target User Group:
Our target user group for this project consists of those who, like Gabe’s friend, are very plugged into their electronic communication and media devices and want to use them while exercising. With cellphones becoming more and more advanced and commonplace, it is becoming necessary for many of us to interact with and respond to them on an immediate basis. At the same time, though, they are becoming increasingly expensive and fragile. As such, one of the needs of the target group is to ensure that their phones are safe and not in fear of falling and having a cracked screen. Young users especially, who are used to always being plugged in, also want to be able to perform simple tasks like taking calls, sending texts, and maybe changing the music playing while they are working out. A gesture based, hands free manner to interact with their devices is especially useful for these users because it is simple and would allow them perform a task without requiring the exact and careful motor skills that touching the screen does.

Problem Description and Context:
As stated earlier, we first thought of the problem when a friend of Gabe’s experienced it himself. Those who exercise, especially in the gym, often multitask while doing so, but risk getting distracted and damaging their phone or themselves. On treadmills in particular, users often need some external stimulation besides staring at the wall ahead. We want to design a solution that allows the user to interact with their phone in a safe, fun way. We believe that a major cause of difficulties with using phones on a treadmill is that phones have a relatively small interactive surface, which requires more attention to get right, especially while moving quickly. Our system would need to have simple interactions and a large interactive surface to resolve these issues. It would also need to function in some of the most commonly attempted activities while running, for example, talking on the phone and reading. The interface could then be safe enough for use while running, but also give the user the entertainment or productivity that they are looking for.

Technology platform:
We chose to use a hybrid platform of both Arduino and Kinect for our application. We felt that Kinect was the best form of detection for the touch-free interaction with a large interface and variety of simple interactions, which is key to the solution of the problem described above. Additionally, from lab, we have discovered that Arduino has sufficient means of recognizing analog input. We need to be able to process the signals detected by Kinect in some way; Arduino is an appropriate means of translating this into the mechanical interactions on a touch-screen.

Team Deep Thought

GROUP MEMBERS:

  • Neil Chatterjee (neilc@)
  • Alan Thorne (athorne@)
  • Vivian Qu (equ@)
  • Harvest Zhang (hlzhang@)

PART I:

  1. Kinect DJ system: Uses gestures and body motion from skeleton data to control music mixing.
  2. Air guitar: a flex-sensor based air guitar for Xbox games.
  3. Automated third arm: Kinect or Android-based robotic arm for assisting in building, soldering, etc. A functional helping hands tool.
  4. Kinect Haptic Vision: Vibration based on Kinect’s forward vision to help blind people see by feel from depth sensor data.
  5. Shower rave: LED filter through water streams to color the stream optically; add music and make it into a whole dance system.
  6. Posture Adjusting Shoes: using weight sensors and motors, readjust the inside bottom surface of the shoes to improve posture.
  7. Sychronized Firework Launcher: Arduino and smartphone controlled synchronized fireworks launcher for private fireworks shows
  8. NFC Information Platform: NFC controlled information booths throughout a building, accompanied by a database on an app and a map. Get close to get info
  9. Kinect Gesture Analyzer: Observe which physical tics people use while giving talks and critique.
  10. Turn Signal Clothing: Bicycle shirts that have LED turn signals for safety.
  11. Kinect room analyzer + home decorator: Kinect scans a room and then uploads a mesh of the surrounding area to alter in photoshop.
  12. Smart Clothing — Breathalyzer: measures people’s health; Drunkard’s shirt that measures pulse, temperature, and BAC and reports safety.
  13. Laser tag T-Shirts to create a more visual experience.
  14. Real-life angry birds with tracking data and projectile’s simulated with Kinect motion.
  15. Symmetric face properties: Analysis of a person’s face to reveal how symmetrical they really are.
  16. Kinect-controlled quadcopter/RV car. Kinect motions feed data to an Arduino to remotely control these electronics.
  17. 3D Kinect scanner that rotates an object on a platform to create a point cloud of the object.
  18. Reflex games that use an arduino to interpret response time.
  19. Triangulating gunshots using Raspberry Pis and sound sensors spaced throughout a city, room, building, etc.
  20. Arduino swiffer/roomba that automatically navigates the floor of a room to clean.
  21. Arduino door opener that contains a security presence to allow access for some people
  22. Nest like system that allows general control of the homeostasis of a house.
  23. Security cam from bike stealers that texts you if your bike has been stolen.
  24. Arduino oscilloscope that measures voltage and current by using variable resistors. Additionally, Processing would create a plot.
  25. Fire safety ALERT button that sends texts to everyone in a hall.
  26. Backpack dog follows you around by tracking your motion with a Kinect.
  27. Predicting lightning strikes using ozone sensors.
  28. Methane Interpolator: determine where cows are or a fart detector that uses an assembly of these in an enclosed area.
  29. Automatic garbage can: to follow drunk people around in case they can no longer handle themselves.
  30. Electric Bike Conversion optimizes power usage during a bike and can provide uphill assist.
  31. Laptop controller for when you’re lying down (hard to see / use keyboard / balance weight).
  32. Automatic page turner for books.
  33. Planetarium (lasers) that uses servos to project images onto a screen.
  34. Transparent screens (lasers) that uses servos to project lasers onto an opaque glass
  35. Keyboard pants that allow anyone to type on themselves.
  36. Giant touchscreen made out of an assembly of smaller phones.
  37. Phone tag: Mobile game that allows you to play tag with your cellphones.
  38. Bluetooth phone glove on your hand so that you can talk with your gloves.
  39. Measure foot pressure and adjust for correct walking position to improve posture.
  40. Theremin instrument simulated using the Kinect.
  41. Bluetooth backup camera to be implemented in cars for a cheap solution to alternatives.
  42. Auto-instagram shots uses Arduino to easily and quickly upload certain images to Instagram (ethernet shield).
  43. Home automation wake up system to integrate a morning routine into an electrical system
  44. Nightmare detector that monitors respiration, pulse, and temperature to isolate REM cycles and interpret the presence of a nightmare.
  45. Face 3d models with pin array uses an array of FSRs to create a point cloud of someone’s face.
  46. Hug meter (emotion tracker with Kinect) lets you know who needs a hug in a room.
  47. Lighting based on music input (visualizer)
  48. Lie detector based on voice. Arduino does FFT to isolate irregularities in the person’s voice.
  49. Underground meal swipes market uses an Arduino and a magnetic card reader to reprogram a prox.
  50. Covert t-shirts pay attention to vibrations and secret gestures to convey information without speaking.
  51. Fingerprint replication uses an Arduino and a fingerprint scanner to create a fingerprint mesh for replication.
  52. Sound detecting system that generates mood/ambiance music to rhythm of sex (or any other rhythmic activity)
  53. Using a Kinect or raspberry pis to track a speaker on stage (example application — spotlight).
  54. Using a Kinect to scan someone’s BMI, offering customized health information (ex. safe amount of alcohol one individual could imbibe)
  55. Leap motion controller virtual keyboard that allows typing on any surface.
  56. Kinect a cappella system which uses spatial data and skeleton data to allow a person to create synthesized songs by themselves (or with others!), recording multiple music tracks and overlaying them.

PART 2:

  • A Cappella Solo Live (#56): Mainstream computer-based electronic instruments and music making machines are still few and far between; there are custom setups designed for particular artists and there are non-programmable, dedicated hardware devices that accomplish much of the same functionality, but they are usually better left to recording engineers instead of performers and certainly are an obstruction while performing on stage in front of a live audience. As several of our group members are musicians and performers, this is also something we are personally quite enthusiastic about. It is also modular enough that a sizeable chunk of functionality can be implemented in this course, and further features can be integrated later.
  • Alternative: Triangulation of sounds using multiple Raspberry Pis and sound sensors (#19): Initially, the idea was developed to have real-world significant applications, like triangulating gunshots in warzones or other critical sounds throughout a city, room, building, etc. It’s really interesting because, even though the technology already exists, using raspberry pis would effectively make it the cheapest available technology of its kind. This would allow the product to be used in a wide-range of applications previously not possible!

PART 3:

  • Target audience: professional musicians, amateur musicians, “Youtube artists,” and other Kinect owners who want a live, interactive synthesizer and looper for experimentation. With an unobtrusive, floor-mounted Kinect connected to a small laptop (that can be used as a monitoring screen) in front of the user, the system can be used in concert halls as a live performance tool. Alternatively, the same system set up in a living room (perhaps connected to a home theatre PC) allows for an entertaining quasi-game, and an easy way to record and create music for publishing online. This could gain traction among the many talented Youtube artists who incorporate various technologies to create great covers and original music.

 

  • Problem description/context: Looper-synthesizers (usually used with guitars or synth tracks) exist in several forms today, including iOS apps and hardware devices, but the iOS take on loopers (VoiceJam) is very much an experimentational and maybe recording tool instead of a live performance capable setup, and hardware-based loopers are specialized equipment that present a pretty steep learning curve for those not well versed in how they work; they are also quite expensive. However, they are great fun, and we expect that many people would love to have the functionality if presented in an intuitive way that requires little specialized equipment. Thus the barrier to entry is lowered by using the Kinect + computer based approach, and the experience of interacting with the machine is less a distraction from the performance and more an integral part of the choreography itself.

 

  • Appropriate technology: The modularity and flexibility of this system means that it can be used to perform live (laptop + Kinect), practice and experiment in a living room (HTPC + TV + Kinect), or any of a variety of other configurations. Kinect is small and highly portable, making it trivially easy to set up at performance venues (especially compared to things like speakers, stage extensions, etc). Implementing all of the backend logic in software allows us to add functionality as we choose, and the Kinect is a relatively inexpensive I/O device that is also perfectly suited to the job; it can track skeleton movements as well as z-axis movement for easier and more precise gesture analysis, which we wouldn’t get from a normal camera. Finally, we can include Arduino-powered elements that also interface with the same computer running the backend software, which would allow us to add functionality that the Kinect can’t handle, such as individual finger flex movements.

Possible sketches: