Group Num­ber: 4
Group Name: Team TFCS
Group Mem­bers: Collin, Dale, Farhan, Ray­mond

Sum­mary: We are mak­ing a hard­ware plat­form which receives and tracks data from sen­sors that users attach to objects around them, and sends them noti­fi­ca­tions e.g. to build and rein­force habits and track activity.

Previous Posts:

P6: https://blogs.princeton.edu/humancomputerinterface/2013/05/06/p6-team-tfcs/
P5: https://blogs.princeton.edu/humancomputerinterface/2013/04/22/5642/
P4: https://blogs.princeton.edu/humancomputerinterface/2013/04/08/p4-team-tfcs/
P3: https://blogs.princeton.edu/humancomputerinterface/2013/03/29/p3-2/
P2: https://blogs.princeton.edu/humancomputerinterface/2013/03/11/p2-tfcs/
P1: http://blogs.princeton.edu/humancomputerinterface/2013/02/22/964/

Final Video

Changes from P6

– Added a “delete” function and prompt to remove tracked tasks
This was a usability issue that we discovered while testing the app.

– Improved the algorithm that decided when a user performed a task
The previous version had a very sensitive threshold for detecting tasks. We improved the threshold and also used a vector of multiple sensor values to decide what to use asa cutoff instead of only one sensor.

– Simplified choice of sensors to include only accelerometer and magnetometer
This was a result of the user testing which indicated that the multiple sensor choices vastly confused people. We simplified it to two straightforward choices.

– Updated text, descriptions and tutorial within the app to be more clear, based on user input from P6
– Updated each individual sensortag page to display an icon representative of the senor type, simplified the information received from the sensortag in realtime, and added a view of user’s progress in completing tasks

Goal/Design Evolution

At the beginning of the semester, our goal was to make an iPhone application that allowed users to track tasks with TI sensortags, but in a much more general way than we actually implemented. For example, we wanted users to decide which sensors on our sensortag–gyroscope, magnetometer, barometer, thermometer, etc–they would use and how, and we would simply assume that users would be able to figure out how best to use these readings to fit their needs.  This proved to be a poor assumption, because it was not obvious to nontechnical users how these sensors would be used to track tasks they cared about.

We quickly reoriented ourselves to provide not a sensortag tracking app but a *task* tracking app, where the focus of the app was in registering when users took certain actions–opening a book, taking a pill from a pillbox, going to the gym with a gym bag–rather than activated the sensors on certain sensortags. Within this framework, however, we made the model for our application more general, exposing more of how the system functions by allowing them to set up sensors for each task, rather than choose from a menu of tasks within each application. This made our system’s function easier to understand for the end user, which was reflected in our second set of interviews.

Critical Evaluation
Our work over the semester provided strong evidence that this type of HCI device is quite feasible and useful. Most of our tested users expressed an interest in an automated task-tracking application and said that they would use Taskly personally. Still, one of the biggest problems of our implementation of a sensor-based habit tracking system was the size and shape of the sensors themselves. We used a sensortag designed by TI which was large and clunky, and although we built custom enclosures to make the devices less intrusive and easier to use, they were still not “ready for production.” However, as mentioned above, this is something that could easily be fixed in more mature iterations of Taskly. One reason to believe that our system might function well in the real world is that the biggest problems we encountered–the sensortag enclosures and the lack of a fully-featured iPhone app–are things we would naturally fix if we were to continue to develop Taskly. We learned a significant amount about the Bluetooth APIs through implementing this project, as well as about the specific microcontroller we used; we expect BLE devices, now supported only by the iPhone 4S and later phones, will gain significant adoption.

The project ended up being short on time; our lack of iOS experience (initially) made it difficult to build a substantively complex system. The iPhone application, for example, does not have all of the features we showed in our early paper-prototypes. This was partly because those interfaces revealed themselves to be excessively complicated for a system that was simple on the hardware side; however, we lost configurability and certain features in the process. On the other hand, we found learning new hardware platforms (for both iOS and the SensorTag) to be something that could definitely be accomplished over the course of weeks, especially making use of previous computer science knowledge.

One final observation that was reinforced as a result of our user testing was that the market for habit-forming apps is very narrow. People were very satified with the use cases that we presented to them and their recommendations for future applications for the system very closely aligned to the tasks we believed to be applicable for Taskly. Working on this project helped us recognize the diversity of people and needs that exist for assistive HCI-type technologies like this one, and helped us gain a better idea of what kind of people would be most receptive towards systems where they interact with embedded computers.

Moving Forward

One of the things we’d most like to improve upon in later iterations of Taskly are custom sensortags. The current sensortags we use are made by Texas Instruments as a prototyping platform, but they’re rather clunky. Even though we’ve made custom enclosures for attaching these sensors to textbooks, bags, and pillboxes, they are likely still too intrusive to be useful. In a late iteration, we could create a custom sensor that uses just the bluetooth microcontroller core of the current sensortag we’re using (called the CC2541) and the relevant onboard sensors like the gyroscope, accelerometer, and magnetometer. We could fabricate our own PCB and make the entire tag slightly larger than the coin-cell battery that powers the tag. We could then 3D print a custom case that’s tiny and streamlined, so that it would be truly nonintrusive.

Beyond the sensortags, we can move forward by continue to build the Taskly iPhone application using the best APIs that Apple provides. For example, we currently notify users of overdue tasks by texting them with Twilio. We would like to eventually send them push notifications using Apple Push Notifications Services, since text messages are typically used for communication. We could also expand what information the app makes available, increasing the depth and sophistication of historical data we expose. Finally, we could make the sensortag more sophisticated in its recognition of movements like the opening of a book or pillbox by implementing Machine Learning data to interpret these motions (perhaps, for example, using Weka). This would involve a learning section where the user performs the task with the sensortag attached to the object and the system would learn what sensor information corresponds to the task being performed.

Another thing we need to implement before the system can go public is offline storage. Currently the sensor only logs data when the phone is in range of the sensortag. By accessing the firmware on the sensortag, it is possible to make it store data even when the phone is not in range and then transmit it when a device becomes available. We focused on the iOS application and interfacing to Bluetooth, because the demonstration firmware already supported sending all the data we needed and none of us knew iOS programming at the start of the project. Now that we have developed a basic application, we can start looking into optimizing microcontroller firmware specifically for our system, and implementing as much as we can on the SensorTag rather than sending all data upstream (which is more power-hungry). A final change to make would be to reverse the way Bluetooth connects the phone and sensor: currently, the phone initiates connections to the Bluetooth tag; reversing this relationship (which is possible using the Bluetooth Low Energy host API) would make the platform far more interesting, since tags would now be able to push information to the phone all the time, and not just when a phone initiates a connection.

iOS and Server Code

https://www.dropbox.com/s/3xu3q92bekn3hf5/taskly.tar.gz

Third Party Code

1. TI offers a basic iOS application for connecting to SensorTags. We used it as a launching point for our app. http://www.ti.com/tool/sensortag-ios

2. We used a jquery graphing library for visualization. http://www.highcharts.com

Demo Poster

https://www.dropbox.com/s/qspkoob9tggp6yd/Taskly_final.pdf

Dale, Colin, Farhan, Raymond – 4

We decided to build a Glove that controls the pitch of a song as it is being played. It was a real time interactive control on the music which allowed the user to create dubstep like effects on the music stream. The final system implemented the desired control well and mapped the up and down gestures to the pitch of the music. One improvement could be a more continuous mapping to the pitch, instead of the up and down motion corresponding to step shifts in the pitch, we could have a more fine control using the movements of the hand. We would also like to add gestures for controlling the wobble effect on the music (a left right twist of the wrist or such).

Instrument 1 – Scream Box (aka The Egg from Harry Potter):

The first instrument we designed was a box that starts making high pitched noise when you open it. As you try to cover the top with your hands, the pitch of the sounds goes down and to shut it off you close the lid of the box. The aim was to make a system that is a standalone musical object that has a very real-world mapping to it – you try to cover  as much of the box to make it quieter. The device uses a photo-sensor to measure how covered the top of the box is.

Instrument 2 – Touch Sensing Piano:

The aim was to recreate a piano like instrument with capacitive sensing aluminium foils as the keys. The mapping to create sound is natural, and the use of capacitive sensing makes the interaction feel natural and unobstructed.

Instrument 3 – Pitch Controlling Glove:

This instrument tries to map physical gestures to pitch control on music. The device uses an accelerometer to recognize motions of the hand and maps them to controlling the up of down pitch of a song. This can be used to recreate the “dubstep” effect on songs.

Final Instrument: Pitch Glove

We decided to refine the pitch glove because it afforded a very novel kind of interaction with the music and allowed control over a parameter of the music with gestures. The refining process was mainly dealing with mapping the accelerometer data to correct gestures and making the piping to the pitch control software work.

Parts List:

1. Arduino Uno
2. Processing Software

Instructions:

1. Download the code for the processing of the music.
2. Attach the Arduino either to your wrist using a strap, or hold it in your hand.
3. Enjoy the music move with your gestures.

Code:
Arduino Controller

const int groundpin = A2; // analog input pin 2
const int powerpin = A0; // analog input pin 0
const int xpin = A5; // x-axis of the accelerometer
const int ypin = A4; // y-axis
const int zpin= A3; // z-axis
void setup()
{
//initialize the serial communications:
Serial.begin(9600);
// Provide ground and power by using the analog inputs as normal
//digital pins. This makes it possible to directly connect the
//breakout board to the Arduino. If you use the normal 5V and
//GND pins on the Arduino, you can remove these lines.
pinMode(groundpin, OUTPUT);
pinMode(powerpin, OUTPUT);
digitalWrite(groundpin, LOW);
digitalWrite(powerpin, HIGH);
}
void loop()
{

if(analogRead(zpin) > 440)
{
  Serial.print("h");
}
else if(analogRead(zpin) < 280)
{
  Serial.print("l");
}
else
{
  Serial.print("n");
}

}

Music Controller

// Parts of this code were based on Sampling_03.pde from Sonifying Processing tutorial

import beads.*;
import processing.serial.*;
AudioContext ac;
SamplePlayer sp1;
Gain g;
Glide gainValue;
Glide rateValue;
Glide pitchValue;
Serial myPort;
float rate;
float pitch;
char serialVal;
int bufferSize = 0;

void setup()
{
  String portName = Serial.list()[0];
  myPort = new Serial(this, portName, 9600);

  size(512, 512);
  background(0); // set the background to black
  stroke(100);
  line(width/2, 0, width/2, height);
  line(3*width/4, 0, 3*width/4, height);
  line(0, height/2, width, height/2);
  text("", 100, 120);

  ac = new AudioContext();
  try {  
    sp1 = new SamplePlayer(ac, new Sample(sketchPath("") + "music.mp3"));
   }
  catch(Exception e)
  {
    println("Exception while attempting to load sample!");
    e.printStackTrace();
    exit();
  }

  sp1.setKillOnEnd(false); // we want to play the sample multiple times

  pitchValue = new Glide(ac, 1, 30);
  sp1.setPitch(pitchValue);

  rateValue = new Glide(ac, 1, 30);
  sp1.setRate(rateValue);

  gainValue = new Glide(ac, 1, 30);
  g = new Gain(ac, 1, gainValue);
  g.addInput(sp1);
  ac.out.addInput(g);

  ac.start();  
}

void draw() {

  //read value from serial port.
  if ( myPort.available() > 0) {  // If data is available,
    serialVal = myPort.readChar();         // read it and store it in val
    bufferSize++;
  }

 if(bufferSize > 10)
 {
   myPort.clear();
   bufferSize = 0;
 }
 float dRateValue;
 if(serialVal == 'h')
 {
   dRateValue = 0.01;
 }
 else if(serialVal == 'l')
 {
   dRateValue = -0.01;
 }
 else
 {
   dRateValue = 0;
 }

  println(serialVal);
  // calculate the mouse position on screen as displacement from center, with a threshold
  // TODO: feed in the accelerometer values here
  //float dRateValue = ((float)mouseX - width/2.0);
  //float dPitchValue = ((float)mouseY - height/2.0);

  //if (abs(dRateValue) < 10) dRateValue = 0;
  //if (abs(dPitchValue) < 10) dPitchValue = 0;
  /*
  if(serialVal == 10)
  {
    dPitchValue = 0;
  }*/

  // adjust the rate and pitch depending on the X and Y position of the mouse, drawing a trace
  stroke(100);
  point(rate*width/10.0 + width/2.0, pitch*height/10.0 + height/2.0);
  rate += dRateValue;
 // rate = 2*dRateValue/width;
 // pitch = dPitchValue/height/10;
  stroke(255);
  point(rate*width/10.0 + width/2.0, pitch*height/10.0 + height/2.0);

  // print and set output
  println("Rate: " + rate + "; Pitch: " + pitch);
  rateValue.setValue(4.0*rate);
  pitchValue.setValue(-2.0*pitch + 1.0);
  sp1.setPitch(pitchValue);
  sp1.setRate(rateValue);

  delay(10);
}

void mousePressed() {
  rate = 0;
  pitch = 0;
}