I. Your group number and name

Group 25 – Deep Thought

 II. First names of everyone in your group

Neil Chatterjee Harvest Zhang Vivian Qu Alan Thorne

III. 1-sentence project summary 

AcaKinect is a voice recording software using the Kinect for gesture-based control, which allows the music users to record multiple loops and beat sequences and organize them into four sections on the screen, giving the users an more efficient and intuitive music recording interface.

IV. 1 paragraph describing 3 tasks you have chosen to support in this working prototype 

The first task (easy) is to create a simple one-track loops using gestures, by raising one hand above the head. This creates a loop in one of the four sections of the recorded voice. The second task (medium) is to create multiple loops in different sections of the screen, and even multiple loops in one section of the screen using the same gesture from the first task. The user does this by repeating the first task in the same and in different sections. The tracks will all loop over each other. The third task (hard) is to complete the second task with the addition of deleting tracks, by raising one arm and pointing the other arm to the elbow and swiping outward. This is the first part towards the creation of a fully editable sequencing and recording system.

 V. 1 short paragraph discussing how your choice of tasks has changed from P3 and P4, if at all, as well as the rationale for changing or not changing the set of tasks from your earlier work.

We have scaled back the set of tasks from P3 and P4. For our paper prototype we wanted to make sure that users could do everything the software had to offer. For this prototype we wanted to ensure that our core functionality worked properly. We put off implementing a master recording functionality in order to enable track deletion. The music recording process is more expressive and customizable if the users are able to delete tracks: we wanted our prototype to reflect that. We also have had some difficulty implementing a master track and so we could not include it in our tasks for now. In our paper prototype, we also had a “settings menu” where you could adjust the overall recording volume and EQ. Since this wasn’t essential to creating a minimum working prototype, we didn’t include it since it added additional complexities we would have to manage. We hope to be able to fully implement all the functionalities in our paper prototype by our final demo.

VI. 2–3 paragraphs discussing your revised interface design

We changed the display of the beat count (for a total of 8 beat loops) from a timing bar at the bottom of the screen, to simply print out a beat count. This was easier to implement and may actually be simpler for the user to understand. We will find out during the prototype testing how helpful this format of displaying the beat count is. We were also unable to get the section panels to successfully display transparency, so we changed the display to instead show a bar at the bottom of the section which the user are standing in.

From our own preliminary testing, we found that the Kinect is extremely sensitive in terms of gesture detection. So our “delete track” (raise one arm and swipe other arm) gesture often gets mistaken for a “start loop recording” (raise one arm) gesture since both involve raising a single arm. The beginning of the delete motion could start recording unintentionally. We plan on eliminating the gesture overlap, and make them clearly different. We have not yet implemented this change, but expect to before user testing on the hi-fi prototype.

Additionally, the GUI is incomplete. Currently, we are relying on the depth image from the Kinect to show the figures on the screen. In our paper prototype, we had a simple body outline which showed the person on the screen. Ideally, we would do something similar for our hi-fi prototype and outline the bodies of the users using OpenGL/Box2d to create a clean, simple interface. Adding color would be awesome too. See Section VIII for an attempt at creating a better UI — we were unable to get the timing for the beats exactly right, so we delayed implementing the better UI for the final demos. One important part of the UI which we have not yet implemented is displaying a rectangle in each panel corresponding to the loops which the user has recorded. This was due to some difficulties managing the visualization in Processing. We hope to add this in the upcoming weeks.

Storyboards:

Sketches for the still-unimplemented parts of the system:

VII. 3–4 paragraphs providing an overview and discussion of your new prototype

We have implemented the following functionality:

• Visually track user using Kinect skeleton data
• Built gesture-based functionalities for recording multiple loop, deleting a loop
• Bring down the sound recording latency to a manageable level
• Heads up beat/recording status display

The main thing we learned from the paper prototype testing was that the sound playback of the recorded loops was essential for the user to understand and fully utilize the system, which is why we focused on loop recording instead of the master recording for the hi-fi prototype. To have accurate and good sound playback, we needed to display to the user the number of beats per loop. We did this by printing out a beat count using text in the upper right hand corner (See Section VIII). In addition, we wanted to add visual indicators when the loop recording was about to start, during the recording, and when it finished. To do so, we displayed messages in the upper left hand corner — “get ready…” when the loop recording was about to start, “recording” when the loop is recording, and no message when the recording is finished.

We left out the following features: master track recording, and a settings menu. As a reminder, the master recording would be able to combine all the repeating loops that are playing back on screen plus any additional sound that the user adds during the recording time, and save to a single track. We weren’t able to implement this because it requires further investigation. It is not essential for a minimum working prototype. From paper prototype testing, we found that users got confused between “master track recordings” and “loop recordings,” so we decided to prioritize loop recordings in our working prototype and eliminate the confusion for the users. We aim to successfully add this master track functionality later (without confusing the user). For the settings menu, again the EQ and sound level adjustment is not part of the core functionality, so we are saving it until the more important parts work.

Wizard-of-oz techniques: We’re pretending that the master recording is working properly when in fact it is not yet. Our current plan of action is for users to use their imaginations to pretend it is recording. We may also use paper indicators (with text like “Master Track is recording…”) to help users understand, and test out the best ways to signal the users between loop and master track recordings. We implemented functionalities in order of importance to the main use of the software. We ranked the priorities: 1) get loop recording working and synchronized, with sound playback; 2) loop deletion; 3) show user position on screen; 4) show beat count and recording status. Todo: UI improvements (body outline and loop track blocks) and non-implemented functionalities (master recording and settings menu).

External Code and Libraries Used:

• Minim Library (sound recording and processing)
• SimpleOpenNI framework for reading in Kinect data (and Skeleton library)
• For body outline and improved UI (not fully functional): followed the Kinect Physics Tutorial for the body outline, which utilizes the blobDetection, toxiclibs, PBox2D, and org.jbox2d libraries which are all freely available on the Internet.

VIII. Video & Images of Working Prototype

Videos:

A demo of our working prototype.

The non-functioning improved UI. Including in the openGL and polygon drawing libraries for drawing the figure caused additional latency (the beat count was not consistent, sped up during beats 2 and 3 and 5 and 6; this is really bad, since musicians need very constant tempo). We will try and work out the bug for this in the future so we can use this cool UI!

Photos: