Group 20 – Name Redacted

Brian, Matt, Ed, and Josh

Project Summary: Our goal is to teach students the fundamentals of computer science without the need for expensive and elaborate hardware or software.

Description of the Test Method:

Consent Form:

To obtain consent, we explained the three tasks to the participants.  We made sure that they understood that their participation was completely voluntary and that at any moment they could stop participating.  We asked the participants to sign a simple consent form that can be found here.

Participant Selection:

Since the user group of our project are students with limited computer science background, we got three participants who had only take introductory computer science courses.  One of our participants had taken COS 109, and the two other participants have taken COS 126 only, and are in non technical majors.  We wanted our participants to have some experience with computer science so that we would not need to teach TOY in a thirty minute block, but also not enough experience to have learned assembly language.

Testing Environment:

We tested our project in a classroom setting.  All four group members helped go through our written scripts and watched as the user interacted with our system.  We  only have one participant at a time to make the participants feel more comfortable.  We attached pieces of tape to the AR Tags so that the participants could tape the tags onto the board.  We used a blackboard to emulate a projector and we drew the output.

Testing Roles:

Matt led the discussion and task on binary, Ed led the discussion and task on the TOY Program, and Brian led the discussion and task for the tutorial.  Leading the task included saying our scripts as well as “being” the projector and drawing the appropriate output on the board.  Josh said the demo script and obtained consent from each of the participants.  He also recorded notes during the testing phase.  We all helped with the write up and discussion after the testing.  Here is a link to our demo and task scripts.

Summary  of Results:

We ran our demos with three separate users who each had little experience with computer science. Overall the demos went very smoothly, and after our introductory explanation of each task, users had little trouble interacting with our system. We also were able to get good feedback from our users on things that work well in addition to things that could be improved.

In general, users thought the interface was simple and intuitive, and they were easily able to use it after being given a brief block of instruction. In both the binary and toy program demos, users said that they would like to see feedback as to what is actually going on when the program is running or how the numbers are being converted. Some users were also confused about where exactly they were supposed to place the data cards in relation to each other. In the toy program, some users were unsure if the results of the instructions were supposed to be displayed immediately or after the run data card was put up.

Discussion of Results:

We gained some very valuable insights from testing our tasks with three different users.  Firstly, all three of the users would like a more in depth number base lesson.  All three suggested outputting how to get from binary from a decimal number (i.e. 1 + 2 + 8 = 11 for 1011).  Before these testers, we did not really consider changing the binary lesson, and we think that we can make some great improvements in the future with our binary lesson.  Looking back on our introductory CS days, these testers are absolutely correct in terms of how to think about changing numbers between different bases.  For our tutorial task, none of our testers could really provide too much insight in terms of learning how to teach students but they did like the level of detail that we provided the teacher in terms of learning about TOY.  In fact, one of the testers suggested that a similar level of detail be output when the students are coding.

Going along with this, there was some confusion about what the arguments should be for each command.  It is also a pain to write out the code for a program and then find out that the commands are all wrong.  We might be able to add a component to our program that outputs the various arguments for the six commands.  One of our testers asked if it is “mov src dest” or “mov dest src” which is not at all obvious even for computer science majors (since there are different standards).  Since we are accustomed to the intel format, we just assumed that everyone would see the command as “mov src dest”, but that is not a good assumption.

Future Test Plan:

It is time to start building the higher-fidelity prototype.  Our idea requires a significant amount of coding, computer vision, and design decisions for the output of the projector.  It is thus very important that we start creating our actual prototype if we hope to finish before Dean’s Date.  Some of the difficult design decisions will come from the coloring scheme that we use and our display of error messages.  In the future, we will test our prototype with an actual projector  and then we can get valuable feedback about how we display information.  Also, since we want our product to appeal to middle school students, it is important that our project looks nice, and we cannot provide a good visual reference for testers if we are just drawing on a whiteboard.

We have already made significant progress on reading the tags from a webcam and projecting an image.  In the next couple weeks we will show the projected images to other students to receive aesthetic feedback as well as feedback on the overall design decisions.  We will be a good place then to make these changes to the projected output.