Group Number: 1

Group Members: Kuni Nagakura, Avneesh Sarwate, John Subosits, Joe Turchiano, Yaared Al-Mehairi

Project Summary: Our project is a serv­ing sys­tem for restau­rants, called Ser­vice­Center, that allows wait­ers to effi­ciently man­age orders and tasks by dis­play­ing infor­ma­tion about their tables.

Introduction:

Here, we describe the methods and rationale behind a user study geared to aid us in the design of our serving system. The goal of our system is to present  pertinent information to servers in restaurants that allow them to task manage more efficiently. The prototype being evaluated is a single laptop connected to a display projector, which serves as the motherboard. The laptop acts as both an order input interface for the servers and a kitchen-side order status interface for the kitchen staff (this task  was performed by a group member). We chose to use a laptop as the basis of our hardware because the primary purpose of our testing was for functionality and usability, and any interaction with a device in our system is a mouse-screen interaction, which a laptop provides well. Furthermore, the laptop allowed us to display customer information on a large screen, which is how we envision our motherboard should look like in our final design. As we have discussed, the purpose of our user testing was to gain insight in future design improvements. This being the case, our user study consisted of three tasks, ranging from easy to hard that covered the main actions that a server would perform  using our system, described in the sections below.

Implementation and Improvements:

Our P5 post can be found here: http://blogs.princeton.edu/humancomputerinterface/2013/04/22/p5-grupo-naidy/

We do not have any new changes to our implementation, as we were preparing for P6.

Participants:

The participants in our research and testing were all members of the university community. They were selected purely on a volunteer basis without any specific requirements; we did give them a quick demographic survey in order to gain a deeper perspective on who we were working with and how they would relate to our target audience. All of our test subjects were familiar with computers and used them regularly; one of our test subjects had some prior experience working at a restaurant. Overall, we found our participants to be related only tangentially to the target user group, but since the primary purpose of our testing was for functionality and the tasks were relatively simple, we found this was sufficient. In particular, we had one user who had experience in the service industry–his post-testing interviews were incredibly helpful. Otherwise, our subjects did not have waiting experience.

Apparatus:

The testing was conducted in the dining room of one of Princeton’s eating clubs, at times when meals were not being served. We found this setting to be ideal, since it nearly replicated the environment of a restaurant and provided the necessary equipment (plates, glasses, a pitcher) without having to impose on any third party. A laptop computer was used to run the prototype system, and it was connected to a projector to show the order display on a large, highly visible screen – more or less how we imagine the final product to be used. The computer ran both the order display interface, into which the test subject entered new orders, as well as the “kitchen staff” interface, which was used to update the orders. The kitchen interface’s data was sent over a simulated wireless connection to better replicated intended use circumstances.

Tasks:

In our work­ing pro­to­type, we have cho­sen to sup­port an easy, medium, and hard task. For the easy task, users are sim­ply asked to request help if they ever feel over­whelmed or need any assis­tance. This task is impor­tant to our sys­tem, because one of our func­tion­al­i­ties is the abil­ity to request assis­tance from wait­ers who may have a minute to spare and are watch­ing the motherboard.

For our medium task, we ask users to input order infor­ma­tion into the sys­tem for their table. Once users input orders, the moth­er­board will be pop­u­lated with orders and the kitchen will also be noti­fied of the orders. This is a medium task since this is sim­ply inputting information.

Finally, for our hard task, we ask our users to uti­lize cus­tomer sta­tuses (ie. cup sta­tuses, food orders) to deter­min­ing task orders. This task cov­ers both ensur­ing that cups are always full and that the pre­pared food is not left out for too long. The motherboard’s main func­tion­al­ity is to gather all this infor­ma­tion in an easy to read man­ner, allow­ing wait­ers to sim­ply glance at the cus­tomer sta­tus to deter­mine what order they should do things.

Procedure:

The testing procedure was designed to place the test subject in as accurate a simulated restaurant environment as was possible.  The tasks that can be accomplished with the system were required to be performed interspersed with each other, as they would be in a real restaurant, rather than completely divorced from each other.  Three of the group members sat at different tables and each played the part of 3 to 4 customers.  The tester acted as the waiter for about 15 minutes while the customers ordered a standard, prescribed combination of appetizers, entrees, and deserts.  The tester was responsible for entering the orders into the system and for bringing the “food” (actually paper labels) to the table after it had been prepared and plated.  Each dish took a specific length of time before it was ready.  Periodically, generally about twice a test, the tester would have to fill the water glasses at each table since the motherboard indicated that they were empty.  Participants generally did a good job of announcing both what they were doing, ex. “I am refilling the water glasses,” and what they were thinking, ex. “It’s confusing that red means the water glasses need to be refilled but that the food isn’t ready yet, so I don’t have to do anything.”

Test Measures: 

“Turn around time’” – time between dish being put out by kitchen and being delivered. This was measured by hand – electronic measurements were dependent on users correctly using the system, making the metric interdependent on other variable factors (discussed later)

Post-Test ratings by users (Self reported Likert scale 1-5. The results of these questionnaires are provided in the Appendix):

• “Recoverability” – Whether the system made it easy to recover from mistakes. We felt that this was important because in a system where “online” decisions are being made, the system must be especially robust to user error.
• “Easiness to Learn” – Since this system would be deployed in existing restaurants, the restaurants would want to minimize the time that users (waiters) would take to adapt to the new system and start working at their maximum efficiency. Thus, we felt this an important metric to measure.

Results and Discussion: 

Testers had some trouble interpreting the graphical display on the motherboard.  In particular, they suggested that the color coding of various events could be made more intuitive and that the presence of empty water glasses could be indicated in a more obvious way than the number of full or half-empty ones.  Basically, the display of things that need to be urgently addressed should grab the user’s attention and colors should be consistent. If red is to indicate attention for cups, it should not indicate “not ready” (and thus requiring no attention) for orders.  Perhaps in the future, the red circle that contains the number of empty water glasses can be made to flash.  On the bright side, users seemed to have a fairly easy time entering orders into the terminal for the kitchen to prepare. One thing that we were interested in seeing was whether or not order input would be a hinderance to servers.

Overall, the testers were fairly efficient in the use of the system. We had wanted to put a “heavy load” on the testers to see how they would respond to a lot of activity, and we did this by scheduling what we thought was a high number of orders (9 “orders” comprising of about 25 dishes) over the course of 10-12 minutes. The average “turn around time” of the users was quite low. We decided to split the data into instances where the waiter was in the “kitchen” area when the dish came out and when the waiter was not. Unsurprisingly, when the waiter was around, the average turn around time was negligible. When the user was not in the kitchen, the average across users was about 30 seconds per order.

However, we realized after the fact that we did not have a “control set” of  turnover times, which would ideally be turnover times from waiters working in a real restaurant environment. We noticed that waiters had the most trouble when sets of orders from different tables came out at the same time, and in a real restaurant setting, there could be more tables, and thus more of these challenging situations. Our setup used only 3 “tables” of guests placing orders, but it may have been more accurate to have a larger number of tables placing orders with a lower frequency. We also noticed that none of the users decided to call for help in this trial. This is most likely because we used paper plates during the trial, which allowed users to pick up as many plates as needed when delivering orders. The physical space we were using to test was also quite small, allowing users to move back and forth between patrons and the “kitchen” very quickly, allowing them to deliver a large amount of plates quickly without too much delay. Since this system is meant to help with actions whose durations are very short, environmental influence on action time can be proportionally very significant, and care must be taken to recreate an accurate environment to get meaningful data.

Users seemed to figure out very quickly how to enter the data into the interface for a new order, and by the time they were entering their second or third full order they had become quite efficient with the system. However, we noticed that several test subjects seemed to forget about the requirement to delete the orders after they were ready and delivered. Since there are a limited amount of order slots for each table, any order input on a full table would not go through. This resulted in at least one mistake where an order was lost in translation from paper to prototype when the user did not look up at the screen immediately to check that the data had been entered successfully. We have also considered making the ready orders disappear automatically to avoid this problem, but this raises the additional problem of the system needing to know when the plate has been taken out. In this case, at the very least a warning to tell the user that a table is full of orders would be a good addition to the prototype.

Appendix:

All surveys, forms, and scripts are located in this dropbox folder. All the collected raw data is also located in this folder.

Group Number: 1

Names: Yaared, Kuni, Avneesh, Joe, John

Project Summary: Our project is a centralized serving system, called ServiceCenter, that allows waiters to efficiently manage orders and tasks.

Description of Test Method:

To obtain informed consent from the test subjects, we first asked if they would be interested in helping to test our system for a COS class.  Once they assented, we explained that they would be playing the role of a waiter in a restaurant and interacting with our system and that the testing itself would take about ten minutes plus set up time.  We then presented them with the consent form that can be accessed here, and they signed it. We also read the demo and task scripts to them in order to explain what they were going to be doing through the test. The demo and task script can be accessed here and here.

All three test subjects were affiliated with Terrace F. Club.  The first was a member of the kitchen staff, so he had experience serving food to others.  He was a relaxed and friendly waiter.  This subject expressed interest in our prototype system when he saw us setting up, and we were able to show him how it worked in detail by using him to test it.  The second subject was a student who has spent multiple summers working as a waitress in the restaurant at a country club.  Since she was used to demanding customers, her style was more professional and focused.  She was chosen based on her experience as a waitress who was used to working in a high pressure environment. The last subject was a male student.  He had a unique approach to customer service, but was an enthusiastic participant and waiter.  We chose him because we wanted to test the system on an inexperienced waiter/waitress.

We chose to run our low-fidelity prototype in the lower dining hall of Terrace F. Club. The lower dining hall is split into two sections: a solarium and a seated area. The solarium was where we set up our Motherboard – a screen that displays order information, order status, and cup status for all tables in the restaurant. For the actual board, we used a white board with the table floorplan for the restaurant drawn on. The customers sat in the seated area, where our test users took orders and served the food. When serving food, the waiter on duty served plates with labels on them, specifying what dish it was. We used cups, jugs, cutlery, and updated the Motherboard using dry-erase pens to simulate our system for this low-fidelity prototype.

For our low-fidelity prototype test, we divided roles amongst our group as follows. John, Yaared, and Joe acted as customers in charge of ordering for their respective tables, while at the same time making any test run observations from the viewpoint of the customer. Avneesh acted as our main test run observer and took notes. At the same time, he was available to help complete any tasks if the waiter/waitress requested help. Kuni managed the Motherboard and the “kitchen”. Given an order, Kuni would update the appropriate section of the Motherboard, put out completed orders to be served by the waiter on duty, and again update the Motherboard, thereby simulating the transfer of information between the kitchen and floor.

For our testing procedure, we decided to stagger 4 different groups of customers of varying size (i.e. 4 tables). The first group (i.e. John) enters and is served by the waiter. John orders for a large group of 7. Now, the second group (i.e. Joe) enters and is served by the waiter. Joe orders for a small group of 2. ‘5 minutes later’, the third group (i.e. Yaared) enters and is served by the waiter. Yaared orders for a medium-sized group of 4. The second group (i.e. Joe) then asks for the bill and leaves after paying. Now, the fourth group (i.e. Joe) enters and is served by the waiter. Joe orders for a medium-sized group of 3. The respective groups finish their meals, ask for checks, pay and then leave. We thought that staggering 4 groups of customers of varying size in quick succession would be a good way of simulating a situation in which servers may be hard pressed. Since our system is supposed to allows waiters to efficiently manage orders and tasks, we thought simulating a situation in which servers may be hard pressed, i.e. dealing with a lot of tables in peak hours, would best test the effectiveness of our system (after all this is the problem we are trying to solve!).

Summary of Results:

Our users understood and used the system well throughout our prototype test. All of our tasks were completed—calling for help was very simple and only happened when our users had to bring out multiple plates of food at once. Our other two tasks—checking customer status and determining task order—were performed regularly throughout the 10-minute period for every test run. An incident occurred where there was confusion over what a customer had ordered when two customers had placed similar orders. Another major incident was when our first subject was confused about which tables corresponded to the figures on the board. At times, Kuni, who was operating the Motherboard, had trouble keeping up with the speed of orders being placed and preparing the paltes. Otherwise, the user testing went very smoothly.

Users rarely (less than 5 times) looked at the motherboard for more than 10 seconds. Most of the time, they would glance in passing and quickly assess what their next task was. The only time they seemed to stop to look at the board was when there were multiple tables with ready plates. Throughout the test, there were times when the customer had to remind the waiter/waitress that their cup needed refilling. Overall, Kuni processing all the information and updating the board while also preparing the plates seemed to be the biggest bottleneck throughout the testing process.

Discussion of Results:

The most common “problem” faced by the test subjects seemed to be deciding how to order food deliveries to tables when multiple tables had food ready. One test subject volunteered the suggestion that orders should display an “estimated time left” that would show the rough time until each dish would be ready (the estimate could be determined in a variety of ways.) This could prove helpful in planning for future trips to tables when there is some down time. Also, it could be helpful to automatically sort the orders on the screen. All orders could be given a time value, positive for orders that have been sitting out (for how long they have been out), and negative for orders not yet ready (for how long until they will be ready). The orders could then be automatically sorted in ascending order by time. This would place them in priority order for the waiter/waitress.

We also realized that the dedicated board updater could be a bottleneck in a real restaurant setting even in the workings of the real system. If there are a lot of orders with many modifications and many tables being served, then the input of all these orders will most likely be the most time consuming part. But we realize that this is a worse case no matter what the system is. If we dedicate a person to input, then this could alleviate the problem. The dedicated input person would take the task from waiters/waitresses and possibly become quicker at inputting the information. Users were overall enthusiastic about the system, and they didn’t need too much explaining beforehand to understand how the system worked. We were happy that users did not need to scrutinize the board to figure out what the status of each table was—the layout and concept seemed to be clear.

Future Test Plans:

There are several areas of our low-fidelity prototype that could use extra testing before we move on to the high-fidelity prototype; many of these were brought to light during this testing session. As it is, we have already tested the usage of the information screen both in terms of entering data and reading data. However, we did notice during the testing procedure that simulating the information screen without a computer created a major bottleneck – Kuni had as much work to do in each task procedure as the actual outside participant. In order to better simulate the actual speed of the system (since speed and efficiency are at the core of our goals), we might have to find a faster method of displaying the data without computerizing it, if we want to continue to test with low-fidelity. Finally, we would also want to incorporate the input interface in the next round of testing. Since in our system, we would have a dedicated worker inputting the orders into the system, in our test Kuni essentially played this role by directly writing the orders onto the board.  In the creation of the paper prototype for the input interface, we went through some brief pseudo testing, going through 2-3 iterations of the input interface. But for future testing, we would have another subject input the orders as the simulation runs.

There are other areas which we found difficult to test but which could have yielded useful information. For one, we were not actually able to do testing in an open restaurant setting with waiting staff, instead using the dining room of an eating club with students with waiting experience as an alternative. Testing in the real setting would yielded the ideal results, but we are unlikely to find a restaurant willing to participate. Another smaller area which was difficult to emulate in low fidelity was the entrée timer (i.e. the timer which counts down the time until the food in the kitchen is done being prepared). It was simply infeasible for the board manager to be constantly updating the time, and somewhat less useful anyway since we used wait times of 1-2 minutes instead of 15-30 minutes as is common in actual restaurants. Testing this aspect might be done with an actual timer or stopwatch in the future.