Caught in the act: Video system for mapping behaviors (Royal Society: Interface)

By Catherine Zandonella, Office of the Dean for Research

Studies of animal behavior have come a long way from the days when scientists followed their subjects around with pen and notepad. But although cameras have replaced clipboards, evaluating the resulting videos is still a cumbersome process.

To make that job easier and more comprehensive, researchers at Princeton have built a computerized system that analyzes videos to reveal what animals do, how often and for how long, and then generates an easy-to-understand map of the behaviors.

By looking at the map, the researchers can tell what sorts of movements the animal did without having to spend time watching the video. They used the system to track the behavior of fruit flies (Drosophila melanogaster) and found that it accurately detected behaviors such as grooming a leg or waggling a wing.

Map that indicates the type of movements of a fly.
Princeton researchers have developed a computerized system for evaluating animal behavior. A video system records an animal, in this case a fruit fly, for one hour. Then a computer program analyzes the video to create a map of the behaviors. The map can reveal, for example, the differences in movements between male and female fruit flies. (Image source: Joshua Shaevitz)

The researchers’ goal was a system that could create accurate records of behaviors for use in studies of the mechanisms behind those behaviors – in other words, the genes and brain circuits that govern movements. Knowing which genes and neural circuits govern behavior will help answer basic scientific questions and could shed light on the mechanisms behind conditions such as autism.

Described in the journal The Royal Society Interface, the system was built for monitoring Drosophila flies, which are commonly used in studies of genes, but the researchers say that the system can also analyze the movements of other creatures, including worms, mice and humans.

The new system is a significant advance over current approaches because it takes note of all the animal’s activities, not just behaviors that seem important to researchers, said Joshua Shaevitz, associate professor of physics and the Lewis-Sigler Institute for Integrative Genomics, who led the study. “We don’t know which behaviors are important to a fruit fly, or a mouse,” he said. “Instead, we ask the computer to find behaviors that are frequently repeated, which tend to be the ones that are worth studying.”

Researchers need reliable ways to catalog behaviors in studies of how genes and neural pathways control behavior. A common experiment involves deleting a gene in an organism such as a fruit fly or worm to look for any resulting loss of function or change in the organism’s behavior. But without reliable behavioral data, researchers cannot make firm conclusions about the role of the deleted gene.

With the new system, a high-speed camera records the animal for a given period of time, and the computer sorts the resulting video frames. For the current study, the researchers recorded individual fruit flies for a period of one hour, resulting in an enormous amount of data. The challenge was how to sort the frames and translate the images into data that could be evaluated.

“We have to figure out how the body parts of the fly are positioned in relation to each other at each point in time, and then find a way to tell how those body parts are moving, while keeping the amount of data manageable using the available computing power,” said Gordon Berman, an associate research scholar in the Lewis-Sigler Institute for Integrative Genomics. “Then you have to project the information into some form that allows you to understand the data.”

The researchers accomplished this goal by writing a computer program that converts the information about a fly’s position and movements into mathematical descriptions that are then grouped according to their similarities. The computer places these groupings on a two-dimensional map, so that different regions of the map represent different types of movements.

A computerized system for evaluating animal behavior.
Individual fruit flies were filmed for about one hour, resulting in over 300,000 video frames. Then, a computer program converts the fly’s position in each frame into mathematical descriptions that represent movements. The computer then groups the movements on a map, so that one region of the map represents leg movements while another represents wing movements, and so on. By looking at the map, the researchers can tell what sorts of movements the fly did, such as grooming its front leg, grooming its head, or waggling its wing. (Image source: Joshua Shaevitz)

To verify that the system worked, the researchers collected data from 50 male and 50 female flies. They identified hundreds of behaviors, including ones that had never been documented and others that were consistent with findings from human observers. For example, some of the findings consistent with previous observations were that male fruit flies kick out their legs when grooming, whereas females don’t, and that young females are more active than young males. “Males and females do things slightly differently, and we can pick up on that right away,” said Shaevitz.

Male fruit flies kick out with their legs when grooming:

whereas females do not:

The team is now expanding the capabilities of the method, Shaevitz said. “The technique that we have developed can also be used to study patterns of behavior in humans,” Shaevitz said. “There are several conditions that are diagnosed on the basis of behavior, such as autism. Having a more quantitative way of measuring behavior could improve the accuracy of diagnoses.”

The study co-authors included William Bialek, Princeton’s John Archibald Wheeler/Battelle Professor in Physics and the Lewis-Sigler Institute for Integrative Genomics, and Daniel Choi, a graduate student working with Shaevitz.

The work was funded through awards from the National Institutes of Health (GM098090, GM071508), the National Science Foundation (PHY 0957573, PHY 1066293), the Pew Charitable Trusts, the Swartz Foundation and the Alfred P. Sloan Foundation.

Read the abstract

Gordon J. Berman, Daniel M. Choi, William Bialek, Joshua W. Shaevitz. Mapping the stereotyped behaviour of freely moving fruit flies. The Journal of the Royal Society Interface (2014). DOI: 10.1098/rsif.2014.0672 Published online 20 August 2014 http://rsif.royalsocietypublishing.org/content/11/99/20140672