Fig. 1.

a 3D imaging of mouse pose dynamics. MoSeq uses depth cameras to image the 3D pose dynamics of mice, which are then used to identify behavioral syllables and grammar. b Plotting the imaged 3D behavioral data over time (compressed here using the random projection technique, top row, grey) reveals that behavior self-organizes into blocks (apparent as vertical striations in the imaging data). Plotting the mouse’s height at each point along its spine (middle row) similarly reveals the block-like dynamics of the mouse’s behaviors during the experiment. Based upon the intrinsic structure present in the data, MoSeq uses a probabilistic modeling approach to identify the complete set of behavioral syllables expressed within the experiment, and then takes advantage of this information to label each frame of 3D video (bottom row, indicated as colored blocks). Each discovered behavioral syllable is a brief, reused, and stereotyped motif of action (bottom); in a typical 30-min experiment in a featureless bucket approximately 40 such syllables are identified that encapsulate 95% of the mouse’s behavior. c Behavioral state maps generated by MoSeq depicting behavioral syllables (nodes, diameter is proportional to syllable usage) and transitions (edges, thickness is proportional to transition likelihood) encapsulate all behavior expressed within a given experiment captured by the camera and can be used to predict future behavior from present actions. d Exposing mice to stimuli (or manipulating genes or neural circuits, not shown) causes changes in the overall usage of individual syllables during an experiment. Here, mice were exposed to the fox odorant TMT, which causes fear-like behaviors in the mouse, including avoidance of the odor source. Exposure induces a wholesale behavioral state change in the mouse, which can be captured as differences in syllable expression (asterisk indicates syllables that pass a statistical test for difference with air-exposed mice). e Plotting out behavioral state maps for control and TMT conditions (as in c), and then subtracting these state maps identifies new behavioral trajectories through syllable space that are induced by exposure to the stimulus. Upregulated temporal connections between syllables are shown in blue, while downregulated connections are shown in red. The new behaviors induced in mice by TMT—including freezing and avoidance—are encoded by trajectories through the blue part of this state space. Figure (parts b-e) adapted from [7] Neuron 88(6), Alexander B. Wiltschko, Matthew J. Johnson, Giuliano Iurilli, Ralph E. Peterson, Jesse M. Katon, Stan L. Pashkovski, Victoria E. Abraira, Ryan P. Adams, Sandeep Robert Datta, Mapping sub-second structure in mouse behavior, 1121–1135., Copyright 2015, reprinted with permission from Elsevier