Storytelling and Visualization

Abstract

Throughout history, storytelling has been an effective way of conveying information and knowledge. In the field of visualization, storytelling is rapidly gaining momentum and evolving cutting-edge techniques that enhance understanding. Many communities have commented on the importance of storytelling in data visualization. Storytellers tend to be integrating complex visualizations into their narratives in growing numbers. In this paper, we present a survey of storytelling literature in visualization and present an overview of the common and important elements in storytelling visualization. We also describe the challenges in this field as well as a novel classification of the literature on storytelling in visualization. Our classification scheme highlights the open and unsolved problems in this field as well as the more mature storytelling sub-fields. The benefits offer a concise overview and a starting point into this rapidly evolving research trend and provide a deeper understanding of this topic.

Keywords: storytelling; narrative; information visualization; scientific visualization

Source: MDPI, https://www.mdpi.com/2078-2489/9/3/65/htm
This work is licensed under a Creative Commons Attribution 4.0 License.

Introduction and Motivation

"We believe in the power of science, exploration, and storytelling to change the world" - Susan Goldberg, Editor in Chief of National Geographic Magazine, from "The Risks of Storytelling", October 2015.

"In a world increasingly saturated with data and information, visualizations are a potent way to break through the clutter, tell your story, and persuade people to action" - Adam Singer, Clickz.com, "Data Visualization: Your Secret Weapon in Storytelling and Persuasion", October 2014.

Throughout history, storytelling has been an effective way of conveying information and knowledge. In the field of visualization, storytelling is rapidly developing techniques that enhance understanding. By visualization, we refer to "interactive data visualization" as defined by and described extensively by Ward et al.. We note that there are many references to interactive visualization throughout the survey. For example, the term "interactive", appears over 40 times throughout the text. We also note that animated transitions are a major theme in the survey and receive their own section in the text. See Table 1. Many communities have commented on the importance of storytelling in data visualization. Storytellers tend to be integrating complex visualizations into their narratives in growing numbers.

Table 1. Our classification of the storytelling literature. The y-axis categories fall into who-authoring-tools and user-engagement, how-narrative and transitions, why-memorability and interpretation. See Section 1.2 for a complete description.

1. Linear: A story sequence path in linear order is prescribed by the author.
2. User-directed path: The user selects a path among multiple alternatives or creates their own path.
3. Parallel: several paths can be traversed or visualized at the same time.
4. Random access or other: There is no prescribed path. There is currently no literature prescribing random order. Therefore we replace this with a column called "overview".

Literature Search Methodology

1. IEEE EXPLORE Digital Library
2. ACM Digital Library
3. Visualization publication data collection
4. the annual EuroVis conference
5. the Eurographics Digital Library

Related Work

Ma et al. state that a story that is well paced exhibits deliberate control over the rate at which plot points occur. They present a selection of scientific storytelling visualizations from NASA related work and describes various examples.

The Scientific Visualization Studio (SVS) at NASA uses storytelling visualization to investigate observational data collected by instruments and sensors and make it more suitable for consumption by the public.

The science museum presents visualization to the public with complex and abstract geographic phenomena at extreme size scales for explanatory animations. The science museums provide further interpretation through labels, videos, and live demonstrations. See Figure 3.

Figure 3. Ma et al. show the interactive software used at the Exploratorium in San Francisco. The purpose of this software is to educate users on the process of how tides, currents and rivers combine in the estuary of San Francisco bay. A touch-screen is used to place floats into the virtual water so that the user can see the effects of the current on the float. Users can watch the effects of predicted tide and river flow cycles on the floats trajectory. Other contextual information is provided as an animation alongside the visualization.

Storytelling enables the user to interact with geographic data such as the Earth's climate or the collapse of a star by using a story model, such as story nodes or story transitions. Ma et al. is based on previous scientific visualization work at NASA, based in the scientific research center and scientific museum and describe how visualization can be used to tell a good story, and tell it well. This is a topic that the scientific visualization research community paid little attention to at that time.

Tong et al. published a storytelling visualization survey paper as a short paper in abridged form. It contains no image or paper summaries. This is a full-length, comprehensive, extended version of that survey. It is approximately double the length.

Authoring-Tools for Storytelling and Visualization

Authorship refers to writing or creating a book, article, or document, or the creator of a work of art according to The Oxford English dictionary, especially with reference to an author, creator or producer. For our purposes, we will adopt a definition of author described by Rodgers, "An author is best described as an individual solely responsible for the creation of a unique body of work". Hullman et al. state, "Story creation involves sequential processes of context definition, information selection, modality selection, and choosing an order to effectively convey the intended narrative".

An author is best described as an individual solely responsible for the creation of a unique body of work. Presenting the findings of a qualitative study of undergraduate writers at The City University of New York, Hullman explores student perspectives on models of authorship, the relationships between these models and student experiences of authorship in different writing situations, and proposes the importance of distinguishing between the multiple models and definitions of authorship and the rhetorical contexts associated with each. Rodger develops a qualitative study of 800 students on the definition of authorship and their rhetorical contexts over a one-hour interview. Students defined authors as "[people] who see writing as being beyond a hobby", and as a term that should be applied only to those individuals for whom writing is "something he or she has to do", "a career", or "an act that will lead to something being published".

All papers in this section focus on authoring-tools for storytelling. Wohlfart creates new volume visualization stories for medical applications. Gershon and Cruz present general storytelling for information visualization. Kuhn, Lee and Plowman all develop unique creator tools for storytelling visualization.

It is important to note that our survey is not simply a list of papers. Individual papers are summarized according to a special methodology. This process connects related papers together such that the connections and relationship to previously published literature is made clear.

Authoring-Tools for Linear Storytelling

The literature in this sub-section focus on visual designs for authoring in a linear style that is prescribed, automatic, or semi-automatic (as opposed to interactive) or decided by the users. In other words, creators are provided with assets to formulate a linear story.

Gershon and Page state that storytelling enables visualization to reveal information as effectively and intuitively as if the viewer were watching a movie. They introduce the concept of storytelling and presents advantages of storytelling.

One example presents a situation in which a number of enemy positions surround a school with children trapped inside as de facto hostages as the crossfire fills the space overhead and both sides move toward confrontation. Gershon and Page is based on previous work of Denning and explain the usage of storytelling in information visualization.

Lu and Shen propose an approach to reduce the number of time steps that users required in order to visualize and understand the essential data features by selecting representative datasets. They design a flexible framework for quantifying data differences using multiple dissimilarity matrices. A new visualization approach that filters data analysis results, which is achieved by measuring the degree of data similarity/difference and selecting important datasets that contain essential data features. See Figure 4.

Figure 4. This figure shows the system architecture from Lu and Shen. It integrates the information of data analysis and a single 3D data visualization method for users to explore and visualize overall time-varying data contents. (a) A time-varying dataset; (b) Selected data features; (c) The distribution of time steps; (d) 3D visualization method; (e) Overall time-varying data contents. Image courtesy of Lu and shen.

They interactively select representative datasets that include a significant portion of features of scientific data, whose data distribution requires more analysis than time sequence, reduces the amount of data to necessarily visualize and still keeps the essential data information. This can be used to improve the efficiency of time-varying data visualization.

An interactive storyboard is used to visualize and explore the overall content of time-varying datasets through composing an appropriate amount of information that can be efficiently understood by users.

Lu and Shen is based on the previous work of time-vary visualization and design a general method for comparing data dissimilarities. They do not require a dense sampling frequency to capture the object evolution and their work is not limited to specific feature models, such as geometry or interval volumes, and their attribute designs.

Storytelling, in the context of this article, deals with the core of information visualization by extracting relevant knowledge and enhancing its cognition. Cruz et al. present generative storytelling as a conceptual framework for information storytelling. They create stories from data fabulas using computer graphics as a narrative medium. Data fabulas are a set of time-ordered events caused or experienced by actors.

A story is formed by characters. It involves the representation of the fabula's actors and the definition of a temporal structure. The engine transforming a fabula into story consists of two models. The event model creates a story timeline and an action model creates a set of actors behaviours. For example an empire's decline visualizing western empire's decline in the 19th and 20th centuries. See Figure 5.

Figure 5. Cruz et al. show the British hegemony and the newly independent South America in 1891. Each empire and independent territory is a circle whose area is proportional to that entity's land area. Former colonies are unfilled circles with rims in the corresponding empire's color.

Cruz et al. is based on previous work of narrative theory and presents generative storytelling as a conceptual framework for information storytelling.

Authoring-Tools for User-Directed and Interactive Storytelling

A large body of research has been carried out for authors wishing to create their own user-oriented or interactive stories. This literature focuses on interactive, user-driven authorship (as opposed to automatic or semi-automatic authorship). Storytelling is a relatively new form of interactive volume visualization presentation. Wohlfart explores the usefulness of storytelling in the context of volume visualization. He presents a story telling model and divides the concept of volumetric storytelling into story authoring and storytelling constituents. He presents a volumetric storytelling prototype application, which is based on the RTVR (real time volume redering) Java library for interactive volume rendering. See Figure 1. The storytelling model contains a range of hierarchy levels, in top-down order, which are: story node, story transitions, story action group, story action atoms. See Figure 1. The story nodes form the corner marks of the story and store the state of the whole scene. Story nodes are connected by story transitions, each consisting of one or multiple story action groups. Each story action group stores the scene changes relative to its preceding action group (or story node).

The story authoring process contains two steps: a story recording process and a story editing process. The outcome of this recording process is a raw prototype of a story told through volume visualization. In the story editing step, this raw story is refined until the final story outline is reached.

This process presents a volume visualization following the storytelling model. And the key feature is interaction, including viewing interaction, representing interaction and data interaction. Wohlfart and Hauser also discuss the paradoxical integration of storytelling and interaction, also called the narrative paradox.

Figure 1 shows an image sequence taken from a sample linear volumetric story visualized with their prototype. The distinct story nodes refer to the key events in the story, which provide an overview first, then details on specific features in the dataset, and at the end a conclusion made by the story author. The necessary story transitions are represented as orange arrows from one story node to the next and are animated in the prototype application. The story consumer may take over some story parameters (e.g., camera angle) already during playback or at the end of the story to further investigate the dataset.

This story guides the observers through the visualization, puts the contained visual representations into context with each other and finally introduces them to important features in the data. See Figure 6. Wohlfart is based on previous work of volume visualization and interactive visualization and combine these concepts together to develop a storytelling model for volume visualization.

Figure 6. The top two images show an overview of the CT scan data presented by Wohlfart and Hauser. A partial clipping reveals both the skin layer and bone layer, but shows the full set of data. The middle shows a zoomed view that isolates eye swelling in the image (left), and a filtered view that exposes some blood effusions in the swollen region. The bottom offers a comparison of the non-injured eye with the injured one and shows the cause of the swelling which is attributed to a tripod fracture just below the eye. This design offers the user a macro overview as to lay the foundations of a story background then narrows the scope to view the focal point of the image. Image courtesy of Wohlfart.

Geological storytelling is a novel graphical approach for capturing and visualizing the reasoning process that leads to a geological model. Lidal et al. present a sketch-based interface for rapid modelling and exploration of various geological scenarios. The authors present a concept that handles sketching processed over time and a novel approach for externalizing the mental reasoning process. The process can be presented and evaluated. The geological storytelling model contains three main parts. See Figure 7.

Figure 7. Lidal et al. present a sketch-based interface for rapid modelling and exploration of various geological scenarios. The sketch-based interface is split into two windows. The Story Tree (left) which shows a tree graph representation of all the geological stories, and the Canvas (right) which shows the sketching interface which utilises a pen and paper interaction to record geological sedimentary data. A geological story is built using horizontal lines to separate different geological layers, vertical lines to show fault systems, and polygons for highlighting large sedimentary layers. The user can navigate through different geological stories with the story tree and then inspect the geological elements of that story.

The canvas is a sketch-based interface where the geologist can draw the geological story on a 2D seismic slice backdrop, utilizing a pen and paper interaction style. The StoryTree is a tree graph representation of all the geological stories, each with its own subtree of story nodes. Individual story nodes can be selected for editing in the canvas. One or more complete story trees can be selected for playback or comparative visualization in the InspectView. The InspectView serves two purposes. First, it is a view where a story can be played and evaluated. In addition, multiple stories can be played synchronized for a side-by-side visual comparison.

The data-based story is recorded in SketchStory as a sequence of charts in XML files. The charts are linked with specific sketch gestures. The presenter draws an example icon and then draws a sketch gesture for chart invocation. Sketchstory recognizes the gesture and creates the corresponding chart. Lee et al. is based on previous work for storytelling of information visualization and sketch-based interaction, and develops the SketchStory system to enhance storytelling in a presentation.

Lidal et al. is based on a previous storytelling model for scientific visualization and develops a storytelling model for geological visualization.
Lee et al. present SketchStory, a data-enabled digital white board to support real-time storytelling. It enables the presenter to stay focused on a story and interact with charts created during presentation. See Figure 8.

Figure 8. Lee et al. show an example of SketchStory in information visualization presentation. (a) A presenter sketches out a chart axis; (b) SketchStory completes the chart; (c) The presenter interacts with the charts.

Storytelling is one of the most impactful ways to teach, learn, and persuade. Lundblad and Jern present geovisual analytics software with integrated storytelling. It can be applied to large spatial-temporal and multivariate data through dynamic visual user interfaces.

Using a scatter plot matrix gives the analyst a good overview of all correlations between the selected indicators. The analyst can use the scatter plot matrix as an overview and then steer the scatter plot for interesting detailed combinations over time. See Figure 9. The distribution plot presents a special visualization technique that displays the variation within individual European countries. The Motala River map is visualized for different stories divided into different layers, such as a glyph layer, stream layer, polygon layer and background map layer. It shows the local and total water flow, and water path from source to ocean.

Figure 9. Lundblad and Jern show Vislet aimed at a comparative visualization using linked Scatter Matrix and Scatter Plot to analyze national correlation between 6 indicators between 1960 and 2010 from the World Databank.

Lundblad and Jern is based on the previous work of the storytelling concept  and work of web-based geovisual tools, integrates storytelling with geovisual analytics software.

Authoring-Tools for Parallel Storytelling

In this category of literature, authors create stories in parallel. In other words there may be multiple authors working in parallel i.e., simultaneously for the final outcome. This is opposed to a single author as in the previous subsection.

GeoTime events are recorded in x, y, t, coordinate space. This is used in observation analysis and can make a major contribution to a storytelling model. Eccles et al. presents the GeoTime stories prototype that combines a geo-spatial map with narrative events to produce a story framework. See Figure 10. This system provides functions for simple pattern detection in simultaneous movement activity. These functions look at possible interactions between people within the narrative, the speed at which they travel, and the type of location that they visit. Narrative text authoring enables the analyst to create and present stories found within the data. The story window displays this data as well as discovered patterns. The system enables multiple stories to connect together if they follow a linear flow. Also simultaneous narratives can be shown in a single image for a direct comparison.

Figure 10. Eccles et al. show a GeoTime visualisation instance. The l-axis represented by height is temporal. x- and y-axis represent the geospatial location Here you can see a taxi driver's route over the course of a few hours. Each pick up and drop off is labelled and the route is mapped on the x- and y-axis using the map.

This system uses a similar approach to Sense.us. Instead of using a blog-type discussion workflow for adding text, Geotime is designed for authoring a single story and annotations are integrated into the data itself.

The CodeTimeline visualization by Kuhn and Stocker enables developers who are new to a team to understand the history of the system they are working on. Designed to show a development team's tribal memory, the software offers a partial replacement for exhaustive documentation. See Figure 11.

Figure 11. Kuhn and Stocker show the CodeTimeline collaboration view. Colors denote different user contributions and each line represents the life of files in the code. Sticky notes are added so the users can learn the history of the code beyond the file evolution.

A collaboration view presents visualizes code ownership and historical patterns in collaboration. A sourcecloud flow view presents a word cloud of added and removed vocabulary between software releases. Lifetime events can be added by users as a frame of reference in each of the visualizations. This method of linking also enables new users to learn more about the history of the software development. These events can be include anything from email threads to pictures of the team during work.

Prior to Kuhn and Stocker, Ogawa presents "software evolution storylines" and "Code Swarm", which focus on the interactions between developers on projects but do not focus on telling a story about the software history. Codebook, a concept presented by Begel et al., outlines a social network that connects software engineers with their shared code base. It encourages interaction with their code and others, enabling a broader understanding of the project they share with other developers.

User Engagement

The literature in this category addresses an important but less developed research topic, namely user engagement. In other words, who do we engage with storytelling and how can we engage an audience?

Mahyar et al. address how prior research in different domains define and measure user engagement. They discuss existing frameworks for engagement from other related fields and propose a taxonomy based on previous frameworks for information visualization.

They present five levels of user engagement in information visualization. See Figure 12.

Figure 12. Mahyar et al. present five levels of user engagement in information visualization.

1. Expose (Viewing): the user understands how to read and interact with the data.
2. Involve (Interacting): the user interacts with the visualization and manipulates the data.
3. Analyze (Finding trend): the user analyze the data, finds trends, and outliers.
4. Synthesize (Testing Hypotheses): the user is able to form and evaluate hypotheses.
5. Decide (Deriving Decisions): the user is able to make decisions and draw conclusions based on evaluations of different hypotheses.
   

Their work is based on previous work of Bloom's taxonomy and adapts it to information visualization.

User Engagement for User-Directed Visualization

The literature in this subsection focuses on interactive, user-driven visualization for user engagement. Engagement specifically focuses on each user's investment in the exploration of a visualization. Boy et al. use low-level user interaction e.g., the number of interactions with a visualization that impact the display to quantify user engagement. They present the results of three web-based field experiments, and evaluate the impact of using initial narrative visualization techniques and storytelling on user-engagement with exploratory information visualizations. The main contribution of their work include: the design of three web-based experiments on user-engagement information visualizations. They hypothesize narrative elements should effectively engage the user in exploration of data and analysis the result. They conclude that storytelling does not help engage users in visualizing their experiments.
Boy et al is based on previous work on narrative visualization and user-centred metrics. The negative outcome of their study clearly indicates that more future work is needed to investigate whether or not storytelling increases user engagement.

Narrative Visualization and Storytelling

Narrative structures include events and visualization of characters. An example narrative can be a simple interface that presents trends in keywords over time. Narrative visuals contain the transition between events. It involves "using a tool to visually analyze data and to generate visualizations via vector graphics or images for presentation" to decide "how to thread the representations into a compelling yet understandable sequence". Plowman et al. report that a narrative specifically refers to the macro-structure of a document in contrast to the term story which refers to both structure and content. This structuring of evidence, combined with the choice of appropriate rhetorical strategies, is referred to as "the art of storytelling" among literary scholars. Research in narrative visualization points to visualization features that afford storytelling including guided emphasis and structures for reader-driven storytelling. It also includes the principles that govern effective structuring of transitions between consecutive visualizations in narrative presentations, and how different tactics for sequencing visualizations are combined into global strategies in formats like slideshow presentations. We separate transitions into their own section, Section 5 and Section 6, because of their importance.

A narrative can be seen as a macro-structure which creates global coherence, contributes to local coherence and aids recall through its network of causal links and signposting. The focus of Plowman et al.'s research is how students make sense of their learning with multimedia by constructing their own narratives in conjunction with the narrative guidance. The design elements presented by the software constitute narrative guidance and can be a combination of features specific to interactive media, such as the need for clear navigational procedures, with features associated with traditional media, such as recognizable narrative and a clear relationship between tasks and the macro-narrative.

Plowman et al have developed three versions of Galapagos as a research tool, based on extended observation of students using commercially available CD-ROMs in schools.

The linear version is designed in such a way that students are led through the eight sections of the CD-ROM in sequence and it is closest to a traditional narrative as presented in educational television. The linear version presents a high degree of narrative guidance and little opportunity for learners to decide their own narrative path so they have relatively little control. The resource-based version offers no guidance through the CD-ROM sections and leaves students to define their own route. There is very little narrative guidance offered and learners have to construct a narrative by making decisions about sequence, so there is a high degree of user control and heavy use of the menu to decide the route. The guided discovery (GDL) version offers guidance in breaking down the task by providing paths through the material, questions to stimulate enquiries, and direction to specific resources. The GDL version was designed to offer a balance between narrative guidance and support for narrative construction and this is reflected in a more even balance between user and system control. Learners are able to determine sequence and their course of action but are offered guidance in doing so. Most sections (seven) are accessed by the GDL users because the guide encourages them to be interactive in their approach and to use the material to support their response.

All papers in this section develop methods or structure on how to improve narrative storytelling visualization. Viegas et al. present methods for improving data memorability. Fisher et al. present ways for tracking narrative events over time. Segal and Heer investigate the design of narrative visualizations and identify techniques for telling stories with data. Hullman et al. design the structure of a visualization to present storytelling. Figueiras studies how to incorporate narrative elements as storytelling elements. Again, these papers may cover more than one topic in Table 1. The borders between categories are not 100% black & white. We place papers in the category reflective their main focus.

An overview of the visualization methods used in storytelling for visualization can be found in Figure 2. We include it in the section on narrative visualization since this is where the most research has been done. We can observe that most of the visualization designs used are familiar.

Narrative Visualization for Linear Storytelling

The literature in this sub-section focus on narrative visualization using linear automatic or semi-automatic approaches (as opposed to interactive approaches). The research here involves tools and techniques with an emphasis on how stories are created.

Hullman et al. describe a system called contextifier, which automatically produces custom, annotated visualizations from a given article. The system architecture contains four main sections. A news corpus consists of a large set of news articles. A query generator identifies the most-relevant company in the article. An annotation selection engine integrates selected features into an annotation. And the graph generator generates line graphs using annotations and series. The flow of information can be seen in Figure 13.

Figure 13. Hullman et al. show the architecture of contextifier (left) and illustrate Parallelism in sequencing in the NYT Copenhagen (right).

Hullman et al. is based on previous work in storytelling in visualization and Kandogan's automatic annotation analytics. It develops a system that can automatically generate custom, annotated visualization from a news article of company. Hullman's work places more emphasis on providing background information or perspective on the data than Kandogan's.

Hullman et al. outline how automatic sequencing (the order in which to present visualizations) can be approached in designing systems to help non-designers navigate structuring decisions in creating narrative visualizations. Their focus is on how linear, slideshow-style presentation can be optimized using knowledge of sequencing styles and strategies by incorporation.
Hullman et al. argue that analysts using narrated data presentations could be aided by tools for identifying effective sequences for visualizations. They conduct a qualitative analysis of the structural aspects of 42 examples of explicitly-guided professional narrative visualizations. One example is shown in Figure 13. They propose a graph-driven approach for finding effective sequences for narrative visualizations informed by their analysis, including defining data attributes for transitions, labelling, and maintaining consistency. The result suggests a need for more sophisticated global constraints than simply summing local transition costs to determine the best path through a graph of weighted visualization transitions. This paper is based on previous work of narrative sequencing and narrative visualization, and demonstrates that narrative sequencing can be systematically approached in visualization systems.

Amini et al. identify the high-level narrative structures found in professionally created data videos and identify their key components. They derive broader implications for the design of an authoring tool to enable a wide audience to create data videos. Amini et al perform two studies to enhance understanding data videos. They conduct a qualitative analysis of 50 data videos from 8 reputable online sources, and observe that data video categories are also hierarchical and can be further decomposed into units: sequences that put forward different points contributing to a single category. They design a series of workshops to observe how professional storytellers create data video storyboards. They observe the creation process is non-linear and iterative. Amini et al. is based on previous work on storytelling and storytelling in information visualization.

Bach et al. develop graph comics for data-driven storytelling to present and explain temporal changes in networks to an audience. Bach et al. present six steps to guide graph comics design. See Figure 14.

Figure 14. Bach et al. present graph comics for data-driven storytelling.

They first collect diagrams, comic literature, and pictures within comics to understand traditional comics structure. The second step is to find possible visual encodings that can represent graph objects, their properties, and the possible changes which they may undergo. They design principles that define when certain visual marks and their attributes can be used and when not. They exploit their design challenges and the structural principles to create comics. They contact two domain experts to collect external feedback and present a qualitative study to check if graphics comics are readable by a wider audience. Bach et al. is based on previous work on network exploration and data-driven storytelling.

Narrative Visualization for User-Directed and Interactive Storytelling

The literature in this subsection focuses on interactive, user-driven narrative visualization (as opposed to automatic or semi-automatic). In other words, the papers focus on techniques that enable users to create narratives interactively. Viegas et al. summarize two methods of visualizing email archives with the aim of improving memorability of the data. Both focus on the higher level patterns of the user's email habits. The original goal was for these visualizations to uncover social patterns in the archive, but the resulting visualizations caused the user to be more reflective of the data as opposed to analytic. They look at data points and want to recall the story behind it, even share the visualization with friends. See Figure 15.

Figure 15. Viegas et al. show the PostHistory visualisation. On the left is the calendar view, showing 365 squares to represent each day of the year (This image only shows data up until May). Size corresponds to the volume of email sent on that day. The colour highlights a specific recipient that has been selected in the contact panel (left). The contact panel shows all the contacts the user has been corresponding with over the year. A contact can be selected to highlight their interaction in the calendar view.

For visualizing email activity, the two axes stand for time, and the dyadic relationship between user account holder and each human interaction. Pattern recognition includes interaction frequency, interaction rhythm, interaction balance, and archive size. The visualization interface includes two main panels; the calendar panel, showing email intensity, and the contacts panel, showing the names of the people being interacted with. When the user clicks on a day square in the calendar panel, the contact panel highlights the names of the people communicated with that day. A name can be clicked on in the contacts panel and each day where that person had corresponded will be highlighted in the calendar panel. The contact panel can be viewed as an animation transitioning through the year of data. The email header data is used to derive the social context of the communication. Five different relationship types are classified. This can be either directly between correspondents or through mutual recipients in group emails. The Social Network visualization looks at each message and evaluates the role of the user (through the email address used i.e., work, school or personal) and makes connections regarding the interaction accordingly. This data is visualized as an animation that evolves over time. Each second represents one day in the archive. A clustered word cloud is used to display the data. Previous visualizations of online social interaction data have been focused on unravelling the data from the researchers' perspective, whereas these visualizations are for the benefit of the user.

Hullman and Diakopoulos state that narrative information visualizations are a style of visualization that often explores the interplay between aspects of both exploratory and communicative visualization. This work contributes to information visualization design and theory by providing insight into the types and forms of given rhetorical techniques in narrative visualizations, and the interaction between those techniques and individual and community characteristics of end users. The authors study how rhetorical techniques are used in visualization. They then investigate the resulting effects of these techniques on user interpretation. The authors collect 51 professional narrative visualizations e.g., from the New York Times and BBC. Each visualization is "coded" using theory form semitics, statistics, decision theory, and media and communication studies. The visualizations are categorized according to a selection of rhetorics information access, provenance rhetoric, mapping rhetoric, procedural rhetoric, and linguistic rhetoric. Their work provides a taxonomy of specific information presentation manipulations used in narrative visualization. See Figure 16.

Figure 16. Hullman and Diakopoulos demonstrate how data can be window dressed to change the viewers opinion of it. These two images visualize the same data but each illustrator has different intended outcomes. The top image shows an unstructured, complicated graph of conflicting colors and shapes, clearly intended to confuse and obstruct the data, whereas the bottom lays the data out in a simple fashion using consistent shapes and colors.

In the mapping America visualization example, The United Stated Census represents a nation wide attempt to provide an objective view of the demographic of the country. Ethnic group is mapped to color, samples are shown on a map and a single ellipse represents 200 people. The poll visualization summarizes the accuracy of political poll predictions from several years and polling agencies in a small multiplies presentation of vertical line graph. Colored bars representing the political parties are drawn to connect data points positioned on the y-axis according to the amount of time prior to the election and on the x-axis according to whether the predictions fell over (to the right) or under (to the left) a centred vertical line representing complete accuracy (or error of zero).

Hullman and Diakopoulos is based on the previous work of Segel and Heer which makes an initial step towards highlighting how varying degrees of authorial intention and user interaction are achieved by general design components in narrative visualization. This work examines the design and end-user interpretation of narrative visualizations in order to deepen understanding of how common design techniques represent rhetorical strategies that make certain interpretations more probable.

A visualization with a narrative is set apart from a visualization without through both its structure and its content. A narrative-based visualization attempts to create a natural flow whereby the data has an obvious progression and therefore permits easier understanding and memorability.

Figueiras takes professionally produced visualizations as case studies to analyze how to incorporate narrative elements as storytelling elements. By presenting prototypes of storytelling in selected case studies, Figueiras presents a design study and model for narrative visualization by using storytelling techniques.

In the "How many households are like yours", users can choose the primary residents and secondary members of a household, then get the number and percentage of households. Figueiras introduces short stories describing different kinds of families instead of having only one article about types of families. This technique engages the user with a focus on creating empathy.

"What does China censor online?" is a tag cloud that only has a title and text shaped on a map of China. Figueiras introduce a tooltips pop up when a user clicks on one region, which provides more detailed information. See Figure 17. Tooltips provide additional context in the form of text which help explain the possible reasons for censorship.

Figure 17. Figueiras shows a visualization of Chinese online censorship enhanced with storytelling. An interactive feature is added so that the user can click on an instance of censorship to learn more about it. This supplies context to the user and also may draw an empathetic response from the user.

"Death Penalty Statics, Country by Country" figure is a static visualization with different size of bubbles representing the number of death sentence rulings. Figueiras designs an interaction such that when a user chooses a year, a graph displays the number of death sentences handed out that year, which provides extra temporal information and a redesign into a story.
The following Narrative Strategies are described:

1. Context: Providing context to a visualization enables the user to make sense of the data using additional information. Without a sufficient amount of context, less meaning can be derived from the data, whereas the addition of context gives the user more information to explore the data and begin to understand features found within it. This is made easier by the development of interactive visualizations and the ability for users to choose what layers of information they see.
2. Empathy: Although not often associated with information visualization, it has been found that emotive/empathetic visualizations are more memorable and more enjoyable for the user.
3. Time Narrative: Utilizing the temporal nature of data in visualization allows users to mentally map the data by adding a sense of story flow. This improves user memorability and aids in the understanding of the data.
   

Figueiras is based on previous work of storytelling and narrative visualization, and develops a model to add storytelling in narrative visualization.

Storytelling aims to simplify concepts, create emotional connection, and provides capacity to help retain information. Figueiras presents the results of a focus group study on collecting information on narrative elements. She then suggests strategies for storytelling in visualization.

Sixteen participants are asked to study 11 information visualizations of different types and different characteristics (interactive, non-interactive, introductory text, accompanying article, and audio narration). Then they are asked to rate visualizations in terms of comprehension, navigation, and likability. See Figure 18. The participants give high scores to all visualizations, particularly to interactive visualizations. The study suggests that a good storytelling visualization is well-structured and interactive with audience preferences. The results of the user study suggest that interactivity, the option of drilling-down, context, and a sense of relatability and importance for users to feel engaged.

Figure 18. Figueiras shows the visualizations used in the focus group study and the elements that compose them.

Figueiras is based on previous work of narrative visualization, and storytelling visualization. The author uses a focus group to examine storytelling effects in information visualization and storytelling visualization.

Nguyen et al. develop a new timeline visualization, SchemaLine, to gather, represent, and analyze information. They then use a preliminary study to evaluate its effectiveness. See Figure 19.

Figure 19. Nguyen et al. present the SchemLine system.

The system contribution includes: a visual design for an interactive timeline that groups notes into schema determined by the analyst; an algorithm to automatically generate a compact and aesthetically pleasing visualization of these schema on the timeline; and a set of fluid interactions with the timeline to support the sensemaking activities defined in the Data-Frame model. Their work is based on previous work of timeline visualization and sensemarking with timeline.

Narrative Visualization for Storytelling in Parallel

In this category of literature, the structure of events is layed out in parallel. The research here focuses on tools and techniques that create multiple narratives at once, in other words simultaneously. These can be useful for groups.

Information visualization systems enable users to find patterns, relationships, and structures in data which may help users gain knowledge or confirm hypotheses. The most basic element in a narrative is a character. An event occurs through the interaction of a set of characters. In this paradigm, a scene consists of a chunk of events, a story consists of a sequence of scenes, and a world model is made up of a set of stories. Akaishi et al. propose several methods for visualization of chronological data based on the narrative structure of a document. Akaishi et al. map each narrative component (world model, story, scene, event, character onto elements of a document, set of stories, sequence of scenes, part of sentence, sets of terms). The system features a decomposition unit and a composition unit. A set of stories is stored in a database by the decomposition unit. In the database, each story is divided into scenes, forming a world model. Appropriate scenes are selected and used by the composition unit to compose a new story. When a user accesses the information, the software provides the results as a story. The story is presented in various ways.

The dependency relationship among terms forms a directed graph, called a Word Colony. In a Word Colony, interdependent terms are embedded into the same node. The strength of term dependence is mapped onto the distance between nodes of terms, and term attractiveness is mapped onto the size of node. To visualize this relationship, Akaishi et al. use a spring model graph, which is a visual overview of a document. NANA represents the content of a document as a topic sequence and topic matrix. Topic sequence is regarded as the graphical plot of a document and topic matrix represents the relationships among several topic changes. Akaishi et al. support users' efforts to find desired parts of documents and to guess the context (plot) of the document.

Narrative is a simple interface that straightforwardly presents trends in keywords over time. Fisher et al. present narrative as a way of presenting temporally dynamic data. In this case, narratives help the user by tracking concepts found in news stories that change over time. Fisher et al. show how to piece together complex information and examine multiple variables, See Figure 20.

Figure 20. Fisher et al. show daily references to four US presidential candidates from 1 January to 26 March 2008. Time passes along the x-axis for each candidate; number of mentions of the term along the y-axis.

The first step is based on a business analysis task to find trends and public relations. In this case study, the requirement is to find out how a topic has developed over time and to see the evolution of the latest and most interesting stories. The system design includes data acquisition, temporal visualization, using other tools for correlation, understanding readership, and adding feature in narratives. The narratives project is based on Microsoft's Live Labs which provides real time data acquisition. Temporal visualization enables us see how a small group of words evolves over time relative to one another. By analyzing the form of correspondence and understanding readership, additional features can be added into the narrative project. Fisher et al. is based on previous work in topic detection and tracking, and temporal visualization, and presents narrative as a new technique in visualization.

Narrative Visualization Overviews

Segel and Heer state that storytelling is revealing stories with data and using visualization to function in place of written story. The Oxford English Dictionary defines a narrative as "an account of a series of events, facts, etc., given in order and with the establishing of connections between them". Heer et al. investigate the design of narrative visualizations and identify techniques for telling stories with data graphics and challenges with the salient dimension of visual storytelling. They describe seven genres of narrative visualization: magazine style, annotated chart, partitioned poster, flow chart, comic strip, slide show, and video. See Figure 21. They also discuss directions for future reader-centric research.

Figure 21. The figure shows the seven genres of narrative visualization presented by Segal and Heer. These vary in terms of the number of frames and the ordering of their visual elements. A video, for example has a strict ordering and high frame number, whereas a 'Magazine Style' poster may have a few frames in one image that are not strictly ordered. These genre elements dictate if a story is author-driven or reader-driven. Author-driven content uses a linear ordering of scenes and has no interactivity. Reader-driven content has no prescribed order to scenes and a high level of interactivity with the reader.

In the New York Times visualization on steroid usage in sports, one larger image and line chart are combined with small images, line charts, and bar charts to illustrate the usage of steroids status over 30 years. The visualization incorporates visual highlighting and connecting elements leading viewing order. The year is mapped to the x-axis, the amount of steroids is mapped to the y-axis, and different colors represent different players.

In the New York Times visualization on budget forecast, a progress bar is used to describe the accuracy of past White House budgets predictions. The time is mapped to x-axis, and budget situation is mapped to the y-axis.

The Afghanistan nation-building development project example is a interactive geographic visualization with details on-demand sliders that present the status of Afghanistan nation-building development projects. Opium cultivation is mapped to the color, and countries are shown on the map. Time can be changed from 2005 to 2009 by dragging the control bar.

The Gapminder visualization uses animated bubble charts to show possible detrimental effects on a person's ability to follow trends. Continent is mapped to color, region is mapped to each bubble, and size is mapped to bubble size, and position is mapped to average yearly income.

The Minnesota Employment Explorer shows how mouse-hover provides details-on-demand, double-clicking an industry triggers a drill-down into that sector while an animated transition updates the display to show sub-industry trends. Color represents different industries, the x-axis represents the time, and the y-axis represents employment.

Segel and Heer is based on previous work of narrative structure, visual narratives, and storytelling with data visualization and observes the storytelling potential of data visualization and drawn parallels to more traditional media. This paper identifies salient design dimensions, clarifies how narrative visualization differs from other storytelling forms and how these differences introduce both opportunities and pitfalls for its narrative potential.

Static Transitions in Storytelling for Visualization

A transition refers to the process or a period of changing from one state or condition to another according to the Oxford English Dictionary. In the visualization literature, transitions may be the focus of visualization and include both dynamic and static which are alternatives of presenting visualization. Static visualizations are those that do not rely on animation. Transitions may be considered part of narrative storytelling. However, we designate the literature here in its own category to reflect the importance of transitions and to keep related literature on this topic together. Several research papers focus on the transitions in storytelling. This is why they are separated into a special group.

In this section, the visual designs of transitions is generally static. The authors focus on presenting the trend of data along timelines. Robertson et al. evaluate three approaches of using bubble charts and attempts to discover which one works best for presentation and analysis. Tanahashi and Ma presents a storyline visualization which consists of a series of lines, from left to right along the time-axis. Liu et al. design a storyline visualization system, StoryFlow, to generate an aesthetically pleasing and legible storyline visualization. Ferreira et al. propose a method of visualizing a large amount of taxi data consisting of both spatial and temporal dimensions.

Static Transitions for User-Directed and Interactive Storytelling

The literature in this subsection focuses on interactive user-driven transitions. The user creates static transitions interactively, i.e., using a process they have some control over (as opposed to automatically).

TaxiVis proposes a method of visualizing a large amount of taxi data consisting of both spatial and temporal dimensions. This approach examines trends over time as opposed to individual taxi trips, visualizing data from a day in length, up to a year. Seasonal events such as Thanksgiving and Christmas can be compared in a like-for-like fashion. See Figure 22.

Figure 22. The (top-left) image shows the trips rendered on the map. However the cluttered view can be reduced by employing a level-of-detail approach (top right) which takes a subsample based on the order in which the trips occurred. The (bottom-left) image shows a density heat map of the taxi trips whereas the (bottom-right) image averages out the data in each region to make a regional density heat map.

Time selection widget allows the user to change the time frame of the visualization. Maps server as the canvas for the visualization. A graph of the raw data with time plotted to the x-axis and frequency of taxi trips on the y-axis. To reduce clutter, a density heat map is used. This can either be as points on the map or averaging out the data within regions on the map.

Taxi behaviour is a popular focus of research. Among others, Veloso et al. explored patterns and trends in taxi ride data looking at the relationship between pick up and drop off points. Liao et al. developed a visual analytics system to error check GPS data streamed from taxis.

Static Transitions for Parallel Storytelling

In this category of literature, the static transitions are shown in parallel. In other words, many transitions can occur simultaneously. Robertson et al. define a trend in data as an observed general tendency. The most common way to see a trend in data is to plot a variable's change over time on a line chart or bar chart. If there is a general increase or decrease over time, this is perceived as a changing trend. Robertson et al. propose two alternatives to animated bubble charts for visualizing trends in multiple dimensions and describes a user study that evaluates the three approaches for both presentation and analysis. In conclusion, Robertson et al. state that traces and small multiples work best for analysis.

The gapminder trendalyzer uses a bubble chart to show four dimensions of data, life expectancy is mapped to the x-axis, infant mortality is mapped to the y-axis, population is mapped to bubble size and continent is mapped to color.

An alternative multi-dimensional trend visualization provides the user with the ability to select particular bubbles such that the animation shows a trace line for the selected bubble as it progresses. See Figure 23. In a small multiples visualization, countries can be clustered based on position, size, and location. They are further grouped by continent and ordered alphabetically within each group. Robertson et al. is based on earlier work by Tversky et al. and Baudisch et al. Previous work is limited to small data set sizes (200 samples or less). Their work focuses on presentation rather than analysis and relies on animation to show trends over time.

Figure 23. Robertson et al. show the trace lines of the graph animation. The traces visualization shows bubbles at all x and y locations throughout the time frame. This is a conversion of an animation into a static image.

Visual Storylines, by Chen et al. is designed to summarize video storylines in an image composition while preserving the style of the original videos. Chen et al. present a new visual storylines method to assist viewers in understanding important video contents by revealing essential information about video story units and their relationships. The first step of the algorithm is to extract the storylines from a video sequence by segmenting a video into multiple sets of shot sequences and determining their relationships. See Figure 24. The second step is to visualize a movie sequence in a new type of static visualization by using a multi-level visual storyline approach, which selects and synthesizes important story segments according to their relationships in a storyline.

Figure 24. Chen et al. presents video shot clustering algorithm combines both visual and audio features to generate a meaningful storyline.

Chen et al. is based on the work of video summarization and first clusters video shots according to both visual and audio data to form semantic video segments.

Storyline visualization is a technique that portrays the temporal dynamics of social interactions by projecting the timeline of the interaction onto an axis. Tanahashi and Ma present a storyline visualization which consists of a series of lines, from left to right along the time-axis, that converge and diverge in the course of their paths. Algorithm overview is shown in Figure 25. The layout is based on a set of horizontal slots that divide the screen space along the y-axis. Each of these slots has the capacity to accommodate blocks of interaction sessions as long as they do not overlap in time. Rearranging lines takes the slot-based layout of interaction sessions derived from a genome and determines the order of the line segments in each interaction session and its alignment in order to reduce unnecessary wiggles and crossovers. In order to prevent such misleading effects, it is critical for the layout computation to include the removal of unnecessary white space to determine the final layout. Tanahashi and Ma is based on the idea of XKCD's hand-drawn illusion "Movie Narrative Charts" and develops an algorithm for general storyline visualization.

Figure 25. Tanahashi and Ma present the overview algorithm of generating storyline visualizations.

Storyline visualizations, aim to illustrate the dynamic relationships between entities in a story. Liu et al. design a storyline visualization system, StoryFlow, to generate an aesthetically pleasing and legible storyline visualization. It supports real-time user interaction, hierarchical relationships among entities, and the rendering of a large number of entity lines. The layout pipeline consists of four steps: relationship tree generation, session/line ordering, session/line alignment, and layout compaction. In the first step, StoryFlow creates a set of dynamic relationship trees for different time frames, in which the relationship trees are used to order sessions and entity lines. Next, sessions/lines between successive time frames are aligned to maximize the number of straight lines in the layout. Finally, a quadratic optimization algorithm is performed to obtain a compact storyline layout. See Figure 26.

Figure 26. Comparison of King Lear using both methods of layout; (a) StoryFlow; (b) previous method by Tanahashi and Ma. The StoryFlow layout presented in this paper focuses on minimising white space and efficiently ordering the story lines to ensure the most concise visual representation of a story. Intersecting lines represent interaction between characters and major events in the story are labeled to add clarity to the visualization.

Liu et al. is based on previous work of Tanahashi et al. Liu et al. add support for real-time interaction, hierarchical relationships, and a large number of entity lines.

Animated Transitions in Storytelling for Visualization

Gonzalez and Cleotilde define animation as a series of varying images presented dynamically according to user actions, in ways that help the user to perceive a continuous change over time and develop a more appropriate mental model of the task. The results of their study show that decision making performance is highly contingent on the properties of the animation user interface such as image realism, transition smoothness, and interactivity style, and also sensitive to the task domain and the user's experience. Values of accuracy, time, ease of use, and enjoyability for the two types of images, transitions, and interactivity styles indicated that realistic images, gradual transitions, and parallel interactivity produced better decisions. Decision making accuracy, time, ease of use, and enjoyability in animated interfaces are influenced by the form of image representation, the transition effects, and the form of interactivity. This research supports the idea that to be an effective decision support tool, animation must be smooth, simple, interactive, and explicitly account for the appropriateness of the user's mental model of the task. Gonzalez and Cleotilde review selected empirical investigations from the literature in education, psychology, and HCI which suggest that animation may make interfaces easier, more enjoyable and understandable, and study the effect of animation on decision making.

Animated Transitions for Linear Storytelling

The literature in this sub-section focuses on animated transitions using automatic, or semi-automatic approaches (as opposed to interactive techniques to animated transitions).

Heer and Robertson investigate the effectiveness of animated transitions in traditional statistical data graphs, such as bar charts, pie charts, and scatter plots. A visualisation framework called DynaVis is created to test the effectiveness of animation on the user's preference and information retention. Graph animations are used to keep viewers engaged and to promote creative thinking about the data. See Figure 27.

Figure 27. Heer and Robertson show the process of transition for a scatter plot to a bar chart. The top path starts by stretching the points to size and then moving to the right location, whereas the bottom path moves the dots first, then resizes and reshapes them.

The software displays animated transitions of statistical data graphs. Sorting and filtering animation provide the user insight into the composition of the data. All transitions take place over a time frame rather than instantaneously so the user can see exactly how the visualisation has changed. Animations between different graph types are implemented by morphing the data from one shape and size to another. Statistically significant differences in user preference were found between static graphs and animated graphs. Animated transitions can improve graphical perception. This is reflected in the findings of the user experiments testing recall and understanding. However, not all transition scenarios are found to be significantly different.

Heer and Robertson is based on the previous work of Bederson and Boltman  but builds upon it by testing different transitional events.

Animated Transitions for User-Directed and Interactive Storytelling

The literature in this subsection focuses on interactive, user-driven transitions. The user or users create animated transitions interactively (as opposed to automatically as in the previous section). Bederson and Boltman examine how animating a viewpoint change in a spatial information system affects a user's ability to build a mental map of the information in the space. Based on a user-study involving a spatial map of a family tree, animation is found to improve subjects' ability to learn the spatial position of family members within the tree without a speed penalty.

Two different family trees of nine individuals are presented to two groups people with animation and without animation. The subjects were given three kinds of tasks; navigation of family trees, exploratory family trees, and reconstruction of family trees. The speed and accuracy of performance are recorded. In this experiment, there is a statistically significant improvement in accuracy of the reconstruction task over that of other tasks. Animation resulted in fewer task errors. See Figure 28.

Figure 28. Bederson and Boltman show the ordering effects when presenting an animated and non-animated graphic. If the animated graphic is shown first then there is little difference in recall error, however, if the animation graphic is shown second then the recall error is significantly higher for the non-animated graphic. Image courtesy of Bederson and Boltman.

Bederson and Boltman is based on Gonzalez and Donskoy and Kaptelinin which address the relationship between animation and users' understanding. Compared to previous work, Bederson and Boltman focus on animation of the viewpoint. The design of the experiment is to change from a single in-between frame to several in-between frames.

Akiba et al. introduce an animation tool named: AniVis for scientific visualization exploration and communication. This tool can turn the results of data exploration and visualization into animation content and the users can create a complex animation sequence by combining several simple effects.

Parameter-space blending operator creates intermediate frames between two instances of frames I₁ and I₂ by interpolating their respective parameters. If I₁ and I₂ do not overlap in time, they generate intermediate frames by interpolating the parameters of the last frame of I₁ and the first frame of I₂. Otherwise, they generate intermediate frames by interpolating the parameters of their corresponding frames.

An image-space blending operator creates the animation content between I₁
and I₂ by interpolating their respective image frames. Similarly to parameter-space blending, if I₁ and I₂ don't overlap in time, they generate intermediate frames by blending the last frame of I₁ and the first frame of I₂. The effect is that the last frame of I₁ gradually fades out as the first frame of I₂ gradually fades in. If I₁ and I₂ overlap, they generate intermediate frames by blending.

A playback operator lets users repeatedly loop through one or more consecutive instances of interest.

A MRI head data case study focuses on highlighting a brain tumor. The animation is comprised of four pieces of dynamic content. The first is a spatial overview that rotates the volume data 360 degrees along the y-axis. The second piece is a spatial exploration in which the user customizes the view. The third is a parameter-space blending between a spatial exploration and a slicing, which reveals a tumor's inner structure. The parameter-space blending highlights a tumor by varying the opacity while zooming in on the region of interest. See Figure 29. A hurricane data case study has five components. The first is a caption showing the animation's content, blended with a spatial exploration that zooms in on the data. The second piece is a temporal exploration to show early time steps. The third is a variable overview that browses through three data attributes: vapor, wind speed, and cloud. The fourth piece is a temporal exploration to show later time steps. The fifth is a spatial exploration that zooms in on the hurricane's eye.

Figure 29. Akiba et al. show the AniVis animation tool displaying MRI scan data. By blending the two layers of data together, a new layer of information is revealed (middle image).

Akiba et al. is based on previous animation support and an animation enhanced system and develops template-based visualization tools for animation.

To explore the challenge of gradually moving from interest to insight, Nagel et al. propose the term staged analysis. Invoking temporal and theatrical notions, they define staged analysis as a carefully choreographed process of breaking up a complex whole into its component parts and purposefully preparing the manner of their appearance. In the context of visualization, the concept of staging typically refers to animated transitions broken up to be more easily observed. They build on top of this notion of staging and extend it to a guided analysis process.

As we can see, the literature on transitions is spread amongst information and scientific visualization. Table 2 shows an alternative classification of the literature divided up into information, scientific, and geo-spatial visualization. We can see that most of the storytelling research focuses on information visualization.

Memorability for Storytelling and Visualization

Memory refers to the faculty by which things are remembered; the capacity for retaining, perpetuating, or reviving the thought of things past according to the Oxford English Dictionary. Memorability is an important goal of storytelling. A good visualization technique engages the viewer's attention and increases a story's memorability.

All papers in this section evaluate the effects of visualization on memorability. Bateman et al.  explore the effects of embellishment on comprehension and memorability. Saket et al. illustrate that map-based visualization can improve accuracy of recalled data comparing with node-link visualization.

Borkin et al. develop an online memorability study using over 2000 static visualizations that cover a large variety of visualizations and determine which visualization types and attributes are more memorable. They investigate a domain at the interface between human cognition and visualization design.

A visualization taxonomy classifies static visualizations according to the underlying data structures, the visual encoding of the data, and the perceptual tasks enabled by these encodings. It features twelve main visualization categories and several popular sub-types for each category. Borkin et al. run memorability tests via Amazon's Mechanical Turk with 261 participants and gather memorability scores. The result in memorability comparison test demonstrates that there is memorability consistency with scenes, faces, and also visualizations, thus memorability is a generic principle with possibly similar generic, abstract features. The result in visualization attribute tests illustrates that higher memorability scores were correlated with visualizations containing pictograms, more color, low data-to-ink ratios, and high visual densities.

Borkin et al. show that visualizations are intrinsically memorable with consistency across people. Visualizations with low data-to-ink ratios and high visual densities (i.e., more chart junk and "clutter") were more memorable than minimal, "clean" visualizations.

Memorability for Linear Visualization

The literature here shows and tests visual designs in linear order. Users are asked to compare the visual designs (e.g., standard bar charts) verses embellished bar charts. In other words, users are tested on their ability to recall one visual design at a time in linear fashion.

Bateman et al. examine whether embellishment is useful for comprehension and memorability of charts. Bateman et al. compare plain and embellished charts, and conclude that a user's accuracy in describing the embellished charts is no worse than for plain charts and that their recall after a two-to-three week gap is significantly better.

Fourteen embellished charts are selected from Nigel Holmes' book Designer's Guide to Creating Charts and Diagrams, and converted to plain charts. See Figure 30. Twenty participants are presented a chart on a slide, alternating between embellished and plain versions. Participants are required to perform two tasks (reading and describing task and recall task) after five-minutes and after 2–3 weeks. The eye-gaze and task performance of participants are recorded for analysis. This study shows that there is no significant difference between plain and embellished versions for interactive interpretation accuracy and recall accuracy after a five-minute gap, but after a long-term gap, recall of both topic and detail of chart (categories and trend) is significantly better for embellished charts. Participants saw the value in message more often in Holmes' charts than in the plain charts.

Figure 30. Bateman et al. compare two different levels of graphical embellishment of the same data. The top graph is an embellished image but still retains the recognisable features of a bar chart. The bottom image replaces the bars with a silhouette of a person next to a drink where the height of the drink corresponds to the height of the original bar. This method also uses the addition of color to emphasize the data.

Previous studies have suggested that minor decoration in charts may not hamper interpretation, and work in psychology has shown that the use of imagery can affect memorability, but there is very little work that looks at how chart imagery can affect the way people view information charts.

Borkin et al. present the first study incorporating eye-tracking as well as cognitive experimental techniques to investigate which elements of visualizations facilitate subsequent recognition and recall. They design a three-phase experiment (See Figure 31) and evaluate the performance of recognition and recall. The conclusion includes visualizations with more memorable content can be memorable 'at-a-glance'. Titles and text are key elements in a visualization and help recall the message. Pictograms do not hinder the memory or understanding of a visualization. Redundancy facilitates visualization recall and understanding.

Figure 31. Borkin et al. design three-phase experiment to evaluate viewer performance of recognition and recall.

Borkin et al. is based on previous work on perception and memorability of visualization and eye-tracking evaluation visualization.

Memorability for Parallel Visualization

In this subsection, users are presented with a large number of relation data in parallel (as opposes to one at a time). Users are tested on their ability to process relationship data in parallel (all relationships simultaneously). This is distinct from memorability for linear visualization where recall focuses on one visual design at a time in linear order.

Saket et al. illustrate that different visualization designs can effect the recall accuracy of data being visualized. Compared to a node-link diagram, a map-based visual design is more effective.

Two datasets are examined. A book dataset (small) and LastFM dataset (large) are transformed into a node-link diagram and node-link group (map-based). See Figure 32. Three phrases are performed to examine the difference between node-link diagram and map-based visualization. In phase 1 participants examine two kinds of visual design without task with unlimited time. Phase 2 asks participant to study two kinds of visualization with six tasks in a required time. Phase 3 asks participants to recall what they read in phase 1 and 2, complete 6 tasks similar to phase 2, and 3 new addition tasks. The result of the experiment illustrates that recalling map-based diagrams is more accurate than recalling node-link diagrams, but no faster. The participants spent more time on map-based visualizations than node-link visualizations.

Figure 32. Saket et al. show two visualization of the same data: node-link diagram and map-based diagram.

Saket et al. is based on previous work of visualization memorability and a recalling experiment.

Unsolved Problems and Conclusions

This survey provides a novel up-to-date overview of storytelling in visualization. The most important recent literature is included and discussed. Since storytelling in visualization is a recently new subject, we expect an increase in research in the coming years. Moreover we believe it will evolve into a popular topic in the field of visualization.

By reviewing Table 1 and Table 2, we can see storytelling visualization focuses on information visualization more than scientific visualization, which conveys that more challenges are left unsolved in this field. However, by refining a storytelling model for scientific visualization, the implementation of storytelling in scientific visualization could increase in the future. We can also see that storytelling in visualization concentrates more on exploration than on presentation. Like Kosara and Mackinlay state: "visualization techniques address the exploration and analysis of data more than presenting data".

In future work, there are many directions and unsolved problems. Storytelling will gain importance in data presentation and data exploration. Here is a summary of some unsolved problems in storytelling for visualization.

• It is clear that objective measures of user-engagement is a relatively unexplored area of research. Can we derive a mature classification of user engagement activities? Is user engagement something we can clearly define?
• Data preparation and enhancement: Virtually no one has addressed the challenge of data preparation and enhancement for storytelling. Moreover, is storytelling data best captured or derived from an existing data set or software system? Can a standard data file format be developed?
• Narrative visualization for scientific and geo-spatial visualization: Why has there been such an imbalance of research narrative visualization for information visualization but virtually none for scientific and geo-spatial visualization.
• Transitions for scientific visualization: The benefits of static transition versus dynamic transitions in visualization still remains relatively immature.
• Memorability for visualization: What are the key elements for making a memorable visualization? This is still an immature research direction.
• Animated transitions for geo-spatial visualization: Animated transitions for geo-spatial visualization remains an open research direction. This is surprising given the popularity and importance of geo-spatial visualization.
• Interpretation for scientific information, and geo-spatial visualization: Currently no papers to our knowledge focus on the topic of effective interpretation of stories, this topic remains largely unexplored.
  

The classification of literature, we present makes it clear that many future research directions remain open in storytelling and visualization.