literature review
“As far as we know no (or very few) research has been held about immersive reading in a digital environment.”
—Columbo, 2011
My academic research looked at the design research surrounding digital books and transmedia storytelling. Design research on digital books has focused almost entirely on extractive reading, reading to learn or extract specific pieces of information. On the other hand, Transmedia storytelling, at least from a popular perspective, has been particularly effective in engaging fans in an immersive story experience. I think my contribution can lie in-between these two. By using methods and ideas from transmedia storytelling and applying them to the digital reading experience I hope to build an experience that immerses readers even deeper in a story.
Early Lit Review
My literature review started wide. I was particularly interested in transmedia storytelling and the media and cultural phenomenon surrounding it. As I narrowed my focus and scope I looked more thoroughly at digital reading and specific case studies and projects within commercial and design literature.
Transmedia Storytelling
I also used my literature review to solidify and detail my own understanding of transmedia storytelling from a variety of experts including Henry Jenkins, Cristy Dena, and Frank Rose.
Hybrid book concepts
Along with digital books and transmedia storytelling, I am interested in current work with hybrid books. My primary research showed how much readers are still attached to the analog reading experience. Therefore, I looked into how researchers were currently working to bridge the gap between physical books and digital augmentations. A number of interesting concepts emerged.
Click on the images to view details of each project.
survey of iPad fiction
I completed a survey of enhanced fiction apps on the iPad. I looked at stories that were designed exclusively for the digital format and those that were originally published in text and were ported to a digital format. I think the opportunity area here, especially looking at immersive reading experiences will be in using augmentation to enhance the story and story world. For my prototype I will in all likelihood use an existing text, but I would hope the solution could be expanded and used to create unique works as well. I want to stay away from gamifying reading because, while and interesting experience, I think it can take you out of the story, and the book becomes more about completing tasks and playing rather than fully understanding and immersing yourself within a text.
Review Process
I chose to focus on adult fiction because a considerable amount of commercial and academic research has already been completed around non-fiction and children's books. I first completed a broad survey to identify what types of enhanced books are published in the iTunes store. Of these apps I chose five native iOS stories and five books originally published traditionally and released as enhanced editions for the iPad to review further. I found books to review primarily through iTunes searches, Kirkus Reviews, and searches on technology websites like Wired.com.
I collected the most recent 100 reviews of each book app from iTunes, open coded the reviews, and used grounded theory to draw conclusions. For apps that had fewer than 100 reviews I collected all the reviews that were available.
survey + interviews
“When I read, I'm transported away to another world. I get high off this—figuratively, of course.”
—Anonymous survey response
Survey
My survey was set up with four main sections: demographics, reading habits, likert scales to rate features that could be included in an interactive or transmedia book, and finally open ended questions.
The data that emerged from the survey was really helpful, especially the long from responses to the questions, "What is sacred about your reading experience? What would you hate to lose?" and "What would it be awesome to do with enhanced digital books?"
Interviews
I conducted 10 semi structured interviews to learn how participants currently read and interact with books, and how they would prefer to interact with multi-media or interactive books in the future. I kept the study purposely vague and asked participants to talk about their reading habits in general: fiction, non-fiction, long form, short form, for work/school, or for pleasure. This allowed me to get a broad range of responses from people who may interact with books differently.
Card Sorting
I brought cards with me to each interview session, each with a different way in which an interactive or multimedia book could be augmented. Participants were asked to sort the cards on a matrix with two axises, from boring --> really cool, and I wouldn't use this --> I would use this. It was great to hear people talk about why they wouldn't and wouldn't want stories to be augmented in different ways, and see which cards people agreed on, and which were polarizing.
Media Use
Each participant was asked to map their media usage throughout the day. They annotated this worksheet, and thought aloud as they completed it. I asked participants to describe why and on what devices they complete each activity throughout the day. This activity gave me a better understanding of the context in which they read, and how reading activities compare to those surrounding other media.
synthesis
Affinity Diagraming
Over the course of two days I affinity diagramed all of my interview and card sorting data and used that to form initial findings. The video above is a three minute time lapse of my process. Along with this exercise I complied my survey data, commercial review findings, and literature review. All of this research went into forming the findings and principles detailed below.
Reader Motivations
Three different motivations for pleasure reading emerged from my interviews and card sorting. Some participants were also required to read for classes, that motivation was not included in this framework as it was required reading and not chosen for pleasure. Unlike traditional user types, people often fall into multiple categories based on their motivations from book to book. Therefore I am referring to them as motivations.
PREFERRED Enhancements
This was by far the most difficult framework to put together, and to this day I think the most unclear. It aims to represent how interested participants were in each category of enhancements. The categories relate loosely to the reader motivations, but do not match up exactly. Two main points I would like to call attention to are that "Play" features were almost always qualified as needing to be in service of the story, which is why play exists entirely within immerse. And social features were not particularly interesting to card sorting or survey participants.
Design Framework
I use the following framework to help define my concepts. They can also be used to define existing books/apps.
The Delivery
Is this book contained within a single object, as are current books, or distributed like transmedia stories—across platforms and media?
The Object
Is the object on which the user is consuming the story digital or analog? An app, an augmented physical book, or a hybrid of the two mediums?
The Story
Has the story been crafted to be experienced in an augmented or transmedia way? Or is this a system or platform by which we could augment stories that have already been written and traditionally published.
prototype
Second screen Prototype
For my experiential media class I decided to take a stab at prototyping a hybrid physical digital reading experience. The prototype was based loosley on findings from my research, primarily that there are still a large number of people who prefer to read analog books and that nearly all of my participants in my card sorting and survey indicated they were particularly interested in interactive story visualizations like maps and character charts.
I decided to build a second screen experience inspired by the work being done with tablets and TV interfaces. I wanted to provide context relevant information based on the readers current page. I thought that the best way to do this was with a physical book and tablet interface.
The prototype used Makey Makey to recognize pages in a copy of The Hobbit and update a second screen interface with context relevant content, namely a map that kept track of the location of the characters and updated with key plot points as the story progressed.
Prototyping this system, while I received good feedback, also helped me realize that a second screen is likely not the best way to deliver augmented content for a physical book. While the content was well received, it quickly became apparent that it was awkward to juggle two devices (a book and a tablet) while reading.
poster presentation
Poster Presentation
At the end of fall semester we had to put together a poster presentation of our research to date. I wanted my poster to give a clear picture of the breadth of research that I conducted, and demonstrate the direction I would be taking next semester. I received positive feedback on my poster design from my my reviewers, and feedback that will help me move forward in the spring.
After the presentations we had an open house where the public could come in, look through the work, and ask questions. I think this was even more productive than the presentation itself. It was much more laid back and I got good feedback on my prototype and research. I really wish I had brought business cards because there were a couple of people I probably should have tried to stay in touch with. Oops, I will remember next time.
initial concepts
Concepting
From my research and poster feedback I generated a number of concepts. The top four are listed below. I took these concepts to readers to get feedback and narrow these down to the concepts I prototyped.
Augmented E-book
This concept takes the ebook format to the next level. Supplemental material is available from every page, layers of annotations from the author or fans can be overlaid on the text, and the book provides custom summaries if you haven't read in a while.
Book light projector
A micro projector clips to the top of your book and project supplemental content into the margins of a custom printed novel. Projections can be story material like maps or character charts or you can upload, download, and share custom layers from other readers.
Serialized Transmedia
This episodic story is released chapter by chapter. In between releases readers can sign up to receive updates from a variety of additional sources like characters or fictional news outlets, keeping readers connected with the story even when they are away from the core text.
Social Interactive story
This story is designed to be shared with friends. When a group signs up, each reader selects a perspective through which to read the story. As the story progresses it unfolds like a mystery game, readers are given choices and build the story together.
Projection Mapping Prototype
My projection mapping prototype primarily responds to my research nding that many readers prefer the experience of reading an analog book to reading on a tablet.
Feedback on my second screen prototype indicated that having two devices led to a strange juggling experience while trying to hold a physical book and a mobile tablet. Projection mapping remedies this by combining the digital content and physical book into one hand held device.
Book Design
The book was custom designed to enable appropriate additional content to be shown on each page. Most pages had large top and side margins, while a couple had a full page of text on the left side of a spread, and an empty page on the right hand side. This allowed custom content to project comfortably into the margins.
Physical Installation
Building the physical installation was an exciting challenge for me. I have done little physical prototyping before this project but am very interested in designing and prototyping hybrid physical and digital systems.
The prototype was built as a table-top installation to minimize complications and avoid having to track the book in space. My nal installation used a webcam mounted to the side of the table to read a bar code on the book, and a pico- projector mounted above the table in a lamp shade to project the supplemental content. Both devices were connected to a laptop under the table running processing.
Content Development
After getting the page recognition working I developed the content to be projected. I wanted to demonstrate how di erent types of content could be displayed but didn’t have the time to build in interactivity that would allow readers to switch layers. Therefore, I used different pages of the book to display di erent layers. I developed pages for most of the layers I included in the mobile prototype. Illustrations, new character introductions, audio, publisher annotations, and a new layer, hand written notes shared between two friends.
Illustrations
Audio played automatically when users came to a page that included a song.
Shared conversation with friends
Publisher Annotations
Layered Library Experience
About the Layered Library
The Layered Library platform layers crowd generated multimedia content onto existing books. This relieves the burden of augmentation from publishers, and engages fans and readers with their favorite works.
Layers are collections of multimedia content tied to specific locations within a published text. These layers can be anything from fan illustrations shared with the world to a book club’s private discussion of di erent parts of a story. Layers can take the form of comments and annotationson a text or deeper information about the story such as character or location descriptions.
Fans, scholars, authors, and publishers can all contribute layers to a text and readers choose only the layers that interest them, enabling each reader to have a customized experience of a story. Many fan communities currently exist to talk about their favorite stories and franchises.
By tapping into this collective intelligence and tying the information to speci c locations within a text the Layered Library makes the wide variety information generated by fan communities accessible to all readers exactly when it is relevant.
The Layers
Library
When a reader opens the application they can see all of their imported books, as well as noti cations from friends and options to import or download new books.
Book details
Traditional book details including author, title, and synopsis are displayed, as well as all of the layers a reader has downloaded or created. From this page readers can download more layers, or create and share a new layer.
Layer details
Each layer includes details of its author, content and rating. Layers can be downloaded, shared, or recommended to friends. Premium layers, created by publishers or brands, could also be purchased.
Annotation layer
Annotations appear to the left hand side of the text, the background can be color coded to represent which layer is being shown. When an annotation is opened, the text in the story that it relates to is highlighted.
Graphic Layer
Graphics rst open like annotations, but readers can tap on the thumbnail to view a full sized gallery.
Character annotation layer
Character annotations give basic character details and biographies. The reader can also tap to search for the character in Macro View and see on which pages they appear.
Audio layer
Audio layers open simple audio controls at the top of the page which are static even as the reader turns the page.
Macro view
An experimental view of the entire book, readers can see which pages have annotations, or where major plot points and battles take place. This view can also be used for quick navigation.
Horizontal reading mode
When reading in horizontal mode, annotations appear next to the text and do not obscure the map.
Interactive Map
The interactive map updates as a reader moves through the book, tracking the location of characters and the locations where major plot points occur.
Evaluative Research
Journal Study
My mobile prototype was the focus of my evaluative research. Because it is the most throughly designed and developed, I was able to evaluate it for desirability, viability, and usability. My physical prototypes were evaluated informally through demos and group feedback sessions.
To evaluate my mobile prototype I provided five research participants with a working prototype of three chapters of The Hobbit and 6 different layers. They were given a journal study to ll out asking them to reflect on their experience with the prototype and then participated in a 30 minute post-use interview where I asked in depth questions about their experience and how they could see the system scaling in the future.
Research questions
- Is the Layered Library a desirable platform?
- Which layers did readers find most engaging and interesting?
- Which layers did readers interact with most and/or for the longest time?
- What other types of books would readers want to read in this system?
- What other types of layers would readers want to add to books?
Overall Design Feedback
Overall, reactions to the system were overwhelmingly positive. Every participant said they wanted to keep updated on the project and that they would be interested in continuing to read using a system like this, most said they would pay for the application. This, more than anything, gives me confidence that I designed an effective and desirable system. However, a few high level issues negatively affected the experience.
The inclusion of layers brought a level of distraction to the reading experience that I had hoped to avoid. One participant explained it well, “Sometimes I skipped reading in my curiosity to see what [the layers] said—then I’d go back.” It was never my intention to invite distraction into the reading experience, although, in hindsight, I am not sure how I imagined I wouldn’t when I pulled additional content into the system.
When designing, I thought, that by minimizing the visual representation of additional content, readers would be less distracted while in the ow of reading. However, this didn’t seem to be the case, multiple participants told me in their post-use interviews that they opened up every annotation, because they were curious to see what it would say.
One participant recognized the problem and proposed a solution for herself, “I wonder if I used this system for real if I would feel too pressured to look at the annotations right away. I might turn them o and then review them at the end of each chapter.” I wonder if a feature like this could be incorporated into the interface, for example, asking a reader when they start a new book if they would like to see annotations in-line, or at the end of each chapter.
In this prototype, I included a feature that would allow the user to hide and show layers, but none of the participants used it. I would like to spend some additional time conceptualizing and designing different ways to make interacting with layers less distracting, especially for users who are reading a book for the first time.
The nal big picture issue that was raised centered around finding layers to download. While this was not an explicit part of my prototype, the question came up during one of my participant interviews, as well as during my faculty poster review. Currently, layers are displayed in a single group, and, while I have not yet designed the layer search experience, it became clear that layers would need to exist within some categorization structure.
As I demonstrate the system for more people, it becomes more and more clear that the possibilities for this system are ever growing. This will make organizing layers both challenging and of utmost importance. I am not sure yet what this experience will look like, but the questions and feedback I received will deffnitely help me design that use case.
Conclusion + Reflection
This project was an immense undertaking and an unparalleled learning experience for me. I learned about reading, about design, about prototyping as well as countless other topics. But along with my own learning, I believe that this project and the process that I followed has generated a number of unique contributions and could impact how we think about reading, content generation, and the development of reading experiences.
The solution itself represents a new way to read books, augmented with an unending variety of content. The content production and distribution model, a platform that leverages the collective intelligence and artistic talent of fans, scholars, publishers and authors, is unprecedented in current digital literature. And finally, my reader centered design process provided insight into how readers engage with books now and how they would like to engage with them in the future, leading to a far different outcome than I may have generated using a traditional authoring process.
This project was quite an adventure. I didn’t really know where I was going to end up, but I am happy with both the journey and the destination. Despite moments (or weeks) of self doubt and dissatisfaction with my work I made it through with a project I am very proud of. Honestly, it is not the project I thought I would end up with, but I guess that is a testament to the design process and staying con dent that everything will work out in the end, as it did.