The NWP Multimodal Assessment Project

Few assessment areas have been honed and refined as much as writing assessment. Not that writing assessment is free from controversy—far from it. But in comparison with other focuses for assessment—say critical thinking or creativity—writing assessment has its share of well-worn procedures, rubrics, designs for scoring, and sense of priorities.

But many of these refined processes are challenged by new kinds of texts, especially multimodal texts that may combine modes like text, image, sound, and gesture or collaborative texts that leverage the Internet to include the contributions of many authors through comments or collaborative writing.

So how might our ways of talking about text change if we think about assessment of multimodal texts? To explore this question, the NWP created a Multimodal Assessment Project Committee (MAP).

The MAP Committee, co-chaired by Carl Whithaus of the University of California, Davis, and Elyse Eidman-Aadahl of the NWP, began its work in Fall 2010.  Members include Tara Porter, UC-Davis; Sandra Murphy, UC-Davis/Area 3 Writing Project; Kris Blair, Bowling Green State University; Joe Wood, Area 3 Writing Project; Chuck Jurich, High Desert Writing Project; Lanette Jimerson, Bay Area Writing Project; Will Hochman, Southern Connecticut State University; Becky Rupert, Hoosier Writing Project; and Danielle DeVoss, Michigan State University. The effort was supported by the John D. and Catherine T. MacArthur Foundation’s Digital Media and Learning project.

In this resource, we hope to share our efforts…meeting by meeting…and invite your comments and contributions. In the navigation box at the left, you will see our work develop over the growing pages of the resource, each with its associated discussion thread. We welcome your contributions and encourage you to follow this resource for news and updates about our work.

Exploring Ideas about Multimodal Assessment at the 2010 NWP Annual Meeting

After exploring the dimensions of the problem in our Fall 2010 meeting, we invited members of the NWP community to join us through a session at the 2010 NWP Annual Meeting in Orlando, Florida.  In the session, participants looked at a range of multimodal student products in relation to two typical and popular rubrics for writing: the 6+1 Trait Rubric and the Washington State Writing Assessment Rubric. Participants at different tables were given different pieces of student work and asked to think about how the rubrics did or didn’t speak to important areas of practice for the kind of writing the student was attempting. The student work covered the ground from Voice Threads to digital stories and short videos to non-digital compositions that leveraged text and image. Some of the work we looked at is linked below, although we also looked at several pieces that were not digital and are not available for linking.

In general, participants noted that the rubrics named areas of writing performance that were still very relevant for multimodal texts, but in each case very important areas of practice were invisible. Also, the language used on the rubrics was ‘text specific” and therefore didn’t seem to work for kinds of texts closer to film or visual arts. We wondered how we could create a language that cut across areas with different artistic traditions.

At the session we invited participants and others to share their thoughts after the session through the discussion at the left.

Cross-walking Between Frameworks and Student Work

After our work at the NWP Annual Meeting in Orlando, the MAP Committee met in Davis, California in February 2011. Our goals were to review our overall assessment framework in light of feedback we’ve received and to reflect on the implications of design in tools like rubrics and graphic representations.  What do different designs signal about what is valued in an assessment context and how threads of value inter-relate?

Reviewing Our Assessment Framework

We looked together at the feedback we received on the emerging framework that we shared at the NWP Annual Meeting. There was much rich discussion at the Annual Meeting, and the committee got a real sense from the group how teachers would want to participate and use all or part of our work. That led to some refining of the domains, or areas, in our framework.

At the February meeting we focsued on five consensus domains that many felt were critical for assessing and improving our work as creators of multimodal texts. We asked ourselves, how can we grow and develop over time as creators of these texts. The domains below were our answers (with names that may change):

• Context: we can grow in our awareness of and attention to the context/situation in which we are working and for which we are composing. This includes the rhetorical context of audience, purpose, and occasion; circulation context of the distribution environment; expectations for form and genre; etc.; the assignment if there is one or expectations of a discourse community
• Artifact: we can grow in our capacities to design and produce the pieces themselves, including developing our capacities to manage the technical elements and affordances in the medium of composition, and to use them for effect in ever more powerful ways
• Process Management: we can grow in our abilities to plan, implement, and assess our work; to find and manage resources and digital assets that we use in composing; to reflect on our performance and to collaborate effectively (particularly given the fact that so many multimodal projects are produced by teams)
• Substance/Content: we can grow in the quality and sophistication of what we are communicating in our work; we can improve the quality and power of the ideas or content, the credibility of the information, the depth of the story or argument
• Habits of Mind: we can develop, in an ongoing way over time, dispositions and capacities that will serve us well in life as a productive and creative individual.

Looking at Student Work

As is typical for the Committee, we put the framework up against a collection of student work to see how it would hold up in dialog with student work. We looked at Google Lit Trips produced in Google Earth by elementary students in a class setting; we looked at student films by elementary-aged youth in an after-school program; we looked at digital stories by high school students presenting their digital lives to teacher education students; we looked at a video remix taking on last summer’s Levi’s ‘go forth’ commercials; and a delightful set of deliberately bad Powerpoints. We also looked at a Twitter stream where students responded to The Things They Carried in character and a set of student posters and Shutterfly books.

This video remix, linked below, was created a college-aged student who wanted to speak back to the Levi’s commercials that were airing the summer of 2010. In this remix, the student strips out the audio, adds a found recording of Tuli Kupferberg reading his poem “State Of The Union”, and adds highlighting of elements of the text over the images. To see his video, follow the link at the bottom of the page. It was the only public student product, so it’s the only one we can share with you. Enjoy!

Planning to Test the Framework in Practice

The MAP Committee came together in April for a meeting at CCCC in Atlanta. Our goal was to network with others at CCCC and to refine our framework. Continuing to build on our meetings, the feedback we received at the NWP Annual meeting, and our review of student work, we felt we had developed a sufficiently solid working framework that we could take it out into practice for testing. In developing our framework, we believed assessment could and should address recurring elements of multimodal composing. We identified five recurring elements: context, artifact, substance, process management and technical skills, and habits of mind.

1. Context is the world around the artifact, around the creation of the artifact, and how the artifact enters and fits into the world. Authors-designers-writers attend to the context of a multimodal artifact when they make decisions about genre, along with considerations of where and how communication enters this world. Authors-designers-writers consider the constraints, affordances, and opportunities, given purpose, audience, composing environment, and delivery mode.
2. The artifact is a consumable product that can stand-alone. It contains a coherent message with a clear focus created through an appropriate use of structure, medium and technique. The artifact identifies the connection between resources, composers, and ideas. The artifact may demonstrate habits of mind such as innovation, creativity, and critical stance.
3. As a domain, substance accounts for the overall quality and significance of the ideas developed as writer-designers and audiences experience an
   artifact.  The substance of a piece is related to an artifact’s message as well as the contextual elements of purpose and audiences.  However, the domain of “substance” encourages evaluators to consider elements beyond artifact and context.
4. Process management and technical skills refer to the skills, capacities, and processes involved in planning, creating, and circulating multimodal
   artifacts.  Creating multimodal products involves the technical skills of production using the chosen digital tools, but it also includes larger project management skills as well as the ability to collaborate with others in diverse situations.  Over time, individuals learn to more effectively control the skills and processes of producing and circulating digital content.
5. Habits of mind are ways that writer-designers engage in active learning.  These habits can be cultivated both inside and outside of school.  Habits of mind are patterns of behavior or attitudes that reach beyond the artifact being created at the moment.  They develop over time and can be nurtured through self-sponsored learning as well as teacher-facilitated activities.  Examples include
   creativity, persistence, risk-taking, mindfulness, and engagement.

Sharing the Framework

As with any framework, ours notwithstanding, its benefit to a wider audience is only theoretical until the framework is shared with educators and students engaged in multimodal composing. Elyse lead the path to sharing our framework with educators at the Digital Youth Network in Chicago. The discussion resulted in a critical element not addressed in our current working framework: How do we assess performance techniques such as speaking skills or musical skills in multimodal composing? Several multimodal products, such as digital storytelling and voice threads, require composers to perform with their voices. Composers record their voices multiple times in order to produce the appropriate pronunciations, inflections, flow and tone.

Over the summer, we planned to engage in conversations with summer youth programs, professional groups, and policy makers to see if this framework of domains included the range of areas of assessment interest that our publics name. Our framework is a guide and as such, some committee members will engage in conversations about the entire framework, while other members will discuss one or two elements.

We welcome you to join the discussion. How would you use our framework to discuss the development of a multimodal project? Are there elements that need to be added? Where does performance fit within our framework?

Beginning Our Study of Multimodal Composing

We began our work in October 2010 when we held our initial meeting in Davis, California. After reviewing our charge from NWP and the MacArthur Foundation, we began with what seemed to be a very simple problem:  How do we talk about the array of multimodal products which authors are creating today?  How do we create a shared language for talking about what is valuable in digital videos, VoiceThreads, Word docs, websites, twitter streams, and the other very diverse sets of texts?  We wanted to create a framework that would account for, honor, and accommodate these new ways of composing texts.

We began trying to answer these questions by applying existing writing assessment rubrics to some sample multimodal texts.  We used Washington State’s Content/Organization/Ideas and Conventions Rubrics (HsCoSSS), NWP’s Analytic Writing Continuum (AWC), and the 6+1 Trait rubrics.  We found that although elements of these rubrics translated to the new products, the digital videos, VoiceThreads, and kinetic type Flash productions we looked at could not be fully assessed using those rubrics—parts of those traditional writing assessments did apply, but an equal number of categories did not work.

We continued our work by reading and responding to pieces by the New London Group, Cheryl Ball, Anne Wysocki, Paul Prior, Anne Herrington, Charles Moran, Eve Bearne, and Carl Whithaus.   We were also able to look at the page-proofs from Because Digital Writing Matters. We used these readings to see how teachers and researchers were talking about multimodal writing.

Finally, we wanted to recognize that there were other “fellow travelers” at work on questions of how do we assess and value multimodal writing.  We examined the TDR Final Film Rubric, Digital Storytelling Rubric, and Bearne’s What Does Getting Better at Multimodal Composition Look Like.  These rubrics captured more of the multimodal aspects of the compositions we were examining, but they did not fully describe the dynamic processes of composing the student samples that we were looking at.  The meeting ended with a shift from creating a rubric (or series of rubrics) to creating a framework or set of domains.  We were not looking only at assessment but at the interactions among assessment, curriculum, and pedagogy.  We wanted to create tools that would ultimately enhance learning.