Developing Leadership and Site Capacity Through Program Evaluation and Research

At the South Coast Writing Project (SCWriP) in California we found that our research with NWP’s Local Site Research Initiative increased our site’s capacity for doing research, as expected. What we didn’t expect, however, was how much our research helped us to more effectively fulfill our primary mission of supporting teachers—in a singular way that couldn’t have happened by reading and applying others’ research.

During the three years of the study, our group—which consisted of site directors and teacher-consultants of the South Coast Writing Project and doctoral students at the University of California, Santa Barbara (UCSB)—often found ourselves surrounded by multicolored whiteboard notes, transcripts of teachers’ comments, and samples of student writing. Some of us might not have called ourselves “researchers” to begin with, but we were all researchers in the end.

We were examining the effects of our professional development program IIMPaC. “IIMPaC” is an acronym for the program’s five basic elements: inquiry, inservice workshops, models, practice, and coaching focused on the teaching of writing. The program operates in partnership with low-performing schools that serve low-income populations with a substantial proportion of English language learners. The study compared data from teachers participating in IIMPaC with data from nine teachers and their students from matched schools—surveys, interviews, classroom observations, and collections of teacher and student work—to assess the effects of participation in IIMPaC on teachers’ classroom practices and their students’ learning.

We were especially interested to see if the practices presented in the program actually showed up in the classrooms of the teachers who had participated.

• Was writing celebrated in the classrooms?
• Was there emphasis on revision?
• What instructional techniques and strategies were used?
• How much time was spent in lectures, groups, pairs, and individual activities?
• Were recent examples of student work on display?

One of our initial conclusions as we worked through the data was that some of the practices we would expect to have emerged were proving difficult to see. For example, while we saw a teacher describing writing as a process, the teacher’s revision activities focused more on grammatical correctness and editing than on developing a richer piece of writing. This insight resulted directly in moves to improve the staff development program’s next session—making more explicit the practices we would hope to find in the classrooms of the teachers we were serving by, for example, including techniques that illustrated more clearly the distinction between revision and editing.

Instead of relying on someone else’s “research-based” knowledge to guide our practice, we gained insights that helped us improve our work in local area schools from the knowledge generated through our own research.

Learning the Research Process

In addition to strengthening the professional development we were offering, through our participation in the research process we were developing valuable capacities as individual teacher-consultants and as a local site. Research team meetings like the one mentioned above formed the heart of our collective work: we read transcripts of interviews; discussed common threads; designed interview protocols, surveys, and classroom observation guidelines; and argued among ourselves about the significance of what we were seeing.

At these meetings we worked our way through five major stages of the research process:

• logistics and planning
• research design
• data collection
• analysis
• presentations and publications.

At each phase, we found research to be a messy and delightful process; through trial and error we learned better what to look for and how to look for it as we devised, with generous support from the Research Unit at the National Writing Project, better tools for our research.

Specifically, we learned to

• develop and refine a logic model
• select good writing prompts and design and administer writing pre- and post-tests
• create, modify, and use research instruments
• select and collect student and teacher documents.

Because we were part of a national initiative through our participation in the Local Sites Research Initiative (LSRI), teacher-consultants from our site who had not participated in the initial research study were also able to participate through attending NWP’s national scoring conferences held in Denver.

We did a lot of writing too, taking accurate field notes and writing those notes up for other members of the team to review, as well as writing grant applications and major research reports for the National Writing Project.

Finally, we gained a deep appreciation for the importance of and the difficulties involved in including a comparison group in a study. Not only did we pay stipends to teachers who agreed to be a part of the comparison group, we also offered staff development workshops to their schools free of charge following the completion of the study. Thus their participation in the research became a first step in our building an ongoing relationship with more teachers in our area.

What Active Participation in the Research Means

We found that conducting this systematic and reflective inquiry, while demanding, was remarkably consistent with the frameworks we share with our students and the teachers we serve. As teachers we believe that the best way for students to learn is through their own inquiry, finding answers through active participation in the research and the writing processes; further, the delivery of information from experts can never generate the kind of learning experiences we desire for our students or ourselves.

Perhaps more important, we found that research could be conducted in ways that were consistent with our most important values as NWP teacher-consultants: namely, enhancing the professional standing of teachers, fostering trust and long-term relationships with local teachers and schools, and subordinating the research agenda to the welfare of the kids and the job the teachers have to do. Our experience showed us that we could indeed conduct research while staying focused on these values.

Of course, conducting applied, basic, and teacher research is nothing new to the NWP. Every writing project site works to measure growth in writing in some way, and, according to Miles Myers in The Teacher-Researcher: How to Study Writing in the Classroom, as early as 1978 NWP publications were supporting teacher research and the study of writing in classrooms. In the present educational climate the NWP’s university-school partnership model provides an opportunity to conduct valuable research in schools; however, many site directors and teacher-consultants would not identify themselves as researchers, even though they will, or perhaps even now do, need to begin thinking about program evaluation.

Taking part in the LSRI provided us with opportunities to present our work and findings at annual meetings of the NWP, and at meetings of the Urban Sites Network. Even now, two years after the completion of our study, the work continues to bear fruit: the research team has continued to work together to write articles about what we learned. For example, the publication of an article—“Beyond Strategies: Teacher Practice, Writing Process, and the Influence of Inquiry” in NCTE’s English Education—provided SCWriP’s staff development director with additional evidence, to share with administrators in our partner schools, of our site’s strong connection to UCSB and our commitment to scholarship and reflective practice.

Taking part in this work can be a gentle way of including teachers and teacher-consultants in research conversations through active participation as partners, while also increasing the effectiveness of the professional development services we provide.