Many of my twitter followers have been very interested in the work we are doing the New Literacies Research Lab here in Connecticut. Currently we are working on an IES grant to develop valid and reliable measures of online reading comprehension. So I have been asked to document a few of the challenges and obstacles we face.
Synthesis
I am sure by now most of you are familiar with Leu's model of online reading comprehension: Question, Evaluate, Synthesize, and Communicate. Of these skills synthesis has always been the hardest for us to measure. This plays out in both anecdotal evidence and our data. First how do you make evident something that happens in the head (for you cognitive folks) or in the act of doing ( for those more situativley inclined)? Second in all of our factor analytic patterns we have not developed a model that separates synthesis from communication. As we begin our cognitive labs of our items we are determined to get a measure of synthesis.
A Little Background
I figure we can start with Bloom. When in doubt with assessments its a great place to start. Bloom and his crew (1956) placed synthesis among the higher order thinking skills.It had to do with the assembly of knowledge: putting parts into whole. When Krathwol revised the taxonomy his team renamed synthesis create and moved it to the top of the pile.
I like that it where it belongs. Nothing makes me cringe more when teachers equate synthesis to citations. It is an act of creation. You take multiple streams of information and combine them into something new (Thanks NCTE definition of 21st century literacies I really enjoy that phrase. For us this has been especially hard to capture in a comprehension assessment.
I recently sat in our Scientific Advisory Board meeting and got to listen in as Spiro, Pearson, Kirtsch, and Klienman had a lively debate on synthesis. They all agreed it wasn't simply finding detail A and detail B to make summary statement AB. That was way too old Bloom. The SAB wanted synthesis to look more like A says this B says this so therefore the answer is C.
Thus the act of creation. Of course what isn't accounted for in this model is prior knowledge and unique experiences people brind to knowledge assesmbly. Rand Spiro kept reminding us of this point. So many times new knoweldge comes from such non-linear paths. What we know is often out of happenstance. Once again how do you measure this in an assessment of online reading comprehension.
Preliminary Results
In our first round of cognitive labs we immediately noticed the difficulty of measuring synthesis. We started with one screen in surveymonkey that had students take notes on all the websites they found and then combine them into one summary sentence. Kids hated it. They wanted to take notes as they searched for info or judged websites.
So we encouraged notetaking throughout the task and just made synthesis a final statement. The problem this time was brevity. Their summary statements were short, but their final communication showed evidence of them integrating many details they read. Also if we scored synthesis in the final communication some students who could combine ideas, and make them their own ,might lost points for not being able to use a blog, wiki, email or discussion board (our communication tools). Also what about prior knowledge? Should students not get a scorepoint for using what they already knew? So we needed to change the model again.
Our Latest Iteration
The scientific advisory board also suggested we needed to push the social aspect of our assessment and make it an authentic web experience (I would argue taking notes and using that information is authentic, but that's for another time). So we tried to accomplish two things at once: increase authentic task and embed synthesis. In our next round of cognitive labs we are trying three new ideas: a testlet embedded in instant message, a testlet that uses both surveymonkey but uses instant message for synthesis, and then a third version with your standard Word Document for taking notes. It will br interesting to see how these three versions play out.
Future Iterations
I know many of you are suggesting that instant messages is so 2000 and late.I guess that is the nature of the beast we are trying to study. The instant message interface is just a way to simulate a two way communication embedded within our assessments. We have a talented group of programmers working on a response capture object. The latest idea is to make the assessment look like a social network. I am excited about this idea. I do worry that in chasing temporal and chique validity we may threaten actual validity. Have you ever used Facebook to solve a common problem or investigate an issue? I would still argue that discussion boards (for groups) or popular editorial blogs are where the debate around issues is centered. I guess there are specific fan pages out there that could serve as launching pads. It will be a wonderful line of investigation. If we go this route there will be wonderful opportunities to capture synthesis, even if it is still an incomplete model.
Friday, April 16, 2010
Saturday, April 10, 2010
Online Content Creation: Digital Writing and Digital Storytelling
I am excited by all of the interest in digital storytelling. Many wonderful colleagues are pushing the field forward. I finished Troy Hicks's book The Digital Writing Workshop and just ordered Dana Wilber's book Iwrite. These recent efforts are meeting the needs of teachers who all want to use multimodal composition to meet the needs of digital learners. These are exciting times.
I am a little worried, however, that we also need a pedagogy for teaching digital expository and persuasive texts to join such a strong emphasis on narrative storytelling.
Digital Texts
In almost every field students will need to create digital texts to inform and persuade. I recently read that IBM uses thousands of wikis for technical guides. Journalism is quickly shifting to online environments. Finally entreprenuers have to build a web presence using social media tools.
At the same time, however, I see very few schools preparing students for a world where they have to communicate information using digital tools. My hypothesis is simple. You can teach students traditional writing skills in online environments, but you can not teach digital writing skills in pen and paper environments.
During our Internet Reciprocal Teaching lesson many of our lessons were embedded in the persuasive writing curriculum (CT is one of the few states that actually cares about writing on state assessments, we came across many of the differences between offline and online reading. Our students were working on both their critical evaluation skills and persuasive techniques. We started with Mumia Abul-Jamal, a Philadelphia man whop has been convicted of murder. Some contest this claim. We started with the Wikipedia article and other expository texts. We looked at the text structure and design options. Next we looked at websites from both perspectives-guilty and not guilty. The students quickly noted the design issues such as image, font, and color the authors made.
We repeated the same lessons usings zoo's. The students had to decide if zoos were cool or cruel. You would be amazed at how fast a picture of a sad monkey can persuade a student. They had to learn to read the images and understand design choice. There is no way these skills can be taught with paper and pencil!
Classroom Ideas
Having students research an issue and look for articles from a variety of perspectives is an importatn start. I wish we continued and had students use different writing tools such as websites, wikis, and blogs to create persuasive texts. The focus of the study, however, was on comprehension and not composition.
There are classroom ideas teachers can use. One of the most exciting ideas has turned into Ian's dissertation study. He is having students create hoax websites (think the fake product lessons we have done for decades). First the students look at webites and develop a list of markers of reliability. Then using Iweb the students create their own websites with different levels of sincerity.
Another easy lesson, similar to our Mumia Abul-Jamal lesson, is to have students choose an issue and create a website to persuade. The final phase III lessons we did were similar to this approach. We had students choose an issue to make the world a better place. Sure many students focused on dress code and bad school lunches, but others addressed issues such as dating violence, drugs, and crime. The students had to create a website or online presentation on the topic. We began by storyboarding the websites and focusing on design issues. Only then did we actually begin to write copy.
Will it work?
Is my hypothesis correct? Does instruction in digital writing improve measures in offline writing? There is no evidence out there and it is a line of inquiry that interests me. Connecticut would be a unique testing ground. Persuasive writing eighth grade could be used as dependent variables in an ANCOVA model with 7th grade scores as a covariate. CAPT scores in tenth grade could be used for group differences. Of course I would have to make a measure of argumentative web design.
An important element to teaching critical stances necessary for deep comprehension is to have students develop “perspectives on perspectives” (Lankshear & McLaren, 1993 p. 33). One method to developing critical literacy online is to have students learn about the design of websites (Burbules, 1995). Having students select materials for a page, linking to websites, and using the affordances of web design to formulate arguments may teach both argumentative writing and online reading comprehension. The more someone knows how credible arguments are designed the more they are aware when it is done and when it could be done (Burbules, 1995).
As stated I would hypothesize that instruction in traditional argumentative writing (Fulkerson, 1996) would not lead to an increase on scores of a measure of argumentative web design or online reading comprehension, but instruction in argumentative web design may increase scores on measures of both online reading comprehension and measures of argumentative writing.
Conclusion
I do not want to downplay the importance of narrative writing as we discuss digital storytelling. I firmly believe that creative writing is key to improving technical, expository, and persuasive writing. In fact my favorite educational authors blend their genres. That said I am worried that a strong research agenda in digital writing is not developing as quickly as the research surronding digital storytelling.
I am a little worried, however, that we also need a pedagogy for teaching digital expository and persuasive texts to join such a strong emphasis on narrative storytelling.
Digital Texts
In almost every field students will need to create digital texts to inform and persuade. I recently read that IBM uses thousands of wikis for technical guides. Journalism is quickly shifting to online environments. Finally entreprenuers have to build a web presence using social media tools.
At the same time, however, I see very few schools preparing students for a world where they have to communicate information using digital tools. My hypothesis is simple. You can teach students traditional writing skills in online environments, but you can not teach digital writing skills in pen and paper environments.
During our Internet Reciprocal Teaching lesson many of our lessons were embedded in the persuasive writing curriculum (CT is one of the few states that actually cares about writing on state assessments, we came across many of the differences between offline and online reading. Our students were working on both their critical evaluation skills and persuasive techniques. We started with Mumia Abul-Jamal, a Philadelphia man whop has been convicted of murder. Some contest this claim. We started with the Wikipedia article and other expository texts. We looked at the text structure and design options. Next we looked at websites from both perspectives-guilty and not guilty. The students quickly noted the design issues such as image, font, and color the authors made.
We repeated the same lessons usings zoo's. The students had to decide if zoos were cool or cruel. You would be amazed at how fast a picture of a sad monkey can persuade a student. They had to learn to read the images and understand design choice. There is no way these skills can be taught with paper and pencil!
Classroom Ideas
Having students research an issue and look for articles from a variety of perspectives is an importatn start. I wish we continued and had students use different writing tools such as websites, wikis, and blogs to create persuasive texts. The focus of the study, however, was on comprehension and not composition.
There are classroom ideas teachers can use. One of the most exciting ideas has turned into Ian's dissertation study. He is having students create hoax websites (think the fake product lessons we have done for decades). First the students look at webites and develop a list of markers of reliability. Then using Iweb the students create their own websites with different levels of sincerity.
Another easy lesson, similar to our Mumia Abul-Jamal lesson, is to have students choose an issue and create a website to persuade. The final phase III lessons we did were similar to this approach. We had students choose an issue to make the world a better place. Sure many students focused on dress code and bad school lunches, but others addressed issues such as dating violence, drugs, and crime. The students had to create a website or online presentation on the topic. We began by storyboarding the websites and focusing on design issues. Only then did we actually begin to write copy.
Will it work?
Is my hypothesis correct? Does instruction in digital writing improve measures in offline writing? There is no evidence out there and it is a line of inquiry that interests me. Connecticut would be a unique testing ground. Persuasive writing eighth grade could be used as dependent variables in an ANCOVA model with 7th grade scores as a covariate. CAPT scores in tenth grade could be used for group differences. Of course I would have to make a measure of argumentative web design.
An important element to teaching critical stances necessary for deep comprehension is to have students develop “perspectives on perspectives” (Lankshear & McLaren, 1993 p. 33). One method to developing critical literacy online is to have students learn about the design of websites (Burbules, 1995). Having students select materials for a page, linking to websites, and using the affordances of web design to formulate arguments may teach both argumentative writing and online reading comprehension. The more someone knows how credible arguments are designed the more they are aware when it is done and when it could be done (Burbules, 1995).
As stated I would hypothesize that instruction in traditional argumentative writing (Fulkerson, 1996) would not lead to an increase on scores of a measure of argumentative web design or online reading comprehension, but instruction in argumentative web design may increase scores on measures of both online reading comprehension and measures of argumentative writing.
Conclusion
I do not want to downplay the importance of narrative writing as we discuss digital storytelling. I firmly believe that creative writing is key to improving technical, expository, and persuasive writing. In fact my favorite educational authors blend their genres. That said I am worried that a strong research agenda in digital writing is not developing as quickly as the research surronding digital storytelling.
Subscribe to:
Posts (Atom)