Little Progress...

      This week has been overwhelming.  I honestly haven't had the chance to put in the time that I truly want to with my research.  Health issues, being mommy, and a full time job have really taken over my schedule.  All that being said, I have thought A LOT about my project and my next steps.  I need to continue coding each ERQ.  Hopefully, I can get that caught up as soon as possible and then the incoming ERQs should be pretty easy to code.  In addition to that, I need to finish scribing my interview.  I did a partial scribe of each of them for a previous post, but I need to get the rest of it down.  After following my classmates' blogs, I am a little disappointed that I did not video tape them.  I don't think I would have had the personnel to get that done anyway, but I do wish that I had the body language to see again. Some students were holding a basket in front of them, others played with pencils, while some were completely open and willing to express their true feelings.  I may choose to change this method next time. 
      As I continue this work, I am finding that it is actually fun.  I am enjoying digging deeper into different situations and learning more about my students and students in general. Hopefully we will have full weeks from now on so that I can have a good amount of data to really see how effective the RAP method is. 

Coding

Below is the beginning of my coding efforts that I explained in the last post.  After meeing with Dr. Bowers-Campbell Saturday, I think I am going to add another category for graphic organizers.   You can see that both students below used a venn diagram to compare and contrast stories.  I know teachers use graphic organizers to teach students to organize their thoughts, but we are wanging students to learn to put their final answers into sentence form.  As you can see, I am tracking when they do not restate the question with and "R" and don't answer all parts with a "MP" for missing parts.  When they not prove their answer I am marking a "PA."   One interesting observation in  the example by Kelsey below is that she got a "4" which is full credit, but she was still missing part of RAP.  She also chose to use a graphic organizer instead of using sentence form.  Since we are still in the learning stages of RAP she was able to earn full credit, but after the method has been taught more we will expect students to use sentence form instead to receive full credit for their work.

  Kelsey earned a 4, but still had missing parts.

 This student gets a writer as part of her IEP.

 Tallying...the beginning stages.

   I'm sure I will continue tweaking this process, but this is astart.  I am surprisingly excited to dig into this.  I just wish my schedule allowed me the time to get it done without stressing out. 

Let the coding begin...

I sat down today to begin coding (Bogdan & Biklen, 1982) writing samples only to realize that I left most of my samples at school. So, I used the 10 or so to get a start. I am very glad I thought to do this. I am using the codes "R" for restating if a student fails to restate the question in their answer. "MP" stands for missing parts if students did not answer all parts of the question. Finally,  I am using "PA" when students fail to prove their answer using information from the text or using their personal connecrions. I am also keeping tallies for each category as I write the codes on each piece. So far, all three categories are equal. One interesting observation I've made is that students who earned full credit for their answers still missed parts from RAP. I look forward to diving into the other samples and seeing the trends that emerge.

Resource: Bogdan, R.C. & Bicklen, S.K. (1982). Qualitative research for education: An introduction to theory and methods. Boston: Allyn and Bacon.

Questions, Questions, Questions

As I sit to examine the data I have I have many questions running through my mind.  A few of them I have mentioned in the past and a few more are new.  Bogdan and Biklen (1982) recommends that researchers "force [themselves] to make decisions that narrow the study" (p. 146).  I understand my primary focus and what I ultimately want to know, but I am having trouble narrowing exactly the data I need to accomplish this goal.  Bogden and Biklen also say that researchers often collect more data than they need.  Although I believe I have a lot of data, I am wondering if I need more qualitative data.  I conducted the interviews with my focus groups.  Other than that, I don't really have any other qualitative data. I have kept every ERQ from the participants.  I believe I need to go back to those and start coding for trends (Bogden & Biklen, 1982; Rossman & Rallis, 2003).  I would like to see if students are struggling with restating, are they leaving out parts, or can they not prove their answer.  What exactly is missing?  I initially thought that I could just use the scores from the RAP rubric, but now that I am thinking about it, I could gain more data by finding specifics within the ERQs.  I will be honest and say that that task seems scary and totally overwhelming to me, but in order for this study to work, I must just do it.

In addition to qualitative data, I am still struggling with the best way to record averages.  Since students are completing the ERQs in their classrooms and 2 grade levels are involved, not every studetn will complete and ERQ every week.  I need to decide the best way to document the average info.  I will continue to ponder this. Any ideas will be welcomed!

Resource: Bogdan, R.C. & Bicklen, S.K. (1982). Qualitative research for education: An introduction to theory and methods. Boston: Allyn and Bacon.

Benchmarking Data

1. Benchmark scores using “Kentucky Extended-Response Scoring Guide”

·         4^th grade: 3 2 4 4 1 4 1 2 4 4

Ø  Mean: 2.9 (Range1-4)

Ø  Median: 2 or 3

Ø  Mode: 4

·         5^th grade: 4 1 3 3 2 3 4 3

Ø  Mean: 2.9 (Range1-4)

Ø  Median: 2 or 3

Ø  Mode: 3

·         4^th and 5^th Combined: 3 2 4 4 1 4 1 2 4 4 4 1 3 3 2 3 4 3

§  Mean: 2.9 (Range 1-4)

§  Median: 2 or 3

§  Mode: 4

            Benchmarking scores were found using the “Kentucky Extended-Response Scoring Guide” that is used to score extended- response questions on the KPREP to evaluate extended-response answers without having an “attack” strategy (Kentucky Department of Education, 2012).  Students receive a score from 0 to 4 based on completeness and accuracy.  Students in both fourth grade and fifth grade scored an average of a 2.9 on their benchmark extended-response question. 

              

2. Average Benchmark Scores using the RAP Rubric

·         4^th grade

§  Read/Restate the Question: 0 0 4 4 2 3 2 0 3 4

Ø  Mean: 2.2 (Range0-4)

Ø  Median: 2

Ø  Mode: 0 and 3

§    Answer All Parts: 3 2 4 4 1 4 2 2 4 4

Ø  Mean: 3 (Range1-4)

Ø  Median: 2 or 3

Ø  Mode: 4

§  Prove Your Answer: 1 1 4 4 1 4 1 1 4 4

Ø  Mean: 2.5 (Range1-4)

Ø  Median:

Ø  Mode: 1 and 4

·         5^th grade

§  Read/Restate the Question: 3 0 4 2 4 1 4 3

Ø  Mean: 2.6 (Range0-4)

Ø  Median: 2

Ø  Mode: 4

§    Answer All Parts: 4 1 3 3 2 3 4 3

Ø  Mean: 2.9 (Range1-4)

Ø  Median: 2 or 3

Ø  Mode: 3

§  Prove Your Answer: 2 0 3 1 2 1 4 1

Ø  Mean: 1.2 (Range0-4)

Ø  Median: 2

Ø  Mode: 1

            Each benchmark extended-response question was also scored using the RAP rubric that assigned a number from 0 to 4 for each part of RAP based on evidence of each part of RAP as well as accuracy with the question.  The data above indicates that fifth grade students have a better grasp on restating the question in their answers than fourth grade students prior to being taught the skill.  Both grade levels displayed almost the same ability to answer every part of the question with fourth graders scoring a 3 and fifth graders scoring a 2.9.  The fourth graders, however, displayed a much stronger ability to prove their answer with a mean score of 2.5 compared to 1.2 for the fifth graders.

Interesting findings...
I have to admit, I was surprised by the amount of 4s students earned on the benchmarking assessments.  I am thrilled that our "novice" students are displaying such knowledge, it just wasn't what I was expecting to see.  I am very excited to see how they continue to work with the RAP method in hand. 

New questions...
I am still struggling with exactly how to handle all of the data.  I know that some teachers (myself included) have been doing some extended-response questions together as we teach the RAP method. Should I include those scores in my data?  I know that the instruction must be scaffolded and doing some together is an imporant step.  If I use those scores, what is the best way to document that?  I guess those are all questions I need to continue pondering. 

 

 

 

 

 

 

 

A little more info about my participants...

Here is a little more "formal" information about my participants.  

1. 15 participants (N=15)

·         7 fourth graders

§  1 female

§  6 males

·         8 fifth graders

§  5 females

§  3 males

Fifteen fourth and fifth graders (N=15) participated in the RAP research group.  Of the 7 fourth graders, 1 was female and 6 were male.   Of the 8 fifth graders, 5 were female and 3 were male.  All participants earned a novice on the Kentucky Performance Rating for Educational Progress (KPREP) during the 2012-2013 school year.    

2.    Gap Groups

·         Free or reduced lunch: 48.5% of the school

·         46% of the participants have an IEP for speech, reading, and/or math

       Within this group of novice students, every student was identified in one or more gap groups.  Forty-six percent of the novice students were identified as special education students and have an individualized education plan in speech, reading, writing, and/or mathematics.  One participant was of Asian ethnicity, but was not considered to be part of the gap group for ethnicity.  All other students were white/Caucasian.  48.5 % of the school received free or reduced lunch. Nine of the fifteen students were males with just six females.

3. Students in tier 3 Response to Intervention Groups

·         4^th Grade: 3 out of 7

·         5^th Grade: 4 out of 8

            Of the 15 participants 47% are in a tier 3 RTI group for reading. 
 

Interviews Finally Underway!

      I FINALLY got to conduct interviews today!  It was an interesting experience.  I thought the kids would be really open and be willing to share their thoughts, but that wasn't exactly the case.  I voice recorded each interview and that seemed to cause the students to clam up and not be themselves.  I had to ask more questions than I was hoping to get more information, but the answers they provided were honest and to the point.  
      I conducted 2 different interviews.  The first was with four 4th graders and the other was with four 5th graders.  The students came to my room, which all but 1 student has had groups in so I was hoping they would feel "at home."  It was just the participants and myself in the room.    We sat around a bean shaped table by the window where I always run my groups.  I tried to keep the atmosphere relaxed and make the students feel comfortable by chatting a little about life before we got started.  They shared a few fun experiences from their snow days with their peers.  Then, I explained that I would be recording our interview and the fun attitude changed.  The light-hearted conversation turned to serious business, their faces went straight, and their chattiness ceased.  I was honestly a little disappointed, but I guess talking about testing and writing isn't really that fun of a conversation at any point.
      The goal of the interviews was to determine how students feel when answering extended-response questions and about how they feel about testing in general.  Both groups gave me very similar answers.  Here is part of the script from my fifth grade group.

*(All names are pseudonyms.)

Teacher:  How do you feel about answering extended-response questions?
Multiple students: Nervous
Zendaya: I just don't like taking stuff like that.  I don't like taking an extended-responses.
Teacher: You don't like taking tests with extended-responses? 
Zendaya shook her head no.
Teacher: Why do you not like taking tests with extended-responses?
Zendaya: I, I just always get them wrong.
Teacher: Do you think you know the answer and putting it on paper is hard?
Zendaya: I know it in my head, but when I put it down it comes out wrong.
Teacher: Does anyone else feel that way?
Alexander: Yes. (In an almost angry tone.)
Teacher: Do you want to elaborate? 
Alexander: No.  (In a short tone, holding his head down, playing with a pencil)
....
Teacher: Does anyone worry about punctuation, spelling...
All 4 students: Spelling!
After a few seconds of silence...Teacher: Do you think when you are trying to spell a word you lose your thoughts?
All 4 students: YES
Reagan: When you have to spell words and I spell them and I get it back I have lots of words circled and rewritten.
Teacher: Do you think that discourages you the next time you have to take the test?
Reagan: (Hesitating) No...
....

Teacher: Do you feel nervous about multiple choice questions?
All 4 students: No, no...
Reagan: I like it more when you have choices.
Alexander: I guess on multiple choice the easiest part is vocabulary.
Teacher: The vocabulary is the easiest, but you still have trouble with the rest of the test?
Alexander: Mmmhmm..


There were small conversations inbetween these snippits, but I learned the most from these three sections.  You can tell from the first conversation with Zendaya that she had trouble just getting her thoughts out verbally.  "I don't like taking an extended-responses."  I knew what she meant, but she just couldn't get it out.  It was similiar to her answer.  She knows what she wants to say, but getting it down on the paper is a different story.

In the second part all students expressed a frustration with spelling.  They all expressed that spelling caused them to lose their train of thought.  I was happy to hear that Reagan wasn't discouraged by her mistakes.  The look on her face when she told me "no" was almost as though she was worried she was letting me down.  I quickly reassured her that I am happy she isn't discouraged and that I want her honest opinion.

The final conversation about multiple choice questions was interesting because all but Alexander expressed that multiple choice quesitions are not a problem.  Alexander does seem to have a very negative attitude toward school in general and writing has never been his strength.  

I look forward to chatting with these students again after RAP has been taught more. 

An Organized Start...

    On yet another snow day, I am beginning to work on ways to organize the data I have coming in. At this point I have collected around 2-3 extended-response examples from the participants' teachers. I have thought of many different ways to document the scores. I have taken benchmark data using the rubric that will be used for scoring the ERQs on the KPREP at the end of the year. (The link to this rubric is http://education.ky.gov/aa/distupp/pages/k-prep.adpx.) I plan to do this 2 more times throughout the study.  I am also scoring the samples using a rubric (see below) specifically designed to test the RAP method. In addition, I will be having teachers score a few samples with the RAP rubric as well to ensure validity of the study.  I created the sheet below to document scores in each of these areas.  Each student will have his/her own sheet. I am envisioning, but haven't created yet, a table that will document an average score for each week. That will be my next adventure in this process.

    I am feeling a little more confident about this process, but a little worried that my data will be scarce with the crazy amount of snow days. I guess I'll continue on and see what comes together.

Another snow day and a bit of stress...

You would think that a snow day should be relaxing and enjoyable and in some ways they certainly are. Unfortunately, missing so much school has put my life into a tizzy!  I need some structure. I LOVE being home with my little guy, but he needs structure too.  As I am trying to get caught back up and start taking a look at the data I have been collecting I am realizing that it is a lot to take in.  I am trying to take Rossman and Rallis' advice and write everything down, but trying to organize it in the most beneficial way is a little overwhelming.  I guess I just keep documenting and rework it as I need to.  More on this process to come...