What impact does different levels of "rhetorical reflection" have on the quality of student reflections and the resulting revised drafts?

What type of reflective prompts work best for post-freewriting-draft revision? What type of reflective prompts work best for revision between the second and third draft? What type of reflective prompts work best for transfer between a finished essay and a new assignment?

Problem: The problems expressed by TTU of generic reflections that don't engage students or perform anything significant for students in terms of learning.  How can you create prompts that work better to get students really reflecting in a deeper metacognitive way?

Research Approach

Pragmatism /Postpostivism
--consequences of actions, real-world practice oriented

Knowledge Claims
(methodology)

Experiential/  Quantitative?

Strategies of Inquiry

Quasi-experimental

Methods

Content analysis, possible datagogic analysis,

Sample/Sampling

TTU context—
Three different kinds of prompt series (3 prompts per series)
30 1301 and 30 1302 students (10 students/per level/per prompt series)
Textual data—Data from one essay cycle (probably the second or third essay in the semester)
1st draft, peer response, reflection A, 2nd draft, peer response, reflection B, 3rd draft, reflection C.
These special prompts would be fed to the different cohorts via TOPIC and students may not even know they are doing a reflection for the study.

SAC context—
A more modest study could be done with my own students. It would be hard to do three different prompt series, but possible. I will have two sections of 1301. I could ask for a third and then do a different prompt series for each section. Or I could get one 1301 and one 1302 class and divide the prompts within classes. With only 25 or less students, though, my sample size for each prompt series will be smaller.

Data Analysis

Use the "Irvin Reflection Scale" (a rubric development to rate reflections based upon Anson's Halliday scale, Beach's elements of self-evaluation, and other sources) to score reflections in certain areas. Likewise, subsequent drafts would be rated on the level and effectiveness of revision using another scale (I'm not sure what?) and correlated with grade. 
Possible correlations done through datagogic analysis of #of words, use of certain keywords?, and other information that could be pulled from the database.

Data analyzed by researcher with portion of the data evaluated by a second person for inter-rater reliability check.

Possible Results

The end result would be a score for each type or reflective prompt.  I would be able to say that "this" kind of reflective prompt tends to result in this type of reflection and eventual revision. For classroom purposes, teachers should seek to make their reflective prompts "this" way for maximum effect.

Assessment

This study is sort of like another study I read about on the effect of different kinds of writing prompts (I can't recall the exact study). The idea is to find the practical sweet spot for how to craft reflective prompts for the best result in student learning and production. 

Problems with the study are multiple—
The reflection and revision scales to "score" the data are extremely problematic. Could I ever develop a scale that could reach a level of acceptable inter-rater reliability? Evaluating reflections is a notoriously slippery endeavor.
WGRA?—if not many people are using this kind of classroom practice who will care which kind of reflective prompt works better?  The bigger questions are does it work and how does it work.

Also, developmental issues could complicate this study. What if students "performance" on the different kinds of prompts is affected by development issues? At least the data would all be relative.

If I did this study on my own students, I would have more control. There is no guarantee I would be able to institute this study within the TTU curriculum.

The question I have for this study is how to make the practice teach the theory (rather than it be about testing the theory in practice). It in spirit should be an action research study—but is this the kind of study legitimate for a dissertation? (I don't think so.)

Permutations

If I had funds to pay more "scorers" I could increase the sample size from TTU considerably.

Could include a student survey also to capture some student perceptions of different reflection prompts?  Probably a single survey at the end of the essay cycle.