Jul 2, 2014 - the unit of work, plus detailed lesson plans, powerpoints for each lesson and ... The comparison group wer
GRAMMAR AT GCSE: Exploring the effects of a contextualised grammar pedagogy on reading and writing at KS4
Centre for Research in Writing Annabel Watson, Debra Myhill and Ruth Newman
1
JULY 2014 GRAMMAR AT GCSE Centre for Research in Writing Annabel Watson, Debra Myhill and Ruth Newman EXECUTIVE SUMMARY The project •
This quasi-experimental project investigated the impact of a particular approach to reading and writing instruction which embeds attention to grammar within a pedagogy focused on developing students’ understanding of the choices available to writers.
•
Twelve GCSE English classes across four schools were organised into either an intervention or a comparison group. The intervention group were taught a three week scheme of work founded upon the principles of the pedagogy noted above, while the comparison group were taught various schemes (according to the usual practice of their teachers) which focused on the same GCSE assessment objectives as the intervention.
•
An abridged sample Edexcel GCSE English reading and writing non-fiction examination paper was used to measure reading and writing attainment of both groups before and after the intervention period.
•
Two students and the teacher from each intervention class were interviewed twice during the delivery of the intervention, each time following on from lesson observation.
Key Findings •
The intervention had a statistically significantly positive impact on student reading and writing outcomes, with a more significant impact on reading than on writing. This is in accordance with previous investigations of the pedagogy’s impact on writing (Myhill, D.A., Jones, S.M., Lines, H. and Watson A. 2012; Myhill, D.A., Jones, S.M., and Lines, H. 2013).
•
The intervention had a significantly positive impact specifically on students’ responses to reading questions which required language analysis, and on the sentence structure, punctuation and spelling scores of the writing test.
•
In the reading test, the benefits were greater for classes which had a higher ability profile, but the statistical tests did not find that ability, gender, pupil premium status (or
2
free school meal where schools used that measure), or EAL status had a significant impact on the effect of the intervention. •
Evidence from the student interviews and reading tests indicates that the use of metalinguistic terminology helped some students to articulate their thoughts more clearly when talking and writing about texts, particularly when analysing language and discussing the effects grammatical structures.
•
This evidence also indicated that in order to use metalanguage effectively, students needed to appreciate the context and purpose of the texts which they analysed. Some students struggled to understand that grammatical structures or devices don’t have generic or static effects.
•
Evidence from the writing tests indicates that in the post-test intervention group students were more focused on crafting sentences for effect and were experimenting with a wider variety of sentence structures.
•
The student interviews suggest that grounding discussion of effect in students’ own writing appears to particularly support their understanding of linguistic and grammatical choice, perhaps because those choices originate with them, suggesting the potential of writing for reading.
•
Participating teachers expressed the opinion that the scheme encouraged students to engage more ‘consciously’ with the writing process, thinking more carefully about the options available to them and more about the effect of the techniques applied, instead of writing a ‘stream of consciousness.’
Key Implications for Teachers •
Explicit attention to grammar can foster student reading and writing development when it is contextualised within lessons which focus on reading and writing.
•
Students can discuss the effects of language or grammatical patterns without using metalinguistic terminology, but if teachers can support the use of metalanguage they may be able to express their ideas more clearly or precisely.
•
When metalinguistic terminology is used, the focus should be on exploring how the linguistic features examined work in texts (including students’ own writing), not on simply learning definitions.
•
Students need particular support to understand that linguistic structures or devices do not have generic ‘effects’ regardless of context: they need to explore the impact of
3
language and grammar in the specific context of the texts which they are reading and writing. •
The highest achieving students demonstrated an understanding that writers make choices: they had moved away from a view that ‘good’ writing should contain a prescribed list of features towards an developing appreciation that language isn’t applied in predetermined or correct ways but is a matter of choice and effect in context.
•
Students may demonstrate inaccuracy when experimenting with new grammatical structures within their own writing.
Further Research •
This study is the first to apply this approach to the teaching of reading as well as writing: the finding that reading improved more significantly than writing suggests that this would be a fertile area for further investigation, particularly investigating the impact of this approach to linguistic analysis for GCSE English Literature as well as Language.
•
The reading focus of this study has been on language analysis. There remains the potential to use the pedagogical approach to teach literary essay writing, which may have a different, possibly beneficial, impact on GCSE reading attainment which is assessed through the writing of literary essays.
•
While there was a statistically significant positive effect for the intervention, the size of the effect (i.e. the actual improvement in scores) was small, due to the fact that the intervention period was short (three weeks) and the test used to measure progress was very broad (an abridged GCSE exam paper). It would be beneficial to work with teachers and their classes over a longer period of time (e.g. multiple schemes of work) to embed the principles of this pedagogy more securely and assess whether the impact might be greater over a longer timescale.
•
Similarly, further research could investigate how the approach might be embedded across a whole department, looking at the full range of student ages and abilities.
4
CONTEXT A previous national study, conducted by University of Exeter research team, had found that embedding relevant grammar meaningfully within the context of writing had a significant effect on the attainment of writing for students in the intervention group (Myhill et al 2012; Jones et al 2013).
A follow-on study (Myhill 2013) designed to investigate the impact of a similar
intervention on weaker writers, also yielded a statistically significant positive result. The fact that this approach closely interweaves reading and writing activities, where analysis and exploration of the linguistic features of texts and their impact on the reader feeds into imitation of and experimentation with the same linguistic features in students’ own writing, raised suggestions from teachers that this pedagogy, designed to support writing development, may also have a beneficial impact on students’ reading attainment, specifically in enhancing their ability to analyse language and discuss the effects of words and linguistic structures. Previous studies had also focused on KS3 (year 8 in particular) so we wanted to explore how this pedagogy might support students working at KS4, particularly in relation to the non-fiction element of the English Language GCSE. The current study, reported here, therefore set out to investigate the impact of the grammar for writing approach developed by the Centre for Research in Writing at Exeter University on student reading and writing attainment at GCSE. The research question was: Does explicit teaching of contextualised grammar at KS4 improve pupils’ attainment in reading and writing non-fiction?
THE STUDY DESIGN The study design followed as closely as possible the overall design of the initial large scale study and subsequent investigation of weaker writers to allow for comparisons between the data sets. The key difference in the current study was that pre and post tests were used to measure reading as well as writing attainment. Student Sample: Throughout the report, the intervention group are referred to as Group 1, and the comparison group as Group 2. The student sample comprised twelve classes of year 10 students in 4 schools in the South West of England. A breakdown of the classes is shown in tabular form below (Table 1). The initial sample comprised 161 students in the intervention group and 147 in the comparison group. The
5
groups were reasonably matched in prior attainment, as shown by KS4 target grades in figure 1 and by pre-test attainment in figure 2; however, it should be noted that the intervention group had a slightly higher ability profile overall, with more very high ability students (target grade A), while the comparison group had more very low ability students (target grade E) due to the inclusion of one small low ability class (C3comparison). The gender balance between groups was good (Table 1), although the intervention group had more girls. SCHOOL A 1 B 1 C 1 2 3 D1 2 Total
Intervention Class Boys size 22 12 29 14 22 15 27 7 31 30 161
10 14 72
Girls 10 15 7 20 21 16 89
Av GCSE Target C C D B B C
COMPARISON Class Boys size 28 13 31 10 23 14 26 15 11 8 28 15 147
75
Girls 15 21 9 11 3 13
Av GCSE Target C B C C E C
72
Table 1: the student sample at the outset of the research
6
Figure 1: GCSE target Grades by Group
Figure 2: Overall Pre-Test Means by Group The Intervention: The intervention was founded on the pedagogical principles of the earlier studies, but was explicitly linked to current GCSE assessment objectives, outlined below. It was planned as a nine lesson unit, designed to be taught over a period of three weeks. Non-fiction text-types were chosen to mirror the unseen pieces which might be used in any current GCSE English Unit 1 examination, so covered print media (newspaper and magazine articles), and online media (emails and online newspaper editorials). A decision was also made to use a satirical fictional work (The Zombie Survival Guide by Max Brooks) due to the motivating subject matter and the fact that it is written as a parody of non-fiction survival guides, using exaggerated non-fiction conventions. The scheme was designed to focus on how non-fiction texts use linguistic and literary structures to engage their audience, and the aims were explained as follows:
7
The purpose of this scheme is to develop students’ ability to analyse non-fiction texts in detail, and to write their own. There is a particular focus on building understanding of a repertoire of linguistic and grammatical structures, and on moving from analysis of patterns in real texts to use of these patterns in students’ own writing, with consideration of purpose and effectiveness throughout. The final assessed outcome of the unit was a written assignment in which students were asked to write a magazine article about a new scientific invention, after analysing a BBC Focus magazine article about astronaut hibernation. The intervention group were given the medium term plan for the unit of work, plus detailed lesson plans, powerpoints for each lesson and other resources. The comparison group were given the GCSE assessment objectives. The intervention unit of work is included as Appendix A.
8
9
GCSE Assessment Objectives
Related Learning Objectives
Reading
Reading
•
AO2iiDevelop and sustain
•
interpretations of writers’ ideas
headlines and why it is effective.
and perspectives. •
Understand the typical underlying structure of news
•
Explore how effective headlines are created through
AO2iii Explain and evaluate
unusual, striking, visual and dramatic combinations
how writers use linguistic,
of words and images.
grammatical, structural and
•
presentational features to
Explore how adjectives, nouns and verbs are used to create effective description in a newspaper article.
achieve effects and engage and
•
Explore the effectiveness of expanded noun phrases
influence the reader.
•
Analyse the style of viral emails, including presentational features, punctuation and narrative
•
Explore how topic sentences are used to create cohesion and appeal to a reader
•
Explore the linguistic and grammatical features of scientific non-fiction
Writing
•
AO3i Write clearly, effectively
Writing
•
and imaginatively, using and adapting forms and selecting
create effective description in a newspaper article •
vocabulary appropriate to task and purpose in ways that •
Use expanded noun phrases to add detail to a description
•
Understand how to imitate the style of viral emails
engage the reader.
effectively in your own writing, focusing on
AO3ii Organise information and
presentational features, punctuation and narrative
ideas into structured and
•
sequenced sentences, paragraphs and whole texts,
Explore how topic sentences are used to create cohesion and appeal to a reader
•
Explore the use of sentence patterns to create
using a variety of linguistic and
effects: parentheses, parallel structures and minor
structural features to support
sentences
cohesion and overall •
Explore how adjectives, nouns and verbs are used to
•
Use the grammatical and linguistic conventions of
coherence.
scientific non-fiction to shape your own writing
AO3iii Use a range of sentence
effectively
structures for clarity, purpose and effect, with accurate punctuation and spelling.
10
Training the Teachers: The teachers in the intervention group attended one full day training event at the university where they were introduced to the pedagogical principles informing the intervention, and given a detailed introduction to the unit of work and its resources. They were also then given an individual half day of training during which researchers worked alongside them to tailor the scheme for the needs and abilities of their specific classes. Pre and Post Test Measures: The pre and post test measures used an edited version of a new Edexcel sample GCSE reading and writing non-fiction examination paper. The paper was edited to make it fit an hour long lesson by removing the second reading extract and the questions relating to it, along with its associated writing question. This left students with one extract to read, five reading questions to answer and one writing question to respond to. The focuses of the reading questions were as follows: Q1. Comprehension and information retrieval (3 marks). Q2a. Comprehension (1 mark). Q2b. Analysis of language (1 mark). Q3. Selection of quotations and analysis of language (4 marks). Q3. Analysis of language (6 marks). The writing question was an extended piece of non-fiction writing which was given two marks, one for overall composition, and one for sentence structure, punctuation and spelling. The same test was used for both the pre and the post test. The tests were blind marked by a team of Edexcel examiners using the mark schemes and scoring procedures used by Edexcel in their GCSE examination marking, with a sample cross-moderated by a Team Leader. Examiners did not know whether they were marking pre or post tests, nor whether the papers belonged to intervention group or comparison group students. Classroom observations: There were two lesson observations in each school during the three week intervention period, and only the intervention classes were observed. The purpose of the observations was principally to record how teachers used the intervention materials and how students responded. The observation data was captured through an observation schedule which recorded timing and activities, the teacher’s input, student responses, and general observational comments.
11
Teacher Interviews: Each teacher was interviewed twice during the intervention period, after a lesson observation. The purpose of these interviews was to determine teachers’ perspectives on the efficacy of the pedagogical approach in general and the intervention scheme in particular for their students, as well as to provide some evidence of the impact of the materials on the teachers’ own developing understanding of the pedagogy. The interview schedule is outlined below.
Teacher Interview Schedule Main Construct: What is the impact of the pedagogical support materials (SOW and resources) on the teaching of grammar? 1. How does this scheme of work compare to what you’d normally do when teaching non-fiction at KS4? (interview 1 only) 2. What are your opinions of the scheme and its impact on pupils’ learning? (follow up issues probing for explanation where necessary) 3. Have you come across any challenges in teaching this unit? (note, these may be personal to teacher e.g. subject knowledge, may be related to the class e.g. student responses, or may be flaws in the scheme or resources.)
Student interviews: Two sample students, a boy and a girl, were identified from each intervention class for interviewing after each lesson observation. These students were selected by the class teachers on the basis that they would enjoy and benefit from the experience, and were invited to take part on a voluntary basis. The purpose of these interviews was to ascertain what kind of learning about writing the unit of work was promoting. The interviews took the form of a writing conversation, about a GCSE writing sample provided by the researcher in interview 1, and about a piece of their own writing produced during the unit of work in interview 2. During these interviews, students were invited to discuss the authorial choices made in the written work, focusing on language, sentence structure and overall structure, as well as to reflect on their learning in the observed lesson. The interview schedule is reproduced below.
12
Student Interview schedule Section 1: Main construct: Understanding of the learning objectives Q1. What do you think the teacher was trying to teach you in that lesson? Follow up questions relating to specific episodes / tasks in the observed lesson and explicitly drawing out perceptions of learning or lack thereof.
Section 2: Main construct: impact of grammar teaching on pupils' metalinguistic understanding Interview 1: Stimulus: A grade GCSE sample Q1. Tell me about the words the writer has chosen. What do you think is effective? Is there anything you would improve? Q2. Tell me about the sentences. What do you think is effective? Is there anything you would improve? Q3. Tell me about the writing overall. Anything you think is effective? Anything you would improve? Probe knowledge and understanding of grammatical concepts as they arise. Probe ability to discuss the effects of words and grammatical patterns. Probe how precisely they can discuss writer decisions, choices and effects.
Interview 2:Stimulus: example of own writing from the scheme. Q1. Tell me about this piece of writing. What is it about? What were you trying to do in this piece of writing? Q2. Tell me about the words you’ve chosen. What do you think is effective? Is there anything you would improve? Q3. Tell me about the sentences. What do you think is effective? Is there anything you would improve? Q4. Tell me about the writing overall. Anything you think is effective? Anything you would improve? Probe knowledge and understanding of grammatical concepts as they arise. Probe ability to discuss the effects of words and grammatical patterns. Probe how precisely they can discuss writer decisions, choices and effects.
13
THE STATISTICAL RESULTS From the initial sample of 308 students, the final data set for statistical analysis was 240, with 124 in the intervention group and 116 in the comparison group, representing an attrition rate of 22%. This high attrition rate and may relate to the fact that the post-test took place during the last week of the spring term (and in one school on the last day). The attrition rules out students who were absent for either the pre-test or the post-test or both, but it is worth noting that students who were present for the pre and post tests and thus included in the data were sometimes absent during the intervention period. The descriptive statistics indicate that the two groups scored similarly on the pre-tests, but with slightly higher overall means for the intervention group. The intervention group gained in both reading and writing scores in the post test by .5 in reading and .5 in writing, while the average for the comparison group declined by .4 in reading and by .8 in writing. Suggested reasons for this decline are discussed later. Group
Focus
Number
Pre-test Post-test score: mean score: mean Reading Intervention 124 4.1 4.6 Writing 10.6 11.1 Reading Comparison 116 3.8 3.4 Writing 10.2 9.4 Table 2: the mean scores of the two groups pre and post test
Figure 3: Line graph illustrating relative performance of the two groups on the reading test.
14
Figure 4: Line graph illustrating relative performance of the two groups on the writing test.
When this pattern of performance is analysed at class level (Table 5), it illustrates that variability between classes. In the reading tests, two of the intervention group and three of the comparison group scored less well in the post-test. In the writing test two each of both the intervention and comparison classes scored less well in the post-test. Only one class scored worse on both posttests of reading and writing(School A comparison group), and the decline in the writing score in this class is considerably higher than any other gains or declines: this suggests that the overall writing results may be skewed by this group, so we must exercise caution when interpreting the results of the inferential statistics. The differences between classes may signal the significance of the teacher and the way they implemented the intervention, or signal contextual differences around the taking of the tests (such as the fact that the post-test happened so close to the end of term). Alternatively, it may echo the findings of the parent ESRC study that the intervention had a stronger beneficial impact on more able students: the highest reading gains for the intervention group were in the classes which scored more highly in the pre-test (School C class 2 and School D classes 1 and 2), although this pattern was not repeated quite so distinctly with the writing scores.
15
SCHOOL
Intervention (Reading)
Comparison (Reading)
No.
Pre Test
Post Test
Diff
No.
Pre Test
Post Test
Diff
A1
17
2.3
2.1
-0.2
21
3.2
2.7
-0.5
B1
20
3.7
3.4
-0.3
24
4.8
5.0
+0.2
C 1
16
2.2
2.4
+0.2
19
4.3
3.4
-0.9
2
26
4.2
5.5
+1.3
23
4.4
3.2
-1.2
10
0.7
0.9
+0.2
22
2.9
3.5
+0.6
3 D1
26
5.8
7
+1.2
2
22
4.6
5.3
+0.9
Table 3: showing the Reading results at class and group level SCHOOL
Intervention (Writing)
Comparison (Writing)
No.
Pre Test
Post Test
Diff
No.
Pre Test
Post Test
Diff
A1
17
4.7
5.9
+1.2
21
7.7
3.6
-4.1
B1
20
10.9
11.3
+0.4
24
16.1
16.4
+0.3
C 1
16
8.4
7.9
-0.5
19
8.0
8.1
+0.1
2
26
11.2
11.1
-0.1
23
9.1
10.3
+1.2
10
3.5
1.9
-1.6
22
10.9
11
+0.1
3 D1
26
14.6
14.8
+0.2
2
22
10.4
12.7
+2.3
Table 4: showing the Writing results at class and group level Whilst the data clearly indicate a greater overall improvement in the intervention group at the level of basic comparison, in order to determine whether the results are statistically significant, inferential tests were used. Because the data represents students ‘nested’ in classes who could not therefore be randomly assigned to comparison or intervention groups, an analysis of covariance (ANCOVA) is most appropriate. It is important to note that this represents a robustly strict approach to the data. When ANCOVA controls for the covariate it also removes some of the treatment effect, reducing the likelihood of obtaining a significant result. Before undertaking an ANCOVA, it is necessary to test that the relationships between the covariate (the pre-test results) and the dependent variable (the post-test results) is the same for 16
each of the groups. To check this, firstly, a test of linearity was conducted to check that the covariate/dependent variable relationship is linear. Figures 5 and 6 below presents the results of the tests of linearity for the reading and the writing tests, which indicate linear relationships. Next, a check for homogeneity of regression slopes was conducted to ascertain if the covariate/dependent variable relationship is the same. Here the data (tables 5&6) show results which are non-significant (p=.534 and p=.219) and thus do not violate the assumption of homogeneity. A final test, the Levene’s test of equality of error variances (tables 7&8) show nonsignificant values of p=.096 and p=.727 and thus the assumption is not violated.
Figure 5: Test of linearity for Reading
Figure 6: Test of linearity for Writing
17
Tests of Between-Subjects Effects Dependent Variable: Reading Post-test Source
Type III Sum of Squares
df
Mean Square
F
Sig.
a
3
171.449
34.073
.000
182.033
1
182.033
36.176
.000
8.563
1
8.563
1.702
.193
Reading Pre-test
399.253
1
399.253
79.345
.000
Group * Reading
1.957
1
1.957
.389
.534
Error
1192.551
237
5.032
Total
5603.000
241
Corrected Total
1706.896
240
Corrected Model Intercept Group
514.346
Pre-test
a. R Squared = .301 (Adjusted R Squared = .292)
Table 5: Homogeneity of regression slopes for Reading
Tests of Between-Subjects Effects Dependent Variable: Writing Post-test Source
Type III Sum of Squares
df
Mean Square
F
Sig.
a
3
1309.524
59.641
.000
Intercept
348.188
1
348.188
15.858
.000
Group
102.632
1
102.632
4.674
.032
3643.967
1
3643.967
165.960
.000
33.302
1
33.302
1.517
.219
Error
5181.828
236
21.957
Total
34572.000
240
9110.400
239
Corrected Model
Writing Pre-test Group * Writing Pre-
3928.572
test
Corrected Total
a. R Squared = .431 (Adjusted R Squared = .424)
Table 6: Homogeneity of regression slopes for Writing
18
a
Levene's Test of Equality of Error Variances Dependent Variable:TotalReadb F
df1
2.795
df2 1
Sig. 239
.096
Tests the null hypothesis that the error variance of the dependent variable is equal across groups. a. Design: Intercept + TotalReada + Group
Table 7: Levene’s Test for Reading a
Levene's Test of Equality of Error Variances Dependent Variable:TotalWriteb F
df1 .122
df2 1
Sig. 238
.727
Tests the null hypothesis that the error variance of the dependent variable is equal across groups. a. Design: Intercept + TotalWritea + Group
Table 8: Levene’s Test for Writing Having undertaken these preliminary tests, the ANCOVA analysis indicates that the different outcomes of the intervention and comparison group is statistically significant for both reading and writing overall scores (see Table 10). The reading result has a very high significance level of p
