representing qualitative research data when writing up a thesis. Examiners have been known to pounce on cherry-picking r
Ema Ushioda Centre for Applied Linguistics
CHERRY-PICKING AND ANECDOTALISM: HOW NOT TO REPORT RESEARCH 17th Warwick International Postgraduate Conference in Applied Linguistics 2014 Pre-Conference Workshop
This workshop will discuss problems and pitfalls in representing qualitative research data when writing up a thesis. Examiners have been known to pounce on cherry-picking reporting practices where analysis seems based on particular data excerpts chosen to illustrate a point, thus verging on anecdotalism. The workshop will consider how analysis and reporting practices can achieve greater transparency and represent the data in more rigorous and systematic ways.
8 out of 10 owners who expressed a preference said their cats preferred Whiskas From a research reporting point of view, what is problematic about this claim?
One swallow does not make a summer
‘Choosing to make selective choices among competing evidence, so as to emphasize those results that support a given position, while ignoring or dismissing any findings that do not support it, is a practice known as “cherry picking” and is a hallmark of poor science or pseudo-science’. •(Richard Somerville, Testimony before the U.S House of Representatives Committee on Energy and Commerce Subcommittee on Energy and Power, March 8, 2011)
‘I call the analysis for this “documentary-style,” and this is where we see cherry picking used as a deliberate strategy. In a documentary, the commentator says something such as, “His death was devastating news for his daughter,” and the daughter then appears on screen, and says sorrowfully, “I was simply devastated.” So it is with cherry picking: Data that support the commentary are deliberately selected to endorse that same commentary. No other data are presented. Although variation is very limited, the article appears neat and convincing.’ (Morse, J. M. 2010. ‘Cherry picking’: Writing from thin data. Qualitative Health Research 20(1): 3)
‘There is a tendency towards an anecdotal approach to the use of data in relation to conclusions or explanations in qualitative research. Brief conversations, snippets from unstructured interviews […] are used to provide evidence of a particular contention. There are grounds for disquiet in that the representativeness or generality of these fragments is rarely addressed.’ (Bryman 1988: 77)
‘[…] qualitative researchers, with their indepth access to single cases, have to overcome a special temptation. How are they to convince themselves (and their audience) that their “findings” are genuinely based on critical investigation of all their data and do not depend on a few well-chosen “examples”? This is sometimes known as the problem of anecdotalism’. (Silverman 2005: 211)
Part of developing a more critical approach to qualitative interviewing is moving beyond the temptation to carve out quotable parts that serve our purposes. (Mann 2011: 21)
How can we guard against cherry-picking and anecdotalism when reporting findings?
What are possible pitfalls to avoid? How can we represent our data comprehensively and systematically?
Maximising confidence in the validity of our analysis and conclusions
CONSTANT COMPARATIVE METHOD Glaser 1965
Basically an iterative (i.e. repeated) process of constantly comparing data, codes, classifications, cases, etc., in an effort to test existing categories, expand these or look for new categories, codes, concepts, relationships, themes, etc., until theoretical saturation is reached and nothing new emerges.
TASK 1 (see page 3 of handout)
Here is a dataset from teachers who have registered for an in-service course on classroom management. The dataset derives from responses to an open-ended question on the registration form asking participants to indicate briefly their reasons for enrolling in the course. Examine the dataset and consider how you would apply the constant comparative method in analysing it. Try out your approach and discuss your analysis with your neighbours.
DEALING WITH DEVIANT CASES ‘Comprehensive data treatment implies actively seeking out and addressing anomalies or deviant cases’ (Silverman 2005: 215)
TASK 2 •Are there any deviant or anomalous cases in the dataset for TASK 1 above? How would you account for them in your analysis?
JUSTIFYING CHOICE OF EXAMPLES ‘Researchers
seldom provide the criteria or grounds for including certain instances and not others. As a result it is difficult to determine the typicality or representativeness of instances and findings generated from them’ (Mehan 1979: 15). TASK 3
•Think of your own research data. When selecting examples (e.g. data extracts, episodes, cases, stories, individuals, themes …) to discuss, what criteria do you use for justifying these choices? •How do you show whether these examples are typical or atypical and how representative they are of your whole dataset?
COMPREHENSIVE APPROACHES TO REPRESENTING DATA Role of quantification
‘By our pragmatic view, qualitative research does imply a commitment to field activities. It does not imply a commitment to innumeracy.’ (Kirk & Miller 1986: 10)
TASK 4 • Read the research paper extract (separate handout) and discuss whether the writer succeeds in showing the relative weight and distribution of the reported factors across her interview dataset. • Think of your own research data. How can you represent your data as comprehensively as possible in your thesis (e.g. in tables, appendices)? What might be suitable approaches to quantifying your findings?
