McKrell, L., Green, A. & Harris, K. (1998). Libraries and Community Development: The National Survey of. Public Libr
SCOPING THE ECONOMIC AND SOCIAL IMPACT OF ARCHIVES
Sarah Horton & Jaqueline Spence Department of Information Studies, University of Wales Aberystwyth
Undertaken for
MLA Yorkshire
March 2006
Project team: Dr Sarah Horton Ms Jaqueline Spence Dr Christine Urquhart Mrs Sian Spink Ms Rhian Thomas
Contact details: Dr Sarah Horton Department of Information Studies University of Wales Aberystwyth Llanbadarn Fawr ABERYSTWYTH SY23 3AS tel. 01970-622189, fax 01970-622190 email
[email protected]
Table of Content Executive summary
4
Acknowledgements
6
Abbreviations
7
1
Introduction
9
1.1
Project brief
9
1.2
Project rationale
9
1.3
Aims
10
1.4
Objectives
10
1.5
Scope
11
2
Methods
12
2.1
Introduction
12
2.2
Literature review
12
2.2.1
Literature search
12
2.2.2
Widening the scope of the review
13
2.2.3
Processing the documents
13
2.3
Measures of usage
14
2.4
Limitations
14
2.5
Presentation of findings
14
3
Developing a Taxonomy of Usage
15
3.1
Introduction
15
3.2
Archive usage trends
15
3.3
Broadening the concept of usage
17
3.3.1
Primary users
17
3.3.2
Secondary users
25
3.3.3
Non-users?
30
3.4
A taxonomy of usage
30
3.4.1
Associated measures of usage
31
4
Developing a Taxonomy of Impact
40
1
4.1
Introduction
40
4.2
Defining impact
40
4.2.1
Impact as difference
40
4.2.2
Impact as a complex outcome
41
4.2.3
Timescale of impact
43
4.2.4
Impact on whom?
43
4.2.5
Types of impact
46
4.3
Defining economic impact
47
4.4
Defining social impact
48
4.5
A taxonomy of impact
53
5
Impact Evaluation: Approaches and Methods
56
5.1
Introduction
56
5.2
Defining impact evaluation
57
5.2.1
Issues and concerns
57
5.3
Impact evaluation: the archive domain
59
5.3.1
Statistical surveys of usage
60
5.3.2
Evaluations of projects and online services
61
5.3.3
Service reviews / surveys
64
5.4
Impact evaluation: the ALM sector
66
5.5
Impact evaluation: the wider cultural sector
70
5.5.1
The Arts
70
5.5.2
Libraries
72
5.5.3
Museums
73
5.6
Impact evaluation in other fields
74
5.6.1
Stated preference techniques
75
5.6.2
Cost-benefit analysis (CBA)
78
5.6.3
Cost-effectiveness analysis (CEA)
79
5.6.4
Multi-criteria analysis
81
5.6.5
Actuarial assessment
82
5.6.6.
Other techniques / methods
85
2
5.7
Summary
85
6
Conclusions and Recommendations
91
6.1
Introduction
91
6.2
Demonstrating ‘value’
91
6.3
Evaluating ‘impact’
94
6.3.1
Social impact
94
6.3.2
Economic impact
96
6.4
Recommendations
97
Bibliography and References
99
3
Executive Summary Aims and objectives This project was commissioned by the Yorkshire Museums Libraries & Archives Council (YMLAC) in order to “increase YMLAC’s and the Archive domain’s knowledge and understanding of the methodologies / approaches available to undertake economic and social impact assessment.” The objectives were to:
Systematically review the published evidence on economic and social impact assessments adopted in the archives domain, both within the UK and internationally
Develop a taxonomy of usage, based on the need to differentiate primary and secondary users, economic and social impacts
Assess the transferability of other schemes developed for the valuation of information or information services to the archives sector
Develop an impact taxonomy that addresses economic / social impacts over time Produce a mapping of impact type against methods, noting any particular problems or opportunities in deploying these methodologies
Taxonomy of usage Usage can be viewed as a continuum which extends from the record creator, via the archivist, to the primary user of the service, the secondary user of research products, and the ‘non-user’ at the other extreme. It also represents a knowledge supply chain, where the archivist adds value to the record in producing archival products (finding aids / secondary resources), and users add further value through generation of research products, which through secondary consumption inform wider society. Primary usage encompasses engagement with archival products and original archives and must be considered in relation to key parameters: the extent of the domain, and form of service (on-site / off-site; mediated / un-mediated; active / passive). The characteristics of primary users – demographics (age, gender, ethnicity, disability), and motivation / purpose – also affect usage and usage patterns. Secondary usage encompasses engagement with the research product of primary use; usage may be active or passive. Its potential scope (in reaching audiences for television and print publication) calls into question the concept of non-use. A taxonomy of use is presented as the basis for the further exploration of the impact of service provision; for purposes of clarity this is in two parts:
Form of service use: primary; secondary Purpose of use: personal (leisure / non-leisure); education (researcher; teacher; student); work (paid / voluntary) Potential measures of secondary usage (television viewing figures; readership and book sales figures) are identified.
Taxonomy of impact The definition of impact is complex, and types of impact are fundamentally inter-related; approaches vary in their scope and intention, but can be most usefully conceptualised in relation to ‘the difference made’. Scales of impact, over time, and on particular constituencies, may be mapped to the GLOs and prototype GSOs proposed by MLA. Short-term impacts are usually associated with learning new things or acquiring new
4
skills, longer term impacts concern changes in individual behaviour or community attitudes. Impact assessments need to consider who is affected, when, and how. Definition of economic impact can be relatively clearly defined in relation to demandside (service operation) and supply-side (service consumption) impacts. Definition of social impact is more problematic, but can be approached thematically in relation to government policy, and conceptually in relation to the generation of social and personal capital. The taxonomy proposed is based on a view of impact as a continuum which builds cumulatively over time, and extends from the individual to associated groups, communities and thus to society as a whole. Dimensions of economic and social impact are identified as operating at points within this:
Economic impact: learning and skills development; enterprise support; regeneration and renewal; and tourism.
Social impact: independence; empowerment; healthy lifestyle; participation; cultural diversity / identity; family ties; care and recovery; social empathy; capacity; and safer communities.
Impact evaluation: approaches and methods The issues associated with impact evaluation are considered, and an overview of existing activity relevant to the archive domain, ALM sector and wider cultural scene presented. The potential of impact assessment methodologies utilised in other fields, such as environmental impact assessment or health economics is assessed, in relation to the following techniques:
Stated preference techniques (including contingent evaluation) Cost benefit analysis Cost-effectiveness analysis Multi-criteria analysis Actuarial assessment
The evidence is not easy to interpolate for the archive sector as the aesthetic considerations and risk factors differ. Such impact assessments may often be used to compare investment options, to aid decision making or to assess service performance. Their potential for mapping against the proposed taxonomies is currently limited and requires more detailed conceptual development.
Conclusions and recommendations Impact assessment for the archives sector has been very limited but the existing evidence at least indicates the type and range of impacts to be expected; there is however insufficient research data of the rigour and scope required for generalisation across the domain. Methodologies that could be applied to the archives domain include SP techniques, which might address some of the difficulties of assessing the impact of secondary usage and the ‘non-user’; however, more detailed case studies are necessary to establish the parameters of value, before these could be applied in practice. An approach adapted from the health sector concept of QALYs could offer a way forward. Thought should also be given to whether the level of investment required in developing complex methodologies can be justified by their eventual end-use. Such techniques are dependent on the perceived ‘public value’ of archives, which highlights the need for further work to raise awareness of the domain and what it does.
5
Acknowledgements The project team would like to thank the representatives of The Yorkshire Museums, Libraries and Archives Council (YMLAC), now MLA Yorkshire, and Yorkshire Archives Council (YAC) who comprised the steering group for this project for their help and support: Claire Dyson, Regional Archives Advisor, YMLAC; Jason Doherty, Director (Policy & Corporate Planning), YMLAC; Judy Burg, University Archivist, University of Hull; and Chris Sheppard, Head of Special Collections, University of Leeds.
6
Abbreviations A2A
Access to Archives (the English strand of the UK Archives Network)
AAC
Archive Awareness Campaign
ABC
Audit Bureau of Circulation
AHP
Analytical Hierarchy Process
AIM25
Archives in London and the M25 Area
ALM
Archives, Libraries, Museums [used to denote sector]
ANW
Archives Network Wales
ATP
Ability to Pay
BARB
Broadcasters’ Audience Research Board Ltd.
BLRIC
British Library Research and Innovation Centre
CBA
Cost-Benefit Analysis
CEA
Cost-Effectiveness Analysis
CER
Cost Effectiveness Ratio
CIPFA
Chartered Institute of Public Finance and Accountancy
CISIR
Cultural Investment and Strategy Impact Research
CV
Contingent Valuation
DCMS
Department for Culture, Media and Sport
DTLR
Department for Transport, Local Government and the Regions
EVRI
Environmental Valuation Reference Inventory
GLLAM
Group for Large Local Authority Museums
GLOs
Generic Learning Outcomes
GSOs
Generic Social Outcomes
GSU
Genealogical Society of Utah
HMC
Royal Commission on Historical Manuscripts
HRQOL
Health Related Quality Of Life
IBSS
International Bibliography of the Social Sciences
IPF
Institute of Public Finance
LIRP
Learning Impact Research Product
LISA
Library and Information Science Abstracts
LUCAS
Liverpool University Centre for Archive Studies
MCA
Multi-Criteria Analysis
MLA
Museums, Libraries and Archives Council
7
MORI
Market & Opinion Research International
NPV
Net Present Value
NRS
National Readership Survey
PRONI
Public Record Office of Northern Ireland
PSQG
Public Services Quality Group
QALY
Quality Adjusted Life Years
RAJAR
Radio Joint Audience Research
SCAN
Scottish Archive Network
SEBs
Social and Environmental Benefits
SOSIG
Social Science Information Gateway
SP
Stated Preference [Technique]
SWMLAC
South West Museums Libraries & Archives Council
TEV
Total Economic Valuation
TNA
The National Archives
VRAG
Violence Risk Appraisal Guide
WTA
Willingness to Accept
WTP
Willingness to Pay
YAC
Yorkshire Archives Council
YMLAC
Yorkshire Museums, Libraries & Archives Council
8
1
Introduction
1.1
Project brief
This project was commissioned by the Yorkshire Museums, Libraries & Archives Council (YMLAC) in order to “increase YMLAC’s and the Archive domains knowledge and understanding of the methodologies / approaches available to undertake economic and social impact assessment.” The commission was to focus on the identification and evaluation of methodologies / approaches appropriate for deployment within the archive domain. Specifically, the investigation was to:
1.2
Adopt a broad definition of the term ‘usage’; encompassing in particular aspects of the secondary usage of archives.
Explore the approaches to economic and social impact assessment adopted by the archives domain, both within the UK and internationally.
Explore approaches to economic and social impact assessment used within other fields which may be transferable to the archives domain.
Rationale
The rationale for instigating this research was based on YMLAC’s (2005, point 2) recognition that: Increasingly, public investment in cultural activity and cultural resources is being evaluated in instrumental terms – according to the perceived economic or social return on that investment. Consequently, there is growing pressure for robust and authoritative data on the impact of cultural activity. Impact evaluation is a recognised focus (Impact Evaluation, 2006) within the current work programme of the Museums, Libraries and Archives Council (MLA), and reflects central government priorities and concerns. As Burns Owens Partnership (2005a, p.2) noted in a recent report for MLA: enthusiasm for evidence-based policy is seen as a hallmark of the current administration, and as part of its apparent commitment to a less ideological age – one more based on ‘what works?’. In addition, the need to prove value for money to a sometimes sceptical public, the diminution in the public’s desire to simply take professionals at their word, and the greater amount of policy-relevant research that is being undertaken worldwide, have all contributed to the pressure of public agencies to produce evidence for their actions and investments. The continuing focus of central government policy on social issues, such as social exclusion, neighbourhood renewal, regeneration and sustainable communities, also points to the relevance of social impact measures, and the need for archives (as with their counterparts in the cultural sector) to demonstrate their relevance to these agendas.
9
YMLAC further recognised that data on the economic and social impact of archives was lacking both locally and nationally. Recent studies, such as Burns Owens Partnership (2005a) have highlighted key weaknesses in the evidence base across the ALM sector as a whole, and Wavell, Baxter, Johnson and Williams (2002, p. 19) specifically refer to the paucity of research into the social impact of archives in the UK. This said, the study is able to build on work undertaken by the library and museum domains, and on crossdomain research carried out in recent years under the auspices of the MLA regional agencies. The library domain in particular was already considering these issues in the late 1990s, when a programme of research and related activities (The Social Impact of Libraries: a research and demonstration programme1), partly funded by the British Library’s Research and Innovation Centre, was underway. This programme recognised the importance of the policy context, which emphasised the role of the public library as a ‘community asset’, and the need to ‘measure’ “the intangible benefits of the library” (Linley & Usherwood, 1998, p.8) and “demonstrate in measurable terms that public libraries offer value for money.” (Harris & Green, 1997) The study also identifies and draws on approaches to economic and social impact assessment used within other fields, (including the arts, the health sector, and environmental and insurance work), where there was apparent potential for application in the archives domain.
1.3
Aims
The aim of the research is therefore to inform YMLAC about the methodologies appropriate for assessing the economic and social impact of archives, with the expectation that the research will have wider application to the archives domain within the UK. The research also aims to increase knowledge and understanding of the comparative benefits and drawbacks of the identified methodologies and approaches.
1.4
Objectives
The resulting objectives were to:
1
Systematically review the published evidence on economic and social impact assessments adopted in the archives domain, both within the UK and internationally.
The outputs from this programme included the following reports:
Kerslake, E. & Kinnell, M. (1997). The Social Impact of Public Libraries: A literature review. (BLRIC Report 85). Loughborough: Department of Information & Library Studies, Loughborough University. McKrell, L., Green, A. & Harris, K. (1998). Libraries and Community Development: The National Survey of Public Library Authorities. (BLRIC Report 86). Liverpool : Library Association, Community Services Group. Matarasso, F. (1997). Beyond Book Issues: The social potential of library projects. (BLRIC Report 87). Stroud: Comedia. Harris, K. (1998). Open to Interpretation: Community perceptions of the social benefits of public libraries. (BLRIC Report 88). London: Community Development Foundation. Linley, R. & Usherwood, B. (1998). New Measures for the New Library: A Social Audit of Public Libraries. (BLRIC Report 89). Sheffield: Centre for the Public Library In the Information Society.
10
Develop a taxonomy of usage, based on the need to differentiate primary and secondary users, economic and social impacts.
Assess the transferability of other schemes developed for the valuation of information or information services to the archives sector.
Develop an impact taxonomy that addresses economic and social impacts over time
Produce a mapping of impact type against methods, noting any particular problems or opportunities in deploying these methodologies.
1.5
Scope
As outlined in the brief (YMLAC, 2005) this report addresses the social and economic impact of the archive domain in the UK as a whole. In delineating the constituency of this domain, the breakdown identified by MLA in their 2004 Overview of Data in the Museums, Libraries and Archives Sector has been adopted. This is reproduced below in Table 1. It should be noted that much of the commentary and data in this report has however been derived from studies that focus on the public sector, and in particular local authority archives. While the findings can for the most part be generalised across the domain, the specific context of privately held archives may introduce additional factors that would need to be taken into consideration.
National
Institutions responsible for the acquisition, preservation and communication of the archives of a nation, broadly defined. Will include the National Archives; National Archives of Scotland and PRONI
Local authority archive services
County or District Council or Unitary Authority archive service
University, higher education and school archives
Held by public or private sector educational and learning institutions
Religious archives
Held by cathedrals, churches or other religious bodies
Museum & gallery archives
May be public sector or independent museums or galleries. If public sector, archive is not owned or managed by a County archive service
Charity & charitable archives
Held by a private organisation with charitable status
Business & company archives
Held by a private company
Private collections (not commercial)
Including historic house archives, professional and learned societies and other private collections
Other
Other archives, for example community archives Table 1 : Archive domain breakdown, taken from Matty (2004, p.3)
Throughout the report the term ‘domain’ has been used to denote archives, libraries and museums as individual types of organisation. As a collective group, the designation ‘ALM sector’ has been adopted, to distinguish this grouping from the wider ‘cultural sector’, which is seen as encompassing the arts, and other leisure activities of a cultural nature.
11
2
Methods
2.1
Introduction
The methods adopted were based on those of a systematic literature review. This reflected the requirements of the brief (YMLAC, 2005), which called for a synthesis of the existing research evidence, based on a comprehensive literature search (see 2.2), and appraisal of the identified evidence. It was also necessary that any search strategy adopted should be transparent enough to allow future researchers to apply the same approaches to check and update the conclusions drawn from the review. This is comparatively easy in tight disciplinary areas but more problematic for this review given the remit to appraise evidence in other related sectors that could be applied to the archives domain. The possible methods for assessing the secondary use of archives further required an overview of the ways data is collected on viewing figures for media and publishing outlets (see 2.3). It soon became apparent that the potential scope of the review, in relation to literature from outside the ALM sector, would place limitations on the depth at which some issues could be addressed, and this is discussed further in 2.4. This section concludes with a brief overview of how the findings of this process are presented in the remainder of the report.
2.2
Literature review
The approach to the literature review worked outward from existing known reviews (notably Wavell, Baxter, Johnson & Williams, 2002, and Burns Owens Partnership, 2005a) and documentation (often located online) provided by organisations that have responsibilities for the archives domain. The websites for MLA (Museums, Libraries and Archives Council), and the equivalent Regional Agencies were notable sources for existing studies and policies relevant to evaluation or impact. 2.2.1
Literature search
To complement this web search for studies on social and economic impact within the ALM sector, the phase one literature search was conducted utilising the following databases:
Library and Information Science Abstracts (LISA)
Web of Knowledge
OCLC FirstSearch
Social Science Information Gateway (SOSIG)
International Bibliography of the Social Sciences (IBSS)
LexisNexis
EmeraldFullText
Google Scholar
Most hits came from Web of Knowledge, OCLC FirstSearch and LISA. There was little of direct relevance on SOSIG or LexisNexis. IBSS produced only background information. LISA was, unsurprisingly, the most productive database for data specific to the ALM sector, and searches on two timescales indicated that there were more relevant items produced between 2000 and 2005, than between 1995 and 1999,
12
indicating a relatively recent research interest in this area. Approximately 60% of all database-located items came from peer-reviewed sources. At an early stage it also became apparent that the search for data that could be synthesised on the social and economic impact of archives would require gathering fragments from larger studies of the ALM sector. A considerable amount of the relevant literature identified talked about the possibilities of impact evaluation, but very few boasted much data for synthesis. There were lessons that could be learned on methodologies from some of the surveys conducted, however, and relevant details of the methodologies used were recorded. 2.2.2
Widening the scope of the review
A picture started to emerge from some of the research commissioned by MLA, covering public libraries, museums and archives; however, the literature search for the phase one literature review also produced items that discussed:
Theoretical ideas around social and cultural capital (for arts organisations, including museums)
Accountability of cultural and heritage organisations
Volunteering, direct employment and indirect employment
Other impacts that related to social and economic impact (such as those on health/welfare, education and culture)
Non-users, and the problems of attitudes, often professionally and organisationally ingrained, that militated against extending the audience
Viability and vitality of arts organisations (related to accountability)
Using the same databases, a second phase literature review used these concepts as a springboard to collect further details about assessment methods for arts and cultural organisations, the health sector, environmental economics, and transport planning:
Definition and measurement of social and cultural capital
Approaches to the evaluation of social and economic impact
Non-users and contingent valuation techniques
Much of this process involved chain searching (following up bibliographic references from located items) as we found no foolproof method of identifying, through other search mechanisms, many of the relevant reports and monographs highlighted by this means. The lack of visibility of many relevant studies, including material generated by MLA, but more particularly reports to government departments (intended to inform policymaking), lent to the review process an unanticipated element of serendipity. 2.2.3
Processing the documents
Titles, and, if available, abstracts, of documents were visually scanned to assess their relevance to the review. Full text of relevant documents was obtained for further scrutiny. The references of all material that provided background information or data that contributed to the review were entered on to an Endnote bibliographic database by one of the research staff, and further notes about the methodological approaches or type of impact identified added by other team members.
13
2.3
Measures of usage
Data on primary usage was mainly drawn from statistical sources readily identified through the prior knowledge of members of the project team. It soon became apparent that very little has been written about the methods for assessing secondary usage, apart from one study produced for the Archives Task Force (Kenyon, 2003). From an anecdotal perspective, secondary usage of archives clearly derives however from products such as books, periodicals, and documentary programmes. Accordingly, searches were made to identify potential sources for locating and analysing relevant data that could offer indicators of secondary usage, such as TV viewing figures (see 3.4.1, Table 6). Methodologies for evaluating the link between the primary usage of archives and such media products remain however to be identified.
2.4
Limitations
What was evident throughout this process was the wealth of material from sociological and other disciplines addressing the conceptual issues surrounding social, cultural and personal capital, and public value. Much of this literature is written for consumption by individuals cognisant of the discourse in such fields; it was not therefore possible, within the confines of this study, to explore these issues in any great depth. Discussion around these issues focuses for the most part therefore, on how these concepts have been addressed in the literature pertinent to the ALM sector. The writing on evaluation and impact assessment is if anything even more extensive, touching as it does many different disciplines, from the work of economists, to environmentalists. Within the remit of this project, it was not possible to produce an indepth review of all studies undertaken in these fields, and what is presented here is therefore a selective overview of the available methodologies.
2.5
Presentation of findings
The identified literature and methodologies have been analysed and synthesised in meeting the objectives of this study; namely in the production of taxonomies of usage and impact, and in the identification of potential methodologies for the evaluation of impact within the archive domain. The remainder of this report presents the findings of this process:
Section 3 explores the concept of usage, extending this beyond the traditional primary user to encompass wider secondary usage, and presenting a taxonomy of use based on forms of service use, and purpose of use. Potential measures of secondary usage are also discussed.
Section 4 considers what is meant by impact, the complexities of its definition, and how the concept has been applied in the economic and social spheres. A taxonomy of impact is presented based on a view of impact as a continuum which builds cumulatively over time, and extends from the individual to associated groups, communities and thus to society as a whole.
Section 5 then reviews the issues associated with impact evaluation, providing an overview of existing activity relevant to the archive domain, ALM sector and wider cultural scene, and outlining the key methodologies utilised in other fields.
Our conclusions and recommendations are presented in section 6.
14
3
Developing a Taxonomy of Usage
3.1
Introduction
The development of the following taxonomies, and the identification and evaluation of the methodologies available to undertake social and economic impact assessment, is predicated on the recognition of the fundamental symbiotic relationship between the two concepts of ‘usage’ and ‘impact’. For the archive domain these are issues that have really only begun to come to the fore in recent years. While archives as a resource have arguably always had some form of ‘cultural identity’ role, the association with meeting community needs is something that has only recently achieved wide acceptance at the service delivery level. This position can be contrasted with the library domain which is perceived in terms of a wider user constituency, and where concepts such as ‘community librarianship’ began to emerge in the 1970s, even if they did not reach full realisation.2 The continuing low profile of archives as a domain means that even though services are seeking to engage with a wider constituency, recognition of what they do is limited. Usage levels are low in comparison with comparable services such as libraries and museums, but are nevertheless at a level that challenges the ability of many services to cope. The brief for this project (YMLAC, 2005, Appendix I) draws on the central belief “that archives have an impact out of all proportion to that experienced by the comparatively modest number of primary users.” This section will therefore explore the context of archive usage, and how the concept of usage in the archive domain might extend beyond the traditional primary constituency. Drawing on this discussion, a taxonomy of usage is then presented, as the basis for further exploration of the impact of service provision.
3.2
Archive usage trends
This report is concerned with the question of how archives can demonstrate their impact on society and the economy. A study of this nature reflects both the political and social context within which archive services operate. Even as recently as fifty years ago such a study would lack relevance both to the nature of the usage of archival resources and to the political context within which services operated. The early development of archive provision was closely associated with the needs of an academic historical constituency; where the research products of this association, were also, for the most part, intended for academic consumption. Furthermore, the concepts of social and economic impact were not well understood; archive services were not faced with the need to justify themselves in these terms, and at local government level at least, were still in a phase of development where their establishment as part of local authority service provision was widely accepted as a ‘good thing’ and not subject to indepth scrutiny. If the pattern of usage had remained restricted to the academic sphere, it would be hard today to justify the required expenditure of resources. However, as statistics from the
2 Black, A. (2003). False optimism: modernity, class, and the public library in Britain in the 1960s and 1970s. Libraries and Culture, 38(3), 201-213.
15
late 1970s onwards demonstrate, there has been a huge expansion in user levels, ironically in a period when curbs on public spending have seriously affected the funding of many services.3 The user-base that this encompasses now extends beyond academia, to reflect the rise in popularity of the heritage dimension of history, and in particular genealogical and local history research.4 This shift in user profile has not always been viewed positively by the profession, and there has been a tendency to be dismissive of family historians in particular.5 However, as the US commentator Bearman noted in 1989 the profession’s “preferred user, the scholar, does not come in great enough numbers to justify archives.” Recent user surveys (PSQG, 2002, p.27) indicate that 63% of UK users are pursuing personal leisure interests, compared with 15.3% engaged in formal education as a student or researcher. Figures from the 2001 equivalent survey (PSQG, 2001a, p.3) delineated between categories in more detail and indicated that just 1.1% of users fell into the category ‘research by academic staff’, although a further 1.5% were engaged in a ‘research degree project’.6 The characterisation of family and local history pursuits as being of marginal importance – reflecting narrow, sentimental even nostalgic personal interests – also fails to take account of underlying social needs and consequent benefits. As Hall (1987, p.5) noted in the late 1980s, local history “can be used as a unifying bond in a community and can serve as a catalyst to bring people together.” Similarly, writing in the context of Canadian provision, Craig (1987-88, p.10) argues that “the phenomenon of the genealogist has been misunderstood”, and that it represents the satisfaction of a “deep social need” to establish “personal bonds with society.” Such views accord with the thinking of Lowenthal (1998) and others that the growth of interest in ‘heritage’ reflects social needs to establish a firm foundation in a world where change is endemic. Although archive usage has expanded considerably over the last three decades, the nature of service provision itself has tended to remain focused on core management and preservation functions, including the provision of on-site user access services. In contrast, public library services have expanded beyond their traditional core ‘bookbased’ functions, adapting the ‘library idea’ to contemporary social needs (Matarasso, 1998, p.38). In this way, although libraries have seen a decline in traditional usage, they have diversified to provide a wider range of services that appeal to, and bring in more users. Archive services have thus far been able to ride the crest of the wave of interest in family history research, and although some services have responded
3
In the local authority context, Forbes, H. and Dunhill, R. (1997) note an average increase of over 20% in visitor numbers between 1992 and 1996, and 130% since 1979. Steady increases in user numbers have been an ongoing feature of service provision, although current trends in online access are beginning to affect on-site use.
4 By 1984 one survey (Smith, 1987) was already indicating that genealogists formed over 51% of local record office users; the 2002 PSQG survey found that 71.8% of users were researching family history (PSQG, 2002, p.40). Indications from the 2004 PSQG survey results (IPF/PSQG, 2004) are that there has been an extension of research interests, but the weighted figure for family history research still stands at 58.9% overall. 5
Tucker (2005, p. 14) indicates that it is only in the last ten years that a greater willingness to document the importance of family historians has emerged.
6
It should be noted that these figures reflect the constituency surveyed (a self-selecting sample, in which the local authority sector appears predominant). As will be noted later, this survey does not delineate its findings on the basis of the constituent sub-domains, and it is likely that user categorisation would be heavily influenced by the nature of the sub-domain; for example, it is to be expected that repositories operating in an academic context would have a relatively higher proportion of academic users.
16
innovatively in meeting access needs, others have almost been overwhelmed by demand. Amalgamation with local studies and other services has widened the scope of service provision in some instances – but capacity issues often place archive services in a reactive position when seeking to engage with user constituencies. While the archive domain occupies a more restricted professional arena than its library and museum counterparts, increases in usage levels have placed archive services in a position to demonstrate the impact and relevance of archives in both social and economic terms. The domain remains, nevertheless, a relatively small player in comparison to libraries and museums, let alone the wider cultural sector. This points to the need, identified by YMLAC (2005, point 4), to espouse “a broad definition of the term ‘usage’” and recognise that the impact of archive use reaches a much wider constituency than simply visitors through the door.
3.3
Broadening the concept of usage
Broadening the concept of the term ‘usage’, as it applies to the archive domain is therefore one of the fundamental underpinning themes of this study. YMLAC (2005) delineated in their brief between primary and secondary usage. We would contend that this is part of a continuum of usage which encompasses the record creator at one extreme, and what in relative terms, might be deemed the ‘non-user’ at the other. 3.3.1
Primary users
Primary use remains the most straightforward and readily apparent aspect of archive usage. YMLAC (2005, Appendix I) offer the following definition as a starting point: Primary user refers to the individual reader, researcher, enquirer or web browser who has direct contact with the archive service. In assessing primary use within the archive domain it is however necessary first of all to establish the parameters of usage. This involves delineating:
The extent of the domain (types and numbers of services operating)
The forms of service provided (on-site delivery, remote enquiries, web-based services, etc.)
These parameters provide the necessary context for the analysis of measures of actual usage. It is important to approach any assessment exercise systematically, establishing at the outset the purpose of the exercise, applying appropriate parameters, and developing effective instruments to gather the required data. As Williams and Procter (2002, p.31) note in relation to existing data collection in the domain, “there is a lack of clarity of the purposes for which data is collected and thus difficulty in carrying out any meaningful analysis.” One consequence of this, as Williams (2003, p.78) further notes, is that “little effective use is made of the vast amount of data that is currently collected.” Having established the parameters within which primary usage is to be assessed, consideration must then be given to the types of data required. At the very least any enquiry is likely to be interested in level of usage, other aspects that may be important, depending on the purpose of the study, might include who these users are (in terms of their demographic profile), their motivation or purpose in engaging with the service, and their satisfaction with the experience. These aspects already feature to one extent or another in the current mechanisms used within the domain for assessing primary usage.
17
Extent of Domain The difficulties of defining the extent of the archive domain have not gone unrecognised, and the implications of this for the evaluation of service provision are acknowledged by a number of commentators. Pickford (2002, p.10) sums up the situation as follows: Until it is clear how many archives services there are, and of what type, detailed statistical analysis and comparison will be hard – if not impossible. Table 1 (section 1.5) outlines one possible categorisation of archive service types, however, as its originators PLB Consulting Ltd. note (2004, p.48) excepting the local authority sector, there is a lack of definitive data on the total population of archives held by other public sector institutions, such as higher and further education. Furthermore: Difficulties in defining what constitutes an archive once one moves out of the public sector create a problem when one is seeking to quantify overall demand patterns across the domain. This situation is also acknowledged by Matty (2004, p.56) who notes that “it is very hard to gain a statistical overview of the domain or to find widely accepted definitions for the different types of services and collections.” Matty’s overview of the available data, does identify some of the means by which the domain might be defined (p.56-7)7 but is forced to conclude that: None of these methods produce watertight or generally applicable categories. More importantly, they do not represent groupings for which coordinated statistical returns are available at national or regional level. Differences in constituency served also make it difficult to generalise about primary usage across the domain. This is an issue that has particular implications for data collection intended to encompass the full domain, since some measures may be not be applicable at sub-domain or even individual service level. Pickford (2002, p.16) suggests that a useful approach might be to classify archives in terms of organisational context and remit; his suggested classification identifies four resulting ‘types’ of service (see Table 2 below). Type
Definition
Archives Services
Archives as main activity of organisation or department
Archives Units
Archival activities as an identifiable secondary function of an organisation with other responsibilities (for example, library or museum)
Small Archives
Archives held peripherally as a minor element of the holdings of a library, museum or similar body
Archive Holders
Archives in the care of creating organisation
Table 2 : Pickford’s template for classifying archives by organisation context and remit
7
Inclusion in ARCHON, recognition as a Place of Deposit for Public Records, TNA ‘approval’, and the categorisations identified by the Regional Archives Strategies.
18
The approach taken by the newly-instituted DCMS / National Statistics Taking Part survey (2005, Appendix Part B, p.23) defines the domain generically in terms of “places that keep archives”, noting that these are “usually called a record office or archive centre.” While grouping archives in this way offers the opportunity to gain a statistical overview of usage across the domain as a whole, the lack of delineation of sub-domain limits further analysis. Forms of Service Provision The report of phase two of the English Archival Mapping Project, Our Shared Past: Developing 21st Century Archive Services (2001, p.17) was able to note that “English local authority archive services have a strong tradition of responsive, well-respected public service”. It is apparent that user satisfaction with these services is high, as demonstrated by responses to the PSQG Surveys of Visitors to British Archives8, where on-site users are asked to rate their satisfaction with key aspects of their engagement with the service, including: opening hours; finding aids; document delivery; microfilm and IT facilities; and copy services. For the purposes of this study however, we are concerned more explicitly with the overarching forms of service provision with which users may engage, rather than the activities that such engagement involves. In this context, it is important to recognise that while on-site use continues to be a significant feature of primary usage, such usage also extends to the services provided outside the walls of the archive service. These services might take the form of outreach activities (exhibitions, talks, school visits, etc.), remote enquiries, or increasingly today some form of ‘virtual’ usage. Changes in technology, particularly developments in ICT have had an enormous influence on the form that service provision may take across the archive domain. The Archives Task Force report (MLA, 2004a, p.24) claims that: The archives domain has led the way in developing innovative approaches to maximising the potential of digital networking to deliver real service benefits to all. Its appendix B lists an impressive range of archive networking achievements, and with its ‘big idea’ for a Gateway to UK Archives, the report promotes further development in this area. Forms of service use might therefore be delineated between on-site and off-site services (mediated and un-mediated), and those which involve the user in active engagement with resources, or where the user experience is passive. There is a difficulty here in defining the boundary between self-directed active research and that which is mediated in some form.9 For the purposes of this study, the following definitions have been adopted (see also Table 3):
8
PSQG (1999a, p.2) 96.7% of respondents rated the archive’s service as either excellent or good overall across the UK; the same figures for both 2001 and 2002 stood at 94.4% (PSQG, 2001a, p.22), (PSQG, 2002, p.14). Comparative figures for 2004 (IPF/PSQG, 2004, p.13) were only slightly lower at 93%. 9
It is accepted that findings aids might (and perhaps should) be deemed as a form of mediation between the original archival source and the primary user; however, for the purposes of this study it was felt to be necessary to distinguish between the self-directed active research undertaken utilising primary archival products such as catalogues to facilitate access to original archival sources (hereafter deemed unmediated), and that which is directly mediated by the intervention of the archivist, through what is termed a secondary archival product.
19
Primary User – a user who engages with an archival product Primary Archival Product – the service product of archival processing of original archival sources (catalogues and other finding aids, physical and online)10 Classified as un-mediated since it presents information on the full range of archival sources, from which the user selects what is required, and moves on to engage with the original sources themselves Secondary Archival Product – the service product of an archivist’s engagement with primary archival products and original archival sources (this encompasses enquiries, and more tangible service products such as education packs, exhibitions, talks, and presentation of digitised resources, but excludes commercial publication) Classified as mediated since it presents information selected by the archivist from the full range of original archival sources Active Use – where the user is actively engaging with primary and secondary archival products (notably finding aids) as the means to facilitate access to original archival sources, through self-directed research Passive Use – where the user experiences engagement with secondary archival products presented through the mediation of the archivist (for example, an exhibition, digitised resource, or talk) On-site – where engagement with archival products takes place on the premises of the archive service Off-site - where engagement with archival products takes place outside the premises of the archive service, within the community, or remotely via the internet
Mediated
Un-Mediated
Enquiries (personal) On-site Services
Use of findings aids to access original archival sources
Attendance at: Exhibitions; Talks; Visits Enquiries (postal; email; telephone)
Off-site Services
Attendance at: Exhibitions; Talks Use of: Education packs; online digitised resources
Use of online catalogues / networks to identify original archival sources
Table 3 : Breakdown of forms of service provision
10
Although there is a sense in which original archival sources might themselves be seen as primary archival products – where archival processing of records ‘produces’ the archive – such sources are seen here as distinct from the resulting service products, namely catalogues, indexes and other finding aids.
20
By definition therefore passive primary use is always mediated in some form; although active use may involve some aspect of mediation (for example, through enquiry processes), it will generally be un-mediated. This might be represented as shown below in Fig. 1, where the passive user process is represented by the thin line linking the user to the secondary archival product via the archivist, and the thick line the process of active use. The line linking ‘enquiry’ to the un-mediated primary archival product represents the link between active use and the mediated elements of the enquiry process; the dashed line at the end of the passive use process signifies the potential for a purely passive user to transform into a future active user.
mediated
SECONDARY
ARCHIVAL PRODUCT Education pack Exhibition Talk Digital Resource
ARCHIVIST
USER
ARCHIVE
Enquiry
un-mediated
PRIMARY
ARCHIVAL PRODUCT Catalogue A2A etc.
Fig. 1 : Representation of passive [
] and active [
] use processes
The existing mechanisms for assessing primary usage do not currently take full account of the different forms of service available, although in part this reflects their methodological approach, and the complexities associated with measuring aspects of remote use. Demographic Profile Government policy initiatives such as social inclusion and cultural diversity have highlighted the importance of profiling the demographic base of usage. For the archive domain the PSQG surveys provide the key current source of data in this area, aside from any user profiling undertaken by individual institutions. These surveys report on demographic indicators such as gender, age, educational profile, ethnicity, and disability. Information on social class is not sought, although the 1998 survey did include occupational data from which some inferences can be made.
21
Matty (2004, p.65-70) provides a useful analysis of the findings from the first four surveys (1998, 1999, 2001, 2002), which can be summarised as follows:
Gender – the ratio of male to female users has remained relatively constant, with neither gender exceeding 55% of all users. On balance however, male users do seem to outnumber females by up to 10%.
Age of archive users – all four surveys reveal that the majority of users are in the older age groups with around 75% being over 45. To some extent this may reflect the fact that the surveys only record individual visits, and thus exclude educational groups.
Education of archive users – evidence from the 2001 survey shows that users have a broad educational background; the assumption that archives cater mainly for the academic community is not borne out, with only 12% having a research degree and 30% a first degree, both of which may be wholly unrelated to their use of archives.
Ethnicity of archive users – data here clearly points to a constituency of almost exclusively white ethnic groups11 ( 98.4% British, Irish and other white).
Disability of archive users – comparison of data from the 1998 and 2002 surveys indicate that some progress has been made in making services accessible to people with disabilities, with an increase from around 2% of all users to 7% with visual, hearing or physical disabilities in 2002. Considers that the predominance of people from older age groups among visitors to archives may be influential here.
These conclusions are broadly supported by the data from the 2004 survey (IPF/PSQG, 2004). It is apparent from these findings that the core demographic of primary usage, can be typified as white, male and over 45. This is often interpreted negatively in the context of social inclusion policy, as demonstrative of a lack of engagement with young people and ethnic minorities in particular. However, as Matty (2004, p.69) points out these figures should be seen in the context of the composition of the UK population as a whole, where the 2001 census figures indicated that 91.31% of the English / Welsh population was white. Furthermore, findings of studies into leisure inequalities and the socially excluded (Roberts, 2004, p.10) indicate that it is “the oldest age groups, not young people (or any especially disadvantaged section of this age group), that have the lowest overall rates of leisure participation.” In this respect, the archive domain’s primary user constituency12 can be seen to be running counter to overall trends, and perhaps fulfilling valuable social needs. These are however complex social issues and more research would be needed to clarify such points.
11 In England and Scotland 98.4% of users were encompassed by the British, Irish and other white categories, this was slightly higher in Wales at 99.3%. The highest rates of ethnic minority use were reported in London (3.4%) and the West Midlands (1.5%). (PSQG, 2002). If statistics from group visits were included (particularly educational groups), a wider range of ethnic minority use might however be evident. 12
The 2001 survey reported that 44.4% of users were aged 70+ (PSQG, 2001a); the 2002 statistics were collected on a slightly different basis, showing that 50.2% of users were aged between 45-65 and 26.7% were over retirement age (65+), (PSQG, 2002).
22
Motivation / Purpose of Archive Use Any typology of primary use also needs to take account of the motivations of users and the purpose served. The key source of data here is once again the PSQG surveys of users, which include data on the main purpose of the archive visit. The categorisation of ‘purpose in visiting the archive’ has evolved over the course of the surveys undertaken; the categories adopted from 2002, including the current 2006 survey are as follows:
Personal leisure / recreation
Non-leisure personal or family business
Formal education as student / researcher
Formal education as teacher
Work in connection with your employment (both paid & voluntary)
This simplifies the categories from the earlier surveys, although these did offer more precise information, delineating specifically between aspects such as media research, research by academic staff, and types of educational project. The types of use the revised categories encompass, in relation to the previous categorisations, are shown in Table 4. Personal leisure / recreation Personal interest or hobby
Non-leisure personal or family business Other
Formal education as student / researcher
Formal education as teacher
Work in connection with your employment
School project (student)
School project (staff)
Media research
Adult education project
Adult education project
Research for business or public body
Taught degree or further education
Taught degree or further education
Professional researcher
Research by academic staff Research degree project Table 4 : Relationship of pre-2002 PSQG categories to those adopted from 2002 onwards
The addition of the question, from 2002, ‘are you researching family history’, highlights the importance of this particular purpose of use, and as results show this remains the dominant motivation for the primary use of archives.13 This will obviously form the major
13 In 2002, 71.8% of users across the UK placed themselves in this category of use (PSQG, 2002, p.40); there is a potential discrepancy here with Q.8 (regarding the main purpose of the visit), which indicates that not all respondents would place researching family history in the ‘personal leisure / recreation’ category (where the figure stood at 63%). This could reflect the fact that a proportion of those engaged in family history research may be professional researchers, and thus class the purpose of their visit as ‘work in connection with your employment.’ Weighted figures from 2004 (IPF/PSQG, 2004, p.36) across the UK are indicative of a widening of research use, with only 58.9% of users undertaking family history research; it remains to be seen if this trend is supported by the findings of subsequent surveys.
23
constituent of the ‘personal leisure / recreation’ category alongside activities such as local history and house history research. The 2001 survey also included questions aimed at investigating the initial motivation for archive use, and whether the use of archives had broadened from this original purpose to encompass other uses. From this data, commentators on the survey (PSQG, 2001b, p.3) concluded that: It is also clear that many (68%) began by getting involved in an activity (such as family history), have since moved on from their original purpose to research other subjects (61%) and most (81%) have gone on to use other repositories. Archives are clearly very good at retaining users and developing their interests and expertise. This development is usually seen in light of the transition from researching family history to house or wider local history, all of which fall within the category of ‘personal leisure / recreation’, but it may also encompass aspects of ‘non-leisure personal or family business’ use, as increased awareness of archives opens up other possibilities for research, and emphasises their evidential value. This category would encompass aspects of the evidential use of archives in relation to property matters, personal status and so on; the pre-2002 categorisations neglected this element which although representing a relatively small proportion of overall use (11.1% in 2002) can be particularly significant in terms of consequences for the individual concerned.14 Another factor that could be influential here is the level of mediation involved, with users often extensively supported by archival staff, as part of on-site service provision. Current Measures of Primary Usage MLA’s Statistics Programme has examined the sources of data available for the archive domain in some detail, and has already concluded (Williams & Procter 2002, p.4) that relevant data is only routinely collected in the local authority sector, and that as a whole (p.33): There is no established infrastructure for the collection and management of data across the archival domain, and there is significant unevenness between the sectors within the domain in the nature and comprehensiveness of data collected. Although the current collection of CIPFA statistics does generate a range of data relevant to the assessment of level of primary usage15, it does not provide information on remote use, or address the demographic profile of users, or their motivation / purpose in using archives. Furthermore, the data is restricted to the local authority subdomain, and past analysis (Williams & Proctor, 2002), (Creaser, 2002), (AKA Partnership, 2003) has revealed underlying inconsistencies, incompleteness, and inaccuracies. AKA Partnership (2003, p.5) therefore conclude that it is “difficult to base much meaningful analysis on the survey, certainly before 1998/99 and to some degree thereafter."
14
Cox and Wallace in their recent book Archives and the Public Good: Accountability and Records in Modern Society (2002) provide a valuable overview of these kinds of uses of archives.
15
The section on ‘Use of Resources’ now includes data on numbers of readers, numbers of exhibitions and displays, numbers of talks / lectures given and attendance at these (with separate data for talks given by archive education officers), and number of enquiries (postal, email and telephone) received.
24
The PSQG Surveys of Visitors to British Archives do address some of these issues, in that they seek to profile the user constituency, incorporating data from the higher education, museum, library, private business and specialist sub-domains as well as from local authority services. Qualitative data on user satisfaction is provided alongside quantitative data profiling the user demographic.16 From 2002 the PSQG surveys also add weight to primary usage statistics, by introducing the dimension of ‘dwell-time’ (asking users how long they stayed at the archive17); figures which can be favourably contrasted with the relatively shorter visits to UK libraries, for example. Significantly, data is also provided on the purpose of primary usage, and on the interaction of the user with the local economy. The PSQG surveys are however the product of a methodological approach that focuses on exit surveys of individual on-site users, and have thus been criticised for failing to address other key elements of the primary user constituency; “a significant omission is that the views of remote users and those of group visitors are not sought." (Williams & Procter, 2002, p.18) Furthermore, although regional breakdowns of data are included, there is no analysis by repository type or sub-domain. This has implications for generalisation given the self-selected nature of the sample, which does tend to be broadly representative of the public sector, and local authority repositories in particular. Nevertheless, as a measure of user profile, it still provides “the only longitudinal quantitative data to be found in the museums, libraries and archives sector.” (Burns Owens Partnership, 2005a, p.29) These represent the only two large-scale sets of data collected regularly within the archive domain through to 2005. The DCMS / National Statistics Taking Part survey, which commenced in mid-July 2005, does provide a further mechanism for data collection, utilising robust approaches to surveying randomly selected households. This has the potential to provide data on levels of use across the population as a whole, however the focus is on profiling users within a very generic categorisation of archive ‘visits’, and does not appear to offer the scope for more sophisticated domain analysis. The picture from these sources can be further supplemented by the findings of ad hoc data collection exercises, such as the mapping projects undertaken in England, Wales and Scotland18, and the activities of individual repositories. Nevertheless, as Pickford (2002, p.9) concludes, the available sources of data, although strong for some areas (notably local authorities) are “very weak or non-existent for others (higher education, business and specialist archives)”. This has real implications when attempting to look across the archive domain as a whole. 3.3.2
Secondary users
The Archives Task Force report Listening to the Past, Speaking to the Future notes how “for the majority of the people of the UK, the significance of archives and the richness they contain has remained no more than a footnote in the histories told in books, in class or on television.” (MLA, 2004a, p.15) It is these invisible, but nonetheless significant aspects of use which this study seeks to make explicit.
16
Surveys typically provide data on age group, gender, educational qualifications, and ethnicity
17
Findings from 2002 showed that over 40% of UK user visits were typically more than three hours in duration (PSQG, 2002, p.20), in 2004 the comparative figure stood at 60.2%, with 32.5% indicating that their length of stay exceeded five hours (IPF/PSQG, 2004, p.17).
18
The two phase survey of English local authority record offices, Our Shared Past: An Archival Domesday st for England (1997-8) and Our Shared Past: Developing 21 Century Archive Services (1999-2000); and the Scottish and Welsh projects The Archival Mapping Project for Wales and An Archival Account of Scotland which also included national institutions, universities and specialist repositories within their remit.
25
YMLAC (2005, Appendix I) defines this kind of ‘secondary’ use in the following terms: Secondary user refers to the individual who has direct contact with the product of the research undertaken by the primary user – the book, TV documentary, lesson, etc. For the purpose of analysis, the parameters of secondary use extend to the public as a whole, access is necessarily off-site, and the delineation between mediated and unmediated use is not applicable. A distinction can be made however, between the active use of research products as part of self-directed research activity, and the passive experience of engagement with research products, usually for entertainment or leisure purposes. In the context of secondary use therefore, the following definitions have been adopted: Secondary User – a user who engages with a research product Research Product – the product of a primary user’s engagement with original archival sources Active – where the user is actively engaging with research products through self-directed research to meet a specified objective Passive – where the user experiences engagement with research products, to satisfy a general subject interest Interestingly, it appears that little attention is paid within the archival literature to the concept of secondary usage. This is not to say that the archival community is unaware of its potential;19 it was highlighted as part of the Archives Task Force review, and Annex F of the report (Kenyon, 2003) specifically identifies indicators that can be used to measure the popularity of history in the UK, on the basis that (p.2): The popularity of local history, family history, oral history and ‘heritage’ is generally underpinned by the content and accessibility of archives. Furthermore, MORI (2003) in their survey of non-archive users (see Fig. 2 below), include within their table of history related activities a number of uses that fall within the definition of secondary usage. This survey (2003, p.2), interviewed a nationally representative sample of 1,953 UK adults; of this number only 10% claimed to have ‘used an archive / record office’ in the previous 12 months. However, other history related activities scored much higher, with watching a historical documentary on TV topping the scale at 60%. These figures broadly reflect the findings from another omnibus survey, Interest in History (MORI, 2002) conducted for the then Public Record Office, which indicated that while only 6% of those surveyed had visited ‘an archive or record office’, 69% had ‘watched a TV documentary on a historical subject’, and a further 29% had ‘bought or read a factual book on a historical subject’. Such findings can be seen as a reflection of the ‘democratisation’ of history, and the evidently increasing popularity of historical televisual programming in particular. Kenyon’s findings (2003, p.58) corroborate this position noting how:
19 The NCA’s Giving Value report (2005) also notes the rise of popular history in the media (3.5.3, p.24); how much of this is based on primary research in UK archives, that is often rendered invisible in the final product. Who Do You Think You Are? is noted as an exception to this, with its consequent impact on archival use.
26
The BBC perceive that there is a sustained interest in history programmes among the public. There may be peaks due to fads or particular events such as the millennium but overall it is steady and probably on the increase.
Top 10 Activities Done in the Past 12 Months Q1 Which, if any of the following activities have you done in the past 12 Months? 60%
Watched a historical documentary on TV
38%
Visited a museum, historical exhibition or something similar
25%
Read a non-fiction book about history
24%
Used or visited a local studies library or history centre Read a magazine about history, local history or family history Looked up history subjects on the internet Read about / researched the history of your local area / town / village Researched your family tree
22% 21% 17% 13% 10%
Used an archive / record office Researched the history of your house / home
5% Source: MORI
Base: All GB Public (1,953)
Fig 2 : History related activity, taken from MORI (2003, p.2)
This phenomenon is not confined to the UK; Nesmith (2004, p.18) writing in the context Canadian archival provision also notes how: Television and the motion picture industry now also bring to millions of people an increasing number of popular dramatizations of historical events, biographies, and novels. Many of these are based heavily on archival materials. These programs made archives more accessible, as well as historical subject matter, since archival documents were often read or displayed on screen. Archives have been at the heart of the success of these programs, as the lengthy credit list of archives at the end of a typical historical documentary indicates. The significance of television in modern life is underlined by Roberts (2004, p.7) who notes that television is one of the big three leisure pursuits (alongside holidays / tourism, and alcohol consumption), accounting for around 40% of all leisure time. The popularity of historical programming must therefore be viewed in this context. Assessment of actual consumption can be determined relatively precisely by evaluating the viewing figures for television broadcasts. The Broadcasters’ Audience Research Board Ltd. (BARB) publishes, through its website, estimates of the audience for television channels and programmes in the UK, and on subscription can provide more detailed breakdowns for particular channels / programmes. Basic data is however freely available; for example, taking as an example the History Channel (a nonterrestrial channel which forms part of a wider subscriber package) and UK History (a
27
free ‘non-terrestrial’ service), where programming is dedicated solely to history, audience viewing figures can be retrieved on a weekly basis. Figures for November 2005 are set out in Table 5 below. Weekly Reach 000s
Weekly Reach % UK population
Share of Total Viewing %
06/11/2005 13/11/2005 20/11/2005 27/11/2005
1498 1658 1786 1633
3.8 4.1 4.5 4.1
0.2 0.2 0.2 0.1
Average
1644
4.1
0.175
06/11/2005 13/11/2005 20/11/2005 27/11/2005
3609 3370 3527 3370
9.1 8.4 8.8 8.4
0.4 0.4 0.4 0.4
Average
3469
8.7
0.4
Week History Channel
UK History
Table 5: Viewing Figures for The History Channel / UK History20
These figures may obviously include some degree of ‘overlap’ in audience and for that reason have not been aggregated. While they may appear low, the very existence of these channels is demonstrative of how in "today's expanding multi-channel television universe, . . . a niche has been created for historical programming." (Epp, 2000, p.58) Furthermore, the commercial basis of most modern broadcasting (and in this case the History Channel) have to be seen to some extent in the context of the viewers “ability and willingness to pay.” (Roberts, 2004, p.7) BARB also provides ‘top 30’ audience listings for terrestrial channels, from which it is apparent that individual instances of historical programming do also attract large viewing figures. In the last three weeks of January 2006, the family history programme Who Do You Think You Are? regularly topped the audience listings for BBC2, with an audience of well over 5 million each week. Information supplied by the BBC to Kenyon (2003, p.58) indicates that the lowest viewing figures for history programmes on the main BBC channels are around 2 million, and a high of 9 million was quoted for one programme. Although, as MORI (2003, p.2) note, watching historical documentaries is also “an activity that nine in ten (89%) of primary archive users” have pursued, these figures still indicate that there is a substantial secondary user constituency for archives among the general population. Anecdotal evidence from individual services also suggests that one effect of secondary use, in relation to television programmes with significant archival content, is actually to generate more primary users. This effect has been most recently noted in relation to the family history programme Who Do You Think You Are?21 20
Source: BARB: Broadcasters’ Audience Research Board Ltd. (2005) Multichannel Weekly Viewing Summary. Retrieved January 30, 2006 from http://www.barb.co.uk/viewingsummary/weekreports.cfm 21
NCA (2005, p.24) notes how the show’s popularity “was translated into increased demand for archival sources at both a national and local level.”
28
Other activities, also identified by MORI, such as reading ‘a non-fiction book about history’, or ‘a magazine about history’ provide further evidence of secondary consumption of archive research products. Kenyon (2003, p.50) indicates that history books comprise around 4.5% of the new titles published each year, and identifies 132 UK history periodical titles, for the majority sample of which estimates of “a total readership of 1,950,000 – 2,100,00 can be assumed.” Kenyon (2003) also identifies other indicators such as figures for membership of local and family history societies, and the popularity of history courses in higher education. The PSQG surveys also give some indication of the intended outputs from primary usage. This question first formed part of the 2001 survey, and from 2002 (in slightly amended form) categorised intended outputs as follows:
In a written report, not for publication
In a publication
In a lecture or talk
In an exhibition
In the media
On a website
Although most of the respondents in the 2004 (IPF/PSQG, 2004, p.24) survey (59.3% across the UK) considered that the output of their research would fall into the first of these categories, a significant proportion were still aiming at wider dissemination, mostly through publication (41.7%), but also in lectures / talks (24.2%). Publication in the media was the lowest category of response at 3.3%, but given the potentially wider audience that such media can reach is nonetheless significant. These outputs form a proportion of the research products for secondary consumption; the survey data also considers the question of with whom they might be shared. This data was first gathered in 2001, and with slight amendment from 2002 now encompasses the following categories:
No-one else (only for my own personal interest)
My family and friends
People in my community, e.g. where I live
Colleagues / students, for formal educational purposes
Business, work or professional clients or colleagues
Here (IPF/PSQG, 2004, p.23) the highest category of response (at 48.6% across the UK) was the second, ‘family and friends’, formal educational purposes also scored highly at 25%, and only 15.7% said they would not be sharing the outcomes of their research with anyone else. What is encouraging about these figures therefore is that in most cases the outputs of the research are being disseminated, and are thus reaching potential new audiences. Overall it appears that the circumstantial evidence for widespread secondary use is considerable. As Epp (2000, p.56) points out, through television in particular: thousands and even millions of viewers become beneficiaries of historical information and, thus, indirect users of archives even though many may not realize the presence of the archival record or the archival institution behind it.
29
3.3.3
Non-users?
With the recognition of the secondary usage aspects of archives, the concept of nonusers actually becomes much more problematic. Potentially everyone who engages with modern media, whether televisual or literary based could be a secondary consumer of archival research products. However, defining a non-user as a ‘non-primary user’ ignores the large constituency of secondary users already identified. Sabin and Samuels (2003, p.8-9) tackle this issue by dividing potential users into five segments, based on their intrinsic motivations (see Fig. 3 below). Of these it could be posited that ‘the indifferent’ and ‘outsiders’ might best be categorised as non-users, although ‘outsiders’ at least may also fall within the category of secondary use. ‘Latent supporters’ are more likely to be secondary users, and ‘personal questers’ are almost certain to be so, and may already have entered the field of primary use. ‘Advocates’ being those who are already regular primary users.
Advocates High interest. See the big picture Personal questers High potential. Have tried some services Latent supporters Need to be shown what’s available 50+ and educational interest Outsiders Self-excluded – too busy or too many alternative pursuits Uninvolved – archives failing to be relevant The Indifferent No intrinsic interest
Sabin-Samuels
Fig 3: Customer segments (Sabin & Samuels, 2003, p.8)
3.4
A taxonomy of usage
In developing a taxonomy of usage there are therefore four ‘dimensions’ that might be considered: demographic profile of user; form of service use; extent of domain; and purpose of use. This accords with questions such as ‘who’ is doing ‘what’, ‘how’, ‘where’ and ‘why’. The following taxonomy draws on the discussion of these issues but seeks to reflect broad typologies of usage and forms of service delivery, rather than user demographics; neither is the taxonomy differentiated on the grounds of subdomain.
30
This decision is based on the need to balance detail with simplicity in order to produce a taxonomy with the conceptual clarity for meaningful domain analysis. Type or purpose of usage is conceptually applicable across the domain, as is form of service use; particular aspects might not apply in all instances, but this is more likely to be at the level of individual services rather than a sub-domain as a whole. Demographic profiles, while providing valuable data for planning, do not provide a useful basis for conceptualising archive use. The taxonomy itself is presented in two parts, one from the perspective of form of service use (Fig. 4), the other focused on purpose of use (Fig. 5) – these need to be viewed in tandem, such that for any given form of service use, the purpose of use can be analysed, and vice versa. Within this taxonomic representation, the broad divisions of primary and secondary usage on which YMLAC’s brief was originally based, are presented in relation to form of service use, with subsequent classification drawn from the discussion and definitions in sections 3.3.1 and 3.3.2; use of archival products (notably primary products) leading on to use of original archival sources. This forms part one of the taxonomy; in the second part, categories of use drawn from existing studies, notably the PSQG Surveys of Visitors to British Archives, have been used to differentiate between types of usage. This approach has been adopted for the sake of clarity; since combining the two parts would produce a taxonomy that would appear unnecessarily complex. These taxonomies also need to be viewed in the context of a value-added knowledge supply chain that extends out through time from the creation of the record. The record creator generates the body of records from which original archival material is selected and processed, and primary archival products (finding aids) are produced which add value in terms of aiding users to locate the information they need. Archive staff may themselves use these products (and indeed the original sources), adding further value for exhibition or other purposes, in producing secondary archival products. In turn researchers use these sources to facilitate access to original archival materials, in order to produce a research product which adds value in the context of their particular purpose, and for the secondary consumers of the product. Such a continuum of ‘valueadded’ usage can be conceptualised as show in Fig. 6. From this it can be seen that the knowledge gained from secondary use of research products may in turn be incorporated into further research products for tertiary and subsequent consumption. For example, an academic researcher may use archival products to access archives, and produce an academic text as the resulting research product. This text may then form part of the active research use of a subsequent academic researcher, who will go on to generate further research products. Similarly in generating any research product the primary, or for that matter secondary, user may well generate transactional records which could constitute future archival sources in themselves. 3.4.1
Associated measures of usage
Measures of usage within these taxonomies will focus on the part one taxonomy, forms of service use. Purpose of use may form a category of data within this, alongside other measures such as the demographic profile of users, and perhaps measures of user satisfaction (as is the case with the existing PSQG surveys).
31
TAXONOMY OF USE Fig. 4: Form of Service Use (Part 1)
ACTIVE Primary (un-mediated): Use of finding aids Secondary (mediated): Enquiries (searchroom-based) ON-SITE PASSIVE Secondary (mediated): Exhibitions; Talks; Visits
PRIMARY USE Archival Products
ACTIVE Primary (un-mediated): Use of online catalogues/networks Secondary (mediated): Enquiries (email; post)
[Archives]
OFF-SITE FORM OF SERVICE USE
PASSIVE Secondary (mediated): Exhibitions; Talks; Education packs; Online digitised resources
ACTIVE Academic texts; Articles; Books; Reports; TV; etc. [For Information-seeking Purposes] SECONDARY USE
OFF-SITE
Research Products
32
PASSIVE Articles; Books; TV; etc. [For Personal Interest]
LEISURE Primary Use: Family / House / Local History Secondary Use: Personal Interest; Entertainment
TAXONOMY OF USE Fig 5: Purpose of Use (Part 2) PERSONAL
NON-LEISURE Primary Use: Evidential Secondary Use: Evidential
RESEARCHER Primary Use: Academic Research Secondary Use: Academic Research
PURPOSE OF USE
STUDENT Primary Use: Educational Project Secondary Use: Educational Project
EDUCATION
TEACHER Primary Use: Teaching Materials Secondary Use: Teaching Materials
PAID Primary Use: Evidential; Media / Professional Research Secondary Use: Evidential; Media / Professional Research WORK VOLUNTARY Primary Use: Evidential; Subject-based Research Secondary Use: Evidential; Subject-based Research
33
TIME
Value-Added Through Human Interaction
ValueAdded
ValueAdded
Through Human Interaction
Through Human Interaction
[ARCHIVE] RECORD Archival Processing Selection Cataloguing Indexing
PRIMARY ARCHIVAL PRODUCT
Archival Use Selection Synthesis Presentation
RESEARCH PRODUCT
SECONDARY ARCHIVAL PRODUCT
Primary Use
Secondary Use
Analysis Synthesis Absorption
Analysis Synthesis Absorption
Fig 6: A Continuum of Value-Added Usage
34
KNOWLEDGE
This is the approach taken by the existing mechanisms for data collection, in this area. The CIPFA statistics focus on the collection of basic quantitative data on level of use, for those forms of service use that can be readily measured within its remit. The PSQG surveys focus on the collection of qualitative data on user satisfaction levels with forms of service use, and include questions that enable the delineation of purpose of use and the demographic profiling of users within this. Current measures of level of usage are therefore able to address aspects of primary use relatively effectively. Measures of on-site use in particular are routinely collected by most archive services and encompass: reader numbers (active un-mediated use); and attendance at talks / group visits (passive mediated use). Simple measures of searchroom-based research queries (active mediated use) could also be adopted. What these figures do not tend to distinguish between is the number of overall users as against repeat visits; this may become a significant distinction as anecdotal evidence suggests that overall on-site visit figures are falling, but numbers of actual users may not be affected.22 Measures of off-site use are more problematic, data on email, postal and telephone enquiries (active mediated use) are routinely collected; some measures of off-site engagement with services through talks and visits (passive mediated use) are also likely to be taken. The CIPFA statistics approach the collection of data on exhibitions in relation to number of instances and appearances, but this does not necessarily indicate reach, and some means of estimating attendance figures might therefore be an appropriate development. Measuring use of online services, catalogues (active un-mediated use) and digital resources (passive mediated use) require technical solutions, such as counts of website hits, but such data can be relatively unsophisticated. Hill (2004, p.139-40) notes the problems associated with counting remote users, but considers that there is “a fair amount of information that can be deduced about them” by utilising the data recorded in web server logs. She gives examples (p.140) of how users of the Archives Hub can be profiled by origin (‘UK academic’, ‘other UK’, ‘overseas’, and ‘other’) based on the domain name of the computer used to access the website. As she also notes however, such mechanisms do not “tell the whole story”, and the use of online feedback forms or ‘new user’ information forms is required to obtain a fuller range of data. Evaluations of online resources such as those undertaken by the Tavistock Institute (Ramsden, Kelleher & Russell, 2002) (Ramsden, Barkat & Kelleher, 2003) of the Archives Hub provide the opportunity to collect more in-depth data for analysis. Indication of the level of non-primary use is also addressed now through the ongoing DCMS / National Statistics Taking Part survey. It is in the area of secondary usage however that solid data tends to be lacking. Some potential indicators of secondary use were discussed in section 3.3.2, notably audience figures for historical television programming, and readership of relevant literary
22 Indications are that the availability of online catalogue information and other genealogical resources is impacting on the genealogical use of archive service in terms of numbers of repeat visits. Users conduct preliminary research online, and may now make only one or two physical visits to the archive service. This situation was vividly illustrated by a presentation from staff at Glamorgan Record Office (Society of Archivists Wales, Meeting and AGM, 25 March 2006), where visit statistics indicate a downward trend, but numbers of registered users have been maintained. Hill (2004, p.139) contends further that this ‘problem’ of falling visitor numbers “should be seen instead as a positive consequence of improving online access to our materials.”
35
publications (monographs and periodicals). The major potential sources of data identified are summarised below in Table 6, however as Starkey (2004, p.4) notes “estimating the size of unseen audiences to media that are consumed in diverse locations” is almost necessarily problematic.23 Nevertheless, the sources of data could be utilised in combination to provide an overview of the potential scope of secondary usage within the UK population. Such measures, while inexact, could add considerable weight to the perceived impact of the archive domain. Methodologies would still be required however for evaluating the level of input into these sources from the primary use of archives, in order to legitimise the link between primary and secondary use.
23
Starkey (2004) reviews the appropriateness of the sampling techniques used by the Broadcasters’ Audience Research Board (BARB) and Radio Joint Audience Research (RAJAR) in particular, and concludes that the “nature of sampling as a means of producing estimates of invisible consumption by audiences is one of an inexact science.” (p.22) He identities inconsistencies that call into the question the representative nature of the samples surveyed and hence the overall legitimacy of the resulting data sets.
36
Table 6: Potential data sources from which indicators of secondary usage could be derived Data Source
Characteristics and potential for use
Cost
BARB provides estimates of the number of people watching television, including which channels and programmes are being watched, at what time, and the type of people who are watching at any one time. Data is collected on all channels received within the UK on both analogue and digital platforms; data is available for reporting nationally and at ITV and BBC regional level.
Further information
Data provided on the website is freely available.
Viewing estimates are obtained from panels of television owning households selected to be fully representative of all television households across the whole of the UK Broadcasters’ Audience Research Board Ltd. (BARB)
The BARB website provides data on weekly and monthly TV viewing summaries; top programmes by channel; and other basic TV facts and statistics, such as levels of TV ownership. Detailed viewing data for all BARB-reported television channels and services is available to BARB subscribers; this includes data sets representing overnight and consolidated viewing figures (the latter incorporating VCR playback) taken from daily panel viewing files and additional lifestyle data on panel constituency.
Detailed viewing data is available by subscription; subscribers pay an annual registration fee, currently (2006) £5,250, and a quarterly subscription fee or licence appropriate to the subscriber’s category of business.
http://www.barb.co.uk/
For the purposes of estimating audiences for historical programming, a range of relevant data can be obtained freely from the website; more sophisticated analysis profiling the viewing audience for example, may require subscription. NRS provides estimates of the number and nature of the people who read Britain’s newspapers and consumer magazines. Currently the Survey publishes data covering some 250 newspapers, newspaper supplements and magazines.
National Readership Survey (NRS)
Based on a sample designed to be representative of the UK population (aged 15+) NRS provides an estimate of the number of readers of a publication and the type of people those readers are in terms of sex, age, regionality and other demographic and lifestyle characteristics. Detailed datasets published monthly, quarterly and annually are available on subscription. The NRS website provides basic ‘top line readership’ data on the major national newspapers and supplements, general magazines and women’s magazines; data is analysed by social grade, gender, and age group (15-44 and 45+), and is presented as average issue readership (AIR) figures for the preceding year (Jan-Dec). Data on general magazines covers two publications potentially relevant as indicators of secondary usage for the archive domain: BBC History Magazine and Heritage Today. Potential audiences for newspaper articles (nationally) can also be assessed.
37
Data provided on the website is freely available.
Detailed datasets are available on payment of an annual subscription, currently (2006) £3,470 (standard), £1,020 (universities, schools and libraries).
http://www.nrs.co.uk/
Data Source
Characteristics and potential for use
Cost
ABC provides independent verification of data for: business to business publications, exhibitions, databases and lists; newspapers and consumer magazines; and website servers and traffic. It acts as an industry watchdog, conducting audits on all titles in ABC membership at least once a year; inspections constitute a complete check that any circulation claims have been made in accordance with ABC audit rules. Audit Bureau of Circulation (ABC)
The ABC website provides access to the latest summary reports of net circulation per issue, broken down by single copy sales and subscription, including data on cover price and subscription rates. Historical trend data is available by subscription to ABC. Data on magazines covers at least two publications potentially relevant as indicators of secondary usage for the archive domain: BBC History Magazine and Your Family Tree. Potential audiences for newspaper articles (in national and regional publications) can also be assessed. TheBookseller.com provides access to book charts from the UK updated on a weekly basis. The UK’s Official Top 50 (based on sales through 7,700 retailers, including internet sales) is freely available through the website. Genre charts, including Top 10 Hardback and Paperback Non-Fiction, are available to Premium members. Subscription to The Bookseller magazine provides a slightly extended range of information; genre charts provide figures for the Top 20 publications, and estimates of units sold are also included.
The Bookseller
Data drawn from Nielsen Bookscan; website data does not include sales figures, and only covers the most popular titles, so of limited use for assessing levels of history publication. Indications of pricing are given. Data reflects weekly statistics only; no historical trend data is provided. An annual Book Sales Yearbook is published which does include subject analysis of History publishing by sales value and volume 24; pricing data, top publishers / imprints, top authors, and data on the 100 bestselling history titles (delineated by value of sales, but incorporating volume data) are also given.
24
Data provided on the website is freely available.
Trend data for various data products is available on subscription; subscription to online historic data currently (2006) stands at £500 per year.
http://www.abc.org.uk/
‘UK Official Top 50’ latest weekly statistics freely available on website. http://www.thebookseller.com Access to other genre charts by subscription (2006: Online £99 per year; Magazine £175 per year).
The Book Sales Yearbook 2004 quotes overall sales of history books at a value of £32 million, and a volume of 2.7 million units.
38
Further information
Data Source
Nielsen Bookscan
Willings Press Guide
Characteristics and potential for use BookScan in the UK collects total transactional data at the point of sale directly from tills and despatch systems of all the major book retailers, providing a continuous retail sales monitoring service for books. Tracks every title sold, not just the bestsellers, and produces a range of information including sales by genre. Is the source of much of the data utilised by The Bookseller and other market commentators – operates as a commercial concern. Could be utilised to track history publication / sales in the UK. Willings Press Guide, Vol. 1 (UK) provides listings (updated quarterly online) for national newspapers, special interest periodicals, journals and directories; incorporates data on the publisher, frequency of publication, circulation figures (where available), summary of content type, and information on readership / target audience.
Cost
Further information
Statistical reports can be commissioned on a commercial basis.
http://www.nielsenbookscan.co. uk/
Single print volume £225 (+VAT for online version)
http://www.willingspress.com/
25
Can be utilised to identify extent of UK history periodical publishing (number of titles) and circulation of same; excludes ‘local periodicals with a very low circulation’.
25
Other bibliographic utilities such as Ulrich’s Periodical Directory (and its online counterpart, ulrichsweb.com) can also be used for periodical resource identification.
39
4
Developing a Taxonomy of Impact
4.1
Introduction
This project is explicitly concerned with scoping the economic and social impact of archives. As a starting point it was therefore necessary to define what was meant by these two terms, and indeed the concept of ‘impact’ itself. This task was complicated by the relatively ‘loose’ application of terminology in general usage, and by the different approaches taken within the literature to the identification of different types of impact, almost all of which are interrelated in some way. This section therefore commences by exploring what is meant by the term ‘impact’, and the key factors that must be considered when seeking to apply this term to archive service provision. The issue of identifying and delineating social and economic impact is then clarified in the context of public sector impact evaluation, with reference to the central government priorities which provide the objectives and criteria against which service delivery is liable to be assessed. A taxonomy of impact is then presented drawing on this discussion, the existing literature surrounding these issues, and the previously established taxonomy of usage.
4.2
Defining impact
At its most basic level ‘impact’ can be thought of as the outcome of ‘usage’. Impacts may however be of many different types and the complexity of the processes at work can make their evaluation extremely difficult. For example, Wavell, Baxter, Johnson and Williams (2002, p.7) note that impact can be:
short, medium or long term;
direct or indirect;
intentional or unintentional;
critical or trivial;
simple or complex.
We will consider here the question of what constitutes impact, and the complexities and issues the concept raises; the question of the existing evidence base for such impacts, and how they might be evaluated will be addressed in section 5. 4.2.1
Impact as difference
We believe that the most useful definition of impact is one that includes the idea that the impact of a service involves an effect of the service on the user. It is easiest to think of this in terms of the difference made. This echoes the views of Wavell et al. (2002, p.7) who consider that impact can be related to the question ‘Did it make a difference?’ Reviews on the impact of library services (Brophy, 2005) (Urquhart, 2005) indicate that such impacts are often assumed to be positive, however the effect of resource constraints in particular, may mean that impacts could unwittingly be negative. The Cultural Heritage Consortium (2002, p.21) similarly note the presumption of positive impact, and pose the question “whether early bad experiences put people off using the services for life – and whether that had any effect on their subsequent development.” Indeed, Brophy (2005) suggests that there could be a scale of impact, from downright 40
hostility and disappointment at the lowest end through mild awareness raising, and changed perceptions to ‘changed action’ at the highest level of impact. 4.2.2
Impact as a complex outcome
It is difficult however to separate impact from the conditioning factors that surround it. When thinking about impact and in particular how impact might be maximised in a service context, it also has to be recognised that there are an enormous range of influences work. As Wavell et al. (2002, p.6) note: These factors need to be understood in order to maximise the actual impact. Some conditioning factors are, to an extent, within an institution’s control given conflicting internal agendas, for instance staff training, allocation of resources, physical environment. Other factors are outwith an institution’s direct control, for instance the personal agenda of visitors and users and the wider economic and social contexts within which the institution operates. The approach taken to dealing with this issue in the literature has tended therefore to focus on situating impact within a conceptual framework of interrelated factors. Linley and Usherwood (1998, p.86-7) working in the context of social audit approaches, developed just such a “framework for an informed value judgement” about the social impact of service delivery. This framework, adapted slightly to reflect the range of factors at work in the specific context of archive service provision, is illustrated in Fig. 7 over the page. It conceptualises ‘impact’ in relation to a range of conditioning factors, including: social objectives, the context of operation, inputs, outputs, and outcomes (where the final outcomes can be equated with social impact). An adaptation of this framework was used by Bryson, Usherwood and Streatfield (2002) in undertaking a social impact audit for the South West Museums Libraries and Archives Council (SWMLAC).26 Wavell et al. (2002, p.7) similarly set impact within a conceptual framework that starts with the aims and objectives of the service (which may be defined by the organisation, and take into account the interests of other stakeholders including users and government). Other elements of this framework include: inputs (the resources the service requires in order to function); the process of what is done with the inputs (cataloguing, exhibition, educational programmes or websites, etc.); and the resulting outputs and outcomes. As they note, it is at the level of outputs – the direct service product of combining inputs and processes – that most professional services are typically assessed, since outputs can provide a quantitative measure of efficiency (for example, number of services provided and number of people provided for, number of reference enquiries answered, time taken to process raw materials). In this framework, outcomes “are the positive or negative engagement with planned outputs by an intended or unintended user” (p.7) and: Impact is the overall effect of outcomes and conditioning factors resulting in a change in state, attitude or behaviour of an individual or group after engagement with the output and is expressed as ‘Did it make a difference?’
26
See section 5.4
41
Fig. 7: Framework for an informed value judgement (adapted from Linley & Usherwood, 1998, p.87)
SOCIAL OBJECTIVES of Parent Authority / Service e.g. Council policy, service mission statements etc.
NEEDS ANALYSIS e.g. Community profiles, marketing activities
CONTEXT e.g. Central Government Policy Economy Environment Organisational Culture
SUPPLY INDICATORS Number of staff Resources etc. Management e.g. Style Culture Structure Quality Marketing
Service Patterns e.g. Centralised Decentralised
OUTPUT INDICATORS Document Issues Number of visits Reference Enquiries etc.
INTERMEDIATE OUTCOMES Skills development People meet VARIABLES
VARIABLES
FINAL OUTCOMES Community Identity Social Cohesion
42
This is very similar to the approach taken by the Cultural Heritage Consortium (2002, p.11-14) in their review of the quantitative time series data available for impact evaluation across the ALM sector. They adopt a framework of inputs (the resources used), processes (whatever is done to the inputs to create something new), outputs (the things the organisation produces), and outcomes (the short to medium term results of applying outputs). They then define impact (p.14) as follows: Impact is the effect of the outcomes on the environment – again, using that term very broadly to include people and society at large – and is usually long-term. Wavell et al. (2002, p.7) also introduce the concept of value, which is defined in terms of the value stakeholders place on a service; something which may not necessarily be monetary, and which could be assessed through methodological approaches such as contingent valuation. 4.2.3
Timescale of impact
Another important element of the definition of impact is the question of timescale. Wavell et al. (2002, p.7) characterise this as follows, noting that: There is a clear distinction between an immediate response to an output, and the impact ‘continuum’ which progresses beyond the immediate interaction through an intermediate and longer term response (e.g. a person reads, becomes employed, contributes to social cohesion). This continuum also reflects how the immediate individual response can then be transferred to a longer-term community impact. In the context of archive service usage, the outcomes of engagement with service outputs will have some form of immediate impact on the primary user. This impact may fall between -2 and 2 along the scale suggested by Brophy (2005), as illustrated by Table 7, which maps impact to the existing characterisations of generic learning outcomes (GLOs) produced by MLA (2004b), and to the prototype generic social outcomes (GSOs) under development by Burns Owens Partnership (2006). The shorter-term impacts will reflect the immediate response of the user to engagement with the service, and therefore may or may not be permanent. Longer-term changes may occur over a couple of years or more after the initial impact, and may reflect cumulative experiences of engagement with the service. Medium-longer term impacts may also reflect secondary usage – engagement not with the outputs of the service per se, but with the products resulting from primary usage. Such impacts may be much wider than the primary impact on the relatively smaller number of primary users who engaged with the service in the first place. 4.2.4
Impact on whom?
If a ‘continuum’ of impact takes place over time then it must be recognised that this also extends to the constituency to whom this impact applies. The impact may be a difference made to an individual, or to a group/community, and eventually to society. Impact within an organisation should also be considered, this may be at the level of internal governance, or extend out to the wider domain or professional partnerships. For example, cross-sector bridge building is a type of impact that may need to be addressed in public sector institutions.
43
Table 7 : Mapping of Impact Scale (Brophy, 2005) to Generic Learning Outcomes (GLOs) and prototype Generic Social Outcomes (GSOs) Scale
Definition
(Brophy, 2005)
(Brophy, 2005)
-2
Hostility (extreme disappointment)
-1
Dismissive (service not worth money/time)
0
Neither positive nor negative – unaware
1 2
3
4
Awareness raised (mild positive impact) Better informed (with relevant information)
Improved knowledge
Changed perception and/or ability
Impact over time / on whom Short-term / Individual Medium-term / Community Longer-term / Society Short-term / Individual Medium-term / Community Longer-term / Society Short-term / Individual Medium-term / Community Longer-term / Society Medium-term / Community Longer-term / Society Short-term / Individual
Medium-term / Individual Longer-term / Community
Medium-term / Individual Longer-term / Community Longer-term / Society
5
Changed worldview (transferable skills acquired)
Longer-term / Individual Longer-term / Community Longer-term / Society
6
Changed action
Longer-term / Individual Longer-term / Community Longer-term / Society
GLO equivalent
Prototype GSO equivalent
(MLA, 2004b)
(Burns Owens Partnership, 2006)
Attitudes and Values (about archives) Attitudes and Values (about archives)
Attitudes and Values (about archives) Knowledge and Understanding (learning facts or information) Knowledge and Understanding (deepening understanding) Skills (knowing how to do something)
Strengthening Public Life (inclusive public spaces and services) Stronger and Safer Communities (tackling fear of crime) Stronger and Safer Communities (contributing to crime prevention) Health and Well-being (encouraging healthy lifestyles)
Skills (key; info management; social)
Strengthening Public Life (enabling community empowerment) Stronger and Safer Communities (supporting cultural diversity; family ties)
Attitudes and Values (self-esteem; attitudes to others)
Health and Well-being (supporting care and recovery)
Enjoyment, Inspiration, Creativity (creativity; being inspired) Skills (social; emotional; communication)
Strengthening Public Life (encouraging participation) Stronger and Safer Communities (improving inter-group understanding)
Attitudes and Values (empathy, capacity for tolerance)
Strengthening Public Life (building capacity) Stronger and Safer Communities (improving inter-group dialogue)
Activity, Behaviour and Progression (doing more; changing ways of managing life; behavioural change)
Health and Well-being (supporting older people; helping children) Strengthening Public Life (encouraging participation; improving service)
44
In the public and private sector, archives are also likely to have an impact on operational aspects such as risk management, and asset management. The public’s perception of organisations as accountable and trustworthy purveyors of a quality ‘product’ can be an important factor in organisational success and profitability; indeed, archives are often used by long-established companies with strong branding, as the means to reinforce their reputation in the marketplace. These types of impacts are something that business archives in particular are likely to need to make explicit. At the top level, such considerations could impact on national or institutional policy making in both the public and private sector. Looking farther into the future, and at the wider population at community / society level, there is a value, rather than an impact, of a service. That value can be the contingent value for non-users, the value of having that service available even if there is no immediate intention of using that service personally. Increasingly however this is being seen in terms of public value, particularly in government circles; the idea that “public services must create a value to the public, in the same way that the private sector creates shareholder value.” (Ray, 2006, p.15) How this works in the cultural arena is already being looked at by organisations such as Demos; Holden (2005, p. 8-10) considers that cultural value consists of three elements:
Instrumental value – the social and economic impact of culture on society
Institutional value – derived from the engagement of cultural organisations with their public
Intrinsic value – the personal value derived by the individual from their subjective experience of engagement with cultural activities
Matty (2006) relates these elements to each other, in an archival context, in the following terms: As the site where the public can engage with documents and other archival material the institution is the means by which the intrinsic can be linked to the instrumental. As Holden (2005) notes, capturing or evaluating how these three types of value work in practice is likely to require different approaches, and points to different constituencies: the public who use services; the professionals who manage them, and the politicians who set the parameters. Holden, and his colleague Robert Hewison (Ray, 2006, p.15)27 also suggest that “debate has been focused too narrowly on the instrumental value of culture and that it is time to engage in more debate with the public over the intrinsic value of heritage.” This would require a shift in emphasis from wider impacts on society, to personal impacts on the individual.
27 Holden and Hewison were speaking at a two-day conference (25-26 January 2006) organised specifically to explore issues around Capturing the Public Value of Heritage.
45
4.2.5
Types of impact
Although this study is explicitly concerned with issues of social and economic impact, the literature reveals that a range of different typologies of impact (many imperfectly delineated) have been identified. Examples of such types of impact, and how they might be characterised, encompass:
Social (in terms of access, inclusion, and participation)
Cultural (in terms of heritage, community cohesion, diversity, creativity and inspiration)
Educational (in terms of skills, knowledge, and application of knowledge)
Economic (in terms of employment (direct / indirect), commercialisation, tourism, and regeneration)
Health (in terms of difference to well-being)
In attempting to unravel these different types of impact, and arrive at a definition of economic and social impact, the policy context provides a useful starting point. This approach is used in many of the ALM sector reports produced in recent years; for example, Cookman and Haynes (2002) in their Strategic Review of Statistical Requirements group their findings under five key policy themes (lifelong learning, inclusion, neighbourhood renewal, modernising government, and building the knowledge economy). Cultural Heritage Consortium (2002), in reviewing the quantitative time series data available for impact evaluation, narrow this down further to three key themes: learning; access and inclusion; and economic impact. By this means, Wavell et al. (2002, p.6) in their wider study of the available evidence for impact evaluation, identify four complex inter-related areas, three of which they define in terms of types of impact: social, learning and economic, together with a fourth related dimension of access, which they consider as “the provision of opportunity, whether physical, emotional or intellectual, to accommodate learning, social or economic wellbeing.” This approach is also informing the work of Burns Owens Partnership (2006, p.1) in developing a social outcomes framework for MLA, where they note the importance of “aligning the sector’s potential social contribution with key drivers of government policy.” What is clear is that different types of impact are fundamentally inter-related. Educational impact, for example, has ongoing implications in the social and economic spheres; Burns Owens Partnership’s current work on developing a social outcomes framework, explicitly recognises this and builds on the Generic Learning Outcomes (GLOs) identified as part of the Inspiring Learning for All framework. Table 7 above offers one interpretation as to how a potential scale of impact (Brophy, 2005), might be related to both the GLOs and to Burns Owens’ prototype Generic Social Outcomes (GSOs). The following discussion recognises the importance therefore of educational impact as underlying aspects of economic and social impact, but will focus for the most part on the economic and social issues to which it contributes.
46
4.3
Defining economic impact
The concept of economic impact is relatively clearly defined within the literature. The Cultural Heritage Consortium (2002, p.19) relate it to government’s economic regeneration goals, noting that the ALM sector could: contribute to overall local, regional and national economic development. They could do this through stimulating and facilitating innovation and creativity, through providing access to intellectual property, particularly things like research findings, patents and standards, and by assisting the transformation to an information- or knowledge-based society. Wavell et al. (2002, p.6) characterise it simply as follows: Economic impact encompasses economic implications for individuals, small or large groups, or for communities in terms of urban or rural regeneration, renewal or sustainability. One recent study (Burns Owens Partnership, 2005b) takes this a stage further and conceptualises economic impact in terms of demand-side and supply-side impacts. The former relates to the direct, indirect and induced economic benefits that are generated by the operation of the organisation. This is a fairly standard approach in economic impact studies, and has been applied to archives by a recent SWMLAC project (Brand & McVittie, 2004). The latter refers to the range of ways in which the resulting products and services benefit the wider economy. These supply-side impacts are characterised by Burns Owens (2005b, p.18) in relation to four main overlapping areas of economic development: learning and skills; enterprise support; regeneration and renewal; and tourism. The constituent elements of both forms of impact are summarised in Table 8.
Demand-side
Supply-side
Direct effects: Those generated by normal commercial operations (paying wages, generating incomes and purchasing supplies).
Learning and Skills Development: Equipping individuals with the skills to enter into and progress in the labour market.
Indirect effects: The ‘knock-on’ effects of the commercial operations of the suppliers of goods and services to the organisation.
Enterprise Support: Supplying individuals and businesses with the information they need to start, grow and become more productive.
Induced effects: The effects generated by the spending of employees locally (e.g. on food and clothing).
Regeneration and renewal: Contributing to a high quality physical and social environment in the renewal of localities. Tourism: Contributing to the image and attractiveness of an area and attracting visitors and additional spending into the economy.
Table 8: Aspects of Economic Impact (derived from Burns Owens Partnership, 2005b)
47
It should be noted that many of the potential supply-side impacts are educational or social in nature and thus intersect with issues of social and educational impact; economic valuations in this context can however be related to the financial costs of failure in social terms. For example, expenditure saved on the criminal justice system, on unemployment benefits, and so on. Parallels can also be drawn with aspects of supply-side impact and the concept of secondary usage, as Burns Owens note (2005b, p.20): The role of archives, libraries and museums as storehouses and distributors of knowledge, artefacts and ideas is particularly important in the so-called knowledge economy . . . The use of knowledge assets to create new products and services is a process of innovation comparable to the way in which new discoveries in the fields of science and technology are translated into consumable goods and services. Much of the discussion of economic impact is based around the contribution made to local economies; Pickford (2002, p.26) attempts to quantify this kind of impact at a UK-wide level, noting that archival activity provides employment for almost 5,000 people, and that the overall spend on UK archives services (public and private) is in the region of £240 million per annum. Pickford also reinforces the point (p.32), that “archive visitors do contribute to the local economy through archival tourism” (an aspect of supply-side impact as defined by Burns Owens, 2005b). Indeed, it is evident from the findings of the PSQG surveys28 that some users travel significant distances to access archival materials, making use of a range of local facilities while visiting an area. Aside from the wider impact of service provision on local and national economies, there is also the question of the potential economic impact that engagement with archives can have on the individual. There is certainly a sense that skills development through use of archive services could lead to improved employment prospects; however, there are very real difficulties in quantifying this impact in relation to other contributory factors.
4.4
Defining social impact
The definition of social impact is much more complex. Wavell et al. (2002, p.6) characterise it as follows: Social impact encompasses inclusion or overcoming exclusion of individuals or groups in terms of poverty, education, race, or disability and may also include issues of health, community safety, employment and education. The literature as a whole indicates that the concept is multi-dimensional to such an extent that it has to be approached at a thematic level, and as has already been noted, government policy provides a useful basis for definition. In a series of reports
28
Data drawn from questions addressing the level of interaction with the local economy (payment for overnight accommodation; eating out locally; using local shops and services; and visiting other places of interest) provide evidence of such impact.
48
from 200429 written under the banner of MLA’s ‘New Directions in Social Policy’ initiative, the range of applicable government policy measures with which ‘social impact’ can be associated is extensive, encompassing among others:
Community cohesion and related community agendas; social inclusion and neighbourhood renewal
Health (with a focus on mental health)
Cultural diversity
This MLA initiative is still ongoing at the time of writing and is concentrating on developing a social outcomes framework, similar to that provided in an educational context by Inspiring Learning for All and its constituent GLOs. The prototype GSOs identified by Burns Owens Partnership (2006, p.2) are focused around the three intersecting themes, illustrated in Fig. 8.
Stronger and safer communities
Strengthening public life
Health and well-being
Fig. 8: Social outcomes framework for museums, libraries and archives (from Burns Owens Partnership, 2006, p.2)
Within each of these three themed areas a range of second tier outcomes are identified, these are set out in Table 9 below, and are intended (p.7) “to articulate how the sector’s contribution in these areas can ‘read across’ to policy priorities and frameworks within the rest of government” rather than provide an exhaustive analysis of possible outcomes. This thinking on social impact is closely related to concepts of social capital, “the web of trust, associations, everyday contacts, cooperative skills, networking which act as ‘social glue’.” (Burns Owens Partnership, 2006, slide 6)
29
Hylton, T. New Directions in Social Policy: Cultural Diversity for museums, libraries and archives.; Linley, R. New Directions in Social Policy: Communities and Inclusion Policy for museums, libraries and archives.; and Weisen, M. New Directions in Social Policy: Health Policy for museums, libraries and archives.
49
Table 9: Constituent second tier social outcomes within each top-level theme (from Burns Owens Partnership, 2006, p.2-6)
Stronger and safer Communities
Health and well-being
Strengthening public life
Improving inter-group dialogue and understanding
Encouraging healthy lifestyles, tackling the determinants of ill-health
Encouraging and supporting awareness and participation in local-decision making and wider civic and political engagement
Supporting cultural diversity and identity
Supporting care and recovery
Building the capacity of community and voluntary groups
Encouraging familial ties and relationships
Supporting older people to live independent lives
Providing safe, inclusive and trusted public spaces and services
Tackling the fear of crime and anti-social behaviour
Helping children and young people to develop and enjoy life
Enabling community empowerment through the awareness of rights, benefits and external services Improving the responsiveness of services to the needs of the local community, including other stakeholders
Contributing to crime prevention and reduction
50
Burns Owens (2006, slide 10) drawing on the work of Aldridge, Halpern and Fitzpatrick (2002) identify three different kinds of social capital:.
Bonding – what binds groups together; reinforcing ‘exclusive’ identities and homogeneous groups
Bridging – what links individuals / groups to other groups; generating broader identities and reciprocity
Linking – links between individuals / groups and other groups with different status / access to resources
Political scientist Robert Putnam (2000, 23), one of the leading proponents of social capital theory, characterises how the first two of these30 work in practice as follows: Bonding social capital constitutes a kind of sociological superglue, whereas bridging social capital provides a sociological WD-40. Aldridge et al. (2002, p.5) consider that these forms of “social capital may contribute to a range of beneficial economic and social outcomes”, but note, as does Putnam, that there are potential downsides, particularly in relation to bonding forms of capital.31 Overall, however, they conclude that the evidence for beneficial effects is impressive. The identification of ‘history and culture’ as major determinants in the development of social capital (Aldridge et al. 2002, p.39-40) also highlights the potential importance of cultural services, and archives in particular. Specific reference to archives in this context is however rare, although Coalter (2001, p.2) does note that: The ‘people-oriented’ nature of cultural services, concerned with personal and social development, can make a substantial contribution to the ‘people-centred’ policy agenda, which aims to develop both social capital (strengthening community networks/capacities) and personal capital (developing skills and confidence). Commentators such as Daly (2005, p.31) note how museums “are viewed as essential to building a sense of community identity”, by undertaking projects that facilitate interaction between individuals and help build social networks. Archives also have the potential to provide a valuable means by which individuals can identify with particular groups or communities, and as such can be seen as contributing to bonding social capital. Indeed, potential social impacts (which can be seen as deriving from the development of both social and personal capital) are emphasised throughout the archival literature, although this is largely in anecdotal terms. NCA’s Taking Part: An audit of social inclusion work in archives (2001, p.7-9) does identify three ‘thematic’ foci (derived from respondents consensus views), which accord well with such concepts:
30
Personal identity and development
Community identity and development
The concept of ‘linking’ social capital is a more recent addition to the discourse in this area.
31
Putnam (2000, p.23) notes that bonding social capital, “by creating strong in-group loyalty, may also create strong out-group antagonism”.
51
Representing communities
This report also relates the uniqueness of the archive contribution in records management terms to the four key indicators of exclusion identified by government: democracy and citizenship; tackling crime; promoting healthier communities; and promoting lifelong learning, educational attainment, and employability. The PSQG surveys of visitors to British archives have focused on similar areas in asking respondents to evaluate archives contribution to society in terms of:
Providing opportunities for learning
Preserving our culture and heritage
Strengthening family and community identity
Supporting administrative and business activity
Supporting the rights of citizens
It is clear that these themes do resonate with users’ thinking; as responses to this question in the two years that it was set show (see Table 10 below).
Archives contribute to society by …
2001 (% respondents agree)
2004 (% respondents) strongly agree
agree
Providing opportunities for learning
65.9
66.7
31.5
Preserving our culture and heritage
82.4
84.7
14.9
Strengthening family and community identity
72.4
45.1
36.7
Supporting business and administrative activity
20.3
20.3
34.8
Supporting the rights of citizens
N/A
33.5
32.6
Table 10: Users’ views of what archives contribute to society (PSQG, 2001a) (IPF/PSQG, 2004)
At a more personal level, Daly (2005, p.36) also recognises involvement in community-based initiatives as helping individuals “develop the skills and confidence to be more involved in the community”, leading in some cases, “to improved understanding and links across ethnic and social divides.” These kinds of individual impacts echo Coalter’s adoption of the term ‘personal capital’. As Aldridge et al. (2002, p.13) note, capital can take many forms; they refer to physical capital, natural capital, cultural capital, financial capital, and human capital as well as social capital itself. Personal capital, as delineated by Coalter (2001, p.23)32 is broadly similar to concepts of human capital. However, human capital, defined by Coleman (2000, p.22) as “changes in persons that bring about skills and
32 Coalter relates the term to: social contact; the development of confidence and self-esteem; education and life-long learning; and health and well-being.
52
capabilities that make them able to act in new ways”, is often viewed in terms of its value as an asset to an organisation; the term ‘personal’ capital orients these changes much more on individual needs. The Cultural Heritage Consortium (2002, p.21) note a range of desirable scenarios – including “academic achievement, success at work, the ability to raise a family harmoniously, an active appreciation of current affairs, or simply general well-being and a calm frame of mind” – to which use of archives, libraries or museums has the potential to contribute. Engagement with such services can be viewed in terms of the development of personal capital (social skills and knowledge) from which these positive social impacts could be derived. The PSQG surveys for 2001 and 2002 offer further support for such thinking; enumerating some possible personal impacts on the user (see Table 11 below). 2001 (% respondents agree)
2002 (% respondents agree)
79
86
Helped generally to increase my abilities, skills and confidence
29.5
30.2
Helped to stimulate / broaden my understanding of history or culture
61.4
58.6
Helped me to develop my job-seeking or workplace skills
5.2
5.9
Helped me to use and develop my ICT skills
18
9.3
Users believe the use of archives to have … Been a useful and enjoyable learning experience
Table 11: Users’ experiences of using archives (PSQG, 2001a) (PSQG, 2002)
The concepts of social and personal capital can be identified with the processes underlying the achievement of outcomes such as the prototype GSOs: stronger and safe communities; increased health and well-being; and the strengthening of public life. Although the development of the GSOs is yet to be finalised, the broad delineation of social impact they embody has been adopted for the purposes of this study, as both a sensible approach to defining the scope of social impact within the bounds of current government policy, and for its likely future adoption by the ALM sector
4.5
A taxonomy of impact
In developing a taxonomy of impact it was therefore necessary to consider aspects such as: timescale of impact; subject of impact; and type of impact. This accords with questions such as ‘who’ is being affected ‘when’ and ‘how’. The following taxonomy (Fig. 9) draws on the discussion of these issues within the broad typologies of economic and social impact identified by Burns Owens (2005b, 2006) in particular. In terms of economic impact the focus is on supply-side benefits rather than the direct demand-side aspects of economic growth. The categorisation of social impact also reflects a simplified terminology where:
53
Independence = the ability of individuals, particularly younger and older people to live independent lives
Empowerment = enabling individual / community awareness of rights and benefits
Healthy Lifestyle = enabling individuals to make informed lifestyle choices
Participation = supporting social and civic engagement
Cultural Diversity / Identity = supporting group cultural diversity / identity
Family Ties = encouraging family / group cohesion
Care & Recovery = supporting group / community care and recovery
Social Empathy = improving inter-group dialogue and understanding
Capacity = enabling the capacity of groups / communities to act
Safer Communities = sustaining safe and secure environments
This ‘taxonomy’ is not presented in a traditional taxonomic form but as a series of cumulative changes affecting individuals, organisations / groups, communities and society over time. This builds on the idea of a continuum of impact, where impacts on the individual percolate out into the associated social group, local community and hence to society. The timescales within which these processes work are not rigid, and at individual level in particular may even reflect a step-change from ‘better informed’ to a change in behaviour. However, at the other levels, effects are more likely to build cumulatively, and may extend over decades or more. As the literature makes clear, both economic and social impact can be either positive or negative in their effects. However, in the context of archives, it is difficult to envisage significant social or economic negative impacts – although it is possible to have a negative encounter with an archive service (e.g. poor service, unsatisfactory answer to an enquiry, closed when one made a visit), it is unlikely that one would leave an archive knowing less than when one arrived, or being less healthy as a result. Whilst recognising that multiple negative encounters may contribute to a cumulative negative impact, and that restricted access to archives (shortened opening hours, no disabled access etc) has an immediate negative impact, these are not included as part of the overall taxonomy, but rather regarded as factors to be considered in the measurement of impact. Likewise there is the issue of whether impact is intentional or incidental and to what extent that is important when gauging the level of impact achieved. If intended impacts reflect government policies or thematic priorities, then it may be important to be able to assess these as a crude measurement of success. Incidental impacts are, by definition, unlooked for and will be intriguing as a result. They may reveal previously hidden relationships and connections that can contribute to increased social capital. Again, the concept of intentional and incidental impacts needs to be considered in the measurement of impact based on the taxonomy defined below.
54
SHORTTERM
Fig. 9: TAXONOMY OF IMPACT
Better Informed
Economic Impact
Improved Knowledge
Social Impact Improved Knowledge
Cultural Diversity / Identity Family Ties Learning & Skills Development
Care & Recovery
Independence Empowerment
Enterprise Support Independence Social Empathy Participation Empowerment Capacity
Economic Benefits
Economic Growth
Healthy Lifestyle Regeneration & Renewal
Participation
LONGERTERM
Social Empathy
Tourism
Improved Health & Well-being
Safer Communities
INDIVIDUAL
Stronger & Safer Communities Health & Well-being
GROUP / ORGANISATION
COMMUNITY
55
Strengthened Public Life
SOCIETY
5
Impact Evaluation: Approaches and Methods
5.1
Introduction
The archives domain, in common with the rest of the cultural heritage sector, is becoming increasingly user-centric. There is now an emphasis on access to services and materials, on the development of the individual and on the growth of communities. This has occurred partly, perhaps largely, in response to UK government strategies and policies, specifically with respect to social exclusion. To demonstrate that archive services strive to meet the aims and objectives of these policies and to provide reliable indicators of levels of impact, a programme of evaluation and measurement is required. To sustain current and future levels of government investment in public sector archive services, the significance of archives in people’s lives and the contribution that they make to the nation and economy needs to be made explicit. Preceding sections of this report have gone some way to identifying and defining the types of impact that fall within the broader definitions of social and economic impact. This section seeks to describe and assess the approaches, methodologies, tools and techniques that can be used to evaluate the level of impact. As well as reviewing current efforts in this area within the archive domain, and ALM and wider cultural sectors, examples are also drawn from the fields of economics, transport planning, healthcare and environmental management, the latter three already having made some headway into the difficulties of evaluating intangibles and non-market goods. Impact evaluation is itself relatively ill-defined, and more than one commentator notes the tendency to focus on easily quantifiable measures, such as inputs and outputs, rather than the more amorphous aspects of outcomes and impact. As Wavell, Baxter, Johnson and Williams found in their study that looks across the ALM sector (2002, p.9), there is “no clear consensus on what constitutes evidence of impact.” They also (p.9) highlight concerns regarding the validity and nature of the evidence that is posited: The research community raise concerns over the quality of ‘evidence’ put forward by the profession in policy documents in the form of personal expressions of conviction or practitioner studies that lack the explicit rigour shown in quality academic research. The nature and quality of evidence varies markedly and much of the evidence is in fact pointing to potential areas of impact rather than actual impact. This view is echoed by the Cultural Heritage Consortium (2002) in a study identifying quantitative time series data, and its potential for impact evaluation within the ALM sector; they found that none of the time series studied came close to measuring impact (p.14), and that the measurements they reviewed “say little about outcomes or impact; rather, they provide a partial measure of the potential of the service to generate impact.” (p.17) The Consortium goes on to note that the PSQG survey is the only one that attempts any form of measurement of social and economic impact, and even here, the data are likely to record only immediate and short-term results, since collection is via an exit survey. In mitigation, however, the report concludes that in taking a quantitative approach, “there are some fairly intractable methodological problems to be faced” (p.20) not least of which is “the problem of defining impact” (p.21), and suggests that a qualitative approach may yield better results.
56
This section will therefore look more closely at how the concept of impact evaluation is defined, and at the issues and concerns it raises, before considering the existing studies relevant to the evaluation of social and economic impact within the archive domain, and approaches and methods from other fields.
5.2
Defining impact evaluation
Wavell et al. (2002, p.7) define impact evaluation as “a management tool used to assess changes that can be attributed to a particular intervention, programme or policy.” They also note however that within the literature definitions are ‘problematic’, and subject to different interpretation; in common with Linley and Usherwood (1998), they therefore define ‘impact’ in relation to the conceptual framework within which it is situated (as discussed in section 4.2.2). In this framework, impact is defined as “the overall effect of outcomes and conditioning factors resulting in a change in state, attitude or behaviour of individual or group after engagement with the output.” (p.7) To what extent outcomes – “the positive or negative engagement with planned outputs by an intended or unintended user” (p.7) – become impact will therefore depend on a range of conditioning factors (including inputs, process, and outputs, and the personal and social contexts of use). It can be seen therefore that impact evaluation will involve the assessment of outcomes, and the extent to which these ‘make a difference’ in the context of the operative conditions of the wider conceptual framework. The consequent perceived value of the service is thus closely related to the extent to which the service is felt to impact on people’s lives. 5.2.1
Issues and concerns
As noted in section 4.2, the concept of impact itself raises a number of questions, such as: impact on whom?; timescale of impact?; etc. These issues have to be considered in developing approaches to impact evaluation, as does the central question of the objective of the process of evaluation itself. As MLA note, “there are huge differences between demonstrating impact to politicians and using impact measures to manage services.” (Impact Evaluation, 2006). Impact evaluation can also work at many levels, from the national, to the individual service project. MLA’s Learning Impact Research Project (LIRP) for example, set out to “develop an understanding of learning and its outcomes and to establish a way of researching and providing evidence of this learning in museums, archives and libraries.” (HooperGreenhill, 2002, p.2) This works at two levels in enabling organisations to be aware of their effectiveness in this area, and thus to work toward improving provision, but also as a means for MLA to provide quantifiable evidence as to the national picture of the impact of learning in its constituent domains. In approaching impact evaluation it has to be recognised that many aspects of impact are, and will remain, relatively intangible. This is an issue that LIRP had to deal with in the context of developing learning outcomes for museums, libraries and archives, as “open, informal and flexible learning environments” where it is “impossible to define specific learning outcomes for each individual.” (Hooper-Greenhill, 2002, p.7) In the field of social impact the problem is compounded, as we have seen, by the lack of an agreed definition as to what this actually constitutes.
57
The fact that impact is not a ‘one-off’ event, also raises the question of ‘when’ to undertake evaluation. In section 4.2.3 it was noted that while service delivery may have immediate effects, some outcomes will unravel over much longer time-scales. In the context of learning Hooper-Greenhill (2002, p.7) notes that: It is difficult to know when to assess the achievement of learning. If learning is a continuous lifelong process it is very difficult to assess the outcome of informal learning at any particular moment in time. If learning outcomes are being researched immediately after a museum or library visit, for example, how do we know that greater learning will not happen after this moment? What is being measured? Impact can be seen in many different dimensions. As the same report goes on to note, it is usually the user who defines the objective of the visit and what they want from it, and criteria for ‘success’ are therefore unfocused or open. Social impacts too can be immensely personal; as Usherwood (1999, p.3) notes in relation to local studies services, an interest in local history could help “people dealing with life changes, such as retirement or bereavement.” These incidental types of impact will vary from person to person, and while perhaps peripheral to the original objective of the visit or research, can actually have the greater impact on the individual concerned. Other commentators working in the field of measuring outcomes have noted the complexities of ‘disentangling’ the various factors at work. The Cultural Heritage Consortium (2002, p.20-21) discuss the very real practical difficulties to be overcome: how to distinguish cause and effect over the long term in a meaningful way; the distortion of time between an individual’s experience and collection of the survey data; how to separate the influence of the archive from other experiences; and how to define what constitutes a positive impact. To this must be added the difficulties “inherent in linking micro-level effects on individuals to the more macro-level of the community” (Burns Owens, 2005, p.8) – the so-called ‘problem of aggregation’. Although Guetzkow (2002), cited by Burns Owens (2005, p.8) argues that “it is axiomatic that, other things being equal, the more widespread or intense the participation of individuals in a particular cultural activity is, the greater the impact will be on the whole community”, approaches to deriving clear evidence of impact at the macro-level are still far from perfect.33 Furthermore, this axiom does not necessarily hold if one accepts that the production of personal capital and social capital as potential outcomes are distinct from one another. Burt (2000, p. 256) distinguishes between human capital as “a quality of individuals” and social capital as “a quality created between people”. His discussion is framed in a managerial context, but his conclusion, that managers with more social capital get higher returns on the human capital at their disposal is interesting, as it posits that the realisation of individual potential (based on human or personal capital) is dependent on a level of pre-existing social capital or ‘opportunity’. Under this scenario, evidence of individual impact does not necessarily contribute to a cumulative social impact.
33
Burns Owens (2005, p.8) list a number of such approaches: simply adding up the percentage of individuals in a population that are affected by something (more being better); looking for the threshold or 'tipping point' whereby individual effects become community effects; looking at the type of networks that people form via cultural activities (ie does it increase their 'linking' social capital?); or looking at how a few key individuals or community leaders are affected and how this feeds into a general climate of opinion. They are nevertheless forced to conclude that such mechanisms are “problematic and contested” (p.8).
58
The question also arises as to how to delineate the macro-level itself. The notion of ‘community’ is itself contested, although as Linley and Usherwood (1998, p.30) note “it is often conceptualised in two ways, that is ‘communities of interest’ and ‘spatial communities’”. This is another aspect to be considered when defining what is being measured. Despite all this the Burns Owens Partnership (2005, p.67) were able to conclude, in their study of the evidence base for museums, archives and libraries that: There is a growing body of evidence which suggests that interactions of various sorts with museums, archives or libraries can have a variety of beneficial effects on individuals. The quality of this evidence, and how far it addresses these kinds of issues and concerns, is considered below.
5.3
Impact evaluation: the archive domain
As yet there has been little work completed specifically on the social and economic impact of archive services, which may be indicative not only of the relatively recent shift in focus of the archival ‘raison d’être’ from the collections themselves to user needs and aspirations, but also of the sheer complexity of determining what the impacts might be, the level at which these can reasonably be ‘measured’ and actually conducting the evaluative work. Increasingly today, in situations where funding for particular projects or programmes derives from external sources, the funding body will however require that some form of project evaluation is carried out, as a means of assessing the ‘success’ of a project in relation to its original objectives. Such studies may encompass indications of the impact of the programme or the service under evaluation. A number of the domain specific studies identified fall into this category, which also corresponds broadly with requirements within the public sector for services to demonstrate accountability and ‘best value’. A more holistic approach to programme evaluation is discussed by Rallis and Bolland (2004), and Grimard and Pagé (2004) consider how programme evaluation ideas have been applied to archive services. Generally, the examples cited have stressed the importance of taking stock of the current situation and deciding how to move forward, rather than a complete impact assessment. Such programme evaluations answer the questions – how are we are doing and what could we do? Impact assessment digs down a little deeper by asking what difference do we make, and, on the basis of that, provides strategic planning indications. Grimard and Pagé (2004, p.123) cite Williams and Proctor (2002) on the data already collected in the archival domain. At present that data collection is not sufficiently extensive, or reliable to adequately serve evaluation purposes. In considering the current state of archive programme evaluation, they recognise that the sector has concentrated until now on statistics compilation and basic activity tracking; information that while useful, cannot really contribute to the measurement of social and economic impact. Reviewing over 30 years of archival literature, they observe that few authors deal with the subject of evaluation and those that do provide only a superficial treatment.
59
Indirectly, they acknowledge the need for assessment of social impact, calling for wide scale surveys on existing and potential users, along with examination of their needs, expectations and goals. Whilst concentrating mainly on inward-looking aspects of archive services relating to performance assessment, Grimard and Pagé nevertheless point to a pressing need for archives to develop “customized – if not symmetrical – evaluation techniques”, in order to describe and measure what they term “the diversity of the contexts and realities of the archival milieu” (p.116), claiming that the archival community needs to develop its own methodological framework. The literature search did however identify a range of domain specific studies with the potential to generate data relevant to the evaluation of social and economic impact in the archive domain. These include: statistical surveys of usage across the domain; evaluations of projects, including online services, operating across the domain; and individual service reviews / surveys. Key examples of the identified studies in each of these areas are reviewed in relation (where appropriate) to: method and approach; findings; validity of data; benefits of method or approach; costs and/or deployment issues. 5.3.1
Statistical surveys of usage
The key sources of data in this area – the annual collection of CIPFA Statistics, and the generally bi-annual PSQG Surveys of Visitors to British Archives – have already been considered at some length in earlier sections of this report. Besides measures of usage, both incorporate data that provides some indication of potential impacts. CIPFA Archive Services Statistics Estimates (Annual) Method / approach
Annual questionnaire distributed to local authority archive services
Relevant findings / type of impact
Statistics collected generally focus on measures of service performance and use of resources, and provide only broad indicators of levels of usage; data on revenue expenditure and income could however feed into evaluation of demand-side economic impact
Validity of data
Quantitative data; some question over consistency of interpretation, and past application, which has led commentators to conclude that the early surveys at least do not provide a sound basis for meaningful analysis. Data restricted to local authority sub-domain
Benefits of method
Standard questionnaire design conducted by central authority CIPFA, provides comparative data across the local authority sub-domain
Costs / deployment issues
Financial costs associated with participation
PSQG Surveys of Visitors to British Archives (Bi-Annual) Method / approach
Exit survey of users; standard questionnaire
Relevant findings / type of impact
Data collected includes measures (means of travel; interaction with local resources; length of stay) from which can be derived indications of the economic impact of users on the local economy, notably with regard to tourism; respondents are also asked to rate their perceptions of service value, which are indicative of potential social impacts (learning & skills development; empowerment; family ties; cultural diversity/identity)
60
Validity of data
Data collected from across domain, but only from individual on-site users
Benefits of method
Standard questionnaire design conducted in association with central authority (IPF Market Research Unit), provides comparative longitudinal data across time, and across domain
Costs / deployment issues
Financial costs associated with participation
DCMS Taking Part: The national survey of culture, leisure and sport (Continuous) From 2005/6 an additional source of data is provided by the new DCMS led household survey Taking Part. This survey is conducted on a continuous basis (commenced midJuly 2005), and is produced in line with National Statistics protocols. The survey, conducted by interview, measures involvement in activities undertaken for recreation or leisure purposes, including voluntary work, but excluding paid work and academic study. It includes a section relating to visits to archives. Method / approach
Continuous national survey (interviews) of adults (16+) living in a representative cross-section of private households in England
Relevant findings / type of impact
Data collected includes measures of frequency of archive visits delineated by demographic profile (age, gender, disability, ethnicity, socio-economic group, urban/rural). Impact is indicative and related to extent of usage
Validity of data
Data collected in accordance with National Statistics protocols; will build to provide a large statistically-valid sample. Only addresses on-site participation. Data is limited to ‘attendance’ only and does not delineate on the basis of type of service visited, or purpose of visit; this limits the level of analysis which can be applied
Benefits of method
Standard interviews yielding longitudinal participation data
Costs / deployment issues
Conducted centrally by government; no direct costs to domain
5.3.2
Evaluations of projects and online services
The literature search identified a number of domain-specific studies conducted as evaluations of projects and services that extend across the domain, notably those provided within an online environment. Such projects / services often draw on sources of external funding, where some form of evaluation of outcomes is required. Five of the studies34 identified were concerned with preliminary evaluation of online services; three of these are reported on below, the others offered only negligible data on the actual impact of the services. For example, the NANURG user evaluation of the four archive network strands A2A, AIM25, Archives Hub and SCAN (Economou, 2002), focused primarily on users’ views of website design and content; potential evidence of impact is confined to the single point that most users could see “the potential of the sites for satisfying, at least some of their research and information needs” (p.16). A further study reviewed here focused on the evaluation of the 2004 Archives Awareness Campaign.
34
Economou, M. (2002); Ramsden, C., Kelleher, J. & Russell, S. (2002); Ramsden, C. Barkat, S. & Kelleher, J. (2003); Mildren, R. (2004); Archives and Records Council Wales. (2005).
61
Summative Evaluation of the Archives Hub (2003) This report (Ramsden, Barkat & Kelleher, 2003) builds on earlier formative evaluations of the Archives Hub undertaken by the Tavistock Institute (Sommerlad & Kellerher, 2000) (Ramsden, Kelleher & Russell, 2002), where the focus was on validation and informing ongoing service development. This final summative evaluation report is however mainly concerned with issues of user satisfaction, value and usefulness, and outcomes / impacts.
Method / approach
Qualitative research, semi-structured telephone interviews (n=18) and online mini-survey (n=15) focusing on: use and usage; value and significance; outcomes and impacts
Relevant findings / type of impact
Mainly a usage survey, concerned with satisfaction and patterns of usage, findings also indicate that the service has had a beneficial impact on promoting awareness and understanding of archival resources; and in areas of learning & skills development (facilitating teaching practice; adding value / increasing knowledge for research use; enhancing professional work; facilitating private research); together with cost-benefits (economic impacts), in relation to the time and money saved over alternative methods of obtaining the same information
Validity of data
Researchers acknowledge low number of respondents (compared to the extent of usage); also noted is the possibility that evaluation data may be affected by the preponderance of certain groups of users as respondents
Benefits of method
Qualitative research explores the nature of the benefits and the actual differences that might be made – providing a check of existing frameworks, and more data about actual impacts, rather than an estimate of the scale of impacts of a particular type
Costs / deployment issues
Online survey relatively simple to organise, qualitative research does not require a large sample but there are time implications for the time required to interview, transcribe, and analyse the data. Consideration must be given to research design in ensuring the validity of the resulting data
Interestingly the authors (p.i) raise the question as to whether response rates were influenced by users not distinguishing the “distinct outcomes and impacts arising from the Archives Hub”, from those derived from the wider set of web-based search resources available for locating archival materials; something they characterise as routine use of a ‘basket of services’ (p.4). Scottish Archive Network: Project Evaluation (2004) Report (Mildren, 2004), providing a critique and evaluation of the technical, financial and preservation implications of large-scale digitisation of archive materials, based on the experience of the SCAN project. Primarily reports on project management aspects and critical success factors, but with some indication of potential impacts.
Method / approach
Case study, incorporates data from: project progress reports; usage statistics; a survey of participating archives (2003); and views of the project from staff, participating archives, members of a user group, GSU volunteers, external advisers, and customers
62
Relevant findings / type of impact
Includes within its listed achievements contributions to genealogical tourism (notes statistics revealing that in 2001 0.5 million visitors to Scotland undertook some genealogy related activity); building capacity within the Scottish archive community; and benefits to the archive customer, through increased access. Commentary on ‘archives and education’ indicative of potential impacts in learning & skills development on users – similar impacts on the development of ‘staff skills and knowledge’ are directly referenced. Usage statistics are reported, from which the extent of potential impact can be inferred
Validity of data
Evaluative report of the experience of conducting the project, its outputs, and perceived outcomes – reflects internal perspectives. Qualitative comment from a range of associated individuals is included, together with some statistical survey data
Benefits of method
Uses existing knowledge base / data generated as part of ongoing project work
Costs / deployment issues
Time and effort compiling report; requires a high level of ongoing project documentation
Archive Network Wales: Project Evaluation (2005) Report of the findings of an online user survey (Archives & Records Council Wales, 2005), conducted as part of an evaluation of the project to determine how well it had met the requirements of the partners, users and other stakeholders. Primarily aimed at exploring levels of user satisfaction and profiling users, but with some indication of potential impacts. Method / approach
Online questionnaire survey (n=43)
Relevant findings / type of impact
Of the 43 respondents, 53% strongly agreed that the website told them something they didn’t know before (learning & skills development), and 30% strongly agreed that the website had changed their mind about something, which demonstrates progression from ‘improved knowledge’ to ‘changed perception’
Validity of data
Demographic profile of respondents related to population as a whole. Online questionnaire – therefore not a random sample, subjective views collected
Benefits of method
With good questionnaire design can obtain a range of data comparatively easily. Benefited from deployment as part of website; which also provides site for publication of findings
Costs / deployment issues
Relatively cheap to set up and to analyse. Consideration must be given to research design in ensuring the validity of the resulting data. Limitations on form of questions means that impact details may be hard to obtain, or verify
Archive Awareness Campaign 2004: Impact Assessment & Evaluation Report (2005) Report (Fulton, 2005) of the various evaluation activities conducted in relation to the 2004 Archive Awareness Campaign (AAC), including: an assessment of the media coverage of the campaign; user evaluation responses to AAC events; evaluation responses from the organisers of AAC events; and evaluation of the impact of the BBC Who Do You Think You Are series and its link to AAC. The latter drew on an internal evaluation undertaken by the BBC, encompassing: viewing figures and associated website traffic; viewer profile; attendance at events
63
organised in association with AAC as part of a BBC National Family History Weekend; and assessment of the series’ impact on interest in family history, and people’s use of archives. Additional comment on the link with the series was also drawn from the evaluation form completed by event organisers. The objectives of AAC focus on three areas: raising general awareness of the relevance of archives; encouraging non-users; and challenging stereotypical perceptions of archives. Evidence of impact is therefore considered in relation to these objectives, rather than the broader social / economic impact of archive use. Findings are relevant in so far as they are illustrative of levels of potential interest in archives (and hence the potential impact in terms of levels of usage).
Method / approach
Evaluation took the form of a number of discrete strands: 1. Systematic assessment of AAC media coverage, including qualitative analysis of the extent to which the media conveyed AAC’s key messages 2. A nationally developed evaluation form targeted at members of the public who participated in events 3. A nationally developed evaluation form targeted at organisers of events 4. Utilisation of data from an internal BBC evaluation of the impact of the series Who Do You Think You Are
Relevant findings / type of impact
User evaluation found that 40% of AAC participants had not visited an archive before, and 45% stated that they would start researching their family history as a result of AAC. Viewing figures for the BBC series indicated that the series had had an ‘immediate and positive impact’ on people’s interest in family history; snapshot surveys of 10 archive services revealing that this seems to have translated into new users (‘physical visits increased by 24% and new users by 36% in the last quarter of 2004 compared to the same period in 2003’). These findings, indicative of increased use of archives, have the potential to generate social and economic impact through learning & skills development
Validity of data
Qualitative comment reflects subjective views of participants. Surveys of physical visits does appear to justify a cause and effect link between the BBC series and increased use of archives; other figures are indicative of interest, and may not necessarily translate into actual usage
Benefits of method
Use of evaluation forms, deployed as part of the project, enables targeted data to be collected from a relevant participant sample
Costs / deployment issues
Use of evaluation forms is relatively cheap to set up, but subsequent analysis (particularly where response rates are high) may be timeconsuming. Consideration must be given to research design in ensuring the validity of the resulting data. Limitations on form of questions means that impact details may be hard to obtain, or verify
5.3.3
Service reviews / surveys
Most domain-specific activity has probably taken place at the level of individual service reviews / surveys. Indeed, it is evident that some services have conducted independent surveys of their users (Buttler, 1996), (White, 1993) but such activity has tended to focus on levels and purpose of use, rather than assessment of impact. Where published, such surveys tend to pre-date the instigation of the PSQG work, and it can
64
be inferred that subsequent to this most services have adopted this national mechanism rather than developing individual survey instruments. Williams, Baxter, Wavell and MacLennan (2003, p.6) in a survey of impact evaluation undertaken in the ALM sector found that of six archive respondents, two had conducted impact evaluations of internally-funded projects, two of externally-funded projects, and three of core services, but that this activity mostly related to Best Value and PSQG involvement, with some website evaluation (p.7). Although this survey only represents a small sample of archive services, the findings can be taken as indicative of the level and type of evaluation activity being conducted. Best Value Reviews The requirement for local authorities to undertake Best Value Reviews of service provision has now been overtaken by the system for Comprehensive Performance Assessment. Nevertheless, between 1999 and 2003 a number of archive services were included within the Best Value Review process either individually or as part of wider service assessments. Data on the approaches taken to the review process, evaluation methods employed, and review findings is not however readily available; the outcomes in terms of Audit Commission reports can still be accessed in some cases, but these focus on assessment of the service, rather than clearly delineating evidence of service impact, and evaluation methods employed. What is clear from the available data is that the extent and depth of evaluation activities varied enormously from authority to authority; even within authorities, as one County Archivist noted different “business units chose a variety of methods to review their differing services.” (Childs, 2000, p.20) Childs sets out the approach adopted within West Sussex Record Office in 1999/2000, where in relation to the ‘Consultation’ requirement of Best Value, data from the relevant PSQG survey was utilised, and a survey of non-users instigated. From the available data it appears that this approach was fairly typical; other methodologies utilised included focus groups (encompassing volunteers, staff and users), and customer satisfaction surveys. Although the findings from surveys such as Best Value Reviews do point to areas of potential social and economic impact, as with much of the data drawn from domainspecific studies the evidence is by no means clear-cut. This reflects the fact that such evidence is incidental to the objectives of the studies, which tend to focus on usage, customer satisfaction, and service-level performance measurement – outputs as much as outcomes, and not reaching the level of actual impact. This situation is likely to change as the domain adopts a more sophisticated approach to demonstrating the value of the services it provides, particularly in the context of the evaluation of externally funded projects. The following example, while exploratory in tone, is illustrative of the kind of approach that can be taken to considering the impact of such projects. The National Archives: There Be Monsters: A Case Study (2005) For impacts that have not specifically been assessed before, a case study approach may be desirable first, with the emphasis on a formative evaluation that explores likely benefits, how these would be accrued and some of the organisational and partnership issues involved. This case study of an arts project (Griffiths & Tallon, 2005) undertaken by The National Archives in partnership with a Mental Health Trust organisation, with funding from the Heritage Lottery Fund, has, in fact, several impacts apart from the health and welfare of the clients of the mental health services involved. Not least in the
65
way the partners had to co-operate to achieve the project goals, and in terms of changing the pre-conceptions of participants from both organisations.
Method / approach
Case study approach, exploratory evaluation involving: questionnaires (qualitative / quantitative); participant diaries; face-toface personal and group interviews
Relevant findings / type of impact
Explored the possible health and welfare benefits on mental health service users (and noted some of the problems) of an arts project that involved TNA and a London mental health NHS Trust. Possible impacts in terms of improved health & well-being; social empathy; learning & skills development; and participation
Validity of data
Qualitative data – quotations used to attest to the validity of the themes presented. Quantitative data was also collected, but not presented explicitly in the report
Benefits of method
Qualitative research explores the nature of the benefits and the actual differences that might be made, and the case study approach identifies the problems encountered as well
Costs / deployment issues
Not specifically stated, but the impression gained is that this type of project could be quite stressful for all those involved, although there was a successful outcome Likely to require good project management. The accompanying qualitative research also requires time (and therefore money)
5.4
Impact evaluation: the ALM sector
This is an area of increasing activity with a range of recent cross-domain studies being instituted by the English regional MLA agencies, a number of which specifically concern themselves with issues of social and economic impact. This section will review key examples of such studies, addressing: method and approach; findings; validity of data; benefits of method or approach; costs and/or deployment issues. EFTEC: Valuing Our Recorded Heritage (2001) Report (EFTEC, 2001) detailing the findings of a project to identify and measure the benefits of preservation of the recorded heritage (defined as the full range of library and archive materials supporting the written and documentary heritage). Utilises costbenefit analysis as a methodology utilising case studies, and presenting a ‘workbook’ for use by preservation managers.
Method / approach
Cost-benefit analysis utilising contingent valuation in pilot case studies of application (Surrey History Centre; Hulton Getty Picture Collection); surveyed ‘users’ and ‘non-users’ (for SHC, n=38/22) on their use of the institution, perceptions and attitudes towards the recorded heritage, and responses to valuation scenarios Suitability of economic approaches also tested through: expert panel survey (preservation managers); two focus groups (non-professionals)
Relevant findings / type of impact
Focus groups recognised personal, community and institutional ‘benefits’ to preserving the recorded heritage in terms of education (learning & skills development); culture (cultural diversity / identity, participation, social empathy) and availability of information (empowerment, capacity, enterprise support)
66
Over half the respondents to the pilot studies considered that they had personally benefited strongly from the existence of the recorded heritage, and a further 31% considered they had benefited from the existence of these resources. Almost all respondents were willing to pay significant amounts to preserve the recorded heritage; and rated this aspect highly in comparison with other cultural heritage categories
Validity of data
Notes that these were pilot studies and results are therefore exploratory; full-scale surveys would require larger and probabilistic samples. Benefits are rather loosely defined and draw on the perceptions of participants. Was able to conclude that the methodology was viable for future application on a larger scale
Benefits of method
Generates willingness-to-pay estimates, supported by survey data on user and non-user perceptions of service value
Costs / deployment issues
Input of time / resources required at research design stage and in implementation; workbook and case studies do provide a useful starting point
The authors note (p.vii) that while “a great deal of information can be brought within this formal framework, cost-benefit analysis can never do more than inform” decisionmaking; it can’t determine ‘the right answer’ on its own. SWMLAC: South West Museums Archives and Libraries Social Impact Audit (2002) Report (Bryson, Usherwood & Streatfield, 2002) detailing the findings of a project to assess the collaborative impact of the ALM sector in the South West in terms of promoting social cohesion, fostering social inclusion and encouraging lifelong learning. Specifically sought to develop an approach to evaluation, based on social auditing techniques, which would be transferable to general use across the UK.
Method / approach
Development of a social impact audit approach utilising: interviews; focus groups; and pre-existing documentation – implemented across 8 individual projects in the South West
Relevant findings / type of impact
Findings from projects categorised under learning; community identity and social cohesion; economic value; equity and access. Respondents perceived that services had value in all these areas; potential impacts therefore include learning & skills development, social empathy, cultural diversity / identity, regeneration & renewal, enterprise support, and tourism
Validity of data
Qualitative data; utilising quotations to exemplify key findings on impacts from the user perspective. Evidence could be seen as ‘anecdotal’, but authors contend that ‘informed value judgements’ can be made based on such qualitative findings. Developed themes but scale of impact hard to assess; archive data not explicitly delineated from other domains
Benefits of method
Highly flexible; potential to generate in-depth data on complex issues. Can be used to identify matches / mis-matches between social objectives and actual outcomes
Costs / deployment issues
Relatively simple and inexpensive; still requires effort in terms of research design and implementation
67
SWMLAC: The Economic Contribution of Museums, Libraries and Archives in the South West: Libraries & Archives (2004) This report (Brand & McVittie, 2004) presents economic statistics for libraries and archives within the South West and Channel Islands for the year 2002/3. The study is primarily concerned with measurement rather than with valuation; detailed statistics on resource use by archives and libraries are related to the rest of the regional economy; analysis does not explicitly address the economic role and value of services.
Method / approach
Collates and presents detailed statistics covering: outputs (users, loans / document retrievals); employment statistics (FTEs and volunteers; and wage costs); and revenue and expenditure. Estimates were assembled from pre-existing data sources (e.g. CIPFA) and primary data collection via postal questionnaire
Relevant findings / type of impact
Uses data to estimate the direct and indirect economic contribution of libraries and archives to the regional economy; no further analysis is provided but this data could be used to inform judgements of economic impact
Validity of data
Draws on data from a range of valid statistical sources, and from a quantitative questionnaire; considers that for some sectors covered figures may be underestimated
Benefits of method
Utilises existing data sources
Costs / deployment issues
Time (and hence money) required to collate data, conduct primary research, and undertake analysis; some economic expertise required in analysis of statistics
The study, does raise the question, in the context of public sector funding and the investment of time given by volunteers, whether these quantifiable ‘flows of resources’ “should be regarded as indicative of the ‘value’ that government and society place upon libraries and archives”. (p.4-5) ALM London: London’s Culture Equation (2005) This report (Burns Owens, 2005b) presents existing evidence of the economic impact and importance of the ALM sector in London, and sets out a strategy for increasing the sector’s contribution to the economic prosperity of the capital.
Method / approach
Collation of existing data on direct demand-side and inferred supplyside economic impacts, drawn from statistical data sources detailing employment, turnover, and usage
Relevant findings / type of impact
Some delineation of archival contribution within aggregated sector; mostly drawn from sources already reviewed (e.g. PSQG) Evidence is suggestive of supply-side impacts in areas of learning & skills development, enterprise support, regeneration and renewal, and tourism
Validity of data
Draws on data from a range of valid statistical sources, but focuses on areas where data is already in existence – aspects such as indirect and induced demand-side impacts are not considered because base data is not available Appears to omit consideration of CIPFA Archive Services Statistics Estimates
Benefits of method
Utilises existing data sources
68
Costs / deployment issues
Time (and hence money) required to collate data and undertake analysis; more complex methodological approaches to the assessment of indirect / induced demand-side impacts are referenced, which would presumably require much more effort to implement
Bolton’s Museum, Library and Archive Services: An Economic Valuation (2005) This study (Jura Consultants, 2005) used the Contingent Valuation technique to demonstrate the consumer surplus attributable to Bolton’s museum, library and archive services. Based on an approach developed for assessment of British Library services, this technique (more usually associated with environmental and transport economics), seeks to ascertain the collective value that individuals derive from the provision of a service, in relation to (p.9):
Use value – value created through direct use of services
Option value – value derived from services being available for future use if the individual requires it
Existence value – value generated by a service by its existence, for both user and non-user
Respondents are asked both what they would be willing to pay to continue to access a service and how much they would accept in compensation to forego it. This study is discussed further in section 5.6.1 in relation to Contingent Valuation methodology.
Method / approach
Contingent Valuation; utilising face-to-face questionnaires (total n=325; archive users n=66) to ascertain a range of information about users, their profile and their valuation of services Additional qualitative data derived from focus groups
Relevant findings / type of impact
ALM domain users and non-users are specifically delineated; archive users place a high importance on services, and while findings indicate that in comparison to libraries and museums users and non-users were prepared to pay substantially less for archive services, this still represented an increase on current funding levels. Evidence from focus groups demonstrated both a lack of understanding of archives, and recognition of their potential role in linking “future generations to the stories of today” (cultural diversity / identity).
Validity of data
Method is statistically sound; questions arise over potential discrepancies between expressed ‘willingness to pay’ and how this would translate at the level of actual economics
Benefits of method
Provides a solid evaluation of the economic ‘value’ of services to local populations
Costs / deployment issues
Method can appear complex to implement (report provides a toolkit for implementation); some time and effort requirement for effective implementation
Economic Social and Cultural Impact Assessment of Heritage in the North East This report (North East Historic Environment Forum, 2005) has been included here as the study does consider archives, alongside other heritage ‘sectors’ (historic houses, archaeological remains, museums, and the built environment.) Analysis is based on information and statistics provided by the heritage organisations themselves, and there are thus limitations and inconsistencies to the data in terms of availability of statistics,
69
comprehensiveness of coverage, and consistency of approach. This information was supplemented by interviews with the main heritage organisations, and the use of case studies. The study is able to conclude from this that heritage has important positive impacts in the following areas:
Economic impact: in terms of employment, tourism spending, urban renaissance, rural regeneration, and in shaping regional image and identity
Cultural impact: in enhancing wider cultural well-being, and in terms of its own intrinsic cultural value, which helps shape regional and community identity
Social impact: in relation to a range of education projects and initiatives to widen access and participation, which are providing a focus for community development and participation.
As an overarching review this form of study makes persuasive reading and the sheer quantity of evidence adds weight to the conclusions; the quality and rigour of some of the constituent data is however questionable. The coverage of the archive domain is however an extremely minor element within the wider study.
5.5
Impact evaluation: the wider cultural sector
Within the wider cultural sector, including the arts, libraries and museums, a range of domain specific and individual service / project studies have also been undertaken. While these do not relate directly to archive services, many of the methods and approaches, and indeed to some extent the findings, are likely to be transferable to the archive domain. The individual studies included here have been chosen as exemplars of the evaluation work that has recently been conducted. It has not been possible to present a comprehensive review of all studies undertaken in the cultural sector, as this would be beyond the scope of the present report. Additional comment on other studies is drawn from the literature on impact evaluation applicable to each field. 5.5.1
The Arts
Arts Council England (2004) offer a wide-ranging review of the available research evidence for the impact of the arts; as with the archive domain it is noted that while many claims are made about the impact of the arts, the available evidence has not always been “rigorously researched or evidenced” (p.3). Research into the arts and employment (economic impacts) is noted, and potential social impacts arising from education projects, work with the health sector, and in the area of criminal justice. Further social impacts relating to the interplay of arts and regeneration projects are also explored. Data on actual projects is limited, with the focus on findings rather than methodology. Newman, Curtis and Stephens (2003) examined whether community arts projects contributed to positive social gains through a review of existing studies. The review only covers studies which met its inclusion criteria; that there was a clear description of methodology with presence of a control group or use of multiple methods of evaluation. Indeed, several studies (n=11) were excluded as the description of their methodology was not judged adequate. The review comments that many of the studies failed to meet rigorous methodological criteria, the main drawbacks being the lack of necessary detail on the sampling, methods and corroborating evidence. As in this study, “considerable methodological challenges” (p.313) are acknowledged. The categories of impact
70
identified were: personal change (which can be linked to participation, and learning & skills development); social change (cultural diversity/identity, social empathy); economic change (employment, regeneration & renewal); and educational changes relating to improved school performance. Snowball and Antrobus (2002) reviewed economic impact studies of arts festivals, identifying the methodological pitfalls associated with such studies, and the methodologies employed. These included the ‘Willing to Pay’ (WTP) method, which may be used to estimate public goods effects, along with contingent valuation methods, as well as other economic impact assessments. Conclusions reinforce the findings of this study, that attempts to measure partly public goods is “fraught with methodological problems” – further to which “even the most accurate economic impact study of a public good (or a good with both public and private good characteristics) will not account for its full value because such a study cannot give a monetary value to the positive externalities which such goods provide.” (p.1318). The Cultural Investment and Strategy Impact Research project (CISIR) (ongoing) A 10-year longitudinal research study on the Gateshead Quayside intended to evaluate the social, cultural, economic and regenerative impact of the Quayside development over the decade 2000-2010.
Method / approach
Programme of four major research projects, including the collection of data on employment, the composition of the business sector, changes in tourism, and the housing market. Changes in social indicators (educational participation/attainment; health; crime; housing) will also be tracked The cultural dimension of this programme comprises: a series of major surveys (Market Research UK) on cultural values / attendance; documentary analysis and interviews with staff, stakeholders, and consumer groups of the key cultural institutions (the Sage and Baltic); data collection on comparative cultural organisations; profiling of cultural organisation in the fields of Arts and Entertainment, Sport and Recreation, and Heritage and Environment
Relevant findings / type of impact
Seeks to clarify social and economic impacts, and the impact of arts investment on the overall cultural life of the area (changes of attitude / aspiration among key communities) Interim findings indicate that “the developments appear to be reinforcing a certain sense of pre-existing local pride.” (Bailey et al, 2004, p.59) – indicative of impacts on cultural diversity/identity. Results of surveys reported indicate that there has been “a radical change in the perception and role of the arts in people’s lives” (p.61) – links with participation
Validity of data
Methodological approaches appear valid; findings are interim and full data is not available
Benefits of method
Longitudinal research offers the opportunity to assess impact over the required timescales; balance of quantitative and qualitative data; multifaceted data provides a holistic picture
Costs / deployment issues
Major longitudinal research of this nature requires considerable investment of time / resources for returns over a longer timescale
The project is attempting to go beyond the collection of statistics about activity levels, attitudes and participation, and establish appropriate measures for the long-term impact of such developments. Interim findings are reported by Bailey, Miles and Stark (2004) who review the cultural dimension of this research programme, and note (p.58) that its 71
overall intention is “to gather a dataset on the impact of the Quayside development on the consumption and production of culture.” Concludes from data gathered thus far that there appears to be a ‘cultural imperative’ at work (p.61), where culture-led regeneration is tapping into pre-existing cultural identities, leading to “the radical reassertion of a rooted identity in new ways.” (p.63) 5.5.2
Libraries
The library domain has also engaged widely with impact evaluation as a concept, a number of studies relevant to the domain have already been mentioned in relation to the ALM sector. This section reviews two recent domain-specific studies, which have the potential to influence future practice, including a study of the British Library where the methodological approach has already been adapted for use elsewhere (Jura Consultants, 2005). British Library: Measuring Our Value (2003) Report (British Library, 2003) of the results of an independent economic impact study measuring the Library’s direct and indirect value to the UK economy, utilising Contingent Valuation techniques. The economic welfare generated has been measured by the size of the consumer surplus (the value gained by beneficiaries over and above the cost to them of using the service). Respondents were asked both what they would be willing to pay to continue to access a service and how much they would accept in compensation to forego it. Levels of investment in time / money to make use of the Library were also explored, as were the potential resource requirements of using alternatives (if these could be found).
Method / approach
Contingent Valuation; utilised face-to-face questionnaires (total n=2000+ selected at random from different groups including direct users and members of the public) to collect valuation data
Relevant findings / type of impact
Findings demonstrate that in terms of economic impact the British Library generates value of c. 4.4 times the level of its public funding Detailed analysis of specific findings are not reported
Validity of data
Method is statistically sound; consider that the results reported are likely to be conservative (since only selected Library services could be addressed). Results exclude non-UK Library use, which is likely to substantial, and to generate reciprocal value for the UK economy
Benefits of method
Provides a solid evaluation of the economic ‘value’ of services to chosen populations
Costs / deployment issues
Method can appear complex to implement; some time and effort requirement for effective implementation in terms of questionnaire design. Large samples are required to generate valid results
Laser Foundation: Libraries Impact Project (2005) Report (PricewaterhouseCoopers, 2005) of the findings of the Libraries Impact Project commissioned by the Laser Foundation to develop and use measures that could be utilised by a library service to identify its impact on four of the shared priorities between central and local government (children, education, health, and older people).
72
Method / approach
Combined use of quantitative statistical data on usage, and qualitative data on user experience / perception; data collection methods are poorly delineated (brief mention of use of statistics, questionnaires, surveys, and interviews) Methodologies were piloted with 7 authorities (comments are provided on their experience of the process of data collection)
Relevant findings / type of impact
Pilot studies yielded clear and measurable contributions in line with government priorities to: the development of adult skills and child literacy (learning & skills development); patient / public involvement in health (healthy lifestyle); and quality of life for older people (independence)
Validity of data
Impact measures appear sound, but less guidance is given on how to actually collect the data
Benefits of method
Combination of quantitative and qualitative data collection, balances hard statistics with qualitative comment Detailed methodologies are provided, as a practical tool for implementation by library staff; these do however focus on the ‘questions to ask’, rather than ‘how to ask them’.
Costs / deployment issues
Provides a template of impact measures, but time / resources would still be required in terms of development of collection instruments, implementation, and analysis of findings
5.5.3
Museums
In many ways, museums have been at the forefront of impact assessment activity. Silverman (1995) discusses the growing importance of ‘visitor meaning making’ for museums, citing some studies and projects that tried to make the museum experience more relevant for the visitors in meeting individual and community needs. Scott (2003) describes preliminary work using an online Delphi Panel35 study, comprising ‘public’ and ‘professional’ cohorts, to examine what perceptions were of the social, personal and economic impacts of museums. Both cohorts recognised how museums built human and social capital, contributed to community development, and had the potential to yield economic benefits (attracting tourism, creating employment). The professional cohort were also deeply aware of the potential for museums to contribute to social change and public awareness, in meeting social inclusion agendas. Scott continues her theme in a paper from 2004, which extends the discussion to impact evaluation itself, and identifies a range of recurrent concerns, which this study has also noted: problems with the definition of impact; the need for more robust methodologies and valid evidence; problems with establishing cause and effect; and the need for more longitudinal research to assess long-term impact. This leads her to conclusions that might equally apply to the archive domain, that although “impacts are claimed, the current methodological and sectoral issues mean that evidence of impact is lacking.” (p.9) A critique of the social impact of museums and galleries as agents of social inclusion (Newman & McLean, 2004) used qualitative research (semi-structured interviews with policymakers, local and national, and key practitioners). One of the identified themes was the need for measurement of social impact, based on indications from government that qualitative and anecdotal evidence needed to be translated into quantitative terms.
35 A method via which geographically dispersed individuals, who have knowledge and opinions to share, can generate ideas and achieve consensus.
73
It is apparent that in approaching impact evaluation museums are facing many of the same issues as the archive domain. As with libraries, a number of actual studies are being taken forward under the umbrella of the ALM sector. In terms of domain-specific work, Johnson and Thomas (2000) review the approach, methodology, results and interpretation of seven studies which have sought to analyse the specific economic impact of museums. Their overall critique notes that while these studies can improve understanding of the processes underlying potential impact, “without similar calculations for other activities in the economy, based on identical methodologies, it is difficult to place museums in any kind of relative position”, and that even the identified studies are hard to compare on a like-for-like basis. One further recent study is discussed individually here as an approach that could be of possible future relevance. Tyne & Wear museums, Bristol’s museums, galleries and archives: social impact programme assessment (2005) Report (AEA Consulting, 2005) presents the findings of a social impact assessment of completed museum programmes in Tyne & Wear and Bristol.
Method / approach
Literature review on social impact evaluation in the cultural sector; interviews with staff and partners involved in the assessed programmes; focus groups / questionnaires with programme participants (n=63); analysis of GLAAM statistics
Relevant findings / type of impact
Identified positive social impacts in relation to the following variables: exploration of ideas, values & dreams; learning something new; increasing the desire to learn; increasing confidence; inspiring creativity; developing skills (learning & skills development); making friends (participation); increasing pride in culture and traditions (cultural diversity/identity); positive impact on health and well-being (healthy lifestyle)
Validity of data
Methodology appears sound; small sample size, may result in bias within the findings; difficulties arising from time lapse between participation in the programme and the assessment exercise (this could also be a positive factor in demonstrating long-term impact)
Benefits of method
Balances quantitative and qualitative data; draws on opinions of professionals and participants in providing a holistic evaluation of programmes
Costs / deployment issues
Time (and hence money) required to collect and analyse data
This study draws some useful conclusions about the social impact assessment process; notably the need to clearly articulate goals, and link programme design and assessment methodology prior to the launch of projects.
5.6
Impact evaluation in other fields
The literature review identified a range of evaluation approaches, methods and techniques used in other fields which may be transferable to the archives domain. The difficulty of measuring the social and economic impact of archives has already been discussed, and the need to collect qualitative data in a rigorous methodological manner has been identified. The measurement of intangible items that are nevertheless recognised as valuable and desirable (e.g. health, intellectual ability) or completely undesirable (e.g. fear of crime, lack of confidence) is the challenge facing those who seek to pin down the social and economic value of archives. Not least in developing the 74
terminology to allow impact to be expressed in a meaningful way; this question does point us however to approaches that place monetary values on intangible goods, since the language of economic value is well recognised among policymakers. In the following sections, many of the examples of research conducted using these methods are taken from the EVRI database. This database “is a searchable storehouse of empirical studies on the economic value of environmental benefits and human health effects” (http://www.evri.ec.gc.ca/english/default.htm) and has proved an invaluable source of information for this report. Likewise the DTLR’s manual of multi-criteria analysis (2001) provided an excellent starting point for a review of techniques. 5.6.1
Stated preference techniques
One method of approaching the problem is to use stated preference techniques. These techniques have been in use for over 40 years and they enable evaluation of the impact of non-market goods i.e. anything which cannot have a market value directly attributed to it. Stated preference techniques evolved as a market research tool to assess and understand consumer behaviour, but their use has become widespread, particularly in health economics. One of the most well-known applications of stated preference (SP) is the Natural Resource Damage Assessment of the 1988 Exxon Valdez oil spill, where SP was used to estimate the cost of loss of wildlife and environment, but it has been used across a variety of subject areas, including water quality, wilderness preservation, air quality, health care and food safety (Stevens, 2005). In a summary guide prepared for the Department for Transport, Local Government and the Regions (DTLR) (Pearce & Özdemiroglu et al, 2002, p.87), the authors note that: …the use of stated preference techniques is a complex but increasingly used means of establishing money values for impacts which do not themselves have observable money values. This would seem to make them ideal for application in the archive environment. Although there are differences in interpretation throughout the literature, with several different approaches, we will concentrate here on two main applications of the stated preference technique; contingent valuation, and conjoint analysis (also known as choice or multi-attribute compositional modelling). Method Contingent valuation is conducted by means of surveys or questionnaires, where respondents are asked direct questions about their Willingness to Pay (WTP) and Ability to Pay (ATP), for a non-market good or service, within a hypothetical market. A recent US study set out to measure the value of leisure, using contingent valuation (Dalenburg, Fitzgerald, Shuck & Wicks, 2004). Respondents were asked to put a nominal value on various leisure activities, in terms of their WTP and ATP. The researchers then multiplied that amount by the number of hours spent doing the activity and the proportion of the respondents who carry out each activity, to give an estimated value of a leisure pursuit. The concept of Willingness to Accept (WTA), what individuals are prepared to accept to forego access to a service, is also utilised by such approaches. Conjoint analysis employs ranking, rating and paired choice tools to provide a deeper analysis than contingent valuation, requiring respondents to make more complex decisions about their WTP, based on comparisons, adjusted conditions and packaged commodities. For instance, in consideration of food safety, a survey carried out in 2001/2002 examined the public’s WTP for better labelling of genetically modified goods
75
(Loureiro & Hine, 2004), using single bounded dichotomous choice contingent valuation. Respondents were asked to state whether their preference was for mandatory or voluntary labelling, or if they were indifferent to the issue. They were then asked to make choices about their WTP for mandatory labelling (through annual taxes) and voluntary labelling (through higher grocery bills), with a range of amounts being suggested by respondents. The survey enabled a hypothetical price to be put on the value of labelling goods, which was otherwise difficult to evaluate. Conjoint analysis measures preferences over whole scenarios, but also measures attitudes to specific attributes within those scenarios, allowing sensitivity analysis based on the presence or absence of single attributes or combinations. Advantages/disadvantages Stated preference techniques allow monetary values to be placed upon non-monetary items, providing a fiscal basis for evaluation that is obviously attractive to policy and decision makers. In discussing environmental decision making, Stevens (2005, p.189) suggests that “most of the value of preservation is often existence (or passive use) value that can only be measured using contingent valuation, CV, or related stated preference methods”. Indications of willingness to pay for the continued existence of a rare orchid versus willingness to accept compensation for its extinction, for example, allows decision makers to estimate the theoretical value of the orchid and to establish the point where willingness to pay starts to decline. The technique also demonstrates preferred choices between options, providing deeper qualitative information about public preferences and how they value goods and services when compared to one another. Their application to the archive environment is clear; if the value of the archive is predicated upon the ‘difference it makes’ to the users of the service, then it is only by measuring public preferences and attitudes that this qualitative data can be obtained. SP techniques seem to offer an acceptable scientific method of obtaining and interpreting qualitative data that, until now, have remained largely anecdotal in the archive domain. The attitudes and preferences of the non-user can be identified and catered for using SP techniques, and there may be ways of adapting SP to include measurement of the impact of secondary usage. However, there is a certain level of debate about the validity of SP techniques, not least from economists and the physical science field. Prieto and Sacristán (2004) are sceptical about the use and validity of preference based measures for the evaluation of health outcomes. Their study, which compared the results of a preference-weighted statistical analysis of a health related quality of life (HRQOL) questionnaire with a nonweighted questionnaire, concluded that “the empirical issues concerning the assessment of HRQOL through preference based measures remains unresolved,” and go on to note: The debate on the convenience or otherwise of using social preferences in the evaluation of health states is far from being solved. In theory, in government-financed health systems, social decisions are responsible for the allocation of resources. However, the supposed objectivity of social preference measures should not neglect the fact that many conceptual, ethical and methodological problems have yet to be solved… Another aspect of SP worth considering is the extent to which WTP statements may be governed by the fact that there is no actual expectation of payment, and therefore participants may inflate their survey answers.
76
Typical uses and specific studies Prieto and Sacristán notwithstanding, the area of health economics does seem to be in the vanguard in its research into the use of SP techniques, mainly to assess patient attitudes towards health care programmes and policy. Johnson, Banzhaf and Desvouges (2000) used a combination of graded pair and discrete choice modelling to establish “credible measures of the economic value of morbidity.” In essence, subjects can choose to remain in a relatively severe condition and not pay any additional costs for treatment outside the government health plan or insurance plan, or they can choose to pay for additional treatments to improve their health… Subjects were given examples of different health conditions and their effects, along with the associated costs, then asked to indicate their preference for one scenario over another, to determine the willingness to pay for health attributes, and to discover the trade-off relationship between cost and health. They conclude from their research that stated preference techniques (especially when used in combination to provide added robustness and validity) provide meaningful estimates of values, “even from subjects who have never personally experienced these conditions.” In a pilot study, Gerard, Shanahan & Louviere (2003) explored the feasibility of using discrete choice modelling to develop policies to enhance participation in breast screening programmes. They discovered clear preferences for different service configurations and identified the most significant service attributes for the respondents e.g. accuracy, desire for privacy, travel time. They concluded that further research and application of SP techniques should be backed up by economic modelling, but also indicated that SP is a sound generic approach and is “highly pertinent in an environment that is becoming increasingly cognisant of the importance of consumer preferences within health care systems.” (p.1084) Other examples of the use of conjoint analysis with positive results include assessment of preferences for health care interventions in HIV testing (Phillips, Maddala & Johnson, 2002) miscarriage management (Ryan & Hughes, 1997) and vaccination against pertussis (Lee et al., 2005). When searching using the key phrase ‘contingent valuation’, the EVRI database provides 889 records of studies, spanning a 20 year period and many different countries. It is impossible to review all these here, but the following is a list of indicative studies from that set of results:
A Contingent Trip Model for Estimating Rail-trail Demand
The Assessment of Households' Recycling Costs: The Role of Personal Motives
Commodity Information and Willingness-to-Pay for Groundwater Quality Protection
The Value of Footpath Provision in the Countryside: A Case-Study of Public Access to Urban-Fringe Woodland
Individual WTPs for Reductions in Cancer Death Risks
77
These studies demonstrate that stated preference techniques are already widely used in the areas of environmental health and protection, general health and welfare, and transportation, to attempt a valuation of non-monetary goods in a variety of contexts. The extent to which these techniques can successfully be applied in the archival setting remains to be seen, however, last year, as noted in section 5.4, a study was conducted jointly by Bolton Metropolitan Borough Council and MLA North West, using the contingent value technique to assess 15 libraries, three museums and the central archive in Bolton. The technique enabled the capture and examination of use, option and existence values for the services, using a statistically valid sample (325 people participated) of discrete user groups and questionnaires to establish WTP and WTA. The resulting report provides an excellent template for conducting similar studies and could be adapted to examine the archive environment, independent of libraries and museums; indeed a toolkit for use by others is included in the report. The study is detailed in its explanation of method, transparent in approach and revealing in its findings about public perception of archives, in comparison with libraries and museums. There is also evidence that stated preference techniques are gathering some currency in the general arts field. A set of articles in the Journal of Arts Management Law and Society volume 34(3) (2004) provide an overview of contingent valuation methods in the arts. A study by Thompson (1998) mentions the use of contingent valuation for a study in Kentucky, designed to assess the value placed by the community on arts performances. A literature review by Noonan (2002) describes ongoing work by the author to catalogue contingent valuation methodology studies. Most fall into the categories of studies on admission fees, preservation studies (to keep a cultural icon or not), alternative funding studies (willingness to pay) and new project studies to estimate values for projects that have a cultural impact. The list includes arts, museums and broadcasting studies. The review points out the problems of the methodology, in that very different answers may be provided by respondents if the questions are altered slightly. This has also been recognised in several environmental impact studies that have used the method, e.g. Bateman and Langford (1997). The methodology can provide answers the arts organisations may not wish to hear, and fair communication of the findings can be difficult. The review suggests application to ‘non-use’ studies in archives. Rushton (2004) also suggests that the method can be used to assess public interest in privately owned cultural property – and that might apply to some archival collections. 5.6.2
Cost-benefit analysis (CBA)
This method is used to identify and evaluate impacts in monetary terms, distilled into a single criterion (Net Present Value (NPV) in private sector economics). CBA emerged in the 1960s, and is now widely used in government to evaluate and make public expenditure decisions. Method Like contingent valuation, CBA uses willingness to pay (amount that those who gain from the service may be willing to pay) and willingness to accept (amount that those who will NOT gain are willing to accept in compensation) as measurements. A wellknown and popular method of analysing and justifying public expenditure, CBA seeks to establish if the benefits of a service outweigh the costs. As an evaluation tool, it could provide purely monetary values for the worth of the service.
78
When applied to environment valuation and project appraisal in the third world, McFarquhar (2000) insists that ‘social pricing’ is an essential element of CBA, used “to capture indirect benefits and costs of a project not part of the financial flows”. In terms of measuring economic and social impact, the applicability of CBA to archive services lies mainly in the identification and measurement of the benefits side of the equation (although a full CBA could provide useful data upon which to evaluate a service’s effectiveness). Advantage/disadvantages CBA would seem to be a good technique for comparing options, particularly where these may be different in scale and budget, transforming social factors into monetary terms, and evaluating the best possible outcome, based on comparable financial data. However, since there is some correlation between income and willingness to pay or accept, CBA becomes difficult to use if income distribution is to be taken into account, both for the weighting of the data and the subsequent analysis of gains and losses across the community. (There is currently no consensus in government on how this can be satisfactorily achieved). Relationships between different impacts may be insufficiently addressed – the social and environmental consequences of an action may be regarded more negatively when taken together, than when evaluated separately. CBA may not be the best tool to measure diffuse social outcomes, since, according to McFarquhar (2000): Distribution [of benefits] over time and between generations excites much debate among philosophers, social scientists and economists who are not agreed even within disciplines. McFarquhar also points out that “money implies consumer choice, not citizen choice”; by reducing everything to monetary terms, there is a risk of pigeon-holing the user solely as consumer, and ignoring the possibility of altruistic motives in choice. He also indicates that CBA is often used to “justify, rather than assess” projects, and is a topdown, rather than a bottom-up approach, which makes it, in his view, difficult to apply at anything other than a national level. Typical uses and specific studies Cost-benefit analysis is widely used in both the public and private sector, specifically in the areas of transport economics, policy planning, health care, urban regeneration and defence economics. There is an extensive and well-established literature base on the subject which we do not propose to review here. However, Stirling (1997), cited in Kontoleon, Macrory and Swanson (2001), provides an analysis of more than 30 published CBA studies on the environmental costs of coal-fired power stations. The study concluded that although individual results appeared to be extremely precise, when combined, they in fact proved to be extremely varied, highlighting the uncertainties and underlying assumptions that exist, even in the most rigorously applied techniques. 5.6.3
Cost-effectiveness analysis (CEA)
This is an economic evaluation technique, used where alternative options to achieve an objective exist, but where the objective itself cannot be valued easily. In this sense, CEA can be used to assess whether services are using the least-cost way of achieving an objective. This can be carried out as a piece of independent analysis, where
79
achieving the objective does not affect any other aspect of the service. However, it is equally likely that one course of action will mutually exclude another; for example, where one cannot do both, is it more cost-effective to produce more on-line materials or to extend opening hours? In mutually exclusive analysis, opportunity cost – the value gained from using resources elsewhere – must be assessed. Non-cash opportunity costs, such as the use of assets, and external costs, such as consumer travel costs, should be included. Method In an independent analysis, the cost effectiveness ratio (CER) for an archive service would be expressed: CER = Cost of providing service Benefits produced For a mutually exclusive analysis, an incremental ratio is used: ICER = Difference in cost between service options Difference in benefits between options For instance, a proposal to digitise a particular archive collection could be considered independently, weighing the cost of equipment, staff, expertise etc. against the value of perceived benefits to determine whether the project is actually worth undertaking. Mutually exclusive analysis would be required if a limited budget was available and the digitisation project was competing with a cataloguing project, a new member of staff, and an outreach initiative. In this case, options are ranked according to their effectiveness (i.e. gross benefits), irrespective of cost, then the formula above is applied. Depending on budget, it could be that it is most cost-effective to implement two small projects, rather than one large one, because the incremental benefits are greater. Advantages/disadvantages This is good technique from a ‘value-for-money’ perspective but it is not always possible to compare across different options where the benefits are not directly comparable. In healthcare, benefits are generally measured in life-years gained / lost, but Phillips and Thompson (2001, p.5) observe that: As long as the outcome measure is life-years saved or gained, comparisons can be made, but even in such situations CEA remains insensitive to the quality of life dimension. In order to know which areas of healthcare are likely to provide the greatest benefit in improving health status, a cost-utility analysis needs to be undertaken using a ‘common currency’ for measuring the outcomes across healthcare areas. In the archives domain, similar issues are likely to occur; the benefits of a digitisation project will be different from those arising from outreach activities and there is a problem with balancing them against each other. When considering least cost actions, it may be the case that service administrators have a different view to the public – least cost to the service may mean more cost to the user – this should be taken into account during the analysis.
80
Typical uses and specific studies The DTLR (2001) indicates that CEA is the most common form of assessment driving government decisions on alternative routes to similar outputs, and cites the Treasury ‘Green Book’ on Appraisal and Evaluation in Central Government. The EVRI database includes several studies that use CEA as a research technique; these span fishing, agriculture, ecology (specifically preservation of threatened species), air and water quality. 5.6.4
Multi-criteria analysis
There is a range of techniques that can be classified under multi-criteria analysis (MCA), including total economic value, multi-attribute utility theory and linear additive modelling. The common denominator of these techniques is that they do not necessarily rely on monetary valuations, although these may be included within the analysis. Method Keeney and Raiffa (1976) developed a set of procedures for multi-attribute utility analysis, consisting of three building blocks to evaluate multi-criteria options:
a performance matrix
procedures to determine criteria independence or inter-dependence
a mathematical function to enable expression of overall evaluation of a decision option
The performance matrix rows describes each option, and the columns describe the performance of each option under the expected criteria. Table 12 shows an example performance matrix as it may be applied to expenditure options in an archive service. The criteria columns indicate some potential desired social and economic impacts, as well as the anticipated cost and the strategic value of each option; in order to fill out this table, agreed scoring assessments must be established for each criterion. These can be static values or scales and may be numerical, alphabetical, binary (yes/no, present/absent), financial or descriptive (scaled from ‘very likely to occur’ to ‘very unlikely to occur’). After additional scoring and weighting of option preferences (e.g. there may be external reasons to prefer options with a high strategic plan rating), mathematical routines can be applied that will provide an overall score for each option under consideration.
Option Cost
Strategic plan rating (1* - 5*)
Increase access opportunity
Deepen individuals’ skills & knowledge
Strengthen the community
Contribute to personal fulfilment & enrichment
Digitisation project Cataloguing backlog New staff member Outreach initiative
Table 12: Example Performance Matrix for multi-criteria analysis in an archive service
81
Multi-criteria analysis is a complex and diverse subject and whilst acknowledging its potential usefulness in determining impact, it is not appropriate or useful to examine the techniques in greater depth here; some approaches are covered in Table 13. Advantages/disadvantages Keeney and Raiffa (1976) provide a model for decision making that is used extensively in both public and private sectors, and allows complex scenarios with multiple criteria to be analysed consistently. However, the DTLR (2001) maintain that although the approach is “well regarded and effective”, for any but the simplest analyses of the performance matrix, “it is relatively complex and best implemented by specialists.” This is likely to make the method expensive to use unless it is within a major regional or national context. Typical uses and specific studies Total economic valuation (TEV) studies have been used in the ecological domain and in Morton’s short paper, he cites around 20 relevant studies that support the evaluation of intangible wilderness benefits, including “passive use benefits, including option, existing and bequest values” that could apply equally in the archives sector (Morton, 1999, p.4). Morton’s model of the TEV of a wildland network lends itself to adaptation, the ‘passive use’ benefits in particular could reflect secondary usage impacts. Willis et al. (2003) use total economic value to inform on the social and environmental benefits (SEBs) of UK forests, providing empirical estimates of both total and marginal SEBs. The research addressed five areas of forestry: recreation, landscape, biodiversity, carbon sequestration and other SEBs, and concluded that the research has a number of policy implications for the distribution and structure of woodland to maximise SEBs. The authors stress the need for more precise assessment and categorisation of visitors to facilitate more accurate and robust recreational benefit estimates, and admit that there are some negative impacts that have not been quantified (e.g. disruption of power supplies due to falling trees). This wide-ranging study encompasses several methods and techniques to obtain substantial and authoritative data sets, which have in turn been used by the Forestry Commission to establish policy and to communicate confidently the benefits of forestry to the wider public. 5.6.5
Actuarial assessment
Actuarial analysis is a predictive technique that identifies the probable outcomes and probable risks associated with particular courses of action. This type of assessment is widely used in the insurance sector, where insurers must evaluate the probability and effect of an event occurring against the risk of insuring against that event. Method Risk scenarios are developed based upon a balance of probability and impact. Fig. 10 below shows four possible event risk scenarios: 1. Low probability of occurrence, with low impact 2. High probability of occurrence, with low impact 3. Low probability of occurrence, with high impact 4. High probability of occurrence, with high impact
82
Scenario 1 is the one that would be most preferred by an insurance company, as it represents the smallest risk, and scenario 4 is the one that would carry the highest risk and, therefore, the highest premium for the consumer. Scenario 3 characterises events such as the attacks on the World Trade Centre, whose likelihood seemed so completely improbable, but which had such a devastating impact, in human, economic and social terms. High probability with low impact provides a relatively stable and measurable risk.
HIGH
4
1
3
PROBABILIT
2
LOW
HIGH
IMPACT
Fig. 10: Example Risk Scenario
Multi-dimensional instances of this basic model are created to build up complex analyses of situations, and the effect of inter-relating factors. In life insurance, the applicant is asked many health and lifestyle related questions that build up a picture of the risk to the insurance company that insuring that individual would entail; their age and diet, whether they smoke, pursue dangerous sports, incidences of hereditary disease in other family members etc. The actuarial assessment calculates the likelihood of mortality over time, based on the risks associated with the individual’s lifestyle, and arrives at a suitable premium. Different types of actuarial assessment in the insurance field include life, property and casualty, all of which will have different actuarial tables reflecting the associated risk elements. Actuarial assessment as an estimating and predictive technique goes beyond the insurance industry, however, and is now in common use in other areas concerned with the well-being of individual and society, such as clinical care, mental health, social welfare and criminology. Advantages/disadvantages Actuarial science has been practised in this country since at least 1848 (when the Institute of Actuaries was founded in London) and as such can perhaps claim a stronger pedigree than other techniques. Underwriters have the difficult task of classifying insurance risks and as Trowbridge (1989, p.62) observes:
83
The cluster of ideas surrounding classification, selection, and antiselection are fundamental actuarial concepts. The statistical element is the sorting of risks into homogenous classifications, and the estimation of the appropriate probability for each; but the psychological component is of at least equal importance. Human beings can be expected to act on their perception of their own best interests, and to select against any system that permits choices. It is this preoccupation with classification techniques with respect to human beings that makes actuarial analysis interesting from the perspective of taxonomies of impact; in a sense, the whole actuarial process is geared towards creating robust measures of impact for which financial values can be calculated. The applicability of the actuarial concepts of classification, selection and anti-selection to the archival environment can perhaps be seen if the risk can be converted to benefit, predicting positive outcomes, rather than negative ones. Another advantage of actuarial science is that it commonly deals with the uncertain, over long periods and recognises the inherent difficulties of accurate calculation and forecasting. As Trowbridge (1989, p.67) goes on to note: Actuarial assumptions often, though not invariably, relate to a long span of time, not infrequently fifty or more years. The ability of humans to predict even short-range future events is severely limited, and forecasting ability diminishes rapidly as the time span lengthens. Predictions are often based on “extrapolation” or “the continuance of present trends,” but neither can be expected to hold up for very long. Nevertheless, such predictions are made and accepted, and significant business is conducted based upon, and/or as a result of, those predictions. To overcome the problems of forecasting based on assumptions, actuaries make adjustments to their calculations based upon experiential data. There are different methods for achieving this, e.g. experience rating and actuarial gain/loss adjustment, but however it is arrived at, this is an example of building an evidence base that becomes more reliable in its forecasting as real data replaces assumptions. In terms of predicting and calculating the impact of archives on people and society, this would be a good model to follow. Typical uses and specific studies The insurance industry is the most typical and widespread example of the use of actuarial assessment and is where the technique has its origins, arising from the human need for financial security and avoidance of risk or loss. Trowbridge (1989), provides a good introduction to the topic in his Fundamental Concepts of Actuarial Science. In addition to insurance, however, the use of actuarial analysis has extended to other disciplines where it is important to predict the likely consequences and impact of human actions, given certain circumstances. Falconi et al. (2001) conducted actuarial analysis using Kaplan-Meier survival curves to help rationalise surgery choices in people with a specific type of pancreatic tumour. The Kaplan-Meier survival curve is an actuarial technique that enables estimation of the proportion of a sample of patients who will survive by any given time, which can then provide an estimated probability of survival to that time for any member of the population from which the sample is drawn. In the case of Falconi et al’s study, the aim was to provide a rationale for operating, based on likely survival rates.
84
Harris, Rice and Camilleri (2004, p.1064) discuss the use of a Canadian actuarial model, Violence Risk Appraisal Guide (VRAG) to predict future criminal violence: The instrument was shown to yield a high degree of accuracy (relative operating characteristic [ROC] area of 0.76) for the development sample in the prediction of a subsequent criminal act of violence over an average time at risk of 7 years They modified the model to include non-forensic psychiatric patients of both sexes to evaluate its usefulness in assessing the risk of interpersonal violence posed by this extended population. The study concluded (p.1063) that “results provide evidence about the robustness of comprehensive actuarial risk assessments and the generality of the personal factors that underlie violent behavior.” Mills, Jones and Kroner (2005) in general are positive about the VRAG model, but feel that adjustments need to be made before it could be generalised to non-disordered populations. Risk assessment of sex offenders is a subject where the actuarial approach is relatively popular and, for example, Grubin (1999) examines several studies which explore the actuarial approach to assessing the risk that sex offenders pose, and the extent to which they may be likely to re-offend. He criticises the actuarial approach for being wholly empirically driven, which, he believes, negates the possibilities for generalisation, and he continues (p.334): A more crucial difficulty arises because actuarial prediction is about groups that, in the context of low frequency behaviors, provides relatively little value in respect to the individuals in those groups. This would seem to suggest some difficulty in using the actuarial approach to estimate impact at the individual level; however this approach may be amenable to estimating more difficult to identify impacts at an organisational or societal level, based on prediction, adjusted by experience. 5.6.6
Other techniques / methods
Table 13 describes (in alphabetical order) some techniques and methods that appeared in the literature, but which did not necessarily merit extended description, or for which the evidence and/or literature was poor. Some are variations on those described above, others can be used independently or as part of an overall evaluation. Still others are interesting for their potential application to the archives domain and merit further investigation, not possible within the confines of this study.
5.7
Summary
It is apparent from the foregoing discussion that the importance of impact evaluation is widely recognised. The fact that studies specifically addressing social and economic impact in the ALM sector have begun to appear in the last few years is further evidence that development within the sector is underway. Most approaches to evaluation are however still focused on measuring outputs; determining the ‘difference made’ by an outcome that itself is difficult to measure, introduces further methodological complexity. The best that can be said of the existing studies is that they offer partial solutions to these problems. The identified methodologies used in other fields may have the potential to address some of these issues, but many of these are complex themselves and present their own
85
difficulties for implementation. The economic models that were analysed for potential usefulness had, in the main, the common objectives of aiding decisions by proffering alternatives, or providing empirical evidence of a service’s performance, or providing investment options. The role played by the evaluation of non-market goods in such models, whilst useful and having potential, cannot be considered to provide a direct mapping to the proposed taxonomies of usage and impact. Indeed, it may be that the considerable work required to create credible measurements will negate the potential for some of these models to be used. We believe that the type of impact measurement being sought i.e. how to measure the social and economic impact of archives upon individuals, communities and societies, does not exist in an immediately consumable form and requires further conceptual development and testing against the taxonomies. How these challenges might be addressed are given further consideration in the Conclusions and Recommendations of this report.
86
Table 13: Summary of data on other evaluation techniques / methods Evaluation tools / techniques Analytical Hierarchy Process (AHP)
Benefit transfer
Budget impact models
Characteristics and potential for use in heritage context
A form of linear additive model which uses a more complex method of deriving option weightings and scores. Drawbacks: seems to be some academic debate about the theoretical foundations of this technique. Where information on the value of goods and services generated in one context is used to value similar goods and services in other contexts. Possibility that this could apply across museums, libraries and archives, but requires more research to assess potential for adaptation. Budget impact models address the question of affordability and complement traditional economic techniques such as cost-benefit analysis. Scenario-based calculation and modelling that concentrates purely on costs and does not address benefits at all.
Use of the tool/technique
Relevant supporting literature
Transport
DTLR (2001)
Environment, ecology
Ready and Navrud (2005)
Bloom, M.A. et al. (2005) Health economics and research
Trueman, Drummond and Hutton (2001)
Hard to see how this could be usefully employed to evaluate economic impact except in a very limited and narrow sense. Largely used for predictive modelling, mapping circumstances or sets of circumstances to the value of an item, with decisions being made on the basis of certain circumstances being present, absent, or in a particular combination. Easy to interpret and understand, decision trees can be reapplied against different sample populations to ensure reliability. Decision trees36
Could be adapted to find out what kind of service the public prefers, given a series of inter-related options. DTLR suggest combining decision tree modelling with appraisal principles to develop contingent strategies based on multi-criteria assessment.
RTI Health Solutions: www.rtihealthsolutions.org
Many generic management areas, but includes economists, policy makers, general managers, IT strategists
DTLR (2001)
Drawbacks: Usually applied to well-defined issues supported by technical data rather than issues where the basis for appraising them is uncertain.
36
For more information on decision trees the DTLR (2001) recommends: Golub, A.L. (1997). Decision analysis: an integrated approach. New York: John Wiley; Targett, D. (1996) Analytical decision making. London: Pitman Publishing.
87
Evaluation tools / techniques
Characteristics and potential for use in heritage context
Use of the tool/technique
Relevant supporting literature
Dominance
Where one course of action clearly delivers more benefits than any other taking all criteria into consideration.
Transport
DTLR (2001)
Fuzzy sets
New(ish) but rapidly developing field. Based on recognition that natural language is not a precise method of discussing issues. Fuzzy sets are designed to make explicit that which is vague, based on membership values lying between 0 and 1. So where a particular issue or criterion is determined to be 0.3, then membership of the set is less likely than if the valuation was 0.8. If, for example, the set comprises factors affecting use of archives, then a valuation of 0.8 would indicate a strong influencing factor, 0.3 a weak influence, although both are actually set members. Drawbacks: academic debate over whether the fuzziness of results actually 37 matches the fuzziness of perceptions they are trying to capture. Not clear whether there are any critical advantages over more conventional methods.
General morphological analysis
A type of analysis that aims to identify and examine all possible combination and configurations of a problem. Initially applied mainly in the natural sciences, Zwicky (1969) developed it to encompass abstractions such as concepts and ideas. Closely allied to taxonomy and classification, the method involves an ndimensional ‘morphological box’, where all values and conditions are input, allowing analysis of multiple ‘solutions’ (often computer-aided). Ritchey (2002) claims this method is ideal for dealing with what he terms wicked problems and social messes. Current widely used in linguistics.
Medicine Not yet widely applied due to relatively recent development.
DTLR (2001)
DTLR say “They are unlikely to be of much practical use in government for the foreseeable future”.
Linguistics IT – user interface design
Card, McKinlay & Robertson (1998)
Botany
Ritchey (2002)
Genetics and molecular biology
Zwicky (1969)
Could be adapted, using defined economic and social values, to assess likely impacts of actions taken by archive authorities.
Hedonic pricing (can be used in CBA)
This is a method widely used to assess impact on environmental and quality-oflife factors such as air toxicity or noise pollution. Could be used directly (to assess if the archival environment is suitable for the users’ purposes?) but could also be adapted to measure more tenuous quality-of-life aspects such as enrichment, well-being, cultural fulfilment?
37
DTLR (2001) Environment Transport
Ecosystem valuation: http://www.ecosystemvaluatio n.org/defautl.htm EVRI (134 cases)
For further reading the DTLR manual suggests: Chen, S.J. & Hwang, C.L. (1992). Fuzzy Multiple Attribute Decision Making: Methods and Applications. Berlin: Springer; French, S. (1988). Decision Theory: an introduction to the Mathematics of Rationality. Chichester: Ellis Horwood.
88
Evaluation tools / techniques
Characteristics and potential for use in heritage context
Use of the tool/technique
Either quantitative (some form of non-financial measurement) or qualitative (descriptions of effect) statements about the effect of a decision/policy upon the environment. Could be adapted for use in the archive environment for situations where there are no alternative or generally accepted valuations of impact.
Transport
Impact statements
Linear additive model
Market valuations
Performance Matrix
Ranking and Outranking
A type of multi-criteria analysis which multiplies value scores by weighting on each criterion, providing an overall weighted score. Used extensively where information contained in performance matrices are weighted and valued based upon qualitative judgements. Many MCA models use some form of interpretation of the linear additive model. Use of market valuations such as house prices (e.g. is your house worth more if it is near a museum or archive? Does this affect your house insurance premiums? Are archives and museums likely to be situated in low/high crime areas?)
Relevant supporting literature
Environment Health economics / research
Extensive UK government use
DTLR, 2001
Environmental economics
DTLR, 2001
Transport
DTLR, 2001
Developed in France and used throughout Europe. No wholesale adoption in the UK. Netherlands working on models which extend this technique to include processing of qualitative data, for use in urban and regional planning.
DTLR, 2001
Drawbacks: could involve extensive research for little payback in terms of reliable data. DTLR claims that the literature on valuing impact using market valuation or hedonic pricing is inconclusive. A component of multi-attribute theory, this can be used to compare the effects of different service factors against the same performance dimensions. For example, staff service to customers and the archive service’s website could both be measured against performance dimensions such as availability, initial query response, follow-up response, general levels of satisfaction. From a CBA viewpoint, this could be extended to include dimensions such as initial investment, ongoing running costs, development costs (in staff and IT), to provide a more detailed picture of return on investment and to identify potential growth areas. In this technique, all options/decisions are ranked using weights against each criteria. Outranking occurs when one option outperforms the others based on the weighted criteria and does not significantly under perform on any single criteria. Can illuminate the impetus for formulating the option in the first place. This is a good method for assessing options that are otherwise difficult to compare, or where additional information may come to light at a later date. Drawbacks: Options performing badly on one particular criterion will be downgraded automatically, irrespective of weighting on other criteria. DTLR sceptical about the widespread public use of this technique.
89
Evaluation tools / techniques
Characteristics and potential for use in heritage context
Use of the tool/technique
Trade-off
Benefits of one course of action compensate for costs/loss of benefits in another
In common use
Value dossiers and strategies
Although there is comparatively little academic literature on this method, value dossiers seem to be a feature of medical research and the drug approval process. They represent accumulated evidence of the benefits of a product, and could easily be adapted for the archives domain. Although not particularly ‘scientific’, systematic compilation methods could demonstrate consistency and validity of data which is likely to be rich in quality. Cf. impact statements
90
Health economics and research
Relevant supporting literature
van Oostenbruggen, M.F. et al. (2005) RTI Health Solutions: www.rtihealthsolutions.org Adelphi Group
6
Conclusions and Recommendations
6.1
Introduction
There are many potential dimensions to the social and economic impact of archives; this study has specifically addressed concepts of usage, impact, and approaches to the evaluation of impact:
Section 3 explores the concept of usage, extending this beyond the traditional primary user to encompass wider secondary usage, and presenting a taxonomy of use based on forms of service use, and purpose of use. Potential measures of secondary usage are also discussed.
Section 4 considers what is meant by impact, the complexities of its definition, and how the concept has been applied in the economic and social spheres. A taxonomy of impact is presented based on a view of impact as a continuum which builds cumulatively over time, and extends from the individual to associated groups, communities and thus to society as a whole.
Section 5 then reviews the issues associated with impact evaluation, providing an overview of existing activity relevant to the archive domain, ALM sector and wider cultural scene, and outlining the key methodologies utilised in other fields.
Many of our conclusions are self-evident within the foregoing discussion, not least that evaluating the social and economic impact of archives is not likely to be an easy task. This is a multifaceted area and it is not possible to render simple answers from the constituent complexities. The original brief for this study (YMLAC, 2005) did however include within its required outputs:
Recommendations for appropriate methodologies for adoption in future economic and social impact assessment work.
The intended outcome of any such recommendations being the means to make clear the impact of archives, and hence their value, to stakeholders at all levels.
6.2
Demonstrating ‘value’
What this study has already shown is that in terms of both primary and secondary usage, archives have evident potential to impact positively on individuals, communities and society. This is not however saying anything very ‘new’; the profession has made these points time and again in a variety of advocacy documents and reports. What is all too often lacking is recall to data which provides the type of ‘evidence’ to which stakeholders and government in particular, will pay attention. This study has noted identified shortcomings with existing quantitative data collection, but although strong arguments are made for the adoption of qualitative approaches38
38
The Cultural Heritage Consortium (2002, p.21) consider that the nature of impact evaluation suggests that a qualitative approach is required: “Indeed it is difficult to see how such issues could be represented in a quantitative fashion.” Bryson, Usherwood and Streatfield (2002, p.9) similarly argue that: "In terms of understanding how our social institutions influence us, quantitative knowledge alone is insufficient. The story is richer and often less clear than the apparent precision of statistical data."
91
the resulting data is often characterised as weak, and even ‘anecdotal’. Usherwood (1999, p.5) argues persuasively that “qualitative data, properly gathered, are valid evidence and should be treated as such by politicians and professionals alike", but it is apparent that such data is not viewed in the same light as hard statistics. There are some important questions that could be asked of this situation; Holden (2004, p.14) goes so far as to say, that measuring the ‘ancillary benefits’ that flow from culture has become “more important than the cultural activity itself: the tail is wagging the dog.” Such considerations do not however negate the need to demonstrate ‘value’, particularly to politicians who ultimately control the financial resources available to the public sector archive domain. Increasingly government is thinking in terms of ‘public value’, itself comprised of the three components:
Instrumental value – the social and economic impact of the domain on society
Institutional value – derived from the engagement of services with their public
Intrinsic value – the personal value derived by the individual from engagement with the domain
Holden (2005, p.10) relates these concepts to the key parties involved: the public who use services; the professionals who manage them; and the politicians who set the parameters. He argues that the balance of power between these groups is changing; Matty (2006) summarises his arguments as follows, contending that: Citizens will no longer be content to receive passively services cooked up between the politician (with his/her view of what the public wants) and the professional (with his/her view of what the public needs). This has obvious implications for how we approach the evaluation of both individual services and the impact of the domain as a whole. As Matty points out, while politicians are concerned with the instrumental value of services, and how they contribute to key government social and economic agendas – something of which the domain is very aware and to which it has sought to respond in recent years – the public tend to interact at the more personal level of the intrinsic and to a lesser extent institutional. They may recognise that there is a value to a service at an instrumental level, but the value they place on a service will be much greater if they can recognise its personal relevance to them. The importance of these issues is emphasised by Ray (2006, p.16) in her summing up of a conference on Capturing the Public Value of Heritage: The concept of Public Value appears to be one that will influence our sector over the next few years, and as other heritage organisations are already beginning to use it to consider the ways they are engaging with the public, and to argue their case for funding, it is important that archivists are engaged with this concept or we risk further marginalisation. Such considerations do however highlight other issues and concerns, notably the profile of the domain – if the question of demonstrating ‘value’ becomes removed from the instrumental field and enters the intrinsic then the focus is on levels of personal engagement with archives – how does the domain demonstrate value to a public where
92
the majority haven’t even heard of archives?39 EFTEC (2001, p.64-5) highlighted this as a problem, both from the perspective the public and preservation managers, who considered that “the general public is not familiar with the concept of the recorded heritage and its many uses that impact upon the daily life of society.” This points to the need to widen awareness of archives in terms of both secondary usage, and at the level of primary use. Many popular history television programmes, “either include archival footage or are based on primary research in UK archives” (NCA, 2005, p.24), however this contribution is not always evident from the broadcast product. Popular history publication is also on the rise, and although as the NCA go on to note this level of general interest in historical subjects cannot be guaranteed to continue, the archive domain needs to do all it can to capitalise on current enthusiasm. As findings from the EFTEC (2001) study revealed, “people are aware of the benefits provided by the recorded heritage [but only] when recorded heritage is clearly explained to them” (Özdemiroglu & Mourato, 2001, p.8). This problem of domain ‘profile’ is well-recognised, and initiatives such as the Archive Awareness Campaign (AAC) are a valuable step in the right direction; however, it is apparent that more remains to be done. AAC is one mechanism for tackling this issue, but the agencies working for archives need to continue to focus their attention on awareness raising strategy. The qualitative / quantitative debate also leaves the archive domain (and indeed the ALM and wider cultural sectors) with something of a dilemma. While quantitative data is, in relative terms, easy to collect, and indeed there is a tendency to focus activity on enumerating ‘outputs’40, this says little about the eventual ‘impact’ and ‘value’ of services. As the Demos report (Holden, 2004), Capturing Cultural Value, notes on its title page: The value of culture cannot be expressed only with statistics. Audience numbers give us a poor picture of how culture enriches us. What does this mean therefore for the archive domain in terms of evaluating the impact of what it does? The writing on evaluation is extensive, and within the remits of this project, it was not possible to produce an in-depth review of all aspects of the process. It should also be remembered that the key to effective impact evaluation lies not only in dealing with the difficult methodological problems posed by the relatively amorphous nature of impact, but also in defining the task in hand. Patton (1997, p.189) considers that organisations need to ‘focus’ their evaluation around basic questions such as:
What is the purpose of the evaluation?
How will the information be used?
What will we know after the evaluation that we don’t know now?
What actions will we be able to take based on evaluation findings?
39
This very issue was raised by the study of Bolton’s museum, library and archive services, where Jura Consultants (2005, Executive Summary) concluded – having utilised contingent valuation techniques across the ALM sector – that archives were probably disadvantaged by the fact that there was “a lack of understanding amongst all ages and walks of life about what archives actually are and what they contain.” 40 A number of studies note the tendency within the ALM sector to ‘measure what can be measured’; “measuring commitment and effort, rather than effectiveness” (Burns Owens Partnership, 2005a, p.63).
93
There remain also fundamental concerns not only about the capacity of the domain to produce evaluations, but also their capacity to use them. Feinstein (2002, p.433) reinforces the points made by Patton, in noting that all “evaluations have a cost but not necessarily a value.” He contends that the value of evaluations depends on their use, and that this use should not be taken for granted. Many of the evaluative studies identified in section 5 of this report were produced as a necessary element of large-scale project work; in small services the capacity to engage with evaluative activities may be seriously limited, and here the value of nationallyorganised work, such as the PSQG surveys should not be underestimated. There is certainly a need to support individual services in undertaking evaluation, and a number of studies have sought to address this area (Bryson, Usherwood & Streatfield, 2002), (Williams, Baxter, Wavell & MacLennan, 2003). However more emphasis needs to be placed on ensuring that evaluation activity is targeted effectively; the question of whether the costs involved in undertaking large-scale evaluation work are justified is something that future researchers may also wish to explore.
6.3
Evaluating ‘impact’
This study has identified a range of potential methodologies which could be exploited in undertaking evaluation of domain impact; this investigation of measurement techniques was extended beyond the cultural sector to encompass health, transport, economics, environmental assessment and forestry, to discover what lessons could be learned from practice in different environments. A range of techniques were examined with respect to these, particularly those concerned with the measurement of intangibles, or nonmarket goods. 6.3.1
Social impact
In the context of evaluating social impacts, it soon became apparent that each technique or method has its own strengths and weaknesses – none seem to enjoy unequivocal acceptance, not even the economic models. Contingent valuation (CV) was looked at in some depth, as it seemed to offer some solutions for the archive domain, but there do seem to be some drawbacks with CV, which are not merely mechanistic. Throsby (2003, p.277) observes that …it has long been asserted that a distinguishing feature of cultural goods is that acquiring a taste for them takes time i.e. they are classed as experiential or addictive goods, where demand is cumulative, and hence dynamically unstable. If these demand conditions do indeed obtain, it can be suggested that CVM will not be able to provide fully-informed WTP estimates for cultural goods. There are also methodological issues with contingent valuation methods, the resolution of which may be costly and time-consuming to put into effect (Frey, 1997); these are mainly concerned with the validity and credibility of data gathered using CV. The authors of pilot studies conducted on Surrey History Centre and the Hulton Getty Picture Collection were at pains to point out that “…the results are, therefore, exploratory and cannot be aggregated for the population as a whole” (Mourato, Pearce, Özdemiroglu & Howarth, 2000, p.100). This type of research, irrespective of the subject matter, is expensive to conduct and requires large samples, repeated surveys and commitment to longitudinal research in order to build up a credible body of data.
94
Nevertheless, contingent valuation is a technique approved by government (Matty, 2006), for the evaluation of non-market goods and as such should probably be retained in the repertoire of research tools for assessing the impact of archives, with the proviso that stringent methodological procedures are necessary. For instance, contingent valuation could perhaps be adapted to provide discrete choice models to assess the impact of archives upon a) physical visitors, b) virtual visitors and c) non-users. However if Throsby’s assertion above holds true for archives, then this adds a perspective to the taxonomies that had not been considered initially; namely the relationship between usage, time and impact. What makes a secondary user become a primary user? Does secondary usage have a part to play in creating primary users or are there other, more important triggers? Do secondary users possess awareness of their consumption of ‘cultural goods’? Is there a progression of primary usage along the continuum, based upon perceived benefits and positive experiences or is primary usage of archives more ad-hoc? If so, how can it be measured? As the investigation proceeded, one common factor emerged; that the measurements, however arrived at, put numerical or economic worth upon the value of a non-market good, rather than on the impact of that good upon the individual, organisation or society. It can of course be argued that the perceived value of a non-market good is an indicator of impact, but nevertheless, it would be difficult to match these measurements against the taxonomies of usage and impact we have constructed. The studies that we found were concerned more with measuring the public’s perception of whether or not a nonmarket good – whether a library, air quality, forests, a cathedral – was something with an acknowledged value (existence value, use value or bequest value). Whilst useful indicators of perception and relative value, and although sometimes providing useful ranking against competing non-market goods, the studies and the methods employed did not measure impact to any great extent. For example, respondents in the Surrey History Centre study (EFTEC, 2001) were asked to rate their level of personal benefit from the existence of recorded heritage and the responses were ranked from “Do not benefit at all” to “Strongly benefit”. This provides an overall impression of benefits, but does not drill down to the level of detail demanded by the taxonomy of impact, and certainly does not tell us about individual experiences. Many of the techniques are primarily aimed at policy and decision making, rather than assessing impact per se; they are generally used to make choices about courses of action and tend to be predictive as opposed to evaluative. There is a sense in which the available economic techniques seem to be facing in one direction, concentrating on the value of ‘what is being produced for consumption’, rather than on the measurement of ‘the difference the consumption makes’, so their application in the archives domain would provide information about the perceived value or worth of the archive services, the staff, the facilities etc, rather than the experiential value or difference made to the individuals or communities as a result of archive use. How the techniques reviewed here could be applied to evaluate secondary usage is another quandary, although evidence of existence value could, at a stretch, be taken to imply secondary usage. A notable exception occurs in health care, which utilises the notion of quality-adjusted life years (QALY), as part of the assessment of the consequences of particular courses of action and their likely impacts on patients. A QALY is a measure of a benefit of a medical treatment, and represents the number of years of life added by the treatment, where perfect health has a value of 1.0 and where discounts are applied for states of health that are less than perfect e.g. a year’s loss of mobility may be represented by
95
0.5, whereas a 6 month period of moderate pain followed by complete recovery may be represented by 0.85. The parallels with the archives domain could make adaptation of such a tool a powerful mechanism for assessing impact – QALYs are concerned with the effects on an individual, which, when studied over time and large enough population samples, can provide reliable, generalisable evidence. When considering the number and type of variables involved in assessing QALYs (e.g. individual medical conditions, available treatments, differing reactions to same treatments, feelings of well-being, relationship between physical and mental health), it may be possible to identify archival counterparts that would be meaningful within the taxonomies. One of the difficulties in assessing the social impact of archives upon individuals is that it is almost impossible to predict – two people can have exactly the same experience, but with vastly differing impacts, depending on a wide range of variables (the person’s expectations, prior life experiences, education, socio-economic group, cultural perspective) and an adaptation of the QALY technique could provide a methodical approach to dealing with those variables. In a similar way, techniques used in actuarial assessment could help establish what the relationships are between the public and their use of archives. For example, in medicine, the relationship between drinking alcohol and liver impairment is documented and the effects are measurable and predictable over time – perhaps it is possible to draw up ‘life tables’ that could in some way ‘predict’ the effect on individuals and/or organisations and/or wider society that could be mapped to the taxonomy of impact. 6.3.2
Economic impact
In seeking to measure and assess the social and economic impacts of a cultural service such as archives, it is inevitable that the economic impacts will be less significant, than, say a Job Centre – the objectives of an archive are inherently more social than economic. A number of studies within the ALM sector have focused on specifically economic indicators (Brand & McVittie, 2004) (Burns Owens Partnership, 2005b), however data sets are often incomplete, or unaccompanied by the required level of analysis. In terms of studies from other fields, many of the methods that we reviewed were concerned with economics, but few discussed techniques for assessing economic impact of non-market goods or services. The PSQG are at the vanguard in this respect in the archive domain, as they do try to elicit information from survey respondents as to the benefits of their visit on the local economy. However, this is assessing the economic impact that the archive user has on his environment, rather than the economic impact that the archive produces on the user. Indeed, it can be argued that the economic impact of an archive faces two ways – its existence value can create impact on the local economy by virtue of its own operational (demand-side) expenditure, and by drawing people to visit it, and its use value can have a (supply-side) economic impact at the individual or community level. Sadly, there is, as yet, no evidence that using an archive will make one rich. However, there are perhaps impacts on an individual’s employment that can be attributed to the use of archives, and there are measurements that could be made of secondary usage in support of media (TV, film, documentary) which form a significant part of the UK economy, and the negative economic impact of the lack of such archival resource. None of the techniques we reviewed dealt with this type of assessment, which could potentially provide a rich seam of evidence for archives.
96
Further work to develop a ‘rich picture’ of the economic interactions of archive services, and the sources of data to inform this, could enable the development of models to allow the economic impact of the domain to be estimated. This kind of ‘hard’ data has its appeal, and the statistical sources may already be available to some extent; the development of the model itself would however require input from economic experts, to ensure its validity. Any such model would be likely to be complex, and raises the issue highlighted by Johnson and Thomas (2000) that without similar data (based on identical methodologies) for other activities in the economy, it is difficult to draw any meaningful conclusions. This also begs the question, given the size of the archive domain, of whether the effort (and cost) required to develop such a model, would be justified by the utility of the evidence generated in relative terms.
6.4
Recommendations
The taxonomies that are proposed here are aimed at addressing, at least in part, Holden’s observation that there is a need for “a language capable of reflecting, recognising and capturing the full range of values expressed through culture” (Holden, 2004, p.9). He continues that “The DCMS have confirmed that there is no ready-made and reliable methodology in place for calculating the economic impacts of cultural institutions” and we wholeheartedly endorse that view, having found no one method or approach that can be used in an unadulterated state for the assessment of the economic and social impact of archives. The Demos report, of which Holden is the author, contains some damning criticism of efforts to date and cites Selwood (2002), who concludes that: Until the collection and analysis of data is carried out more accurately and objectively, and until the evidence gathered is used more constructively, it could be argued that much data gathering in the cultural sector has been a spurious exercise. Although “…the consequences of cultural engagement are too remote in time and space to be a matter of simple cause and effect” (Holden, 2004, p.18), we have attempted to construct understandable, logical and progressive taxonomies of usage and impact within the confines of the brief that YMLAC has set. Difficulties remain, however, in the expression of impact measurement factors. The methodologies and techniques studied have provided some ideas about valid approaches, but the language to express the qualitative and quantitative difference that archives makes to people’s lives has yet to be developed. We would recommend some further research in this area prior to a trial of the taxonomies within a small pilot study. We looked at certain subject areas as possible innovators in the field of non-market good assessment, obtaining valuable insights and finding analogies with the archives domain. However, this issue of defining measurement may benefit from further research into more sociological, educational and behavioural disciplines, where the measurement of changes in behaviour and attitude may be tackled more effectively. Time and resource constraints prevented more than the most superficial forays into this territory. In terms of adaptable methods and techniques, contingent valuation has value, but only if accompanied by rigorous research procedures and appropriate resources to conduct a large enough survey to provide credible datasets. Cost-benefit analysis (CBA) and total economic value will not, by themselves, enable assessment of economic and social impact. We would question whether the cost involved in adapting such complex methodologies to render data applicable to the evaluation of the impact of the archive
97
domain would be justified by its end-use. In comparative terms archives are small players, and it may be that the language of direct economics is not the best approach to elucidating service value. The areas upon which we would therefore recommend that YMLAC concentrate are the adaptation of the actuarial techniques and the development of a QALY-type model for archives use. Both of these require the definition of measurement factors and a language to express cultural impact that is convincing and comprehensible to the domain, to the government and to the public. The concepts of social and economic impact speak to the instrumental value of archives; if commentators such as Holden (2005) and Matty (2006) are correct, then public value will increasingly come to depend on the intrinsic value individual members of the public derive from their engagement with the domain. The taxonomies proposed here should be capable of overlaying both instrumental and intrinsic values, but the terminology for accurately measuring the impact of intrinsic social and economic value must reflect individual measures of personal enrichment. The difficulty will lie in determining how these may be categorised, given the infinite complexity and variety of human perceptions.
98
Bibliography and References AEA Consulting. (2005). Tyne & Wear museums, Bristol’s museums, galleries and archives: social impact programme assessment. London: AEA Consulting. AKA Partnership. (2003). CIPFA Statistics analysis. Retrieved January 20, 2006, from http://www.nationalarchives.gov.uk/archives/psqg/pdf/cipfa_aka.rtf Aldridge, S., Halpern, D. & Fitzpatrick, S. (2002). Social capital: a discussion paper. London: Performance and Innovation Unit. Archival Mapping Project Board. (2001). Our shared past: phase two: developing 21st century archive services. London: PRO. Archives Council Wales. (2003). Archival data collection in Wales. Retrieved February 8, 2006, from http://www.llgc.org.uk/cac/DataColl.pdf Archives and Records Council Wales. (2005). Archives Network Wales: project evaluation. Results of a user survey, July-August 2005. Retrieved February 6, 2006, from http://www.archivesnetworkwales.info/pdfs/onlineuser.pdf Arts Council England. (2004). The impact of the arts: some research evidence. London: Arts Council England. Bailey, C., Miles, S. & Stark, P. (2004). Culture-led urban regeneration and the revitalisation of identities in Newcastle, Gateshead and the North East of England. International Journal of Cultural Policy, 10(1), 47-65. Bateman, I.J. & Langford, I.H. (1997). Non-users’ willingness to pay for a national park: an application and critique of the contingent valuation method. Regional Studies, 31(6), 571-582. Bearman, D. (1989). Archival methods. (Archives and Museums Informatics Technical Report #9). Retrieved February 1, 2006, from http://www.archimuse.com/publishing/archival_methods/ Black, A. (2003). False optimism: modernity, class, and the public library in Britain in the 1960s and 1970s. Libraries and Culture, 38(3), 201-213. Bloom, M.A. et al. (2005). Budget impact of tegasored on a managed care organization formulary. American Journal of Managed Care, 11(1), Sup. S27-S34. Brand, S. & McVittie, E. (2004). The economic contribution of museums, libraries and archives in the South West: libraries & archives. Plymouth: SWMLAC in association with South West Economy Centre. British Library. (2003). Measuring our value: results of an independent economic impact study commissioned by the British Library to measure the Library’s direct and indirect value to the UK economy. Retrieved January 15, 2006, from http://www.bl.uk/pdf/measuring.pdf Brophy, P. (2005). The development of a model for assessing the level of impact of information and library services. Library and Information Research 29(93), 43-49.
99
Bryson, J., Usherwood, B., & Streatfield, D. (2002). South West Museums Archives and Libraries social impact audit. Sheffield: Centre for Public Libraries and Information in Society in association with Information Management Associates Burns Owens Partnership. (2005a). New directions in social policy: developing the evidence base for museums, libraries and archives in England. London: MLA. Burns Owens Partnership. (2005b). London’s culture equation. London: ALM London. Burns Owens Partnership. (2006). MLA: Social outcomes for museums, libraries and archives – prototype social outcomes framework. (Unpublished presentation / report). Burt, R.S. (2000). Contingent value of social capital. In E.L. Lesser (Ed.), Knowledge and social capital: foundations and applications. Boston: Butterworth Heinemann. Buttler, A.L. (1996). The results of a survey of visitor to Warwickshire County Record Office. Journal of the Society of Archivists, 17(2), 175-181. Card, S., McKinlay, J. & Roberston, G. (1991). A morphological analysis of the design space of input devices. ACM Transactions on Information Systems, 9(2), 99-122. Childs, R. (2000). Carrying out a Best Value Review in an archive service. Metropolitan Lines: Newsletter of the Greater London Archives Network, 54, 19-24. Coalter, F. (2001). Realising the potential of cultural services: making a difference to the quality of life. London: Local Government Association. Coleman, J.S. (2000). Social capital in the creation of human capital. In E.L. Lesser (Ed.), Knowledge and social capital: foundations and applications. Boston: Butterworth Heinemann. Cookman, N. and Haynes, D. (2002). Strategic review of statistical requirements: report and recommendations to Resource: The Council for Museums, Archives and Libraries. London: David Haynes Associates. Cox, R.J. & Wallace, D.A. (Eds.). (2002). Archives and the public good: accountability and records in modern society. Westport, C.T.: Quorum Books. Craig, B.L. (1987-88). Meeting the future by returning to the past: a commentary on Hugh Taylor’s transformations. Archivaria, 25, 7-11. Creaser, C. (2002). Local authority archives statistics data refining study. (Report to Resource The Council for Museums, Archives & Libraries.) Loughborough: Library & Information Statistics Unit, Loughborough University. Cultural Heritage Consortium. (2002). Impact evaluation of museums, archives and libraries: quantitative time series data identification exercise. (A report for Resource: the Council for Museums, Archives and Libraries.) London: Resource. Dalenburg, D., Fitzgerald, J.M., Shuck, E. & Wicks, J. (2004). How much is leisure worth? Direct measurement with contingent valuation. Review of Economics of the Household, 2, 351-365. Daly, S. (2005). Social capital and the cultural sector. London: Centre for Civil Society, London School of Economics.
100
DCMS / National Statistics. (2005). Taking part: the national survey of culture, leisure & sport. (Statistical release 15 December 2005). Retrieved February 28, 2006, from http://www.culture.gov.uk/global/research/taking_part_survey/survey_ouputs.htm DTLR. (2001). DTLR: multi-criteria analysis manual. Retrieved February 6, 2006, from Great Britain, Office of the Deputy Prime Minister Web site: http://www.odpm.gov.uk/ pub/252/MulticriteriaanalysismanualPDF1380Kb_id1142252.pdf Economou, M. (2002). User evaluation: report of findings. (National Council on Archives and National Archives Network User Research Group NANURG.) Retrieved February 6, 2006, from http://www.ncaonline.org.uk/materials/nanurg.pdf EFTEC: Economics for the Environment Consultancy Ltd. (2001). Valuing our recorded heritage. (Report for Resource The Council for Museums, Archives and Libraries.) London: Resource. Epp, K. (2000). Telling stories around the “electronic campfire”: the use of archives in television productions. Archivaria, 49, 53-83. Falconi, M., Salvia, R., Bassi, C., Zamboni, G., Talamni, G. & Pederzoli, P. (2001). Clinicopathological features and treatment of intraductal papillary mucinous tumour of the pancreas. British Journal of Surgery, 88(3), 376-381. Feinstein, O.N. (2002). Use of evaluations and evaluation of their use. Evaluation, 8(4), 433-439. Forbes, H. & Dunhill, R. (1997). Survey of local authority archive services: 1996 update. Journal of the Society of Archivists, 18, 37-53. Frey, B.S. (1997). The evaluation of cultural heritage: some critical issues. In M. Hutter & I. Rizzo (Eds.), Economic perspectives on cultural heritage (pp. 31-49). New York: St. Martin’s Press. Fulton, L. (2005). Archive Awareness Campaign 2004: Impact assessment and evaluation report. Retrieved February 6, 2006, from http://www.archiveawareness.com/materials/aacevaluation2004.pdf Gerard, K., Shanahan, M. & Louviere, J. (2003). Using stated preference discrete choice modelling to inform health care decision making: a pilot study of breast screening participation. Applied Economics, 35(9), 1073-1085. Griffiths, S. & Tallon, A. (2005). There be monsters: a case study. London: The National Archives. Grimard, J. & Pagé, L. (2004). Towards program evaluation in archives. Archival Science, 4, 99-126. Grubin, D. (1999). Actuarial and clinical assessment of risk in sex offenders. Journal of Interpersonal Violence, 14(3), 331-343. Hall, B. M. (1987). Social relevance and local studies libraries. Local Studies Librarian, 6(1), 3-7.
101
Harris, K. & Green, A. (1997). Social impact of public libraries. The Library Association Record, 99(4), 184. Harris, G.T., Rice, M.E. & Camilleri, J.A. (2004). Applying a forensic actuarial assessment (the Violence Risk Appraisal Guide) to nonforensic patients. Journal of Interpersonal Violence, 19(9), 1063-1074. Hill, A. (2004). Serving the invisible researcher: meeting the needs of online users. Journal of the Society of Archivists, 25(2), 139-148. Holden, J. (2004). Capturing Cultural Value. London: Demos. Holden, J. (2005). Valuing Culture in the South East. London: Demos Hooper-Greenhill, E. (2002). Learning impact research project (LIRP): developing a scheme for finding evidence of the outcomes and impact of learning in museums, archives and libraries: the conceptual framework. Leicester: Research Centre for Museums and Galleries. Hylton, T. (2004). New directions in social policy: cultural diversity for museums, libraries and archives. London: MLA. Impact evaluation programme. (2006). Retrieved January 10, 2006, from MLA: The Museums, Libraries and Archives Council Web site: http://www.mla.gov.uk/information/evidence/ev_impev.asp IPF/PSQG. (2004). Visitor’s survey 2004. Croydon: IPF Market Research Unit. Johnson, P. & Thomas, B. (2000). The economic impact of museums: a critique. Durham: University of Durham Business School. Johnson, F.R., Banzhaf, M.R. & Desvouges, W.H. (2000). Willingness to pay for respiratory and cardiovascular health: a multiple format, stated-preference approach. Health Economics, 9, 295-317. Jura Consultants. (2005). Bolton’s museum, library and archive services: an economic valuation. Bolton: Bolton Metropolitan Borough Council and MLA North West. Keeney, R. & Raiffa, H. (1976). Decisions with multiple objectives. New York: Wiley. Kenyon, J. (2003). Listening to the past, speaking to the future: the report of the Archives Task Force: Annex F: the secondary impact of archives: indicators and trends. Retrieved January 28, 2006, from http://www.mla.gov.uk/resources/assets//A/atf_annex_f_doc_5491.doc Kerslake, E. & Kinnel, M. (1997). The social impact of public libraries: a literature review. (BLRIC Report 85). Loughborough: Department of Information & Library Studies, Loughborough University. Kontoleon, A., Macrory, R. & Swanson, T. (2001). Individual preferences, expert opinion and environmental decision-making: an overview of the issues. (Paper prepared for the Symposium on Law and Economics of Environmental Policy, August 2001). Retrieved February 20, 2006 from http://www.cserge.ucl.ac.uk/kontoleon_et_al.pdf
102
Lee, G.M. et al. (2005). Health-state valuations for pertussis: methods for valuing short-term health states. Health and Quality of Life Outcomes, 3(17). Retrieved February 20, 2006, from http://www.hqlo.com/content/3/1/17 Linley, R. (2004). New directions in social policy: communities and inclusion policy for museums, libraries and archives. London: MLA. Linley, R. & Usherwood, B. (1998). New measures for the new library: a social audit of public libraries. (BLRIC Report 89). Sheffield: Centre for the Public Library In the Information Society. Loureiro, M.L. & Hine, S. (2004). Preferences and Willingness to Pay for GM Labelling Policies. Food Policy, 29, 467-483. Lowenthal, D. (1998). The heritage crusade and the spoils of history. Cambridge: Cambridge University Press. Matarasso, F. (1998). Learning development: an introduction to the social impact of public libraries. Stroud: Comedia. Matty, S. (Ed.) (2004). Overview of data in the museums, libraries and archives sector. London: MLA. Matty, S. (2006). Making the case: demonstrating the value of archives to our political masters. (Presentation to the NCA Conference, Evidence of our Value – Our Value as Evidence, 21 February). Retrieved March 6, 2005, from the National Council on Archives Web site: http://www.ncaonline.org.uk/materials/simon_matty_address.pdf McFarquhar, A. (1999). Environmental valuation, project appraisal and political consensus in the third world. Planning and Markets, 4. Retrieved February 20, 2006 from http://www-pam.usc.edu/volume4/v4i1a3print.html Mildren, R. (2004). Scottish Archive Network Project Evaluation Report. Retrieved February 4, 2006, from http://www.scan.org.uk/aboutus/report.htm Mills, J.F., Jones, M.N. & Kroner, D.G. (2005). An examination of the generalizability of the LSI-R and VRAG probability bins. Criminal Justice and Behaviour, 32, 565-585. MLA. (2004a). Listening to the past, speaking to the future: report of the Archives Task Force. London: MLA. MLA. (2004b). More about Generic Learning Outcomes. Retrieved January 8, 2006, from Inspiring Learning for All Web site: http://www.inspiringlearningforall.co.uk/uploads/More about the GLO's.doc MORI. (2002). Interest in history: research study conducted for Public Record Office. London: MORI. MORI. (2003). Listening to the past, speaking to the future: the report of the Archives Task Force: Annex D: Non-archive users survey: omnibus survey. Retrieved January 28, 2006, from http://www.mla.gov.uk/resources/assets//A/atf_annex_d_pdf_5488.pdf MORI. (2004). Interest in history: research study conducted for The National Archives. London: MORI.
103
Morton, P. (1999). The economic benefits of wilderness. The Wilderness Society. Retrieved February 10, 2006 from http://www.wilderness.org/Library/Documents/upload/Economic-Benefits-of-WildernessMorton.pdf Mourato, S., Pearce, D., Özdemiroglu, E. & Howarth, A. (2000). Beyond ‘Dusty Archives’: the economic benefits of preserving recorded heritage. Cultural Trends, 39, 87-119. National Council on Archives. (2001). Taking part: an audit of social inclusion work in archives. Sheffield: National Council on Archives. National Council on Archives. (2005). Giving Value: Funding priorities for UK archives 2005-2010. London: The National Council on Archives. Nesmith, T. (2004). What’s history got to do with it?: reconsidering the place of historical knowledge in archival work. Archivaria, 57, 1-27. Newman, T., Curtis, K. & Stephens, J. (2003). Do community-based arts projects result in social gains? A review of the literature. Community Development Journal, 38(4), 31022. Newman, A. & McLean, F. (2004). Presumption, policy and practice: the use of museums and galleries as agents of social inclusion in Great Britain. International Journal of Cultural Policy, 10(2), 167-181. Noonan, D.S. (2002). Contingent valuation studies in the arts and culture. Chicago: The Cultural Policy Centre, University of Chicago. Retrieved February 20, 2006 from: http://culturalpolicy.uchicago.edu/workingpapers/Noonan11.pdf North East Historic Environment Forum. (2005). Economic social and cultural impact assessment of heritage in the North East. London: Ove Arup & Partners Ltd. Özdemrioglu, E. & Mourato, S. (2001). Valuing our recorded heritage. Retrieved March 2, 2006 from: http://www.uni-hamburg.de/Wiss/FB/15/Sustainability/ccemDateien/CCEMpaper.pdf Patton, M.Q. (1997). Utilization-focused evaluation. Thousand Oaks, CA: Sage Publications. Pearce, D. & Özdemiroglu, E. et al. (2002). Economic valuation with stated preference techniques: summary guide. London: DTLR. Phillips, K.A., Maddala, T. & Johnson, F.R. (2002). Measuring preferences for health care interventions using conjoint analysis: an application to HIV testing. Health Services Research. Retrieved February 20, 2006 from: http://www.findarticles.com/p/articles/mi_m4149/is_6_37/ai_97177052 Phillips, C. & Thompson, G. (2001). What is cost-effectiveness? (‘What is . . .? series, 1(3)). Retrieved February 18, 2006 from http://www.evidence-basedmedicine.co.uk/ebmfiles/whatiscosteffect.pdf Pickford, C. (2002). Archives: a statistical overview. Cultural Trends, 48, 3-36.
104
PLB Consulting Ltd. (2004). Users & non-users of museums, libraries and archives: review of available data. London: MLA. PricewaterhouseCoopers LLP. (2005). Libraries Impact Project. Laser Foundation. Prieto, L. & Sacristán, J. (2004). What is the value of social values? The uselessness of assessing health-related quality of life through preference measures. BMC Medical Research Methodology, 4(10). Retrieved January 22, 2006 from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?articl=415546 PSQG. (1999a). Survey of visitors to British archives – October 1999: national results. Retrieved January 20, 2006 from http://www.nationalarchives.gov.uk/archives/psqg/pdf/nationalresults1999.pdf PSQG. (1999b). National survey of visitors to British archives 1999: Public Services Quality Group commentary. Retrieved January 20, 2006 from http://www.nationalarchives.gov.uk/archives/psqg/pdf/summary1999.pdf PSQG. (2001a). Survey of visitors to British archives – February 2001: national report. Retrieved January 20, 2006 from http://www.nationalarchives.gov.uk/archives/psqg/pdf/nationalresults2001.pdf PSQG. (2001b). National survey of visitors to British archives – February 2001: key findings. Retrieved January 20, 2006 from http://www.nationalarchives.gov.uk/archives/psqg/pdf/keyfindings2001.pdf PSQG. (2002). Survey of visitors to British archives 2002 – national report. Retrieved January 20, 2006, from http://www.nationalarchives.gov.uk/archives/psqg/pdf/nationalresults2002.pdf Putnam, R.D. (2000). Bowling alone: the collapse and revival of American community. New York: Simon and Schuster. Rallis, S.F. & Bolland, K.A. (2004). What is program evaluation? Generating knowledge for improvement. Archival Science, 4, 5-16. Ramsden, C., Kelleher, J. & Russell, S. (2002). Formative evaluation of the Archives Hub. London: The Tavistock Institute. Ramsden, C. Barkat, S. & Kelleher, J. (2003). Summative evaluation of the Archives Hub. London: The Tavistock Institute. Ray, L. (2006). Capturing the public value of heritage. Recordkeeping, 7, 15-16. Ready, R. & Navrud, S. (2005). Benefit-transfer – the quick, the dirty and the ugly? Choices, 20(3), 195-199. Ritchey, Tom. (1998). General morphological analysis: a general method for nonquantified modelling. (Adapted from the paper "Fritz Zwicky, Morphologie and Policy Analysis" presented at the 16th EURO Conference on Operational Analysis, Brussels, 1998). Retrieved March 12, 2006 from http://www.swemorph.com/ma.html Roberts, K. (2004). Leisure inequalities, class divisions and social exclusion in presentday Britain. Cultural Trends, 13(2), 1-15.
105
Rushton, M. (2004). Contingent valuation and the public interest in privately owned cultural property. The Journal of Arts Management, Law, and Society, 34(3), 222-234. Ryan, M. & Hughes, J. (1997). Using conjoint analysis to assess women’s preference for miscarriage management. Health Economics, 6, 261-273. Sabin, R.W. & Samuels, L. (2003). Listening to the past, speaking to the future: the report of the Archives Task Force: Annex E: Towards a better understanding of nonusers. Retrieved January 28, 2006, from http://www.mla.gov.uk/resources/assets//A/atf_annex_e_pdf_5490.pdf Scott, C.A. (2003). Museums and impact. Curator: The Museums Journal, 46(3), 293310. Scott, C. (2004). Museums and impact: some issues for museums to consider. Retrieved January 28, 2006, from http://www.fuel4arts.com Selwood, S. (2002). The politics of data collection: gathering, analysing and using data about the subsidised cultural sector in England. Cultural Trends, 47, 13-84. Silverman, L. (1995). Visitor meaning making for a new age. Curator: The Museums Journal, 38(3), 161-170. Smith, B. (1986). Record repositories in 1984. Journal of the Society of Archivists, 8, 116. Snowball, J.D., & Antrobus, G.G. (2002). Valuing the arts: pitfalls in economic impact studies of arts festivals. The South African Journal of Economics, 70(8), 1297-1319. Sommerlad, E. & Kellerher, J. (2000). An evaluation of the HE Archives Hub. London: The Tavistock Institute. Starkey, G. (2004). Estimating audiences: sampling in television and radio audience research. Cultural Trends, 49, 3-25. Stevens, T.H. (2005). Can stated preference valuations help improve environmental decision making? Choices, 20(3), 189-194. Stirling, A. (1997). Limits to the value of external costs. Energy Policy, 25, 517-540. The National Trust. (2005). Socio-economic impact assessment of the project to restore & open to the public the Llanerchaeron Estate. The National Trust. Thompson, E. (1998). Contingent valuation in arts impact studies. The Journal of Arts Management, Law, and Society, 28, 206-211. Throsby, D. (2003). Determining the value of cultural goods: how much (or how little) does contingent valuation tell us? Journal of Cultural Economics, 27(3-4), 275-285. Trowbridge, C.L. (1989). Fundamental concepts of actuarial science. (Revised ed.) Actuarial Education and Research Fund. Retrieved March 10, 2006 from http://www.actuarialfoundation.org/research_edu/fundamental.pdf Trueman, P., Drummond, M. & Hutton, J. (2001). Developing guidance for budget impact analysis. Pharmacoeconomics, 19(6), 609-621.
106
Tucker, S. (2005). Corresponding identities: archivists and family historians. ARC, 190, 14. Urquhart, C. (2005). Assessing impact – let us count the ways. Update, 4(12), 26-28. Usherwood, B. (1999). The social impact of local studies services. Local Studies Librarian, 18(1), 2-6. Van Oostenbruggen, M.F. et al. (2005). Penny and pound wise: pharmocoeconomics from a governmental perspective. Pharmacoeconomics, 23(3), 219-226. Warburton, M. (2005). Playing a part in neighbourhood renewal. London: ALM London. Wavell, C., Baxter, G., Johnson, I. & Williams, D. (2002). Impact evaluation of museums, archives and libraries: available evidence project. (Report for Re:source). Aberdeen: Aberdeen Business School, The Robert Gordon University. Weisen, M. (2004). New directions in social policy: health policy for museums, libraries and archives. London: MLA. Weiss, C.H. (1999). The interface between evaluation and public policy. Evaluation, 5(4), 468-486. White, P. (1993). New approaches to customer care. Archives, 20, 48-50. Williams, C. (2003). Data collection and management in the archival domain. Journal of the Society of Archivists, 24(1), 65-81. Williams, C. & Procter, M. (2002). Snapshot diagnostic of data flow in the archival domain. Liverpool: LUCAS. Williams, D.A., Wavell, C., Baxter, G., MacLennan, A. & Jobson, D. (2005). Implementing impact evaluation in professional practice: A study of support needs within the museum, archive and library sector. International Journal of Information Management, 25(6), 533-548. Williams, D., Baxter, G., Wavell, C., & MacLennan, A. (2003). Impact evaluation pilot web portal: final report. Aberdeen: Aberdeen Business School, The Robert Gordon University. Willis, K.G. et al. (2003). The social and environmental benefits of forests in Great Britain – Phase 2. (Report to Forestry Commission, Edinburgh). Retrieved March 8, 2006 from http://www.forestry.gov.uk/pdf/sebreport0703.pdf/$file/sebreport0703.pdf Yorkshire Museums Libraries & Archives Council. (2005). Scoping the economic and social impact of archives. (Brief and specification for consultancy and contracts for services.) Leeds: YMLAC. Zwicky, F. (1969). Discovery, invention, research through the morphological approach. New York: Macmillan.
107
