Mar 4, 2016 - Billboards were placed around Milwaukee, which received over a million impressions, half of which landed w
Ask-the-Expert Session 8: Summary Discussion Notes Implementing, Evaluating and Improving Your Communication Campaign Dr. Shawnika Hull March 4, 2016
Expert Dr. Shawnika Hull is an Assistant Professor in the Department of Prevention and Community Health. Dr. Hull's research focuses on the processes through which mediated messages affect social and individual determinants of health. In particular, her work explores the ways in which mass media can be used to eliminate disparities in health and wellness, especially in the context of HIV prevention.
Introduction • Using theory is important to drive program evaluation. When your program is based on theory, you will be able to articulate how you expect intervention activities to influence outcomes and what outcomes should/can be measured to demonstrate success. • Using theory sets realistic expectations, including the magnitude of impact expected from the intervention, how fast we can expect change to happen and whether or not we can expect the intervention to be resistant to counter-efforts (for example, the tobacco industry counteracting tobacco control efforts). Case study about Acceptance Journeys: Dr. Hull and her team worked on a communication campaign in Milwaukee to create a culture of acceptance of homosexuality, as the presence of homophobia in a community is a risk factor for HIV/AIDS, particularly for young gay men of color. She and her team created a logic model stating the expected outcomes from the campaign: changes in community-level acceptance that translates to individual-level acceptance, changes in stereotypes and changes in attitudes toward gay people. Billboards were placed around Milwaukee, which received over a million impressions, half of which landed with the intended audience. The ideal method of evaluating this campaign would have been to conduct a longitudinal study, where changes in acceptance were tracked over a long period of time; however, this is time and resource-intensive. Instead, Dr. Hull and her team evaluated changes among randomly selected individuals over five years in Milwaukee, where the campaign was implemented, as well as Cleveland and St. Louis, which are demographically similar to Milwaukee but where the campaign billboards were not posted. The team built the case that the campaign had successful outcomes by triangulating the following: 1) showing the campaign was targeted correctly geographically, as people in #TAPaskexpert
1
Milwaukee reported seeing the campaign more than those in Cleveland and St. Louis, 2) showing that the campaign was targeted correctly demographically, as Black people in Milwaukee reported seeing the campaign more than other racial/ethnic groups in Milwaukee; and 3) showing that those who had seen the campaign were more likely to be accepting of homosexuality than those who had not seen the campaign. Read more about this study in an article written by Dr. Hull and her colleagues in Social Marketing Quarterly: Using Theory to Inform Practice: The Role of Formative Research in the Construction and Implementation of the Acceptance Journeys Social Marketing Campaign to Reduce Homophobia. Collect and monitor data during communication campaigns • Audience question: How do we make communication programs sustainable? o Expert answer: One of the most important ways to make communication programs sustainable is incorporating your audience into the campaign planning, implementation and evaluation activities. For example, as part of a communication program I currently work on promoting Pre-Exposure Prophylaxis (PrEP) for HIV prevention to young gay men of color, we convene a community advisory board meeting biweekly. The research idea was generated from needs identified by the community and all ideas we pursue emerge from the community advisory board. • Audience question: If I spend $500 on buying a radio spot or billboard ad, how should I go about measuring indicators of success? o Expert answer: One of the measures of exposure we used for our HIV campaign was impressions. Impressions are the number of times people saw your campaign. You may also look at targeted impressions, which are the number of times your target audience members saw your campaign. A simpler way to assess exposure is aided recall, which is when you show someone an image from your campaign and ask if they have seen it. You can relate those results to outcomes you may be interested in, such as a dose-response relationship between the exposure to your campaign and the outcomes: are there better outcomes among people who report seeing the campaign? • Audience question: How should we collect the data? o Expert answer: If staff capacity is limited, there are professional evaluation services available. We used Qualtrics, a service that can post your survey and provide you with a sample representative of your intended audience. GFK is another survey company with pre-determined samples that can make a study more robust. Identify limitations of data collected • Going back to the case study, one limitation of campaign evaluation was capturing the effects of secondary media coverage the campaign received during implementation. Journalists wrote op-eds and radio stations and local media covered the campaign, and there was little the research team could do to capture their effects. One way the research tem measured the effects of secondary media coverage was to ask participants that saw the campaign where they saw the campaign and whether they discussed the campaign with others and in what manner (whether it was a positive, negative or neutral discussion). Dr. Hull also reassured participants and echoed Robert Hornik, a leading authority in public health communication, to “respect the process of the campaign, as real campaigns and programs that work are messy.” • Audience question: I'm having trouble connecting the campaign to health outcomes and impact. How should we do this? o Expert answer: This is an age-old question about health communication. Keep in mind that the likelihood you can isolate the effects of your campaign directly to health outcomes is very small. Your best bet is to measure the intermediate outcomes of your campaign, such as changes in
#TAPaskexpert
2
knowledge or confidence. This is why it is important to have those intermediate outcomes delineated in your program strategy or logic model and based on theory. Use evaluation results to improve the communication campaign • Audience question: How might we use evaluation results to improve campaigns and promote sustainability? o Expert answer: You have to ask yourself: Do you have the resources and capacity to keep this program going? Does the program have the ownership and commitment from community members? Does the community value and want to keep this program going? If sustainability of the program is of interest, it is crucial to develop the program so that the answers to those questions are “yes,” by the end of the program. You have to also assess the components of the program that are working or not working and why. Discovering the reasons why some components are not working are crucial to adjust the program and ultimately increase community buy-in. Similarly, discovering reasons why some components are working are crucial to recreating those successes. • Audience question: Can we adapt Acceptance Journeys, the communication program you mentioned in your case study, to our city? o Expert answer: Yes and no. We had this question when Pittsburg approached us about sharing our campaign materials for them to implement in their city. We had to be mindful that our campaign was designed for a specific audience in Milwaukee and used photographs of real community members, and no communication campaign will transfer seamlessly to other audiences. So, rather than give them our campaign materials, we offered to share the process with them: how to convene community members, get their input, create communication materials, implement the campaign responsive to the community’s needs, measure outcomes and interpret them.
Expert and participant-recommended resources: • CDC’s Applying Theory in the Evaluation of Communication Campaigns • CDC’s Gateway to Health Communication & Social Marketing Practice: Research & Evaluation • CDC’s Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide • CDC’s Program Evaluation Steps • National Colorectal Cancer Roundtable’s Evaluation Toolkit: How to Evaluate Activities Intended to Increase Awareness and Use of Colorectal Cancer Screening • Other tools and resources from GW Cancer Institute o Communication Training for Comprehensive Cancer Control (CCC) Professionals 101: Media Planning and Media Relations: This free online training walks participants through the process of media planning, creating media-ready materials and building relationships with journalists. It includes background information, resources and customizable templates, so after completing the training, you will have a tailored media plan and media-ready materials for your cancer control program to use. Sign up for our eNewsletter to receive notice when Communication Training 102: Making Communication Campaigns Evidence-Based is released. o Media Planning and Media Relations Guide o Media/Communication Plan Template
#TAPaskexpert
3
