Evaluation of the Braddock Youth Project

0 downloads 174 Views 784KB Size Report
The evaluation required the collection, analysis and reporting of information about ..... Quantitative data analysis was
Evaluation of the Braddock Youth Project

Evaluation of the Braddock Youth Project Final Report Submitted to KEYS Service Corps

Gibbs Y. Kanyongo, PhD Duquesne University

1|Page

Evaluation of the Braddock Youth Project

Evaluation of the Braddock Youth Project

Gibbs Y. Kanyongo, PhD

August, 2011

2|Page

Evaluation of the Braddock Youth Project

Table of Contents Table of Contents ......................................................................................................................................... 3 Acknowledgments ....................................................................................................................................... 5 Executive Summary.................................................................................................................................... 6 Introduction............................................................................................................................................... 6 Evaluation Objectives ............................................................................................................................. 6 Methodology for the Evaluation ......................................................................................................... 6 Key Findings .............................................................................................................................................. 7 Summary..................................................................................................................................................... 7

Background of the Evaluation .......................................................................................................... 8 Introduction............................................................................................................................................... 8 Need for Evaluation of BYP .................................................................................................................. 8

Youth Programs: A Literature Review .....................................................................................10 Evaluation Models .................................................................................................................................10 Summary of the Literature Review ..................................................................................................12

Evaluation Methods .................................................................................................................................12 Evaluation Design and Implementation ........................................................................................12 Instrumentation ........................................................................................................................................... 12 Survey Design Construction .................................................................................................13 Implementation of the Evaluation Design ....................................................................................14 Data Analysis…………………………………………………………………………………………………………. 14 Potential Limitations of Evaluation Design and Implementation ........................................14

Evaluation Findings .................................................................................................................................15 Gardening .................................................................................................................................................15 Health Practice ……………………………………………………………………………………………………... 16 Media .........................................................................................................................................................17 BYP Junior ....................................................................................................................................................... 17 Service Learning ........................................................................................................................................... 18 Summary of Evaluation Findings.......................................................................................................... 19

Recommendations and Suggestions .........................................................................................19 Building Evaluation into BYP ................................................................................................................. 19 Recommendations Specific to BYP ....................................................................................................... 20 3|Page

Evaluation of the Braddock Youth Project

Stakeholder Review and Utilization of Findings ...........................................................21 References ......................................................................................................................................................22 Appendices ......................................................................................................................................................23

4|Page

Evaluation of the Braddock Youth Project

Acknowledgments I acknowledge and thank Ms. Helen Wachter, the Director of KEYS Service Corps and all the staff who answered every question I had, and afforded me the opportunity to meet with the staff and youths at Braddock Youth Project. Further, I would like to thank Ms. Andrea Arrington and Ms. Jessica Schmid, the Braddock Youth Project Coordinators and all their staff. I would also like to thank all the team leaders who assisted with the administration of the surveys. Finally, I would like to thank all the unnamed youths who accepted the invitation to complete the surveys and answered my questions that yielded a wealth of information for this project.

5|Page

Evaluation of the Braddock Youth Project

Executive Summary Introduction The Braddock Youth Project (BYP) is a KEYS Services Corps summer and after-school program for youths, which is housed in the Office of Community Services under Allegheny County’s Department of Human Services. The program is now in its sixth year, and it provides Braddock area youths with summer and after-school jobs and opportunities to positively impact their community. BYP provides work experience for youths aged 14-18 years that focuses on community development, service learning, video production/media literacy, community event planning and gardening. Over the last six years, the number of youths participating in the BYP summer program ranged from 38 youths (in 2006) to 92 (in 2009). This year, there are a total of 64 youths participating in the program. Evaluation Objectives The evaluation required the collection, analysis and reporting of information about the effectiveness and efficiency of the Braddock Youth Project. Specifically, the evaluation project required collection of data that: Evaluates the effectiveness of BYP in relation to:  the youths’ knowledge of gardening activities  the youths’ knowledge of good health practices  the youths’ skills and knowledge in media production  the youths’ BYP junior activities  the youths’ service learning activities. Methodology for the Evaluation To address these objectives, a comprehensive data gathering process was undertaken that involved both qualitative and quantitative methodologies to provide representative data from all key stakeholder groups. Quantitative data were collected using a survey instrument that was administered prior to participation in the program and after the youths participated in the program. A comprehensive review of the literature and documents related to youth programs helped to refine the data collection instruments. Also, staff at BYP helped validate the instruments by suggesting changes to the wording of the questions to make sure they were age-appropriate as well as content-specific. The instrument yielded fairly reliable pre-survey data as measured by Cronbach’s alpha (. The gardening subscale consisted of 6 items ( .91), the health practices subscale consisted of 7 items ( .87), the media subscale consisted of 4 items (.72), the BYP Junior subscale consisted of 4 items (81), and service learning consisted of 4 items (= .68).For the post-survey data, the service learning subscale ( .43) had the lowest reliability while the health practices subscale (=.91) had the highest reliability.

6|Page

Evaluation of the Braddock Youth Project

Qualitative data collection methodology included:  Review of current and previous program documentation, including annual reports and papers related to BYP.  Individual interviews with BYP youths, interns and staff  Observations Key Findings A total of 25 specific activities offered at the BYP summer program in the areas of gardening, health practices, media, BYP Junior, and service learning were evaluated in this project. Of these, 19 activities (76% of them) showed a significant increase in the youths’ knowledge/skills from the beginning of the program to the end as measured by the preand post-surveys. Of the19 activities, 14 activities showed a medium to large effect size change, indicating that the BYP summer program had a medium to large positive impact on the youths’ knowledge and skill on these activities. These quantitative findings were supported by comments from the youths: “I learned that giving up on my community was not going to make it any better…If it wasn’t for us Braddock would still be the same beat down and cold ghost town…” “I learned how to help others in the community and help out the little children”. “I learned that the community can be better if you help fix it up and make it a better place.” Summary Despite the fact that this evaluation was conducted in a compressed time frame, the evaluation managed to uncover the positive impact that the BYP summer program is making to the youths of Braddock. One of the strongest aspects of the BYP is the fact that the youths have a strong sense of ownership to what they are doing. They genuinely feel like they have to improve their community for the better. This was reflected in comments by the youths and in observation of the youths engaged in different activities during the evaluation period. Another strong aspect of the project is the model they use that is based on the Boston Food Project’s Standards & Straight Talk. This model encourages personal accountability, and the youths seem to have responded very well to this. Despite all its successes, there are certain things that need improvement. One of the recommendations is to make evaluation an integral part of the project where the stakeholders decide up front what they want to evaluate and how to do it. An effective evaluation requires a lot of time to prepare upfront.

7|Page

Evaluation of the Braddock Youth Project

Background of the Evaluation Introduction This report presents the findings and conclusions of an evaluation of Braddock Youth Project, an urban youth summer and after-school program. The evaluation project was commissioned by KEYS Service Corps, an Americorps Program which is part of the Office of Community Services of Allegheny County’s Department of Human Services. Allegheny County is where the city of Braddock is located. KEYS’ mission is to improve the lives of youth educationally and socially, help AmeriCorps members grow professionally, and improve communities in which they serve. The current evaluation project was conducted between mid-June and end of August, 2011. To address youths with critical needs, KEYS developed in-house youth programs in some of the most impoverished areas in the greater Pittsburgh region. One such program is the Braddock Youth Project. According to KEYS, Braddock has the highest number of families with youths living in poverty in Allegheny County. As a result of this, KEYS developed the Braddock Youth Project to provide academic assistance, hands-on service experience, summer employment and mentoring to youths in Braddock. The Braddock Youth Project uses a model that centers on the notion that youths can be agents of positive change in their communities. The program was designed with the belief that young people can change their communities for the better. They challenge youths to identify needs in their community, then create and implement solutions to address them. Some of the specific activities for BYP are community development, service learning, video production/media literacy, community event planning and gardening. The Braddock Youth Project started in 2006, and the number of youths participating has fluctuated between 38 and 92. The table below shows the number of youths who have participated in each of the past six years. Table 1: Number of youths participating in BYP over the past six years Year Number of youths participating 2006 38 2007 49 2008 64 2009 92 2010 64 2011 64 Source: Data obtained from BYP Coordinator’s annual reports. The Need for Evaluation of BYP The effort to evaluate an urban youth program is not a new phenomenon. There have been notable examples of urban youth programs in cities across the United States, and almost all of them engage in some form of program evaluation to measure the effectiveness of the 8|Page

Evaluation of the Braddock Youth Project

programs they offer. Not only does program evaluation provide important data for inhouse decision-making purposes, but it also helps provide justification to stakeholders for the resources that are committed to the program. Over the past decade, structured programming for children and youth during the nonschool hours has expanded exponentially. A confluence of recent research studies and program evaluations backs the publicly perceived notion that after-school programs can positively influence important developmental and learning outcomes. The rapid expansion of the field and the potential of programs to contribute to child and youth development have made defining what high quality programs look like and learning how to improve program quality key challenges facing the field. Yohalem and Wilson-Ahlstrom (2010) in their paper, “Assessing and Improving Quality in Youth Programs”, describe what is known about the relation between youth program quality and youth developmental outcomes. They summarize different quality assessment tools being used in the field, and discuss how such tools are being used to drive systemic quality improvement efforts. They suggest that individuals can continue to investigate the relation between specific program features and youth outcomes, and practitioners can build data-driven continuous improvement systems designed to ensure the delivery of high quality programming. With regard to BYP, they run several summer programs for youths and the objective of this evaluation is to investigate the relation between the different program features and the stated program outcomes. According to its website, the BYP is a summer and after-school program for youths that is housed under the Allegheny County Department of Human Services, in the Office of Community Services, KEYS Service Corps AmeriCorps Program. BYP provides Braddock area youths with summer and after-school jobs and the opportunity to positively impact their community. In response to the need for BYP to accurately measure the impact of its summer programs, an external evaluation was conducted between mid-June 2011 and end of August 2011 for which the findings are presented in this report. Pursuant discussions with the Director of KEYS Service Corps, the Coordinator for BYP, the evaluation process began with a series of meetings with BYP staff, teens and team leaders leading to the design of the survey instruments. The instruments were targeted to measure different outcomes of the BYP program features. In general terms, there are two main reasons for conducting evaluation: first, to review what is being done in implementing a program (a process evaluation), and second, to identify and measure the program’s results (an outcome evaluation). The current evaluation was focused on outcome evaluation in answering the following evaluation question: 1. What is the impact of participation in the Braddock Youth Summer Program on? a. The youths’ gardening skills and knowledge b. The youth’ health practices knowledge c. The youths’ media skills d. Service learning 2. What is the impact of participation in the BYP junior program on? 9|Page

Evaluation of the Braddock Youth Project

a. b. c. d.

Mentoring children Creating and delivering lesson plans for youth activities Showing children positive ways to interact with each other Teach children concrete ways to help the Braddock community

Youth Programs: A Literature Review Larson and Walker (2010) conducted a study investigating the dilemmas faced by leaders of youth programs. They followed leaders in 12 youth programs over a 2–9 month period, which led to the identification of 250 dilemma situations To create and sustain high quality youth development programs, it is important to understand the challenging situations and dilemmas that emerge in program leaders’ daily work with youths. Qualitative analyses identified 5 categories and 12 subcategories of dilemmas that reflected distinct types of considerations (e.g., youth’s personalities, relationships with the community). The analyses also found that the experienced leaders in the study typically responded to these dilemmas in ways that were youth centered and that balanced multiple considerations. It is argued that researchers need to go beyond identifying features of high quality programs, and more fully examine how effective leaders create and sustain high quality in response to the challenging situations of practice. Lee, Borden, Serido, and Perkins (2009) examined perceptions that young people hold regarding their participation in community-based youth programs. Specifically, the study assessed young people’s sense of psychological safety, their relationships with adult staff, their learning of social skills, and how different ethnic groups experience these factors. Data for the study came from a national evaluation study of youth programs. The results indicate that African American youth are more likely to feel psychologically safe and to have positive relationships with adult staff compared to the other youth. In addition, with the exception of African American youth, the associations between feeling safe and having positive relationships with adult staff and the perception of learning social skills were significantly related. Sheehan and Verner (2009) examined youth programs in Chicago. They suggested that youth programs have shown promise in addressing the “new morbidities” of children (e.g., obesity, substance abuse, violence, etc.). More specifically, they examined the Chicago youth programs, which supports the needs of a child’s development. Discussion in the article includes the history of the Chicago youth programs (developed by medical students), the evolution of a youth program model, factors analyzed for program evaluation, and methods of funding and governance. Because Chicago Youth Programs follows a medical model, the key goal of the programs is to improve the health of children. They recommended that further research needs to be completed on the long-term effects and outcomes of the Chicago youth programs. Evaluation Models Building upon existing knowledge about using and recommending participatory evaluation, Cousins and Earl (1992) emphasized the importance of participatory evaluation as a 10 | P a g e

Evaluation of the Braddock Youth Project

powerful approach to improving organizations and enhancing learning. The authors cited a single case study method by Dawson and D’Amico (1985) that examined user participation in the evaluation of a secondary school development program over a two-and-a-half year period. This study involved program staff in evaluation studies. The decision-oriented approach was the approach of choice for Frisbie (1991). He noted that evaluators should use constructivist, qualitative and responsive evaluation methods when students, parents, business leaders and other community members are involved. The key theme in this article was that evaluation theory and practice must continue to evolve if it is to continue to play a role in educational reform efforts. A variety of different evaluation typology is represented in the literature. These mostly fall into the following categories: results model, goal-oriented approach and logic model. The results model is focused on the results of a given program or organization. The goaloriented approach describes whether or not students have met their goals, with the results informing future decisions (i.e., revise, adopt, reject). The most commonly used, the logic model, is an organized way to present and share the understanding of the relationships among the different aspects of the program, the planned activities, and the anticipated results. A simplified logic model for the evaluation of BYP is presented below. Goal The goal of the Braddock Youth Project is to focus the energy and imagination of 14-18 year old youth who live, work and socialize in Braddock to restore the Braddock community Target population Youths 14-18 years old in the Braddock area 97% of the youth live in families with incomes at 235% below poverty level Majority are African American

Theory Based on the theoretical framework that youths can be agents of positive change in their communities Uses the Boston Food Project’s Standards & Straight Talk

Activities Gardening Service learning Video production Mentoring children Helping seniors to be active

Outcomes Improve the youth’s gardening skills Improve the youth’s knowledge of healthy practices Improve the youths’ video production skills and interviewing skills

Context Positive and supportive environment Safe place Giving and receiving feedback every week

11 | P a g e

Figure 1. Simplified logic model for the evaluation of BYP. Adapted from Dr. Espiritu.

Evaluation of the Braddock Youth Project

Summary of the Literature Review The scope of this literature review covers the studies related to urban youth programs and the evaluation of such programs. Based upon the review, it is evident that many items need to be successfully addressed in order to effectively evaluate youth programs. The information provided in the literature review helped to identify key problems and successful methods that will be useful in the design and implementation of the evaluation.

Evaluation Methods Evaluation Design and Implementation According to Hansen (2005), design should be determined by the purpose of the evaluation, the objective of evaluation or the problem to be solved by the evaluated program or agency. Furthermore, in practice other logics may influence the evaluation design processes. To further assist in the development of the evaluation design, surveys and interview protocols, the external evaluator reviewed documents related to BYP provided by a KEYS administrator and BYP staff. A list of the materials provided is attached in Appendix D. The BYP Evaluation Plan: Design Summary Table (Appendix A) displays the overarching questions of the evaluation. The Design Summary Table identifies more specifically the information collected, how it was collected, and how it was analyzed in order to provide the details necessary to guide the evaluation. For each of the questions, a combination of surveys and document reviews from various stakeholder groups were identified as the appropriate data collection tools/strategies for answering the evaluation questions. Instrumentation The construction of the survey and interview protocol followed the evaluation design summary table development. The first step included the identification of the main objectives/outcomes of each program, some of which are as follows: Gardening  Youths should know how to: cultivate raised beds, care for fruits, permaculture and organic gardening, grow seedlings in greenhouse and transplant, harvest vegetables, and cultivate an empty lot. Media  Youths should know how to: operate video cameras; work on a Mac computer, interview people. Health practices  Youths should create and deliver health-related lesson plans, teach children health information in ways they can understand, help seniors to be active. BYP Junior  Students are given the opportunity to: mentor children, create and deliver lesson plans for youth activities, show children positive ways to interact with each other, teach children concrete ways to help the Braddock community. 12 | P a g e

Evaluation of the Braddock Youth Project

Service learning  Youths should be able to: help seniors in their community, design and carry out service activities in their community, and change their community for the better. With the outcomes identified, this helped inform the survey construction described in the section below. Survey Construction The questionnaire was constructed to target each of the five components of BYP. Specifically, the items were aligned with the specific outcomes of each program. The wording of the items was designed to be at the reading level of the 14-18 year old youths. Even though it was important for the evaluator to remain objective, it was also important for people who were involved with the program to be involved with the evaluation— especially thinking through what questions need to be answered, what data will be collected, and how. To ensure this, items were shared with the staff at BYP to provide feedback on the wording and suitability of the items. Some items were deleted, or reworded following their input. This process helped improve the overall content validity of the instrument, and to ensure that they consistently measure the same constructs. For the gardening and media activities, the youths were asked to rate their skill level or knowledge on a scale of 0 to 10, with 0 being ‘no skill/no knowledge’ and 10 being ‘excellent skill/very knowledgeable’. For health practices and BYP junior, the youths rated their knowledge on a scale of 0 to 10, with 0 representing ‘no knowledge’ and 10 representing ‘very knowledgeable’. The service learning component was measured on a 1 to 5 scale, with 1 representing ‘not at all’ and 5 representing ‘very much’. Quantitative data collected by these instruments were fairly reliable as indicated by the Cronbach’s alpha values shown in the table below. The one exception was the post-survey data for the service leaning subscale, with a Cronbach’s alpha of .43. Table 2. Cronbach’s alpha values for each subscale Subscale Gardening Media Health Practices BYP Junior Service Learning

13 | P a g e

Cronbach’s alpha ( Pre-Survey Post-Survey .91 .67 .72 .65 .87 .91 .81 .74 .68 .43

Evaluation of the Braddock Youth Project

Implementation of the Evaluation Design The evaluation utilized a combination of quasi-experimental and survey research designs in which a pre-survey and post-survey were administered. The youths completed the presurvey at the beginning of the summer program, and then the post-survey at the end of the program. A total of 53 youths completed the pre-survey and 50 youths completed the postsurvey. In terms of demographics, over 90% of the youths were African American and about 67% of them were females. Data Analysis Quantitative data analysis was performed using the statistical software IBM SPSS version 19.0. For qualitative data, document review and interview transcript review were performed to identify common themes that emerged. Potential Limitations of Evaluation Design and Implementation There were potential limitations in the evaluation design and implementation as described below: 







Time frame. This evaluation was conducted over a period of two and a half months, from mid-June to end of August, 2011. This truncated time frame posed a number of challenges. The first challenge was that not enough time was available on the front end of the evaluation to lay the ground work for the evaluation. The second challenge was that due to the busy daily schedule of the youths at the BYP, it was not possible to schedule focus groups or individual interviews. With enough time for upfront planning, it would have been possible to build in these interviews at the beginning so that they became part of the whole BYP. This point brings in the next limitation. Lack of a comparison group. This evaluation relied on a single group pre-test, posttest design with no equal group for comparison. The lack of a comparison group makes this design susceptible to certain threats to internal validity. Lack of focus group interviews. Although qualitative data were obtained using document reviews and observations, structured focus groups would have yielded even stronger data to triangulate the quantitative data obtained through surveys. Reliance on survey data. Each time participants respond to surveys, it is possible that the effect of ‘social desirability’ exists. This evaluation used the youths’ responses to the pre- and post-surveys as the primary source of data. Again, due to the truncated time frame, it was not possible to consider other measurement tools like tests or performance assessment instruments.

Despite these potential limitations, this evaluation successfully yielded important findings that would be useful to the stakeholders. 14 | P a g e

Evaluation of the Braddock Youth Project

Evaluation Findings This section discusses the findings of the evaluation. These findings are intended to answer the following evaluation question: 1. What is the impact of participation in the Braddock Youth Summer Program on? a. The youths’ gardening skills and knowledge b. The youth’ health practices knowledge c. The youths’ media skills and knowledge d. Service learning 2. What is the impact of participation in the BYP junior program on? a. Mentoring children b. Creating and delivering lesson plans for youth activities c. Showing children positive ways to interact with each other d. Teaching children concrete ways to help the Braddock community To answer these evaluation questions, the findings are organized according to each program using tables. Specific activities for each program were measured, results of which are presented in the tables. Each table shows the pre- and post-survey mean values and the corresponding standard deviations, the mean difference between the pre and post, the tvalue, the 95% confidence interval and the effect size. While all the numbers in the table are important, for practical purposes it is important to focus on the mean difference and the effect size values. An important thing to keep in mind in interpreting these results is that for gardening, media, health practices, and BYP junior, the data were collected on a scale of 0 to 10. For service learning, the data were collected on a 1 to 5 point scale. Gardening A total of 15 youths completed the pre- and post-surveys for gardening. On the first item, ‘cultivate raised beds’ in Table 3 below, the youths’ knowledge increased 4.27 points from the beginning of the program to the end of the program. Looking at the effect size of 1.27, the youths’ gain in knowledge on this item was more than 1.0 standard deviation. In other words, by participating in the summer program, the youths gained knowledge on the cultivation of raised beds by more than one standard deviation. From the table, it is clear that students gained knowledge in all but one activity in gardening. The only activity on which the youths did not show significant gain was ‘cultivating an empty lot’. Looking at the pre- and post-survey mean values for this item, it is clear that the youths came into the program already having high knowledge on cultivating empty lots (pre-survey mean = 8.53). At the end of the program, their knowledge did not increase much beyond what they already knew coming in regarding cultivating empty lots (post-survey mean = 8.80).

15 | P a g e

Evaluation of the Braddock Youth Project

Table 3. Results for Gardening Mean (SD)

Cultivate raised beds Care for fruit trees Permaculture/organic gardening Grow seedlings in greenhouse and transplant Harvest vegetables Cultivate an empty lot

Mean difference

t-value

Pre-Survey 4.27 (4.25) 4.20(3.36) 4.80(3.88)

Post-Survey 8.45 (1.64) 6.47(2.64) 8.93 (1.49)

4.27* 2.82* 4.13*

4.14 3.06 4.33

95% CI of mean difference 2.05 – 6.48 0.68 – 3.85 2.08 – 6.18

Effect size

5.93(4.06)

9.13(1.19)

3.20*

3.34

1.15 – 5.25

1.14

6.53(3.56) 8.53(1.66)

9.93(0.26) 8.80(1.64)

3.40* 0.27

3.83 0.65

1.50 – 5.30 -0.61 – 1.14

1.80 0.17

1.27 0.82 1.46

n = 15; *Significant at .05 level

Health Practices For health practices, a total of 14 youths completed both the pre- and post-surveys, the results of which are presented in Table 5 below. All health practice activities but one displayed significant increase in youths’ knowledge between the pre- and post-surveys, with the largest increase in knowledge occurring on the item, ‘helping seniors to be active’, while the activity that did not show significant increase was ‘healthy food choices’. The knowledge on the activity ‘helping seniors to be active’ increased by over 300%, while the knowledge on ‘exercise and fitness’ increased by 108% and the activity ‘teaching children health information in a way they can understand’ increased by 106%. The other remaining activities showed an increase in knowledge of between 39 and 85%. Table 5. Results for Health Practices Mean(SD) Pre-Survey 8.14(1.56)

Post-Survey 9.57(0.85)

1.43

3.68*

95% CI of mean difference 0.59 – 2.27

Create and deliver health-related lesson plans Teaching children health information in a way they can understand

5.93(3.22)

8.14(2.07)

2.21

2.70*

0.44 – 3.99

0.75

6.57(2.71)

8.64(1.79)

2.07

3.64*

0.84 – 3.30

1.06

Community outreach Health and healthier habits Healthy food choices Helping seniors to be active

6.14(2.77) 6.79(2.46)

8.00(2.04) 8.14(2.71)

1.86 1.36

3.05* 2.51*

0.54 – 3.17 0.19 – 2.53

0.85 0.67

6.93(2.50) 3.08(1.38)

8.14(2.35) 9.00(1.53)

1.21 5.92

1.46 11.56**

-0.59 –3.01 4.81 – 7.04

0.39 3.22

Exercise and fitness

Mean difference

t-value

n = 14; *Significant at .05 level; **Significant at .001 level 16 | P a g e

Effect size 1.08

Evaluation of the Braddock Youth Project

Media Results for media activities are shown in Table 4 below, which were obtained from 10 youths who completed both the pre- and post-surveys. All the activities under Media displayed significant changes between the pre- and post-surveys. All the activities displayed large effect sizes of at least more than 1.0 standard deviation. The biggest change occurred on the activity, ‘working on a Mac computer’ on which the students’ knowledge increased more than two standard deviations between the pre-survey and the post-survey. Another way to interpret this is, compared to what they knew about Mac computers before coming into the program, the youths’ knowledge increased by more than 100% on all the activities in Media, and actually, for ‘working with a Mac computer’, the youths’ knowledge increased by over 200% over the summer period. Table 4. Results for Media Mean(SD) Pre-Survey 7.50(1.96)

Post-Survey 9.30(0.82)

1.80

3.03*

95% CI of mean difference 0.46 – 3.14

6.30(2.11)

9.00(0.82)

2.70

5.22*

1.53 – 3.87

2.42

Interviewing people

6.00(1.83)

8.90(1.66)

2.90

4.02*

1.27 – 4.54

1.28

Storytelling skills

6.90(1.52)

8.70(1.89)

1.80

3.38*

0.59 – 3.01

1.01

Operating video cameras Working on a Mac computer

Mean difference

t-value

Effect size 1.10

n = 10; *Significant at .05 level;**Significant at .01 level

BYP Junior For BYP Junior, a total of 11 youths responded to both the pre- and post-surveys. Of the four activities measured for BYP Junior, two showed significant increase in the youths’ knowledge between the pre-survey and post-survey. The ‘mentoring children’ activities increased by 102% and ‘create and deliver lesson plans for youth activities’ increased by 132%. The other two activities show children positive ways to interact with each other’ and ‘teach children concrete ways to help the Braddock community’ did not show any increase in knowledge at all. These results are shown in Table 6 below.

17 | P a g e

Evaluation of the Braddock Youth Project

Table 6. Results for BYP Junior Mean(SD)

Mean difference

t-value

Pre-Survey

Post-Survey

Mentoring children

7.55(2.30)

9.09(1.81)

1.55

3.26**

95% CI of mean difference 0.49 – 2.60

Effect size

Create and deliver lesson plans for youth activities Show children positive ways to interact with each other Teach children concrete ways to help the Braddock community

6.45(2.70)

10.00(0.00)

3.55

4.36**

1.73 – 5.36

1.32

9.27(0.91)

9.27(1.10)

0.00

0.00

-

-

8.64(2.01)

8.64(1.91)

0.00

0.00

-

-

1.02

n = 11; **Significant at .01 level

Service Learning The service learning activities were completed by all students who participated in the BYP summer program. A total of 45 students completed both the pre- and post-surveys and the results are presented in Table 7 below. To measure youths’ participation in service learning activities, four items were used. Of the four items, two showed significant change: ‘I am motivated to help my community’ and ‘I help seniors in the community’. Both items showed medium effect size change between the pre- and post-surveys. The other two items, ‘I feel like I can change my community for the better’ and ‘I feel I could design and carry out a service activity in my community’ did not show significant change between the pre-survey and the post-survey. Table 7. Results for Service Learning Mean(SD) I am motivated to help my community I help seniors in the community I feel like I can change my community for the better I feel like I could design and carry out a service activity in my community

Pre-Survey

Post-Survey

4.44

4.76

0.31

2.54*

95% CI of mean difference .06 – .56

2.71

3.42

0.71

2.46*

.13 – 1.29

.37

4.29

4.44

0.16

0.91

-.19 - .50

.13

4.02

4.22

0.20

0.90

-.25 - .66

.13

n = 45; *Significant at .05 level

18 | P a g e

Mean difference

t-value

Effect size .40

Evaluation of the Braddock Youth Project

Summary of the Evaluation Findings The findings presented above showed that of the 25 different activities measured in this evaluation, a total of 19 showed significant positive change between the pre- and postsurveys. This represents 76 percent of all the programs. Of the 19 activities, 14 showed a large effect size, and the other five displayed a medium effect size change. For those activities that were not significant, it can be explained that the youths began the program with high knowledge or high skill level such that due to the ceiling effect, there was little change over the period of the program. Overall, five of the six gardening activities, all of the media activities, six of the seven health practice activities, two of the four BYP Junior, and two of the four service learning activities showed a positive impact on the youths’ knowledge and skill level.

Recommendations and Suggestions In light of the findings of the evaluation, it is recommended that the following actions be undertaken. Recommendations and suggestions are based on the data collected during the course of the evaluation project and the observations made during the evaluation. The recommendations and suggestions will be provided in two broad categories. The first category covers the building of evaluation into BYP, and the second category covers recommendations specific to BYP. Building Evaluation into BYP When designing a new program, most stakeholders do not think of evaluation from the onset. For any evaluation to be successful, it is important that stakeholders decide up front what they want to evaluate and how to do it. At the beginning of the project, it is crucial to spend time thinking through how to build an evaluation into the whole project. Building an evaluation into your program from the beginning will give you a better opportunity to collect the data needed to measure important changes caused by your program. Here are some of the key things needed to build a successful project that incorporates evaluation from the onset:  Identify what information is needed.  Think about how you will collect the information, what data needs to be collected at the program’s start for comparison purposes, and who will take responsibility for gathering information.  Just as you have to specify the goals of your program, you also must specify the goals of your evaluation. Are you primarily interested in how your program operates, e.g., what activities are carried out? If so, then process evaluation would be required.  Or, are you interested in the results e.g. Are program participants learning what you wanted them to learn? If this is the case, then an outcome evaluation would be necessary.

19 | P a g e

Evaluation of the Braddock Youth Project



In most cases, both types of evaluations are required. The most important step is to commit to an evaluation when you start your program. This will ensure that evaluation becomes part of the routine operations of your program.  For a program that is already up and running like BYP, find a way to add an evaluation component as soon as possible. There are ways to add an evaluation component midcourse.  It is important to make sure measurements of the desired outcomes are consistent and accurate. Recommendations Specific to BYP One of the comments from the youths was to have the youths rotate groups rather than stick to the same groups the whole time. I can see the advantages of sticking to the same groups, but exposing the youths to a variety of activities would probably yield benefits that outweigh using the same groups the whole duration of the program. Another observation I made that was echoed by some comments from the youths is the need to have the community more involved. A positive development in that direction is the symposium that was conducted this year for the first time. The symposium provided different stakeholders from different community-based organizations an opportunity to see the fine work that BYP youths were doing. Absent from the symposium though were members of the Braddock community. Involving the community on a significant scale will help reinforce to the youths that what they are doing is being recognized. Another reason to have a strong youth involvement is because the problems in Braddock cannot be solved by the youths alone. While the youths are doing their part, the community is the other piece of the puzzle that needs to come to the table for BYP to succeed in the long run. Community involvement can take many forms but one of the ways to involve them is to have adult volunteers at BYP. When youths work with someone from the same community they can relate to, there is a sense of togetherness, or ‘we are in this together’. It is therefore, a recommendation of this evaluation that BYP consider bigger involvement of parents of the youths and other Braddock community members. Another theme that emerged from the youths and my observation is the issue of communication. From the youths’ perspective, a number of them commented that instructions and directions should be clearly provided for them during activities. Communication as a theme is not only crucial among stakeholders within an organization, but between stakeholders and the evaluator. From my observation in this project, the communication between the evaluator and the stakeholders was not very efficient. A number of times, emails requesting information went unanswered or there were delays in responding to requests. In this current evaluation, efficient communication was even more crucial considering that the timeline for this evaluation project was very limited. For any evaluation to be successful, the evaluator relies heavily on the cooperation of the stakeholders. It is a recommendation of this report that improvement in communication both within BYP and between BYP and outside stakeholders be an area of emphasis moving forward.

20 | P a g e

Evaluation of the Braddock Youth Project

In terms of continuous data collection, it is important for a program like BYP to internally collect accurate data to document their successes, and currently, BYP does a fairly good job in collecting these data. However, the nature of the data has to be more than descriptive data collected using surveys. Data should be collected using Instruments/tests that specifically measure students’ skills/knowledge on the different activities. Moving into the future, I suggest that BYP work with the evaluator or engage some other professionals to help develop instruments/tests to collect data for internal purposes. Once developed, these tests become the property of BYP that they can administer yearly to measure the outcomes of the program. This is not to suggest that survey data collection be discontinued; rather it is recommended that both types of data be collected. Finally, it is recommended that BYP continue doing what it is doing well as demonstrated by the findings of this evaluation. Addressing those areas that have been identified in this report as in need of improvement should make BYP a solid youth program.

Stakeholder Review and Utilization of Findings Ensuring that the findings of an evaluation are utilized fully and effectively is a goal of many evaluations, and utility is indeed one of the four standards of the Joint Committee on Standards for Educational Evaluation (1994). In an effort to judge the quality efforts toward this goal, a checklist of the various aspects of the utility standard and the results are attached in Appendix B. However, while the evaluation tried to maximize the potential usefulness of this evaluation report, it is only one piece of the utilization puzzle. The other piece are the key stakeholders who must ultimately make use of the findings, and therefore this evaluation undertook an effort to solicit their opinions as to how they might use the results and how the evaluation results could be better presented to make them more useful.

21 | P a g e

Evaluation of the Braddock Youth Project

References Cousins, J.B. & Earl, L.M. (1992) The case for participatory evaluation. Educational Evaluation and Policy Analysis. 14(4), 397-418. Dawson, J. A., & D'Amico, J. J. (1985). Involving program staff in evaluation studies: A Strategy for increasing information use and enriching the database. Evaluation Review. 9,173-188. Frisbie, R.D. (1991). New directions in evaluation: A school district perspective. Theory into Practice. 30(1), 46-51. Hansen, H.F. 2005. Choosing evaluation models: A discussion on evaluation design. Evaluation, 11(4). Larson, R. W. & Walker, K. C. (2010). Dilemmas of practice: Challenges to program quality encountered by youth program leaders. American Journal of Community Psychology, 45, 338-349. Lee, S.-A., Borden, L. M., Serido, J., & Perkins, D. F. (2009). Ethnic minority youth in youth programs: Feelings of safety, relationships with adult staff, and perceptions of learning social skills. Youth and Society, 41, 234-255. Sheehan, K. & Verner, H. (2009). Chicago youth programs: Bridging the gap. Pediatric Annals, 38, 161-166. Yohalem, N. & Wilson-Ahlstrom, A. (2010). Inside the black box: Assessing and improving quality in youth programs. American Journal of Community Psychology, 45, 350-357.

22 | P a g e

Evaluation of the Braddock Youth Project

Appendix A Table 2. Evaluation Plan: Design Summary Table (Adapted from Fitzpatrick, Sanders & Worthen, 2004) Evaluation questions

1.

a. b. c. d.

What is the impact of participation in the Braddock Youth Summer Program on the youth’s skills and knowledge in the following areas: The youths’ gardening skills? The youth’ health practices? The youths’ media skills? Service learning

2.

What is the impact of participation in the BYP junior program on the youth’s skills and knowledge in the following activities: a. Mentoring of children? b. Creating and delivering lesson plans for youth activities? c. Showing children positive ways to interact with each other? d. Teach children concrete ways to help the Braddock community?

23 | P a g e

Activities to be evaluated Gardening

Information sources BYP youths, BYP interns, BYP staff

Media

Information needed to answer the question Scores measuring skill/knowledge level before and after participation

Data collection tools/strategies

Data analysis procedures

Pre and post survey

Quantitative analysis on survey data including descriptive and inferential statistics to calculate gain scores

Observation Focus group** interviews

Health Practices Service Learning

BYP Junior

Document review

BYP youths, BYP interns,

BYP staff

Scores measuring skill/knowledge level before and after participation

Pre and post survey Observations Focus group** interviews

Quantitative analysis on survey data including descriptive and inferential statistics to calculate gain scores Document review

Evaluation of the Braddock Youth Project

Appendix B PROGRAM EVALUATIONS METAEVALUATION CHECKLIST (Based on The Program Evaluation Standards) Daniel L. Stufflebeam 1999 This checklist is for performing final, summative meta evaluations. It is organized according to the Joint Committee Program Evaluation Standards. For each of the 30 standards the checklist includes 10 checkpoints drawn from the substance of the standard. It is suggested that each standard be scored on each checkpoint. Then judgments about the adequacy of the subject evaluation in meeting the standard can be made as follows: 0-2 Poor, 3-4 Fair, 5-6 Good, 7-8 Very Good, 9-10 Excellent. It is recommended that an evaluation be failed if it scores Poor on standards P1 Service Orientation, A5 Valid Information, A10 Justified Conclusions, or A11 Impartial Reporting. Users of this checklist are advised to consult the full text of The Joint Committee (1994) Program Evaluation Standards, Thousand Oaks, CA: Sage Publications. TO MEET THE REQUIREMENTS FOR UTILITY, PROGRAM EVALUATIONS SHOULD: U1 Stakeholder Identification Clearly identify the evaluation client Engage leadership figures to identify other stakeholders Consult potential stakeholders to identify their information needs Use stakeholders to identify other stakeholders _ With the client, rank stakeholders for relative importance Arrange to involve stakeholders throughout the evaluation Keep the evaluation open to serve newly identified stakeholders Address stakeholders' evaluation needs Serve an appropriate range of individual stakeholders Serve an appropriate range of stakeholder organizations 9-10 Excellent _ 7-8 Very Good _ 5-6 Good _ 3-4 Fair _ 0-2 Poor U2 Evaluator Credibility Engage competent evaluators Engage evaluators whom the stakeholders trust Engage evaluators who can address stakeholders’ concerns Engage evaluators who are appropriately responsive to issues of gender, socioeconomic status, race, and language and cultural differences Assure that the evaluation plan responds to key stakeholders’ concerns Help stakeholders understand the evaluation plan Give stakeholders information on the evaluation plan’s technical quality and practicality Attend appropriately to stakeholders’ criticisms and suggestions _ Stay abreast of social and political forces Keep interested parties informed about the evaluation’s progress 9-10 Excellent _ 7-8 Very Good _ 5-6 Good _ 3-4 Fair _ 0-2 Poor

24 | P a g e

Evaluation of the Braddock Youth Project

U3 Information Scope and Selection Understand the client’s most important evaluation requirements Interview stakeholders to determine their different perspectives Assure that evaluator and client negotiate pertinent audiences, questions, and required information _ Assign priority to the most important stakeholders Assign priority to the most important questions Allow flexibility for adding questions during the evaluation Obtain sufficient information to address the stakeholders’ most important evaluation questions Obtain sufficient information to assess the program’s merit Obtain sufficient information to assess the program’s worth _ Allocate the evaluation effort in accordance with the priorities assigned to the needed information _ 9-10 Excellent 7-8 Very Good _ 5-6 Good _ 3-4 Fair _ 0-2 Poor U4 Values Identification Consider alternative sources of values for interpreting evaluation findings Provide a clear, defensible basis for value judgments _ Determine the appropriate party(s) to make the valuational interpretations _ Identify pertinent societal needs Identify pertinent customer needs _ Reference pertinent laws _ Reference, as appropriate, the relevant institutional mission Reference the program’s goals Take into account the stakeholders’ values _ As appropriate, present alternative interpretations based on conflicting but credible value bases _ 9-10 Excellent _ 7-8 Very Good 5-6 Good _ 3-4 Fair _ 0-2 Poor U5 Report Clarity Clearly report the essential information Issue brief, simple, and direct reports Focus reports on contracted questions Describe the program and its context Describe the evaluation’s purposes, procedures, and findings Support conclusions and recommendations Avoid reporting technical jargon Report in the language(s) of stakeholders Provide an executive summary Provide a technical report 9-10 Excellent _ 7-8 Very Good _ 5-6 Good _ 3-4 Fair _ 0-2 Poor

25 | P a g e

Evaluation of the Braddock Youth Project U6 Report Timeliness and Dissemination _ Make timely interim reports to intended users Deliver the final report when it is needed _ Have timely exchanges with the program’s policy board Have timely exchanges with the program’s staff _ Have timely exchanges with the program’s customers _ Have timely exchanges with the public media Have timely exchanges with the full range of right-to-know audiences _ Employ effective media for reaching and informing the different audiences Keep the presentations appropriately brief Use examples to help audiences relate the findings to practical situations _ 9-10 Excellent _ 7-8 Very Good 5-6 Good _ 3-4 Fair _ 0-2 Poor U7 Evaluation Impact _ Maintain contact with audience Involve stakeholders throughout the evaluation Encourage and support stakeholders’ use of the findings Show stakeholders how they might use the findings in their work Forecast and address potential uses of findings _ Provide interim reports Make sure that reports are open, frank, and concrete Supplement written reports with ongoing oral communication _ Conduct feedback workshops to go over and apply findings Make arrangements to provide follow-up assistance in interpreting and applying the findings _ 9-10 Excellent 7-8 Very Good _ 5-6 Good _ 3-4 Fair _ 0-2 Poor Scoring the Evaluation for UTILITY Strength of the evaluation’s provisions for Add the following: UTILITY: 26 (93%) to 28: Excellent Number of Excellent ratings (0-7) __3__x4 =__12 19 (68%) to 25: Very Good Number of Very Good (0-7) __2__x3 =____6 14 (50%) to 18: Good Number of Good (0-7) __2__x2 =____4 7 (25%) to 13: Fair Number of Fair (0-7) __0__x1 =____0 0 (0%) to 5: Poor Total Score = 22 22 (Total score) ÷ 28 = .786 x 100 = 78%

26 | P a g e

Evaluation of the Braddock Youth Project

Appendix C Braddock Summer Youth Project Participants Survey Unique ID______________ Thank you for completing this survey. Please answer the following questions as truthfully as you can. There is no right or wrong answers and no one will know how you answered the questions. 1.

Is this your first time in the Braddock Summer Youth Project? ____Yes ____No

2.

GARDENING Using the scale below, please rate your skill level or knowledge on each activity listed below.

0 No skill/ No knowledge

1

2

3

4

5

6

7

8

9 10 Excellent

Very knowledgeable

_____ Cultivate raised beds _____ Care for fruit trees _____ Permaculture/organic gardening _____ Grow seedlings in greenhouse and transplant _____ Harvest vegetables _____ Cultivate an empty lot

3.

MEDIA Using the scale below, please rate your skill or knowledge level on each activity listed below.

0 1 No Skill/ No knowledge

2

3

4

5

6

7

8

9 10 Excellent skills/ Very knowledgeable

____ operating video cameras ____ working on a Mac computer ____ interviewing people ____ Storytelling skills _________________________________________________________________________________________ 4.

HEALTH PRACTICES Using the scale below, please rate your knowledge level on each activity listed below.

0 No knowledge

1

2

3

4

5

6

7

8

____ Exercise and fitness ____ Create and deliver health-related lesson plans ____ Teaching children health information in a way they can understand ____ Community outreach ____ Health and healthier habits

27 | P a g e

9 10 Very knowledgeable

Evaluation of the Braddock Youth Project ____Healthy food choices ____ Helping seniors to be active

5.

BYP, JR. Using the scale below, please rate your knowledge level on each activity listed below.

0 No knowledge

1

2

3

4

5

6

7

8

9 10 Very knowledgeable

____ Mentoring children ____ Create and deliver lesson plans for youth activities ____ Show children positive ways to interact with each other ____ Teach children concrete ways to help the Braddock community __________________________________________________________________________________________ 6. SERVICE LEARNING Answer each of the following questions by circling a.

I am motivated to help my community

Much

1 Not at all

2

3

4

5 Very Much

Very

b. I help seniors in the community 1 2 3 4 5 Not at all Very Much c.

I feel like I can change my community for the better. 1 2 3 4 5 Not at all

Very Much

d.

I feel that I could design and carry out a service activity in my community. 1 2 3 4 5 Not at all Very Much

7.

**What do you think you learned the most this summer? ___________________________________________________________________________

8.

**Is there anything you think should be improved to better meet the goals for BYP? _________________________________________________________________________________________________________________________________________________ _____________________

**Only on post survey

28 | P a g e

Evaluation of the Braddock Youth Project

List of documents utilized 1. 2. 3. 4. 5. 6.

Appendix D

Final AmeriCorps progress Report for 2009-2010 program year BYP program brief for 2010-2011 program year Final report – Braddock Youth Summer project 2009 Final report – Braddock Youth Summer project 2008 Final report – Braddock Youth Summer project 2007 Final report – Braddock Youth Summer project 2006

29 | P a g e