Presentation "Sowing the Seeds: Program Evaluation that Works for You " is at the bottom

Evaluate the presentation Transcript module 'Sowing the Seeds: Program Evaluation that Works for You " and then answer the following questions.

According to Rossi, good evaluation questions establish a criteria for program performance. As such, theymust be reasonable and appropriate, and they must be answerable. That is, they must be clear, observable dimensions of program performance that are relevant to the program's goals and represent domains in which the program can realistically be expected to have accomplishments.

Question 1: What skills do you take from this presentation in terms of how to define goals and objectives?

Question 2: Can you identify a program goal and objective?

Question :3 Can you name a process and outcome objective for that program?

Question4: What might be a relevant criteria to program performance for your objectives?

Presentation "Sowing the Seeds: Program Evaluation that Works for You "

Presentation Transcript at the bottom

Sowing the Seeds: Program Evaluation that Works for You Self-Guided Training Module Please press or click the mouse to begin. Center drawing credit: Josh Jetson, 16, Australia

Introduction Welcome to the Sowing the Seeds self-guided training module on program evaluation.

Introduction The goal of this web-based module is to increase your capacity to evaluate your program. The examples used in this module focus on evaluation of youth programs but the principles of evaluation can be applied to any program.

Introduction Individuals who will find this module useful are program directors, staff members and volunteers, youth participants, parents, or anyone interested in quality programming. Please press or click the mouse to begin.

Orientation This module is an interactive and flexible way for you to learn about the basics of program evaluation. It will take approximately 30 minutes for you to go through the core module. You will also have the option of going in-depth to learn more about some of the topics.

Orientation The module contains slides with descriptions and definitions. Also, you will have the opportunity to test your understanding with different activities. The module uses an example of a youth program to illustrate the steps of evaluation.

Orientation You also can move at your own pace- Just press , click the mouse, or press the forward arrow when you are ready to continue or if you would like to move quickly through a section. Try it now to go to the next slide.

Outline of Training Module This module consists of five parts: I. Introduction to Evaluation II. Goals, Objectives, & Measures III. Data Collection Tools IV. Data Analysis & Reporting V. Evaluation Steps & Planning

I. Introduction to Evaluation So, let's get started. Why do you want to evaluate your youth program?

I. Introduction to Evaluation Here are some reasons for conducting an evaluation. Program evaluation helps you: assess changes in participant knowledge, attitudes, and skills; secure and maintain program funding; improve programming; and justify program continuation and adoption. Are your reasons included?

I. Introduction to Evaluation Whatever your reasons for wanting to know more about program evaluation, this module will provide you with basic information and tools to think about and plan your own evaluation.

Types of Evaluation There are two types, or parts, of evaluation: Process Evaluation Outcome Evaluation &

Process Evaluation assesses what your program does. The process evaluation has to do with what you teach, how many sessions you hold, your attendance rates, the activities you conduct, etc. Process Evaluation

Outcome Evaluation describes how your participants will change as a result of the program. The outcome evaluation has to do with the knowledge, attitudes, and skills you anticipate your participants will gain. Outcome Evaluation

Types of Evaluation As you go through the module and the optional sections, you will learn more about how to conduct the process and outcome evaluations of your program.

Types of Evaluation If possible, it is important to conduct both types of evaluation. The information you get will help you to match up what you actually did in the program (process) with how participants changed as a result (outcome). The combination tells you what worked and what didn't in your program.

Special Note: Youth Participatory Evaluation A growing number of youth serving organizations are working collaboratively with youth participants to design and conduct program evaluations, an approach called youth participatory evaluation. The results can be great for the program and for the youth! For more information on youth participatory evaluation, refer to the resource list at the end of the training module.

Welcome to Part II This part introduces the goals, objectives, and measures (optional). I. Introduction to Evaluation II. Goals, Objectives, & Measures III. Data Collection Tools IV. Data Analysis & Reporting V. Evaluation Steps & Planning

II. Goals & Objectives When you evaluate your program, you measure whether you are running the program and meeting those served (e.g., youth) as you had planned. Your program's goals and objectives organize what you plan to achieve through your program.

II. Goals & Objectives Goals & objectives are important because they are the destination on your evaluation roadmap. Without them, it is hard to know what you want to accomplish and if you are succeeding! As you will soon learn, goals are more broad than objectives. Objectives are more detailed and measurable.

II. Goals & Objectives A goal is the broad and overarching purpose toward which your program is directed. For example, the goal of the YouthWorks program is "to contribute to a culture of free speech and social responsibility."

Goals What is the goal of your youth program?

Objectives Your program's objectives are the more specific things you plan to accomplish. For example, one objective of the YouthWorks program is to involve 25 youth in 10 media workshops over 3 months.

Goals & Objectives-Why both? You may be wondering: What is the purpose of having goals and objectives? It is important to have both because while objectives help you assess short-term steps or milestones in running your program successfully, a goal helps you to strive toward your ultimate vision for the program. Keeping your goal in mind can help you to do new objectives as your program changes and improves over time!

II. Goals & Objectives If you would like to learn more about goals and objectives now, click on this button. If you would like to continue with the core module, click on this button.

Objectives While a goal is general, an objective should be measurable, achievable, and specific. Remember this and you will get MAS (more) out of your objectives! Measurable * Achievable * Specific

Objectives Measurable means that you can set up a practical way to assess your objective. Achievable means your objective is realistic. Specific means that your objective is as detailed as possible.

Objectives Here is an example of an objective with MAS. One of YouthWorks program's objectives is to involve 25 youth in 10 media workshops over 3 months. achievable measurable specific

Types of Objectives There are two types of objectives: Process Objectives Outcome Objectives &

Process Objectives describe what your program will do. A process objective has to do with what you will teach, how many sessions you will hold, your attendance rates, the activities you plan to conduct, etc. Process Objectives

Process Objectives Remember, one of YouthWorks program's objectives is to involve 25 youth in 10 media workshops over 3 months. This is a process objective because it describes what the program will do. Process Objectives

Outcome Objectives describe how your participants will change as a result of the program. An outcome objective has to do with the knowledge, attitudes, and skills you anticipate your participants will accomplish. Outcome Objectives

Outcome Objectives Another objective of the YouthWorks program is to improve participants' cooperation skills over the course of the 12-week program. This is an outcome objective. You will learn later how to make this objective measurable, by identifying how you will know if cooperation skills have improved. Outcome Objectives

Process or Outcome? Can you guess whether the following objectives are process or outcome? Guess and then click the mouse to see the answer. Objective: Instructors will cover material from at least 6 of the 8 lessons in the curriculum guide during one semester. Process Objective

Process or Outcome? Process or outcome? Guess and then click the mouse to see the answer. Objective: All 15 participants will develop 5 pictures in 3 weeks. Outcome Objective

Process or Outcome? Process or outcome? Guess and then click the mouse to see the answer. Objective: The program will have 85% attendance on average during the summer session. Process Objective

Goals & Objectives Review When you are clear about the goal and process and outcome objectives of your program, it will be much easier to evaluate your program. Take the next five minutes to read about the Better Bodies program and develop some process and outcome objectives for the program.

Goals & Objectives Review The Better Bodies program has as its goal, "to promote optimum lifelong physical health through weight management and education among youth." The program involves about 50 youth (ages 9 to 12 years) yearly in an after school program. The program involves two two-hour sessions weekly over six months and includes lessons and hands-on activities focusing on nutrition education, food preparation, shopping skills, physical education, and games.

Goals & Objectives Review The program involves about 50 youth (ages 9 to 12 years) yearly in an after school program. The program involves two two-hour sessions weekly over six months and includes lessons and hands-on activities focusing on nutrition education, food preparation, shopping skills, physical education, and games. You already know the goal of the Better Bodies program. Can you come up with 1-2 process objectives and 1-2 outcome objectives for the program? STOP and take a minute to think about and do down these objectives. Save them to look at later.

Goals & Objectives Review Did you come up with 1-2 process objectives and 1-2 outcome objectives for the program? Save them to look at later.

II. Measures Measures are the pieces of information that tell you if you're meeting your objectives. It is easy to remember what a measure is because it refers to how you will "size up" or measure if you are achieving your program objectives.

II. Measures The idea of "measures" can seem tricky, but we use this concept in our everyday life. For example, suppose you plan to get healthy this year. How would you keep track of whether you are meeting this goal?

II. Measures You might want to collect data in a couple of areas to chart your progress. These categories or measures may be: Number of times per week you exercise for 30 or more minutes Number of vegetables you eat each day Number of pounds you lose each week

II. Measures If you would like to learn about measures now, click on this button. If you would like to continue with the core module, click on this button.

II. Measures Each measure is related to an objective. For example, if a process objective of the Better Bodies program is to involve the participants in 25 minutes of vigorous exercise at each session, then your measure for that objective would be the information that you record to know if youth are really getting opportunities for exercise. What would that be? The number of minutes provided for the youth to do vigorous exercise at each session.

Types of Measures Just like objectives, there are two types of measures: Process Measures Outcome Measures &

Process Measures are types of information that you collect about what your program is doing or has done. A process measure is a piece of information such as what you taught, how many sessions you held, your attendance rates, the activities you conducted, etc. Process measures

Process Measures Remember, one of YouthWorks program's process objectives is to involve 25 youth in 10 media workshops over 3 months. The process measure for this objective is the number of youth in attendance at each media workshop. Process Measures

Outcome Measures are types of information that you collect to see how participants have changed as a result of the program. An outcome measure is a piece of information that has to do with the knowledge, attitudes, and skills your participants accomplish in the program. Outcome Measures

Knowledge Attitudes Skills Definition Participants' understanding of concepts Participants' feelings about topics Participants' ability to perform activities Examples Definitions of dance terms Comfort level with using camera Ability to develop photos independently Components of the color wheel Interest in sports history Ability to assist others with homework tasks Outcome Measures Here are definitions and examples of typical outcome measures.

Outcome Measures An outcome objective of the YouthWorks program is to improve participants' cooperation skills over the course of the 12-week program. The outcome measure for this objective is the change in participants' cooperation skills from the beginning to the end of the program. Outcome Measures

Outcome Measures You may have several options when deciding on the outcome measure for a particular outcome objective. For example, for the outcome measure, change in participants' cooperation skills, you will need to decide:

Outcome Measures how to put a value on "cooperation skills" (so you can tell if there is a change);

Outcome Measures whether you will use a self-reported measure, where the participant reports back on his/her own skills, or whether you will have a third person like an instructor assess the participants' skills; and

Outcome Measures whether you will assess the change in the participant's skills at the beginning and at the end of the program and calculate the change, or if you will just ask about the amount of change made, at the end of the program.

A Measure's Story You may not realize it but the fact that you know what a measure is means that you know A LOT! Let's consider the story of one measure: Change in participants' knowledge of musical genres. Let's start with some questions for you...

A Measure's Story Change in participants' knowledge of musical genres. What type of measure is this? If you said that it is an outcome measure, then you are right! What is the outcome objective, then? There are several ways of phrasing an appropriate outcome objective. Yours should sound something like, "to increase participants' knowledge of musical genres, by, on average, 2 additional genres, from the beginning to the end of the 6-week unit on music through the ages."

A Measure's Story Change in participants' knowledge of musical genres. Now, from where will the information for this measure come? Since we're interested in changes in knowledge we may want to ask each participant to list the musical genres he or she knows at the beginning, and again at the end, of the unit. Then, we can calculate how many more genres each participant could name at the end of the unit, divide by the number of participants, and see if the average is below or above 2 (the number we wanted to meet in our objective).

Measures Review It is now time to practice your measure-writing skills. Stop and refer back to the process and outcome objectives you wrote for the Better Bodies program. Write down a measure for each objective. When you are ready, move to the next slide.

A Final Word about Measures It may be tempting to skip the stage of figuring out and writing down your measures but don't succumb to that temptation! By figuring out and writing down your measures, you ensure you will gather only the information that you need (and not a lot of extra!) to improve your program and to see if it is working.

Welcome to Part III The next part looks at data collection tools. I. Introduction to Evaluation II. Goals, Objectives, & Measures III. Data Collection Tools IV. Data Analysis & Reporting V. Evaluation Steps & Planning

IV. Data Collection Tools Quite simply, you use data collection tools to collect data on your measures. You can use one collection tool to collect data on several measures. Just like the types of evaluation, there are process collection tools and outcome collection tools.

Process Collection Tools Process collection tools provide information on process measures and process objectives. Some examples are: Attendance records Lesson reports Sign-in sheets Activity logs

Process Collection Tools Process collection tools will provide you with measures on your process objectives. For example, consider the following:

Outcome Collection Tools Outcome collection tools provide information on outcome measures and outcome objectives. Some examples are: Survey Participant interviews Focus groups Observations

Outcome Collection Tools Outcome collection tools will provide you with measures on your outcome objectives. For example, consider the following:

Outcome Collection Tools Let's look at some outcome collection tools in more depth...

Surveys Surveys are questionnaires given to to youth participants, instructors, caregivers, and/or audience members.

Participant Interviews Participant interviewsare face-to-face discussions with individual participants, with the aid of a pre-determined set of questions. Interviews are great because you can get in-depth information from each participant but interviews also can be time-consuming.

Focus Groups Focus groupsare discussions with a small group of participants, with the aid of a pre-determined set of questions. You can gather feedback from a large group of youth using focus groups, as long as you ensure that it is a safe place for participants to share their own opinions.

Observations Observationslook at participants' involvement in program activities. Typically, an instructor or outside observer will record participants' ability to perform tasks or behavior on an observation form at different points throughout the program duration.

Portfolios Portfoliosare collections of participant's work (art, photos, writing pieces, etc.) in the program. Portfolios provide concrete examples of participants' products but may not always reflect the effort put into the project.

Surveys in More Depth Since surveys are the most common type of data collection tool, you may choose to learn more about them now. If you would like to learn more, click on this button. If you would like to continue with the core module, click on this button.

Surveys Surveys are questionnaires given to to youth participants, instructors, caregivers, and/or audience members. Often, the questionnaires themselves are called surveys.

Surveys Surveys may be administered before (pre) and/or after (post) the program.

Surveys Surveys are great because you can ask a lot of questions at one time and you can compare pre and post surveys. However, the information is all self-reported!

Survey Tips Good questionnaires are: 1. To the point- They only ask for information that will be used later (for the evaluation, reports, etc.). 2. Developmentally appropriate- Instructions and questions are at the appropriate reading level. 3. Clear- Formatting is easy to follow and text is large. 4. Straight-Forward- They use simple language and common terms. 5. Well-Ordered- The questions flow in a logical order.

Survey Activity Since you will likely construct a survey at some point, here is a little activity to get you thinking about well-constructed surveys. On the following slides are some example survey questions. Think about what might be problematic about each and then click the mouse to see if you are right.

Survey Activity Problem survey question #1: During this program, did you enjoy the classes on nutrition and the classes on exercise? What is problematic about this question?

Survey Activity Problem survey question #1: During this program, did you enjoy the classes on nutrition and the classes on exercise? This question is problematic because you wouldn't know from the responses whether the respondents who said "yes" liked both the nutrition and exercise classes or if they only liked one, and which one! What would be a better way to find out which classes the participants enjoyed?

Survey Activity One way to solve the problem is to split the question into two separate questions: During this program, did you enjoy the classes on nutrition? During this program, did you enjoy the classes on exercise?

Survey Activity Problem survey question #2: a) What was your favorite part of the field trip to the university athletic center? b) Did you go on the field trip to the university athletic center? c) How could the field trip to the university athletic center be improved? What is problematic here?

Survey Activity Problem survey question #2: a) What was your favorite part of the field trip to the university athletic center? b) Did you go on the field trip to the university athletic center? c) How could the field trip to the university athletic center be improved? This set of questions is problematic because their order does not make sense. What would you do differently?

Survey Activity That's right, re-order the questions so they flow sensibly! 2a) Did you go on the field trip to the university athletic center? 2b) What was your favorite part of the field trip to the university athletic center? 2c) How could the field trip to the university athletic center be improved?

Survey Activity Problem question #3: How much fun did you have in this program? A ton A lot Some A little A tiny bit Almost none None What is problematic with this question?

Survey Activity Problem question #3: How much fun did you have in this program? A ton A lot Some A little A tiny bit Almost none None This question has too many possible responses. What is the difference between "A tiny bit" and "Almost none", and does it matter? How would you improve the categories?

Survey Activity With multiple choice responses it is good to have between 3 and 5 categories. Most importantly, make sure they make sense! 3. How much fun did you have in this program? A lot Some None

Survey Review Although it may seem easy to write up a quick survey for your participants, constructing a useful survey takes thought and skill. Keep in mind the survey tips presented earlier and remember to make the questions and answers as clear and to-the-point as possible. Also, remember to avoid survey fatigue- the shorter the survey, the higher quality of your responses, usually!

Data Collection Tools Review Remember, you can collect information on more than one measure using one tool. You can also collect information on one measure from more than one tool.

Welcome to Part IV This part will looks at data analysis and reporting. I. Introduction to Evaluation II. Goals, Objectives, & Measures III. Data Collection Tools IV. Data Analysis & Reporting V. Evaluation Steps & Planning

V. Data Analysis Data analysis involves looking at the information you have collected and coming up with useful statements about your program.

V. Data Analysis Some examples of useful statements are: Numbers of lessons, classes, or units taught Numbers of youth participants Percentage of youth with improved knowledge, attitudes, and skills Quotes by participants, parents, and staff Response of community members

V. Data Analysis In order to analyze your data, begin by tabulating (counting up) the data from your survey, interviews, observation form, or other collection tool. You can then come up with numbers, percentages, and/or quotes that reflect your measures, which also will tell you if you're meeting your objectives.

V. Data Analysis For example, if you have a survey question that asks participants to report how safe they feel at the after school program (0-not safe...5-very safe), you might: add up the number who said they feel a little or very safe; divide that number by the total number of responses and then; write a statement such as (next slide)...

V. Data Analysis "87% of participants feel safe at the Downtown AfterCare Program."

V. Data Analysis The statements you come up with during data analysis should sound like your objectives. For example, if one of your objectives is to have an average attendance of 85% over a 12-session program, then the statement you make, after averaging the attendance rate recorded in attendance logs over the 12 sessions, might be (next slide)...

V. Data Analysis "The program achieved a 91% average attendance rate over 12 sessions."

V. Reporting When reporting the information from your program evaluation, you can summarize your statements in several ways, including: Tables Graphs Quotes Case studies

V. Reporting If you would like to learn more about each way of reporting, click on this button. If you would like to continue with the core module, click on this button.

V. Tables Tables are useful for showing information about your program or the participants, such as: Topics covered in each of the units or themes and number of sessions dedicated to each. Age, gender, grade, or other demographic information about participants

V. Graphs There are several types of graphs appropriate for reporting evaluation information. The most common and useful graphs are line graphs, bar graphs, and pie graphs.

V. Graphs Line graphs are useful for showing changes in attendance, participation, etc. over time.

V. Graphs Bar graphs are useful for comparing two or more groups, such as changes in outcomes for those with lower versus higher class attendance.

V. Graphs Pie graphs are useful for showing parts of a whole, such as the percentage of participants by gender and age.

V. Quotes and Case Studies Quotes are useful to support and explain points made from the data. Quotes from youth may be collected on survey questionnaires, in interviews, or in focus groups. Quotes from parents and community members may be collected at presentations or through evaluation forms.

V. Quotes and Case Studies Case studies describe one participant's story in the program. You may write a case study about an exemplar participant or a participant who has improved during the program.

Report Structure Report Structure Executive Summary I. Introduction 1.Program description 2.Evaluation purpose and overview 3.Report layout II. Evaluation Methodology 1.Goal & objectives 2.Description of methods used III. Process Evaluation Findings IV. Outcome Evaluation Findings V. Conclusions and Recommendations Appendix Written reports are an important way to share what you found in your evaluation with funding agencies, supporters, and board members. On the right is a sample report structure outline.

Welcome to Part V The final part focuses on evaluation steps. I. Introduction to Evaluation II. Goals, Objectives, & Measures III. Data Collection Tools IV. Data Analysis & Reporting V. Evaluation Steps & Planning

VI. Evaluation Steps & Planning A review of this training module provides the steps for your program evaluation.

VI. Evaluation Steps & Planning Before you begin your program: Establish your goal. Establish your process and outcome objectives. Decide on process and outcome measures. Determine methods for collecting data. Create forms for collecting data. Involve program staff and youth. Construct an evaluation timeline.

VI. Evaluation Steps & Planning During the program: 6. Collect data using pre-determined methods & forms. 7. Tabulate preliminary data. Administer a pre-survey to track change.

VI. Evaluation Steps & Planning At the conclusion of the program: 8. Tabulate all data. Decide on reporting modes. Prepare, review, and present evaluation findings and use the information for program improvement.

VI. Evaluation Steps & Planning As you follow the evaluation steps, it may be helpful to record your program objectives, measures, methods, and reporting techniques in a quick-reference table like the one on the next slide.

Program Goal: Objectives Measures Collection Tools Reporting Process Outcome Program Evaluation Planning Table

Congratulations! You have completed this training module in program evaluation. With the knowledge you have gained you can begin thinking about and even planning your program evaluation. Refer back to slides in this presentation for help as you plan and carry out your evaluation.

Congratulations! The final slides provide some additional guidance and resources on program evaluation, including: Tips for preparing reports and presentations for key audiences. Glossary of terms Resource list Frequently Asked Questions about program evaluation and responses

Tips for Reports & Presentations When reporting to current funders: Describe how you applied their funds directly to benefit youth. Provide numbers (of participants, attendance, etc.). Include graphics and tables that make a point. Use narratives to highlight how their support affected one youth. When reporting to future funders: Provide positive impact of previous programming through sound evaluations. Show youth's, parents', and community's desire to have additional programming. Highlight total numbers potentially served. If that number is small, perhaps highlight total number of participant-hours, which shows intensity of the program.

Tips for Reports & Presentations When reporting to parents: Describe observed benefits to youth and their development. Describe how program supported parents' efforts. Outline, with data, how parents can support and/or advocate for program. When reporting to thepress: Be concise - Use short sentences (subject, verb, object). Use personal stories of youth. Tie results into current issues. Tie results to resolving local problems.

Tips for Reports & Presentations When reporting to policymakers: Be concise. Be prepared to use data to provide counterarguments. Tie results to financial considerations (cost per child for program versus other programs or consequences.) General tips on reporting your evaluation information: Know what is important to your audience. Have both numbers and personal anecdotes on hand. Use "social math" - Describe numbers in terms relevant to the audience. Focus demands sacrifice - Make a point instead of just throwing in everything.

Glossary of Terms Focus Group: A data collection tool in which there is discussion by a group of people facilitated by a moderator. Focus groups can provide a lot of information about people's opinions about the program in the short time of an hour or two. Goal: A broad expectation of what you or your program would like to accomplish. Interview: A data collection tool that that gives you information about how one person feels about a subject. In program evaluation, an interview can help you learn a young person's views on a program or how he/she has changed as part of the program. Measure: A measure is a category of information (data) that you can ask about (through logs, interviews, questionnaires, etc.) that will help you to understand if your program is working and if participants are getting out of the program what you want them to learn.

Glossary of Terms Process Evaluation: A type of evaluation in which you assess how the program is functioning and if it is implemented as intended. Process Measure: A measure that helps you to find out if you are running your program like you want (to assess if you are meeting your process objectives). Program Evaluation: Collecting information about a program or some aspect of a program to assess if the program is meeting its intended outcomes/ objectives. Objective: A specific and measurable statement identifying the thing your program will do or what the youth will learn and do that will help you accomplish your broader program goal.

Glossary of Terms Observation: A data collection tool in which you document youth in the program to see what they are actually doing in the program. Outcome Evaluation: A type of evaluation in which you assess how participants have changed as a result of the program. Outcome Measure: A measure that helps you to find out if the program is making a difference in the lives of the participants, such as a change in their knowledge, attitudes, or skills (to assess if you are meeting your outcome objectives). Record: A data collection tool which gathers information during the program, such as attendance logs or curriculum checklists. Records chart the progress of the program (process-related) or the participants (outcome-related).

Glossary of Terms Survey: A survey or questionnaire is a written set of questions on a topic. A survey is one of the quickest ways to get information from a large number of people. Youth Participatory Evaluation: A type of evaluation in which young people engage in assessing their programs.

Resources Tips on Evaluation Innovation Network, Inc. -provides planning and evaluation tools for nonprofits and funders (www.innonet.org) The Evaluation Exchange-shares tools and tips from the program evaluation field (www.gse.harvard.edu/hfrp/eval.html) SurveyMonkey-web service for creating internet surveys, options at no charge are available (www.surveymonkey.com)

Resources Youth Participatory Evaluation Checkoway B. and Richards-Schuster K. Facilitator's guide for participatory evaluation with young people. Ann Arbor: University of Michigan, 2002. Johns Hopkins Center for Adolescent Health. Youth Participatory Evaluation Project (YPEP) Evaluator trainer manual and participant booklet. Baltimore: Johns Hopkins University, 2005. Available at: www.jhsph.edu/adolescenthealth Sabo K. Youth participatory evaluation: A field in the making. San Francisco: Jossey-Bass, 2003.

Frequently Asked Questions (FAQ) Q: We have a lot of parts to our program. We offer clinical services and have youth programming. How do I know what to evaluate? A: It is a good idea to make a list of questions you want to answer about your program and then prioritize based on what makes sense: funding needs, planning issues, problem areas, etc. It is common to not have money or resources to evaluate everything, particularly when you have lots of services/programs. Making this list will help you know what to assess first.

Frequently Asked Questions (FAQ) Q: I use surveys a lot in my programs. But, participants always leave lots of it blank. What should I do? A: Survey completion can be challenging. It is important to consider the length of your questionnaire and what makes sense for the age of you participants. You want to use a survey that is clear, easily understood, and as short as possible to ask your most important questions. Also, consider when you administer the survey. If it is at the end of day, the reason for incompletion may be fatigue.

Frequently Asked Questions (FAQ) Q: I like a lot of the data collection tools described in this module-surveys, interviews, portfolios, etc. How do I know what to use? I don't have money or time to do all of them. A: Ask yourself, "What questions do I really need answered for this program evaluation?" Based on these questions, pick a tool (or tools) where you can collect information to answer as many questions as possible. A good rule of thumb is 2-3 tools. It will be hard to use more than that given the challenge of administration, time, and funding.

Frequently Asked Questions (FAQ) Q: I plan to use a pre-post survey design for my evaluation. But, kids enter my program not at one time, but on a rolling basis. Is it okay to give those kids the pre-survey when they enter, or do I have to do it all at one time? A: You can administer as they enter, but keep in mind that the content of your program evolves. So, it will be difficult to know the total impact of the program on all participants because they may have different experiences. You can take this into account with some sophisticated data analyses if that is available to you. If you administer the pre-survey at different times, you are able to chart change in individual participants, which may be useful information. And, it is always a good idea to document how your program is implemented (process evaluation) so that you can draw accurate conclusions about your program effect.

Frequently Asked Questions (FAQ) Q: I want to include young people in the evaluation planning, but I am not ready to use the youth participatory method that you mention in the module. What can I do? A: It is always a good idea to involve youth perspectives in some way as you plan your evaluation. You can ask a few students not part of the program (e.g., graduates) to review your questions to let you know if you are capturing areas of interest.

Frequently Asked Questions (FAQ) Q: I plan to use surveys and interview some kids in my program. What do I do with all of the information? A: You should have a plan for recording and tabulating the survey data (e.g., using a spreadsheet) and for reviewing notes/tapes from the interviews (e.g., writing down quotes and themes). It is helpful to write an outline for your report (i.e., what do you want to tell your audience) because this guide can help in knowing how to analyze and summarize your data.

Frequently Asked Questions (FAQ) Q: My funder wants numbers for my youth program, but my program doesn't look at outcomes like this. How can I include some numbers (or frame the impact of my program for my funders)? A. Funders often like to report on the numbers of people served by their funded programs. Consider different ways of presenting information from your program - total number of youth served during the year, average number of youth served per session, numbers of communities represented by your youth, etc. When numbers are small, a funder may like to include a case study that shows how one participant represents the experiences of many of the participants.