Questions and Answers of Program Evaluation

5. Discuss the most common reactions to being evaluated and what an evaluator could do to make evaluation efforts less threatening.
4. Is it possible to conceive of a situation in which program evaluation would not be threatening? Think about the characteristics of a program and its staff that would completely welcome and embrace
3. Discuss ways in which the political pressures identified in question 2 might be neutralized or negated.
2. From the class’s knowledge of local agencies, discuss political pressures that might surface and influence (either positively or negatively) any evaluation effort in selected agencies.
1. Brainstorm ways that treatment fidelity could be compromised in a program with loose management.
e. Ownership of the data and editorial authority. (At the end of the evaluation, who keeps the completed questionnaires and other data? You may need the data, or at least access to it, should you
d. The budget needed. (Not only is agreement on the consultant’s fee important, but also there should be a definite budget for such items as travel, supplies, scales or instruments, printing,
c. The evaluation methodology: data, staff, and facilities needed. (Will the evaluation design necessitate the use of control groups? Will random assignment of clients be necessary? Will it be
b. The beginning and ending dates of the evaluation. (At a minimum, it is important to specify deadlines when the evaluation products must be finished.)On some occasions it may also be advisable to
a. The purpose or focus of the evaluation. (Incorporate a list of the questions that the evaluation sponsor wants answered or hypotheses that will be investigated.)
. Using a computer database or Internet search engine, search for a scale that you could use in a project of your choosing. Be sure to identify the concept the scale should measure and the population
3. Using either a scale that you have developed or one with known psychometrics, collect a small sample of data from your classmates or others. Enter the data into a computer with statistical
2. With a partner, attempt to create a scale measuring some concept of your choice. Present the scale to the class for a discussion of its face and content validity.
1. Identify an instrument and read one or more articles on the instrument. Write a short paper on its development, as well as the reliability and validity that you would expect it to have if you were
2. Read one of the articles on the development and validation of a scale referenced at the end of this chapter. Summarize, in a short paper, all the steps the author went through.
1. Skim one of the books containing rapid assessment instruments noted in this chapter. Make a list of at least five scales that you might be able to use in your future practice. Explain how each
6. Suggest one or more variables measured at the interval level that could be converted to nominal or ordinal level variables.Mini-Projects: Experiencing Evaluation Firsthand
5. For various client groups, brainstorm what behaviors might be good indicators of treatment success or failure. Why might they work better than measurements of attitudes or knowledge?
4. If you were asked to evaluate an instrument measuring hyperactivity described in a journal, what information about the instrument would you want to find in the article?
3. Discuss the following item taken from the evaluation instrument Barbara designed for adolescents: When you are an adult, what are the chances that you will be a drinker?________ I am certain I
2. Barbara Daydreamer designed a three-item questionnaire to be used as a preand posttest instrument to measure adolescents’ knowledge of alcoholism as a disease. Later she was surprised to find
1. What is wrong with the following questionnaire items?a. Describe your mother’s condition during her pregnancy with you.b. Yes or No: Have you ever been involved in any accidents?c. Have you been
6. Performance ratings (Do performance ratings of supervisors improve?)
5. Productivity (Does productivity increase?)
4. Accident rates (Are there fewer accidents?)
3. Documentation of records (Are a greater percentage of records in compliance with quality assurance standards?)
2. Operating costs (Are costs lower?)
1. Absenteeism (Is there less absenteeism after the workshop than before?)
In the past 30 days, how many times have you taken a drink the first thing in the morning?
Once you start drinking, do you find it difficult to stop before becoming completely intoxicated?
On how many occasions have you tried to quit drinking but were unable to do so?
On how many days of the past 30 days have you had an alcoholic drink?
10. Are you an alcoholic?a. yesb. noc. undecided
9. Wouldn’t you agree that clients should keep their accounts current with the agency?a. yesb. noc. undecided
8. Approximately how many minutes do you dream each evening?a. under 15b. 16 to 30c. more than 31 minutes
7. Do you have a male relative and a female relative over 55 years of age living at home with you?a. yesb. noc. undecided
6. Do you not make a practice of shopping only on weekends?a. yesb. noc. undecided
5. What is your income?a. $10,000 to $20,000b. $20,000 to $30,000c. $30,000 or more
4. How long have you been a client with us?a. 6 months or lessb. under a yearc. 1 year or longer
. What is your marital status?a. singleb. marriedc. divorced
2. Do you come here often for help?a. yesb. noc. don’t know
1. Please rate the quality of our services:a. excellentb. goodc. poor
6. Discuss a proposal made to Congress to allow a $3,000 tax deduction for expenses incurred by parents who adopt a handicapped or older child. How would you evaluate such legislation using either a
5. During periods of great flux, as when there is a dramatic increase in inflation, sensitivity analyses are important. What are some other situations or factors over which an evaluator has no
4. Discuss whether it is more important to know a program’s cost or its effectiveness.
3. An outpatient counseling program has a long waiting list of potential clients. As a new manager, what actions could you take to reduce the waiting list? Make a list of these and discuss each
2. Discuss whether the following desired outcomes are likely to be monetary or nonmonetary. Would your evaluation be a cost-benefit analysis or a costeffectiveness study?a. Increased employee morale
1. Identify local, state, or national programs for which you would like to conduct cost-effectiveness evaluations. Discuss the program outcomes for these. Is there a choice of more than one outcome
2. Locate a published, randomized controlled trial in a field of your choice, and read it. Assess your ability to understand the terminology and methods used in this report.
1. Browse through professional journals in a field of your choice to find the report of an evaluation using a nonequivalent control group design. Read the article and critique it.
e. Do the effects of this treatment persist over time?Mini-Projects: Experiencing Evaluation Firsthand
. Do clients who receive this therapy improve more than they would, compared to receiving standard accepted treatment?
c. Do clients who receive this therapy improve more than they would, compared to receiving a placebo treatment?
b. Do clients who receive this therapy improve more than they could with the simple passage of time?
a. Do clients who receive this therapy improve?
7. Imagine that you have been trained in a new form of individual therapy which seems very exciting and is reported to produce dramatic improvements in traumatized clients very rapidly. Discuss how
Would it be ethical to offer group therapy of unknown efficacy to clients, without placing such services in the context of a systematic program evaluation?
They did get offered group therapy some months later, when the study was concluded. Why did the authors see the need to have such a control group?
6. Discuss the ethics of the study of Cohen and Fried (2007) described in this chapter, which involved randomly assigning breast cancer survivors to a notreatment condition, instead of providing them
5. Discuss for any specific program the various outcome indicators that might be chosen in a program evaluation. Are some more valuable than others for showing the impact the program is having on the
Who would be the subjects?
What would be the primary outcome variable? What would be the threats to the internal validity? How would you control for these? What would be the data collection procedures? Who would need to assist
4. Have the class identify a local program that would be interesting to evaluate using one of the designs in this chapter. What evaluation design would be used?
3. Discuss the problems of using a self-report questionnaire with smokers in a smoking cessation clinic. What percentage might be motivated to indicate that they had stopped when in fact they were
d. Did the agency staff seem committed to evaluating the outcome of their program?____ YES ____ NO____ Cannot be determined
8. Solomon four-group R O X O R O O R X O R O Note: O ¼ Observation or assessment period X ¼ Intervention R ¼ Randomly assigned group research designs 249
7. Posttest-only control group R X O R O
6. Pretest–posttest control group R O X O R O O
Table 9.2 Selected Types of Group Research Designs Commonly Used in Program Evaluation Studies±±±±±±±±±±±±Name of Design Notation Pre-experimental Designs 1. One-group posttest-only X O
c. Did the agency conduct an outcome evaluation of this program?____ YES ____ NO ____ Cannot be determined
b. Did the program serve as many clients as projected?____ YES ____ NO ____ Cannot be determined
a. Did this program have objectives derived from the goals for the program?____ YES ____ NO ____ Cannot be determined
2. Discuss the advantages and disadvantages of an evaluator monitoring similar programs in different agencies using these four questions:
1. Discuss conducting program evaluations in various community agencies. Do some of the designs in this chapter seem to “fit” some programs better than others? Why?
4. Were these improvements maintained over a fairly long period of time, after the intervention was completed?
3. Were these improvements caused by the group work? (Yes!)
2. Were these improvements due to simply being assessed? (No!)
1. Did kids who received group work evidence improvements in self-esteem, selfcontrol, and behavior? (Yes!)
5. With what kinds of problems or programs would you feel it necessary to have a larger sample that would give you a margin of error of 5 percent and a confidence level of 95 percent, as opposed to a
4. Give an example of a situation when a purposeful sample would be needed.
3. Under what conditions would it be important to have a level of confidence of 99 percent as opposed to 95 or 90 percent?
2. Describe at least one project best investigated by probability sampling.
1. List some evaluation projects for which nonprobability sampling might be appropriate.
3. Design a client satisfaction study for a program of your choosing. Locate or create a suitable instrument, develop the data collection procedures, and draw up a budget and projected timeline. Be
2. Ask the agency where you are working, interning, or volunteering if any client satisfaction studies have been conducted in the past 7 years. If so, review the instrument used, the data collection
1. Collect as many client satisfaction forms as you can from school, work, fastfood restaurants, or other sources. Examine them and determine how many questionnaires have similar or identical items.
8. What are the pros and cons of using a convenience sample of clients for program evaluation activities?
7. What are the arguments for and against using standardized instruments to measure client satisfaction?
6. What open-ended questions would you want to ask consumers participating in a client satisfaction study of a community mental health psychoeducational intervention?Would you think it would be best
5. Knowing what you now know about client satisfaction studies, argue for and against their place in a comprehensive program evaluation.
4. Although it is known that client satisfaction studies typically are positively biased, discuss why this form of evaluation is used so often in evaluating university faculty.
3. What variables could affect response rates of a client satisfaction study besides the survey administration procedures?
2. How would you tackle the problem of variability in response rates within the same agency (as discussed by Baker, Zucker, and Gross, 1998)?
1. Can the problem of positive bias with client satisfaction studies be “fixed”?What could be done to minimize this form of bias in a consumer feedback study?
6. Under what conditions could an SSRD be considered to represent an “experimental”design?
5. Distinguish between the functions of needs assessments, formative evaluations, quality assurance studies, and summative evaluations.
4. Locate an example of a single system design published in a professional journal during the past 5 years. Identify the type of design, the outcome measure, and how well the intervention is
3. Could SSRDs be used in conjunction with consumer satisfaction studies? How?
5. How many cancellations (or no-shows) were there last month? What were the characteristics of those who canceled and gave no notification? (How many were single mothers with small children or
4. Of those clients referred for services last month, how many were referred by the criminal justice system?

Showing 400 - 500 of 564