ECUR 809 Assignment #2

An over funded pilot research project financed by the National Health Research and Development Program (NHRDP) was designed to examine the feasibility and results of an exercise program for pregnant aboriginal women with a history of gestational diabetes. The pilot project led to the spontaneous creation of a side program promoting exercise for all pregnant aboriginal women. This program is the subject of a potential evaluation; however, I found the information given in the article describing the program to be too basic. There were gaps in the details that are needed for a potential evaluator to formulate a plan. What does the client hope to accomplish by performing an evaluation?  Who is the audience for the evaluation? Evaluation could attempt to find answers for a number of questions; for example, did the women find exercise more accessible through using the program or did the program have an impact on the incidence of diabetes among the women who enrolled? Knowing more details would help to focus the evaluation and better assess the program. Gaining more information would be one of the first steps in the evaluation process. Assuming the purpose of the evaluation is to guide the stakeholders in decisions regarding the future of the program and considering Stufflebeam’s advocacy for a decision-oriented approach to evaluation (Fitzpatrick et al, 2004), it seems appropriate to use the CIPP evaluation model.    

CIPP Begins with ‘C’ - context evaluation.  At first glance it would appear that the programs objectives are obvious and stem from the facts: gestational diabetes is common in pregnant aboriginal women; we know that the increased activity is instrumental in the prevention of gestational diabetes; making a program available to promote increased activity levels in aboriginal women will help these women to have healthier pregnancies;  and increasing the activity of pregnant aboriginal women will decrease the incidence of gestational diabetes.  However, after reading the brief description of the program I felt uneasy about accepting the seemingly apparent objective.  I came up with a list of other possible objectives. Did the exercise program decrease the occurrence of type II diabetes? Promote health in the participants? Determine the feasibility of conducting an exercise program for pregnant aboriginal women?, provide a partnership and foster relationships with the aboriginal community?  Since the program was a bit of a spontaneous effort as an aside to a pilot program that was experiencing limited clientele, the context evaluation component of the CIPP evaluation model should not be overlooked or trivialized.  The objectives can be defined more concretely, proceeding through the evaluation to gain a deeper understanding of the program.

As opposed to the Tylerian objectives oriented style approach, where the evaluator seeks purely to answer the question of goal attainment, the CIPP model can be used to examine the program and guide the development through recommendations provided by the evaluator; this helps the stakeholders gain valuable information and insight.  Starting with clarifying the objectives of the program, the context evaluation could include a needs assessment and a gap analysis.  The input evaluation could help the program planners to consider the alternative ways of meeting the objectives.  It could be revealed that there are more effective delivery strategies that would provide a higher cost-benefit ratio.  Investigation into the implementation could look at participation rates and ways to increase them. This component of the evaluation would help to facilitate improvements to the delivery.  Finally, product evaluation could determine the value of the program in it’s many aspects including the overall experience for the women.  Exploration of the long term effects would be beneficial but could be costly; the budget would determine the depth at which each component could be explored and which additional tools could be employed.  Overall, the CIPP model is effective in helping decision makers determine the next step for the program.     

References

Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2004). Program Evaluation: Alternative approaches and practical guidelines. White Plains, NY: Longman.

Klomp, H., Dyck, R., and Sheppards, S. (2003). Description and evaluation of a prenatal exercise program  for urban Aboriginal women. Canadian Journal of Diabetes, 27 : 231-238.

ECUR 809 Assignment #1

PLEA Evaluation Report

I first heard about PLEA when I applied to work as a teacher in the Daughters and Sisters program early in my teaching career.  The Daughters and Sisters program serves young girls that are recovering from substance addictions. The girls receive an intense treatment program which includes detox, counselling and education. I was offered an interview but unfortunately, I had already decided to accept another teaching position and had to turn it down.  I have always wondered what teaching there would have been like.

Starting in 2008, and continuing over a three year period, an external evaluation of the PLEA programs was performed by the McCreary Centre Society (MCS) in partnership with Douglas College and funded by Justice Canada.  Tools used in the data collection included surveys of participants, a small focus group of participants from one of the programs, and phone surveys of the caregivers. The ethics of the project were examined by Douglas College's Research Ethics Board and the evaluation proceeded with the care and propriety required when dealing with young people.

MCS used a repeated measure design, where the same youth were surveyed three times: prior to intake, at the completion of the program and a short time after. It should be noted that the number of youth surveyed decreased from intake, where 261 were surveyed, to post completion, where only 105 youth were surveyed.  Qualitative information collected from a focus group was presented anecdotally throughout the report.

This report outlined a monitoring and outcome evaluation that provided a summative assessment of the program.  As a result of the inquiry by MCS, some formative feedback was collected.  This feedback may have an impact on the development of the program, but the primary purpose of the evaluation was to comment or pass judgement on the efficacy of the program.   It was determined that PLEA was indeed serving their intended population of high risk youth and that they were able to make a positive change in the behaviours of these youth.

The evaluation had many strengths: it was very thorough; it met the needs of the PLEA administration who initiated the process; having an outside agency perform the evaluation prevented bias; and having the support of the local collage students helped to make the project economically feasible.  

However, a weakness in the evaluation was created by the low completion rates of the second and third surveys.  Only 40% of the youth initially surveyed completed the final survey; it is possible that the youth that did not complete the survey did not present improved behaviour. The completion of the final surveys by less than half of the participants could have led to inaccurate conclusions. Although the evaluation answered the question of program efficacy in the short term, the conclusions drawn were generalized, which left me wondering about the actual long term effects of the program.

The very positive way in which the evaluation report was written left me once again wondering what it would have been like if I had interviewed for and accepted the position working with the PLEA Daughters and Sisters program.  The impact of the program in the lives of young girls struggling with addictions was evident.  Then again, the low rate of survey completion speaks to one of the challenges of being involved with "high risk" youth.