The culminating activity for every good course is a project.
Evaluation Proposal for the Youth Educational Services Program at LA Matheson Secondary
Educational E's Technology
Saturday, April 5, 2014
Tuesday, March 25, 2014
YES Program Survey
Rational:
This survey was designed to be included in my YES program
evaluation proposal. I didn't want to
make the survey too long as teachers time is very valuable and a survey that is
too lengthy will reduce completion rates.
Throughout the revision process, the paper survey was kept to under two
pages or one page printed front to back.
The original survey was sent to four teachers to be pilot
tested. Two teachers responded (spring
break may have been a factor) but I am still waiting for additional responses. As
an interim solution, the survey was also pilot tested by one engineer and one home-maker.
Some options for delivery of the survey include email,
school mailbox, in person, or at a staff meeting.
Teachers feel more open and able to give honest feedback if
the survey is anonymous; therefore, the requested demographic information was
kept minimal. The question regarding number
of teaching years will help the YES staff to see if early service teachers need
more support in understanding the program and the benefits of collaboration. After
testing the survey, this question was made categorical to make it easier to
answer as some teachers forgot the number of years they have been in the
profession and to further protect identity. This question was moved from the beginning
of the survey as it did not catch the interest of the respondents.
The final version is as follows:
How long have you been teaching?
0-5 years
5-10 years
10+ years
The question regarding the knowledge of the number of
students in the classroom receiving support caused a number of participants to
stumble. The question required to much
precision and one respondent answered that they did not know. Another respondent suggested that this question
excluded non-enrolling teachers and counsellors. As the intent of the question was to target
the awareness of students receiving support from the YES program and the
communication that surrounds that awareness, the question was changed to a
categorical multiple choice with the option of identifying as a non-enrolling
teacher.
The final version is as follows:
How many students, that you are aware of, in
your classes this year, receive support from the YES program?
ÿ None
ÿ 1-2
ÿ 3-5
ÿ 6+
ÿ I don’t know
ÿ I do not have any enrolling classes
In an effort to include the non-enrolling teachers’
responses to all questions in the survey, the wording was changed for questions
that referred to “students in your class”
Question number ten in the survey, asking participants if
they agree that the YES program helps students stay in school was poorly
worded. Some people viewed that as a
fact, as it is the overarching goal of the program. The intent behind the question was to see if
there was a correlation between collaboration and the respondent’s perception
of the efficacy of the program. The
question was reworded as follows:
I have seen the YES program help vulnerable students stay in
school.
A short thank you was added to the end of the paper version
of the survey.
Saturday, March 8, 2014
ECUR 809 Assignments #4
Logic Model: Youth Educational Support Program
In
inner city schools, subgroups of the student population are “vulnerable” to
failure or “at-risk” of dropping out of school. It is recognized that students
come to school with non-academic problems that affect their learning these
include hunger, social emotional issues, and conflicts at home; many of these students
lack the resources to address their non-academic needs. The Youth Education Support Program (YES) was
developed to help fulfil this need for students in the Surrey School District.
YES provides services to students in the setting of the high school that they
are attending. The aim of the YES
program is to keep vulnerable youth in school through relationship building.
The
four goals of the YES program as outlined in their program plan are as follows:
·
To
make students aware of their responsibilities to others, themselves and their
school
·
To
facilitate meaningful connections with peers, adults, the school and the
community
·
To
keep vulnerable students in school and improve student behavior
·
To
build cooperative partners with teaching, counselling and administration staff
in order to improve student achievement and attitude in the classroom
The
YES program began more than ten years ago out of a partnership between Henry D.
Sheldon High School in Eugene, Oregon and Princess Margaret Secondary in
Surrey, BC. At that time YES was a pilot of an interagency model that was
designed to connect practicum students from local post-secondary institutions
with a high school site for volunteer experience and field study. The use of the practicum students in the
program delivery helps to keep the cost of the program low. The YES program has continued to evolve since
its conception and is currently being offered at 5 inner city schools in
Surrey.
The
YES program is delivered out of a portable situated on the school
property. Before school, at lunch, and
after school students can drop in to the portable where they connect with the
YES coordinator or practicum students. Teachers,
counsellors and administers can refer students to the program; the YES staff then
initiates a connection with the student at some point in the school day and
works to develop a relationship with the student through various activities
over time. The YES staff facilitates
many activities at the school, lunch programs, special events, community
referrals, drop in games, and one-to-one supports.
This
evaluation will focus on the effectiveness of the collaboration between the YES staff and teachers at the LA Matheson
site. Is this partnership helping students experience success and ultimately helping them to stay in school? How can the collaborative partnership be improved?
ECUR 809 Assignment #3
In inner city schools, subgroups of
the student population are “vulnerable” to failure or “at-risk” of dropping out
of school. It is recognized that students come to school with
non-academic problems that affect their learning these include hunger, social
emotional issues, and conflicts at home; many of these students lack the
resources to address their non-academic needs.
The Youth Education Support Program (YES) was developed to help fulfil
this need for students in the Surrey School District. YES provides services to
students in the setting of the high school that they are attending. The aim of the YES program is to keep
vulnerable youth in school through relationship building.
The four goals of the YES program as
outlined in their program plan are as follows:
·
To
make students aware of their responsibilities to others, themselves and their
school
·
To
facilitate meaningful connections with peers, adults, the school and the
community
·
To
keep vulnerable students in school and improve student behaviour
·
To
build cooperative partners with teaching, counselling and administration staff
in order to improve student achievement and attitude in the classroom
The YES program began more than ten
years ago out of a partnership between Henry D. Sheldon High School in Eugene,
Oregon and Princess Margaret Secondary in Surrey, BC. At that time YES was a
pilot of an inter-agency model that was designed to connect practicum students
from local post-secondary institutions with a high school site for volunteer
experience and field study. The use of
the practicum students in the program delivery helps to keep the cost of the
program low. The YES program has
continued to evolve since its conception and is currently being offered at 5
inner city schools in Surrey.
The YES program is delivered out of a
portable situated on the school property.
Before school, at lunch, and after school students can drop in to the
portable where they connect with the YES coordinator or practicum
students. Teachers, counsellors and
administers can refer students to the program; the YES staff then initiates a
connection with the student at some point in the school day and works to
develop a relationship with the student through various activities over time. The YES staff facilitates many activities at
the school, lunch programs, special events, community referrals, drop in games,
and one-to-one supports.
This assignment will focus on
developing an evaluation design for the LA Matheson site. According to the program coordinator, the
program has never been formally evaluated.
Planning a Program Evaluation: Worksheet
University of Wisconsin-Extension • Cooperative Extension • Program Development & Evaluation
UW-Extension provides equal opportunities in employment and programming, including Title IX and ADA.
© 2006 by the Board of Regents of the University of Wisconsin System http://www.uwex.edu/ces/pdande
Steps in Program Evaluation
Manage
the evaluation
§
Human
Subjects Protection
§
Timeline
§
Responsibilities
§
Budget
|
1.
Engage stakeholders |
||||
2.
Focus |
|||||
§
Describe
program-logic model
§
Define
purpose
§
Determine
use/users
§
Determine
key questions
§
Select
indicators
§
Determine
design
|
3.
Collect data |
||||
§
Identify
sources
§
Select
method(s)
§
Pilot
test
§
Set
schedule
§
Determine
sample
|
4.
Analyze & interpret |
||||
§
Process
data
§
Analyze
§
Interpret
data
§
What
did you learn?
§
What
are the limitations?
|
5.
Use |
||||
§
Share
findings and lessons learned
§
Use
in decision making
§
Determine
next steps
|
|||||
Standards of
evaluation:
· Utility · Feasibility · Propriety ·Accuracy |
Engage Stakeholders
Who should be
involved?
Program
Coordinator, Practicum students, Teachers/Counsellors, School Administration,
Students, School Board Members
|
How might
they be engaged?
The
stakeholders, namely the program coordinator, practicum students, teachers, counsellors, and administration will be engaged early on in the evaluation
through discussions, conversations and emails. Working closely with the program
coordinator will help to ensure that the evaluation targets the right areas
and that the results will be utilized.
The teachers, counsellors and administration will be participants in
the data collection process; they will help steer the evaluator to the means
of data collection that will best suit their needs.
|
Focus
the Evaluation
What are you going to
evaluate? Describe program (logic
model).
See logic model (Assignment #4) – The evaluation will focus on the effectiveness of the
cooperative relationships formed between YES support staff and other staff at
the school in improving student achievement and attitude
|
What is
the purpose of the evaluation?
The
purpose of the evaluation is to provide formative feedback to improve the
program delivery and increase the effectiveness of the program activities
towards the goals set out. As a part
of the evaluation process, the evaluator will work with the program staff to
develop systems for information management that can have long-term use. This evaluation will examine the
collaboration between the YES staff and other school staff.
One
of the secondary purposes of the evaluation process will be the development
of a deeper understanding of the program by teachers and funding stakeholders
such as the board members.
|
Who will use the
evaluation? How will they use it?
Who/users
|
How
will they use the information?
|
Program
Coordinator
|
The
information obtained will help the program coordinator improve the delivery
of the program activities.
|
Teachers/Counsellors/Administrators
|
The
information obtained will help these stakeholders make better use of the
service.
|
School
Board Members
|
The
information obtained will help the funders make decisions about the
allocation of funds to the program
|
What questions
will the evaluation seek to answer?
How
can the communication between YES staff and teachers be improved?
|
Sub questions:
How
do the YES staff communicate with teachers, counsellors and administration?
Is
the communication between the YES staff and the teacher, counsellors, and
administration effective?
What
percentage of teachers is in collaboration with the YES staff?
|
Is
the collaboration between teachers and YES staff helping to meet the at risk
students’ needs?
|
Sub questions:
Which
students are participating in the YES program?
What
are the characteristics of an “at risk” student? What are the students’
needs?
Who
else is supporting the students’ non-academic needs?
|
What information do
you need to answer the questions?
What
I wish to know
|
Indicators
– How will I know it?
|
How
can the communication between YES staff and teachers be improved?
|
Qualitative
Data - Observations, Focus group discussions
Quantitative
Data - Surveys completed by the teachers
Look
for possible alternative/additional methods of communication
|
Is
the collaboration between teachers and YES staff helping to meet the
students’ needs?
|
Qualitative
Data - Observations, Case studies, Interviews
Quantitative
Data - Surveys completed by the students, students marks pre-YES intervention
and students marks post intervention
Define
students’ needs and examine if the collaboration between teachers and YES
staff provides for those needs
Needs
being met could be demonstrated by
-
increased attentiveness in class
-
improved academic standing
-
improved classroom
Observations
of students before YES support and after YES support
|
When is
the evaluation needed?
This
particular evaluation will need to be completed by the end of the current
school year but one of the goals of the evaluation is to provide a means of
building data collection for on-going formative evaluation into the program. This might look like a school based review
every two years, overseen by an outside evaluator in a supporting role.
|
What
evaluation design will you use?
An
interactive, participant oriented evaluation approach will be used. Strategies and theories from both
responsive and monitoring evaluation models will be applied.
|
Collect the information
What
sources of information will you use?
Existing information:
|
Very
little formal data exists. Part of the
purpose of the evaluation will be to help the program staff set up an
information management system. There
is currently a collection of surveys filled out by former practicum students
providing an evaluation of their supervision by the YES coordinator and their
experience in the program.
|
People:
|
Interviews,
surveys and focus groups will be the primary means of data collection.
|
Pictorial records and observations:
|
The
evaluator will directly observe the program activities and perform case
studies of the participants.
|
What data collection
method(s) will you use?
x
|
Survey
|
x
|
Document review
|
x
|
Interview
|
Testimonials
|
|
x
|
Observation
|
Expert panel
|
|
Group techniques
|
Simulated problems or situations
|
||
x
|
Case study
|
Journal, log, diary
|
|
Tests
|
Unobtrusive measures
|
||
Photos, videos
|
Other (list)
|
Instrumentation: What is
needed to record the information?
Survey
tools will need to be created
|
Information
management system – track the students and list the services provided.
|
When will you collect
data for each method you've chosen?
All
data will be collected during the program, as the program does not have a
definitive beginning or end.
Will a sample be used?
No
|
x
|
|
Yes
|
If
yes, describe the procedure you will use.
|
Analyze and Interpret
How will the data be analysed?
Data analysis methods:
|
The
majority of the data collected will be qualitative and based on observation a
thematic model will be used. Basic
statistical analysis will be performed on any quantitative data.
|
Who responsible:
|
Evaluator
in collaboration with the Program Coordinator
|
How will the information be interpreted —
by whom?
The
qualitative data will be summarized by the evaluator using a thematic model.
|
The
information gained by the evaluation process will help the stakeholders to
better understand the YES program, specifically, how the relationships
developed between YES staff, teachers and counsellors helps students in the
classroom.
|
What did you learn? What are the
limitations?
As
the evaluation progresses the evaluator will work closely with the
stakeholders to determine their needs and steer the evaluation efforts;
although this is one of the strengths of responsive evaluation it is also a
limitation.
|
The
qualitative data collected will provide an opportunity to gain new insights
but it may also be viewed as subjective; the utility provided will outweigh
this limitation.
|
The
evaluation will rely heavily on qualitative data which has the potential to
be time consuming.
|
Use the Information
How will the evaluation be communicated
and shared?
To whom
|
When/where/how to present
|
YES
coordinator, administration and counsellors
|
The
evaluation results will be presented/discussed in a meeting. The insights gained by the evaluator will
be shared and a list of recommendations will be reviewed at this
meeting.
|
Teachers
|
The
evaluation recommendations pertaining to the teachers will be written in a
brief report and emailed to teachers.
|
Board
Members
|
Recommendations
requiring funding approval will be communicated to the school board via the
school administration – a written report including qualitative data will be
provided.
|
Next steps?
YES
coordinators, practicum students, administrators, teachers and counsellors
will use the insights and recommendations to improve the quality of delivery
for the program.
|
Manage
the evaluation Standards
x
|
Human subject’s protection
|
x
|
Utility – by design, the evaluation will encourage
utility as it is participant oriented.
The involvement of the stakeholders will ensure the evaluation
produces useful feedback.
|
x
|
Management chart
|
x
|
Feasibility – Valuable information can be obtained using
a simple design. This will increase the feasibility of performing an
evaluation.
|
x
|
Timeline
|
x
|
Propriety
|
x
|
Responsibilities – to be shared between the evaluator and
program coordinator
|
x
|
Accuracy – Multiple data collection methods will ensure
accuracy of the information obtained
|
x
|
Budget – The design will be simple to keep costs down.
|
Subscribe to:
Posts (Atom)