table of contents
The Child Protective Services Training Program: Evaluation Component
Needs Assessment and Curriculum Development
The Evaluation Unit supplements trainers' first-hand observations of
trainee strengths and weaknesses with information from
both quantitative and qualitative sources. This information consists of
prior training experience, trainee performance
(described below in the Training section), as well as, when appropriate,
summaries of survey data and focus group
interviews. Needs assessment instruments are sent to all staff
development, caseworker and supervisory staff preceding each
contract year to discern course preferences and needs. Census surveys are
also sent to Commissioners for completion.
Analysis of these data enable the Institute to better project the numbers
and needs of the trainee population. Follow-up
questionnaires are developed for each course offered more than 3 times
during any contract period and sent to all trainees (and a
sample of trainee supervisors) who attend the particular course to assess
the impact of training on job performance.
Information collection strategies are developed in close collaboration
with OHRD and the local and regional district offices.
Thus needs are assessed by examining information from a variety of
sources. Identified training needs generate curricular
development and revision.
Training: Instrument Development and Administration
It is important to document the impact of training on learning, on job
performance, and thus the overall delivery of services.
To these ends, the Evaluation Unit works with training staff to modify and
develop instruments which focus on curricular
training objectives developed in collaboration with OHRD. These
objectives are related to the caseworker competencies and
course objectives as developed by the Institute and the NYSDSS.
- Evaluating Training Pre- and post-tests are developed which reflect 1)
the appropriate application of best practices and
protocols, 2) skill acquisition as well as knowledge (and thus a
performance orientation) and 3) the goal of improving
services to families and children. Demographic, closed- and open-ended
reaction questions are also utilized. Traditional
item analyses are conducted to facilitate test development. Pre- and
post-test data are analyzed using a repeated measures
design.
-
Assessing Impacts Impact is measured by examining trainee performance
on a series of job-relevant tasks, developed in
conjunction with OHRD. Performance on these tasks is operationalized by
trainee responses to a series of multiple-rating
items (Scriven, 1991). Multiple-rating items consist of a general
description, e.g. case scenario, child abuse report,
followed by a set of items to be rated. Multiple-rating items are easily
scored and exceptionally suitable for job relevant
assessment because of their task specificity and their adaptability to
higher order cognitive tasks. Specific items and tasks
are related to the specific courses taught by the Institute. Examples are
as follows:
Is this child at risk? After reviewing a case description, trainees rate
the appropriate risk level using the scales in the Initial
Risk Assessment and Service Plan.
How do you conduct a CPS investigation? After examining an allegation,
trainees rate the relevance of particular
investigatory steps.
These data are supplemented with qualitative interviews. In particular,
caseworkers who participate in the training are asked
the same questions asked in the initial survey. Differences in impacts
are examined relative to post-test performance and
other variables of interest. Assessments are made by triangulating among
the different sources of information. The
evaluation plan itself is reviewed, evaluated and modified as necessary.
Pre-test data and information on trainee demographics is collected
immediately prior to training. Post-test data and trainee
reactions, both close- and open-ended, is collected immediately after
training. Impact assessment is conducted at 2 intervals,
3 and 6 months after training completion.
Data Management
A central data-management responsibility is maintaining appropriate
subject confidentiality: aggregated mastery scores and
demographics are shared with OHRD, along with Core trainee attendance to
ensure successful achievement of the 3-month
legislative mandate for all new CPS workers. It is also important though
to meet OHRD's specific information needs. To
this end, the Evaluation Unit provides OHRD with electronic versions of
trainee and course data. The Evaluation Unit also
continues to develop a data management system sufficiently flexible to
meet OHRD's increasingly sophisticated electronic
information requests with minimal lag time.
The Evaluation Unit is in the process of developing an electronic system
for tracking trainee participation and responses and
reporting requested information to the OHRD. The system is also being
designed to streamline research and evaluation efforts
for internal use. This system will store and process information regarding
the courses offered by CPSTI, tracking course
registration, participation, demographics and reaction responses, in
addition to grading and storing trainee test responses. The
output of this system will be used in quarterly and annual reports, and
the information that is recorded about courses taught
(list of training courses) and participation (rosters) will be passed
along to the State electronically.
In addition to record keeping for administrative purposes, the system will
provide a means of recording information about
demographics, participation, and test and impact responses of individuals
and groups, so that research can be conducted where
appropriate on this information, as part of CPSTIMs overall effort to
improve the quality of training and refine and improve its
training techniques. For example, it will be feasible to single out
questions which are often answered incorrectly, and
determine whether the question is clearly stated on the test or whether
more attention needs to be given in training. An
efficient data management system will reduce the amount of work required
to produce reports and ask research related
questions, in addition to facilitating the administration and grading of
pre- and post-tests.
Reports
The Evaluation Unit prepares quarterly and final reports which are
submitted to OHRD 20 days after the end of each quarter;
60 days after the end of the contract year. Quarterly reports contain
both a cumulating list of training courses and summaries
of trainee demographics, reactions, and the percentage answering at least
70 percent of the post-test items correctly. For each
training session, OHRD is also provided with the pre- and post-test data
summary it has previously requested: means, standard
deviations, 'average percent scores' [means divided by the number of test
items], and 'average pre-/post-test change' [pre-test
'average percent score' subtracted from that of the post-test]. Each
quarterly report is also accompanied by written and
electronic versions of trainee rosters and the list of training courses.
Information provided in the quarterly reports is
aggregated in the final report. This report also presents results from
the impact analyses.
For newly developed or courses in progress, the Evaluation Unit prepares a
separate report describing the results of any initial
research components (e.g., the Intra-Familial Sexual Abuse Dynamics and
Intervention Course, the Foster and Adoptive
Parent Crisis Intervention and Restraint course). The purpose of these
reports is both to summarize data and to disseminate it
(to OHRD and to interested districts, agencies and facilities) including
particular information about best practices and model
approaches that can be used in developing the training curricula. These
reports are designed to provide a forum for discussion
during curricula and training development.