|
|
Agenda
Evaluation Basics for Those That
Think They're Not Evaluators
|
February 6, 2007
|
|
9:00am- 10:30am
|
Introduction to Evaluation Basics
Presenter: Barri Burrus, Ph.D., RTI International
Descriptor: Participants will introduce themselves and learn the role of evaluation in a care demonstration project, why and how demonstration projects are evaluated, and definitions of basic evaluation terms. This will be a plenary session with opportunities for interaction with fellow participants.
The speaker will address the following:
- Grant requirement for independent evaluation
- What can be learned from program evaluation
- Consequences of not evaluating programs
- Basic evaluation design concepts
- Basic data collection and evaluation terms
- Special Note: Participants may be asked to complete a brief pretraining assignment that could be used in this session.
Learning objectives-Participants will be able to:
- Articulate the grant requirement concerning evaluation
- Describe benefits of program evaluation
- Define basic evaluation terms
|
|
10:30am-10:45am
|
Break
|
|
10:45am- 12:15pm
|
Using the Core Evaluation Instruments
Presenter: Olivia Silber Ashley, Dr.P.H., RTI International
Descriptor: Demonstration projects funded in October 2004 and more recently are required to use core evaluation instruments. In this plenary session, results from an assessment of programs' capacity to use these instruments will be discussed, including problems identified, solutions, and best practices for data collection staff.
The speaker will address the following:
- Background on why core evaluation instruments are required
- Capacity assessment methods
- Capacity assessment results
- Revisions to the core evaluation instruments in response to capacity assessment findings
- How participants can apply capacity assessment lessons learned in implementing the core evaluation instruments
Learning objectives-Participants will be able to:
- Discuss how core evaluation instruments relate to program performance measures and upcoming cross-site evaluation
- Describe improvements to the core evaluation instruments and implement these improved instruments
- Apply lessons learned from the capacity assessment in their own data collection activities
|
|
12:15pm-1:15pm
|
Lunch on your own
|
1:15pm-2:45pm
Concurrent Sessions
|
Group 1: Process Evaluation Data Collection
Presenter: Lisa Lieberman, Ph.D., Healthy Concepts
Descriptor: All AFL grantees are required to conduct a process evaluation to describe program delivery. This session will discuss what process evaluation data participants may be asked to collect, how to collect these data, and why measuring intervention dosage is important. This will be an interactive workshop with hands-on learning opportunities and participant involvement.
The speaker will address the following:
- What process evaluation means, grant process evaluation requirements, and why process evaluation is important
- Process evaluation measures and how these are collected (e.g., demographics of the population being studied, attendance data, program delivery adherence to implementation guidelines, intervention dosage, participant feedback/satisfaction surveys)
- Qualitative and quantitative process evaluation data collection
- Consequences of not collecting appropriate process evaluation data
- Special Note: Participants may be asked to complete a brief pretraining assignment that could be used in this session.
Learning objectives-Participants will be able to:
- Define what a process evaluation is
- Describe grant requirements regarding process evaluation
- Discuss benefits of process evaluation
- Identify how their program measures process
Group 2: Avoiding Bias
Presenter: Mandy Sha, M.A., RTI International
Descriptor: As an AFL Care Demonstration Program, each project should have a strong evaluation. Differences in data collection for the intervention and comparison group participants, as well as response bias and social desirability bias, can undermine the quality of the evaluation. This session will focus on collecting data using unbiased procedures, using exercises and role plays.
The speaker will address the following:
- Bias related to communication and research
- How and why to avoid bias in data collection
- Maximizing similarities in data collection for intervention and comparison group participants
- Challenges and solutions
- Special note: Participants may be asked to complete a brief pretraining assignment that could be used in this session.
Learning objectives-Participants will be able to:
- Describe sources of bias
- Describe how program staff who collect data can bias responses
- Apply best practices to improve data collection and evaluation quality regarding these issues
Group 3: Confidentiality
Presenters: Barri Burrus, Ph.D., and Kimberly Leeks, Ph.D., RTI International
Descriptor: This presentation will address data collection procedures with a special focus on anonymous and confidential data. Keeping information confidential, increasing respondents' perceptions of confidentiality, and coding and storage of confidential information will be discussed. This will be an interactive workshop with opportunities for discussion and structured activities.
The speaker will address the following:
- Best practice data collection protocol (based on the proposed protocol for the cross-site evaluation)
- Confidentiality, handling sensitive data, and data collection protocols that increase respondent perceptions of confidentiality
Learning Objectives-Participants will be able to:
- Differentiate between anonymous and confidential data
- Identify ways to ensure confidentiality of the data
- Apply best practices in maintaining confidentiality, ensuring that data are coded and stored to protect individual identities, and increasing respondents' perceptions of confidentiality
|
|
2:45pm-3:00pm
|
Break
|
3:00pm-4:30pm
Concurrent Sessions
|
Group 1: Confidentiality
Presenter: Barri Burrus, Ph.D., and Kimberly Leeks, Ph.D., RTI International
See earlier description
Group 2: Process Evaluation Data Collection
Presenter: Lisa Lieberman, Ph.D., Healthy Concepts
See earlier description
Group 3: Avoiding Bias
Presenter: Mandy Sha, M.A., RTI International
See earlier description
|
|
February 7, 2007
|
|
8:45am-10:15am
|
Communicating with Your Evaluator
Presenter: Lisa Lieberman, Ph.D., Healthy Concepts Research, Inc.
Communication between data collection and evaluation staff is key to rigorously assessing the effectiveness of an intervention. And team communication is critical to the evaluator's ability to accurately report the data collection activities that occurred. The presenter will discuss important issues related to conveying information to your evaluator, working with the evaluator to solve data collection problems, and end of year reporting requirements for evaluators and how data collection staff communication affects the quality of this reporting. This interactive workshop will include opportunities for group discussion and structured activities.
The speaker will address the following:
- End of year evaluation reporting requirements and their importance to the AFL program's vitality
- How data collection staff affect evaluation quality
- An ideal model for communication between data collection staff and evaluation staff, including incident reporting
- Common data collection problems (including low response rates) and how to problem solve with the evaluator
Learning objectives-Participants will be able to:
- Discuss how their data collection activities are linked to the broader landscape of evaluation quality, end of year evaluation reporting requirements, and AFL program performance measures
- Identify ways they can improve communication with their evaluator
- Describe situations that warrant a written incident report for the evaluator
|
|
10:15am-10:30am
|
Break
|
|
10:30am-12:00pm
|
Recruiting and Retaining Adolescents in the Evaluation
Presenter: Constance Wiemann, Ph.D., Department of Pediatrics, Baylor College of Medicine
Descriptor: This plenary session presents eight key strategies learned in the recruitment and retention of 16- to 21-year-old urban women participating in a 12-month randomized clinical trial of a reproductive health program. Implications for programs serving pregnant and parenting adolescents will be discussed. Participants will be provided opportunities to discuss and brainstorm possible applications of strategies presented.
The speaker will address the following:
- Successful recruitment and retention strategies used with young minority women in community-based intervention research
- Educating clinic staff on the rigors of study design
- Facilitating a team effort between clinical and research staff and modifying recruitment procedures, as needed
- Providing prospective participants the option of enrolling by return appointment
- Anticipating a diminishing recruitment pool over time
- Setting a positive recruitment tone at the beginning of each clinic session
- Considering participants' mothers as important points of contact
- Matching communication styles to participant contacts
- Considering a variety of retention techniques
Learning objectives-Participants will be able to:
- Identify at least five strategies for recruiting and retaining adolescents in the program and its evaluation
- Use adolescents to guide recruitment and retention practices
- Apply these strategies to improve recruitment and retention
|
|
12:00pm-1:00pm
|
Lunch on your own
|
|
1:00pm-2:30pm
Concurrent Sessions
|
Group 1: Avoiding Bias
Presenter: Mandy Sha, M.A., RTI International
See earlier description
Group 2: Confidentiality
Presenter: Barri Burrus, Ph.D., and Kimberly Leeks, Ph.D., RTI International
See earlier description
Group 3: Process Evaluation Data Collection
Presenter: Lisa Lieberman, Ph.D., Healthy Concepts
See earlier description
|
|
2:30pm- 2:45pm
|
Break
|
|
2:45pm-4:15pm
|
Integrating the Data Collection Process, Program Evaluation, and What You Have Learned
Presenter: Olivia Silber Ashley, Dr.P.H., and Barri Burrus, Ph.D., RTI International
Descriptor: Participants will learn how the data collection process fits into the overall evaluation quality, evaluation performance measures for OAPP, evaluation report reviews, other DHHS evaluation review criteria, and publication and dissemination of evaluation findings. This will be an interactive workshop that will provide an opportunity for participants to comprehensively assess the information they have learned throughout the training and develop concrete plans directed at translating this information into action-focused strategies that can be applied in their field settings. Interactive group discussions will encourage sharing of information and applications with other participants. Any unmet training needs will be reviewed as well. A synthesis process will be developed in which grantees will discuss their groups' applications and identify any broader unmet needs that could be addressed through technical assistance.
|
|
