Featured In-Depth Evaluation
Each year, the Department of Evaluation and Learning (DEL) undertakes at least one in-depth program evaluation.
In general, these evaluations use a “developmental evaluation” (DE) approach, which is well-suited to complex and evolving social programs like those at JF&CS. The process of DE includes determining an evaluation plan that is appropriate for the maturity of the program; collecting, analyzing, and reporting on data jointly with program staff; and determining next steps for program improvement based on evaluation results.
2014: Social Network Analysis of Client Enrollment at JF&CS
For 150 years, JF&CS has been a multi-service, integrated agency. While this was known to clients, staff, and our community partners, we wanted to better understand the extent to which this integration occurs.
Using “social network analysis
” techniques and data on client enrollments from January to December 2013, we looked at both client movement—the order clients enrolled in multiple programs—and concurrent enrollments—the programs in which clients were simultaneously enrolled. The products of this analysis included several visual representations of agency integration:
This analysis showed that approximately 25% of our active clients utilize more than one JF&CS service, demonstrating a significant level of integration. It also showed that our Center for Basic Needs Assistance acts as a connecting hub within the agency. Individuals and families struggling with poverty face many complex challenges, and JF&CS is able to help them by referring clients seamlessly to and from services that will meet their basic needs. This integration of care supports our clients in achieving the stability they need in order to move from crisis to stability.
For more information, call 781-647-JFCS (5327) or email your questions via our contact us page.
2013: Care Coordination and Coaching
In 2013, DEL collaborated with program leadership to complete an evaluation of the JF&CS Care Coordination and Coaching
program. Care Coordination and Coaching provides case management and coaching services to adults with long-term mental illness and autism spectrum disorders in order to help clients meet their basic needs, attain greater stability in the community, and take steps towards recovery. Services provided by Care Coordination and Coaching include one-on-one sessions with a case manager, clinical support, and care coordination and collaboration with other treatment and community-based providers.
Key findings from this evaluation (released in June of 2013) include:
- Care Coordination and Coaching is a faithful implementation of a recovery-oriented approach to psychiatric case management.
- Care Coordination and Coaching meets 88% (43/49) of applicable standards for a recovery-oriented program, as defined by the American Association of Community Psychiatrists, the Case Management Society of America, and by Farkas, Gagne, Anthony, and Chamberlin (2005).
- The program’s philosophy of flexible and individualized case management is well-suited to addressing the complex affective disorders and co-existing conditions prevalent in this population.
- 66% of Care Coordination and Coaching clients presented with one disorder, 23% with two, and 12% with three or more disorders.
- More than half of clients from July 2012 to June 2013 presented with a primary affective disorder, including anxiety disorder, bipolar disorder, or major depression.
- 75% of clients during the evaluation period showed improvement in an average of four quality of life areas. The most common areas of improvement were living situation, work and school, finances, and health.
- Clients and family members provided positive feedback, noting the program’s helpful impact on their lives; the quality of the staff and importance of the staff-client relationship; and clients’ increased access to other programs, services, and benefits as a result of participating in the Care Coordination and Coaching program.
- More than 90% of clients and family members would recommend Care Coordination and Coaching to a friend.
- While clients reported the most helpful benefit of Care Coordination and Coaching as accessing services to meet their basic needs, family members noted increases in clients’ levels of hope and empowerment.
DEL discussed the findings of the Care Coordination and Coaching evaluation with division and program leadership as well as the service delivery team, resulting in several recommendations for future program and evaluation planning, including:
- Explore the possibility of adding peer services and group experiences for clients (e.g. traditional group therapy, learning classes, socialization groups, etc.).
- Review and implement standardized measures in order to understand the needs and experiences of clients even better.
- Improve the quality of data collected about service delivery by re-designing the staff database system to improve the link between program activities and client outcomes.
For additional information about the evaluation of Care Coordination and Coaching, please contact us.
2012: Lauren & Mark Rubin Visiting Moms®
In 2012, DEL and staff in the Center for Early Relationship Support® completed an evaluation of the Lauren & Mark Rubin Visiting Moms® program. Key findings include:
- Rubin Visiting Moms serves a racially and socioeconomically diverse population, reflecting the program philosophy that all new mothers need support and that postpartum depression crosses all boundaries.
- Overall, participants showed a significant increase in maternal self-esteem and a significant decrease in maternal depression.
- Rubin Visiting Moms was particularly effective for Latina mothers.
- The consistency between respondents’ feedback and program goals was compelling evidence that the intervention was effective.
We believe that all evaluation findings must be considered by both evaluators and program staff to guide program planning. The evaluation of Rubin Visiting Moms informed several implementation changes:
- Rubin Visiting Moms leadership developed a separate team of paraprofessionals to work with highly vulnerable families.
- Rubin Visiting Moms provided enhanced training to staff and volunteers to support the unique needs of families experiencing multiple risk factors.
- Rubin Visiting Moms began to conduct targeted development to support Spanish language services for Latina mothers.
- DEL and Rubin Visiting Moms leadership improved the evaluation process by including new data collection tools and more rigorous follow-up to better understand program impact.
Additional information about the evaluation of Rubin Visiting Moms can be found in the Executive Summary or by contacting us.