Pictured is Artie Maharaj, PhD. and program evaluator at TSNE with her colleague Romita Mitra, Ph.D, Evaluation Research Associate at the MIT Local Innovation Group leading a session at the American Evaluation Association (AEA) 2023 Birds of a Feather session, In this session, participants had a platform to discuss how they are working towards building a culture of monitoring, evaluation and learning within their organizations.
Over the past seven years, I’ve worked as a program evaluator with a variety of stakeholders: educators, policy makers, community members, students, engineers, researchers, and a variety of leaders at nonprofit organizations. In doing the important work of making change, stakeholders also want to know how their work is making an impact. In addition to their own knowledge, organizations —especially nonprofit organizations are asked by funders to share how their programs are making a difference for the people and communities they are working alongside and on their behalf.
I’m excited to share a few tips for how to start using evaluation and learning tools at your nonprofit organization so that you can use data to inform and improve your programs and policies. As I embark on my journey as TSNE’s new program evaluator, after working in the education and international development space — most recently at MIT D-Lab, I look forward to sharing tips and resources about how you can utilize evaluation activities and learning practices to understand and share with others how your organization is making an impact in the world.
Why start incorporating evaluation activities into your work?
So, why evaluate? Evaluation can give us important insights into if and how our programs are working. We might want to know what worked well and what didn’t work so well during the course of rolling out a new program. Evaluation helps us understand if our programs are achieving positive and desired outcomes for the people and communities we are serving.
In addition to understanding if and how our program works, funders and partners want to know if programs are making an impact within specific populations and communities. Results from tracking outcomes and activities can help communicate the type of impact our programs are making in the communities and groups we are serving. Further, learning activities can help a team identify lessons learned and provide an opportunity for us to contribute our learnings with our partners and within the nonprofit sector. Overall, evaluation and learning activities can help us to identify what’s working well and what changes we might need to make to improve our programs and impact the people and communities we are focused on serving.
So, how do I start incorporating learning and evaluation activities?
Assemble a team and get input from key stakeholders (and along the way)
Before developing and implementing an evaluation and learning plan it’s important that you assemble a team that can support this work as well as get input from key stakeholders. Having an opportunity to learn why this program is important and how your program has made an impact can help identify the type of evaluation that may be most useful and well the types of information that would be most useful to the team and community. Your team might come together and start answering the following questions in a work session:
- How might an evaluation and learning activities support our work?
- What are we most interested in learning from implementing evaluation and learning activities into our work?
- What are we currently doing now related to evaluation and learning that we can build upon?
- Whose voices are we missing from our current planning discussions?
Map your program with Change Theory tools, such as a logic model
A logic model is a visual road map outlining how your program works. While there are many variations of logic models, in its essential form a logic model comprises of these following components: resources, activities, outputs, and outcomes. The logic model represents what resources we need to implement activities and what results (changes in attitudes, knowledge, behaviors) we hope participants of our programs will have by participating in our program activities. Although logic models appear to follow a linear path, they are flexible and sometimes people start at the end by defining their outcomes and impact and then work backward to identify the inputs, activities and funding needed for their programs. As you start working with your team to develop a logic model, here are a few questions you might want to brainstorm together before you draft your first logic model:
- Do we have multiple perspectives at the table to draft this logic model?
- How do we envision ourselves using this logic model?
- How would we know that our program is successful?
- What conditions are necessary for this program to work?
Scope your evaluation with guiding evaluation questions
Once you have developed your logic model, you can now begin to scope your evaluation by identifying a few key evaluation questions that will guide your work. It is best to develop evaluation questions in collaboration with members of your team that will use the findings to inform their work or practice. For example, are you interested in understanding how the program is being implemented or is it more about what outcomes are being achieved (Process Evaluation vs. Outcome Evaluation)?  There are so many questions you can ask but the goal here is to start with 2-4 key questions that will inform your team’s work. Here are some examples of evaluation questions you might want to answer about your program:
- To what extent is the program having a positive impact on participants?
- To what extent did the course deliver the content students expected?
- To what extent do trainees report changes in confidence?
- Did participants learn new skills?
- Did the participants benefit from the program? How did they benefit?
After mapping out evaluation questions, the next step is to figure out how best you can answer those questions by mapping out your metrics. The Barr Foundation has developed a great resource on how to develop metrics for your outputs and outcomes within your logic model. Overall, the above three activities will prepare you for developing a focused evaluation and learning plan that can help you begin your journey toward greater impact.
Ready to get started?
The resources below will help you start implementing an evaluation + learning practice at your organization.
Comprehensive evaluation resources
- Community Tool Box  is an online resource developed and managed by the University of Kansas Center for Community Health and Development for those working in community development and social change. Chapters 36- 39 Evaluating Community Programs and Initiatives
- Better Evaluation Online knowledge platform focused on evaluation practices
- University of Michigan collection of evaluation resources and tools
Developing logic models and metrics
- How to create a logic model (Free mini course on Coursera: 1.5 hours)
- Data Playbook: A framework for collecting data and using it to inform action at your organization
Evaluation approaches that center equity
And finally, please check out TSNE’s blog archives on learning and evaluation to learn more!