Evaluation as an Intervention

Today is the day.  Many of you know that I’ve been hinting at this for years, but today is the day that I make my own mark in the sand and tell you about my own “Theory of Evaluation”.  Please note, that much like other theories, mine is an evolution, not a revolution.  I don’t think I’m going to say anything here that is too controversial - unless you really subscribe to a certain viewpoint (which we will get to later), but I thought it was time.

Evaluation as an Intervention

A few blog posts back, I walked you through my growth as an evaluator - starting with a class in organizational development and working to try to understand collaboratives.  There, I introduced you to the idea of evaluation as an intervention - speaking about the engagement of evaluators in program design and use of evaluation data to inform evolution of programs.  Today, I want to make that case stronger and speak more about why I think it is important.

Evaluation is dead?  I don’t think it is, just evolving.

In April of 2014, Andrew Means posted on the page, Markets for Good a controversial post - “The Death of Evaluation” (www.marketsforgood.org/the-death-of-evaluation/).  In it, he takes to task “traditional evaluation” and makes some rather bold statements:

• Traditional program evaluation is reflective, not predictive
• Program evaluation actually undermines efforts to improve
• Program evaluation was built on the idea that data is scarce and expensive to collect

The posting created a bit of a buzz in the nonprofit world as well as the evaluation world.  I heard a great deal of grumbling from fellow evaluators - “how dare he declare this!?!?!?!”  On the nonprofit side, I heard some quiet, “amen”s.  Neither surprised me.  Nor did it surprise me when I talked with a few funders and heard their thoughts - frankly they were as mixed as the nonprofits.  You see, there are still organizations out there, that unlike Mr. Meers, still want to prove (or at least in a more generous viewpoint, find out if) their program worked.  We’ll get to that in a bit, but more interesting is the trend of organizations that want to learn and move forward.  

To fail forward, you need to know why you failed.

In the world of Collective Impact, StriveTogether has promoted “failing forward”.  The only way to fail forward is to had information that the failure occurred in the first place as well as information as to why the failure occurred.  Frankly, in the areas of manufacturing and healthcare, folks are already way ahead of the nonprofit world - concepts such as ISO standards that ensure consistency of work and Six Sigma for failure assessment have been around for quite a while.  We also hear about performance improvement and quality improvement from these areas as well as others.  They have learned that a successful organization needs to be moving forward and improving to be successful.  Incidentally, they all do a great job of marketing around these concepts as well.

Tradition isn’t all it’s cracked up to be.

So, why does “traditional program evaluation” exist?  Why do evaluations that assess over a longer time with the program held constant (or perhaps affected by the environment) still get done?  I can think of a number of reasons, some perhaps good, some not, depending on your values.  Because frankly, it is the purchaser and user’s values that should be affecting what evaluation gets done.  Here are some of those reasons:

• Intent to generalize the program to other environments - here people just want to know how the program works
• Desire to market the program to others or to the current funder - the interest is in sales of the program, not changing it
• Accountability focus - did the program do what it said it was going to do?

“Traditional evaluation” as Mr. Meers calls it, isn’t focused on learning - or at least incremental learning.  It is focused on the “big picture” - did the program do what we thought it was going to do?  As such, it does have its place.  The question is whether that “big picture” is valued or something else.

Programs evolve.  Why can’t evaluations evolve as well?

In my own thoughts on this, I came to an interesting observation.  Programs do not stay put for long.  They tend to evolve.  So, the “big picture”, long-term evaluation project becomes more and more difficult to conduct, because the maintaining a program as it was designed over years becomes more and more difficult to do as well.  While possible, it just isn’t likely that a program will stay exactly the same.  This is one of the reasons why my cousins who focus just on social science research view evaluation as something “dirty”.  They see it as having less control (the friend of research) of variables and as such, the work isn’t as pure.  It is also why researchers try to boost the rigor of an evaluation project - to try to clean up the dirt as much as they can.

But social programs and innovation specifically is very dirty.  There are changes all the time and Mr. Meers, because of his focus, doesn’t value the purity - he values a program that produces results and craves better results (sorry for putting words in your mouth Mr. Meers - I hope I’m right).  And so, I would argue it is time to seriously consider evaluation as an intervention.

Evaluation can’t be predictive, but it can support efforts to improve and by the way, it can use much of the data that is sitting around unrecognized and thus unused.  The trick is to embrace the dirt - such things as local context, environment, and change.  The cycle of collection and reporting also might shift a bit - depending not on the length of a grant or program goals, but more around expectations for change for shorter-term outcomes.  These “performance measures” and the hypotheses associated with them (yes, we are still taking science here) are the information that constitute the intervention.  They are often directly tied to service delivery or at least more closely associated with it.  And, they are meant to be shared with and used by decision makers as things go.  The real intervention occurs where the information is used and the stakeholders adjust the programming.  Another key hallmark of this work is that measures also do not stay constant over time.

Evaluate the system and program as it is, not as it “should be”.

Did I just surprise you?  I think I did!  I actually promoted the idea of dropping and/or changing measures over the duration of the evaluative work!  This isn’t how most evaluation work is done.  Michael Quinn Patton pokes a bit at this with Developmental Evaluation, but I’m saying that no measure is sacred - all can go.  Remember, the goal of this work isn’t to test a program’s impact, or generalize, or for that matter look for accountability - it is to improve the program.  As certain criteria are met and the program evolves, it is quite possible that some, if not all measures, will be replaced over the duration of the evaluation.  Measures that were meaningful at the beginning, become less meaningful.  This addresses Mr. Meers' concerns around evidence of performance and success - if (and it is a very big if) the stakeholders are still truly interested in improvement.  It isn’t the measures or measurer that determines whether a leader is content with their work - rather they only provide evidence for that leader to make an informed decision.  However, if the evaluation is designed to run on rails and get to a specific destination (and only that destination), pack your bags, for that is where it will wind up.

There are some rules to this.

And so, as we think of Evaluation as an Intervention, we have a few “rules” to consider (think of these as decision marks).  Evaluation as an Intervention:

• Is intended for improvement of programs and systems
• Is intended for learning
• Isn’t intended for accountability assessment
• Isn’t intended for those specifically interested in generalizability of their program
• Isn’t intended for “big picture” assessments or marketing
• Can be inexpensive (we will talk more about that in later blog posts)

Is this for everyone?  No, only if you need it.

As for Mr. Meers, I appreciate his view.  It is shared by many, but not by all and it is important to understand the needs.  It boils down to what information is important and to whom.  In the case of Evaluation as an Intervention, it is just that, recognizing and using evaluation to improve a program or system - not just document it.  And, as with all theories of evaluation, there are moments with one theory is more useful than another.  In my own practice, I engage multiple theories to support my work - I just felt that it was time to really define this one and give it a name.

I’ll be writing about this more in the near future as I work through this theory more.  As always, I’m open to your comments, suggestions and questions.  Please feel free to post comments.

Best regards,

Charles Gasper

The Evaluation Evangelist

My Return as a Blogger and My Life as an Evaluator

I was reading through my previous Blog posts over the past few years and I came across a post I made in November 17, 2010 – the title?...  Wait for it…  My Return.  There it had been over a year since my last post.  Well, once again, I’m writing about my return and once again, it has been over a year.

Changes
Today, I shared with many of you a change that occurred in my life.  I had been considering doing something like this for well over a year and finally decided it was time to switch things.  Earlier this month, I started working for the TCC Group.  TCC works with Foundations, Nonprofits, and Corporate Giving.  The institution has over 30 years of experience in the areas of strategy, grants management, capacity building and evaluation.

I was thinking it was a good idea to write about where I have been for the past year (really years) and why I haven’t written.  Then I realized – I have been working as an evaluator for over 20 years, have I ever really told you why I got into the business?

Have a seat, I’m going to share a bit here about how I got here.

Setting the Hook
In 1991 or so, I was an undergraduate student at Santa Clara University.  There, I was working with someone who would greatly influence my life, Dr. William McCormack.  One of the courses was on organizational development and we were required to work with an organization, writing a case study.  That experience drove me to think more about organizational effectiveness and impact – but it took a few more steps to put me on my path.

There was some additional evaluative work done, but in 1995 I found my love.  I was working on a project to assess the impact of a program on the quality of life of the individuals.  When the evaluation was completed and we submitted our findings, the state took our work and changed the program – improving the lives of the people served.  Thousands of people were impacted by my work!  I had found my calling.

Learning that Learning is Important
Along the way, I worked in quality management for a health system.  This experience molded my view on evaluation further, evolving it away from summative, value-focused work to learning and improvement.  Our work improved the experience and health outcomes for patients in hospitals and also resulted in organizational savings for the health system.  We weren’t conducting long-term studies, but instead focusing on short-term outcomes that predicted long-term success.  The mantra of the day for organizational change was, “what can we get done by next Tuesday?”

Movement to Large-Scale Impact and Collaboration
In 2007, I became a Director of Evaluation for a large health-focused foundation, the Missouri Foundation for Health (MFH).  It was time to think differently again – or perhaps better said, to broaden my thought.  Prior to coming to MFH, my focus was on program evaluation (even if the programs were larger).  Now it was time to see how multiple programs (yep, was still focused on programs) could interact with one another to affect larger scale impact.  During my stent at MFH, I also returned to graduate school and while my evaluative practice was informed by program theory, it truly shifted to being theory-based (I’ll talk about that in a later blog).  I started to think about systems in a broader sense, not only seeing how an individual program interacted with a larger system, but how systems change can affect improvement across broad swaths of issues.

Evaluation as an Intervention
About this time I started thinking about evaluation differently.  I recognized that programs and systems evolve over time and that evaluation can better support the effort if it doesn’t stand completely separate.  Provision of shorter-term information tied to program theory can better inform the evolution of programs and identify where the efforts are being effective.  There has been a shift in AEA in recent years, now recognizing that evaluators can be and some think should be involved in program design.  My epiphany in 2010 was my tied to my recognition that my role as an evaluator in a foundation should support such work.

Collaborative Impact
Most recently, I’ve focused on how collaboratives are built and the results of the collective effort.  I’ve been assessing how these efforts combine with and attempt to change simple and complex systems.  You can think of these as multiple programs working in concert.  In reality, the concert often sounds like a bunch of instruments tuning up versus following a score.  My work has focused on how to get the instruments to play together in the same hall, follow an agreed upon score and perhaps follow the baton of an agreed upon conductor.  (We will revisit this analogy in a later Blog post.)  There is a great deal to learn about collaboratives.  Certainly, this is something that has been done for years.  Folks have given it different brand names, but really it is just about learning how groups of people, organizations, civic leaders, and communities can come together to affect systems change.

Which Brings Us to Today
The process of moving to the TCC Group made me contemplate my practice as an evaluator.  In the interest in keeping this post relatively short, I only shared some of the revelations I made.  My journey reflects two key things, 1) my own personal growth in the field and 2) evolution in the field of social change.  To be clear, programs are still important.  Valuing the impact of those efforts is also very important.  However, sustainability of the programs and their supporting organizations, organizational and communal learning, and systems change have become more important as those attempting to affect larger scale change turn away from focusing on just their work to look at the environment around them.

And So It Goes
I hope you enjoyed reading about my evolution as an evaluator.  Much like the programs and systems we evaluate, my practice will continue to grow.  I would be very interested in learning your stories around how your engagement and understanding of evaluation has changed over time.
As always, I’m open to your comments, suggestions, questions and yes, your stories.  Please feel free to post comments.

Best regards,
Charles Gasper
The Evaluation Evangelist

Educating This Generation’s Evaluators

Some of you may know that I am on an incredible learning journey called “Graduate School”. I am nearly a year into this experience and can honestly say that my own thoughts around theory and practice have been influenced this past year. Clearly, the work of Claremont Graduate University as well as Western Michigan and a few other schools are brining a focus and commitment to professional evaluation otherwise not found. They are creating masters and doctoral “prepared” professionals into the world to engage nearly anyone they can in evaluation. If you want to get a sense of how important I think that is, just skim through the titles of my previous postings over the years. I think there is room for more of these institutions around the globe as my own experience of being the ultimate commuter student to pursue my own PhD has taught me. You see, there wasn’t a school nearby with a strong evaluation focused program in which I could expand my own knowledge and expertise. There wasn’t the community of thinkers locally available. So, first with Claremont’s Certificate Program and later my application and acceptance to the Graduate Program – I found my community. However, I think the stars aligned and I was lucky. Claremont and just started the Certificate Program (I was in the first cohort) and if it wasn’t for the vision of the leadership of Claremont’s Psychology Department with Stewart Donaldson at the helm, I would be stuck, wishing.

As you can probably guess, I have an idea… Well, a few anyway.
1) Online programs have a bad reputation in the academic world. There is a viewpoint that they are not as rigorous as residence programs. This viewpoint needs to change. Online participation in residence programs is now possible – my experience is case in point. In fact, there are times that I believe I get a superior experience to the resident in the classroom, having access to a teaching assistant with whom I can discuss thoughts and ideas that occur to me during the class that I wouldn’t want to disrupt the class in their vocalization. Granted, my experience is a bit different than other online experiences – perhaps in the area of requirements. But, with a bit of effort, the technology is currently present to maintain those requirements – even when the student is thousands of miles away from the campus. I suggest that the schools that educate and train professional evaluators examine this idea more closely and experiment.

2) Workshops at conferences, institutes, and the like are good entrées to topics, techniques, and theories of evaluation – but that about covers it. The onus is on the “student” to seek out additional venues of learning, usually books or websites. AEA has done some fantastic things to offer more information to members in the form of AEA365, its Linkedin group, EVALTALK, and others. EVALTALK was my link with the evaluation community, a place to ask questions from time to time and Linkedin as assumed some of that role as well. AEA365 provides great tips and links to useful ideas – but there is still something missing, an organized, progressive training opportunity for evaluation professionals.

On a daily basis, I work with both amateur and professional evaluators. Frankly that differentiation is unfair. I work with folks along a spectrum of evaluation knowledge and skill. I engage academics that have poor evaluation knowledge and skill as well as academics that are highly knowledgeable in this arena. [At some point, I will write more about the differentiation between content experts and evaluation experts – something a good number of nonprofits and funders misunderstand.] I also engage individuals with bachelors and masters degrees in fields not traditionally associated with evaluation or research that are highly knowledgeable and yes there are those with little knowledge in this category as well. Sending all of these people to a workshop to learn aspects of evaluation is not going to go far in improving their abilities. They need more support than that.

My own work in this area is leading me to a coaching model for engaging and training those lower on the evaluation knowledge continuum. In such a model, technical assistance in the more traditional forms of workshops and one-on-one training occurs – but that the “instructor” or “coach” continues to have contact with the “students”, providing continual education as needed for the “student”. Like a player of a coached team, the “student” receives the training and then is allowed to “play” (conduct appropriate evaluation work at their level) with additional mentoring and advice from the coach. Occasionally, the “student” returns for training (again, envision a team practice) for additional skills/knowledge development. We are testing this in a few projects I’m associated with and if you happen to attend this year’s AEA conference in Anaheim (http://www.eval.org/eval2011/default.asp), you are most welcome to catch a presentation sharing our experiences with this in one organization and where our theory of capacity building has evolved.

However… This still leaves a large gap in the education of evaluators – specifically the group I would call semi-professionals. These are the people on the middle of the continuum that have perhaps a master’s degree or even a strong research focused bachelor’s degree. They often have been practicing evaluation for a shorter period of time and if they are lucky, work in an organization with a more experienced and/or better trained evaluator. But often, they are not – and they are looking for additional educational opportunities. They may sign up and attend workshops on topics, but as mentioned earlier, these are just teasers relative to the depth of focus found in a graduate level course on the topic. Oh – and the reason I can speak about this is this was me many years ago and as I mentioned, I eventually got lucky. But until I got lucky and was able to find a program that was a good fit and allowed me to stay in my profession – I did what most of these semi-professional evaluators do. I attended workshops, conferences, and read books, journal articles, and posed my questions on EVALTALK. And honestly, it wasn’t enough. Yet, with exception to a few opportunities, there really is not much out there for the advancement of people falling into this category. Some are early enough in their careers that they can make the move to a direct residence program. In my case, the residence program accommodated me. But, there need to be more opportunities like mine – otherwise, we are leaving the semi-professional evaluators to their own devices with little support.

Do you have ideas to how to build evaluation capacity and knowledge? Please share!


As always, I’m open to your comments, suggestions, and questions. Please feel free to post comments.

Best regards,

Charles Gasper

The Evaluation Evangelist

Evaluation, Standards of the Profession

Well, it certainly has been some time since I posted here. As I mentioned earlier, my goal is to post twice a month. Clearly, I’m not meeting my milestones. After a review of my process, I have modified my plan and expect you will see more posts in the future, meeting that goal.

Today’s topic focuses on the professional side of evaluation. Up till now, I’ve been presenting the rationale for evaluation and its importance in organizational and program development and maintenance. In the past few months, I ran into an issue that has made me think more about the profession of evaluation and standards. Funders and nonprofits are often faced with the decision of hiring outside their organization for evaluation support. How does one pick an evaluator and how do you know if an evaluator is good?

These are great questions and something that I am still struggling with. I would say that the contractors I work with are fantastic. Yet, the only commonality that they share is that they are university based. Otherwise, they have different backgrounds, different skill sets, and different views on how evaluation should be conducted. Last year, I brought them together to share their work with each other and it is something that our Foundation will continue to do. The differences amongst these contractors were rather striking with some noted below:

· Quantitative versus qualitative methods

· Different focus on aspects of program (e.g. sustainability, quality improvement)

· Different background training and focus (e.g. public health, public policy, education)

However, there was a common factor that all shared. They had training in and followed “research” methodologies tied to their backgrounds. While there are some language differences, individuals trained in public health, psychology, social work, and sociology all have taken basic levels of social science research methodologies. As all of the evaluators are university based, they are required to conform to human subjects rules and pass their designs through an Internal Review Board (IRB). That is a very large commonality and it constrains the work that they do. Further, it develops a form of minimum standard for these evaluators.

But evaluators aren’t just based at universities. There are many independent contractors that provide evaluation services. These contractors can come from similar backgrounds to the ones I listed above, but can also have other backgrounds that vary in education (type and level), philosophy, and technique. Those without social science backgrounds may have different standards of “research” that they have learned. Finally, most of these contractors are not subject to some form of IRB. As a result, there is the possibility of greater variation. The purpose of these thoughts are not to speak to the idea of variation, for I believe that it can be both good and bad, depending on the situation, needs of the stakeholders, etc. Rather, I want to look at this issue from a concept of minimum standards.

So, to identify a minimum standard, we need to all agree on what is evaluation. Again, we can argue this as different cultures have different views on this. Instead, let us assume that you and I have are own definitions with the common idea that at the end of the day, we will have the information we want to have. So, I would argue that the first standard of evaluation is really driven by the needs and wants of the primary and associated stakeholders. In my framework, that means the development of a theory-based logic model of some type that links what the program or project or whatnot is doing with outputs and outcomes that we are trying to affect which will in turn inform my team as to what they might want to know. Additionally, there are other strategic needs that can inform the evaluation design and minimum standard for review (e.g. organizational strategic focus, environmental assessment needs)

Once this first standard of informational need is identified, we now have the minimum standard of what we want to know. The next step is to identify the how and what will be done or some sort of methodological standard. This is where things get a big complicated, but can be teased out/cleaned up.

To begin, there is the basic rule of human subject rules that borrows a bit from the Hippocratic oath – “do no harm”. If some harm must come to the participants, then the benefits of what are learned must outweigh the cost and reasonable efforts must be taken to ensure that the damage is addressed. Incidentally, I would propose that the organizations engaged should also be viewed in this manner. The evaluation should not damage the organization and reasonable efforts should be taken to ensure that any damage is addressed. Unfortunately, I have had an experience in which tenets of this rule were not applied at the organizational level (the aspect of informed consent to the evaluation) and some damage was done and worse, ignored. So, the second standard of professional evaluation should be not to harm the individuals, programs, or organizations engaged in the process.

I should clarify, that the manner in which the individuals and organizations go about applying their evaluation derived information can and should be covered under this as well. It is the evaluator’s responsibility to ensure that the organization that receives the information is given every opportunity to correctly interpret the information. However, beyond ensuring that the information is interpreted appropriately, I don’t bind the evaluator.

The third standard would be acceptance and willingness of the evaluator to be bound by the guiding principles of the American Evaluation Association. - http://www.eval.org/Publications/GuidingPrinciples.asp. In essence, the guiding principles cover the first two standards listed above, but I feel them so important as to separate them out. However, the guiding principles also address in general the concepts of systematic inquiry, including education of the stakeholders on methodology and limitations, evaluator competence, integrity, respect for people, and responsibilities for general and public welfare. While membership in the American Evaluation Association does not indicate that the evaluator considers themselves “bound” by these principles, they have been made aware of them in various forms including –the Association’s website and the American Journal of Evaluation.

Earlier this decade, members of the American Evaluation Association discussed the development of constraints to better define an evaluator. Ideas floated included an exam and some sort of certification. Even this year, membership still struggles with identification and development of a tighter, more distinguishing definition of an evaluator. Again, one can find calls for an Association related certification, but given the breadth of what defines evaluation, a single test or even series has been rejected by other membership. Many universities provide training in evaluation and/or evaluation linked skills - http://www.eval.org/Training/university_programs.asp as well as other organizations that provide professional training and in some cases certification. This patchwork of diplomas, certifications, and training provide something in the area of constraint. One will have a better sense of the skills and training of a graduate of the Claremont Graduate University or Western Michigan’s programs, but it requires the person hiring said evaluator to be familiar with the programs. That means that I, as Director of Evaluation for my Foundation, must be familiar with these and other programs. Fortunately, I’ve been a member of the Association for several years and have had a goodly amount of contact with faculty and graduates of these programs. I have not had much contact with faculty and graduates of American University or California State University, Los Angeles. I have known people to attend the Evaluator’s Institute - http://www.tei.gwu.edu/, but am unfamiliar with their work and know little other than the brochures that lap against my inbox on a yearly bases about that training. So, what is a Director of Evaluation for a foundation to do, or for that matter, a Director of a nonprofit, when reviewing a proposal from a potential contractor?

First, know what it is that you want out of an evaluation. What is the information you want to know about the program(s)/project(s) and document it. It has been my experience that when presented with a vacuum, evaluators will build what they can into the evaluation’s structure. While some serendipitous information that could be of value can be discovered, it is far better to give the contractors a sense of what you and organization wish to learn. This information should be incorporated into the request for proposals (RFP). Second, the RFP should also include a requirement that the contractor agree to and adhere to the American Evaluation Association’s Guiding Principles. Finally, request to see previous work from the contractor, to get a general sense of philosophy and style of evaluation.

In review of these documents, think about your organization and the stakeholders of the evaluation. Do the stakeholders value one methodology for garnering information over another? Will the evaluation provide you with what you want to know? Really, the question is - is the contractor and their evaluation design a good fit for you? That fit, agreement in philosophy, focus, intent, and concern is critical. Without that fit, the most rigorous evaluation design that develops all sorts of potentially useful information will lay fallow for a lack of investment of the stakeholders.

Incidentally, I struggle with selection of contractors for our evaluations, much as others do. I value the diversity that makes up the Association and the profession of evaluation, so I oppose stricter constraints on the use of the title of evaluator. However, the above is the “best methodology” I’ve developed to select contractors.

As always, I'm open to your comments, suggestions, and questions. Please feel free to post comments.

Best regards,

Charles Gasper

The Evaluation Evangelist