Complexity - Excuse or Misunderstanding?

Is complexity an excuse or evidence of lack of understanding? Friends, have you ever had the situation where you have asked someone to explain why something happened, or perhaps why they are feeling a certain way and you have gotten the response – “its just too complex to explain”? I wish I could see the wry looks on at least some of your faces or perhaps the silent nods. One of the major responsibilities of an evaluator is to attempt to understand the program or organization they are evaluating. As most of you know, especially if you have read my past Blog posts, I adopt a theory-based framework for my evaluative work. That means that I’m constantly asking not only about the connections between activities and outcomes, but also why the team believes the relationships exist. Often times, I get a response that is very much like the quote above – “our work is just to complex to consider, much as less model!” Yet, the core concept of theory-based evaluation is the idea that we can get to the underlying connections and reasons.

Much like the Kuebler-Ross Stages of Grief (http://en.wikipedia.org/wiki/Kübler-Ross_model), many of the teams I’ve worked with have gone through stages that start with “it is too complex” to “yes, that is what we do and why we do it”. Once we get people to actually agree to engage with us, their models reflect the complexity seen in models such as this.




If you remember your high school physics class and had paid attention during any astronomy class, you will remember that the model above found at (http://farside.ph.utexas.edu/teaching/301/lectures/node151.html) is the Ptolemaic model. To make the complexity of the model work with Earth in the center, the planets need to orbit around a central point as that central point orbits around the Earth in a direction opposite that which earth rotates. The machine necessary to model this looks like something like this, found here (http://remame.de/wp-content/uploads/2010/03/astrolabe_2.jpg):



Note the complexity of the gearing and process to model the complexity. However, there is another step – moving to a simpler model and that requires the team to take a step back and not have a geocentric viewpoint of their own program or organization, but rather to try to look at everything a little bit different. In the case of astronomy, opening to the notion that the center of gravity for our local solar system resulted in a simpler model – something less complex found here (http://biology.clc.uc.edu/fankhauser/classes/Bio_101/heliocentric_model.jpg)




and results in simpler mechanics as can be found here (http://www.unm.edu/~physics/demo/html_demo_pages/8b1010.jpg)



The simpler model allows for a more accurate representation of what is actually happening and then allows for corrections such as the fact that the planets do not orbit the sun in a perfect circle. In organizations and programs, similar moments of clarity allow the team to test deeper assumptions and improve their associated projects.

Now, let’s be honest organizations and their programs, much like true orbital mechanics aren’t simple – there are layers of complexity. However, there is true complexity and there is complexity driven by poor assumptions or inability to stop and look at things objectively. The role of the evaluator is to help break down these viewpoints and help the team see through the complexity they have invented due to their preconceived notions to help them see the true underlying mechanics of their work and its outcomes. The process isn’t easy and in some cases, I’ve found that the work I do is more like a therapist than like a researcher. There can be displays of frustration and anger as the team works its way through understanding their organization or program. And much like some therapy sessions, the team can pretend that there is agreement among them when there isn’t – unifying against the evaluator to avoid the pain of the experience and/or the possible pain of discovering that their world view isn’t as clear as they would like. I will write more about process another day, but suffice to say, opening people to other views can be rather difficult work.

So back to the original question, is complexity an excuse or evidence of lack of understanding? I’ve often found it can be both and with that in mind, the wise evaluator, interested in understanding the theories of an organization or program, will continue to try to get their team to “simply” their model of their theory. It is in that simplification that real and difficult discussion occurs that provide insights as to what the organization or program is trying to accomplish and how.

Also, please note that at no point did I say that complexity isn't a part of everything we do - it most certainly is. However, experience would indicate that when we think about what we do and how we do it, our mental models are significantly more complex than reality. Further, our perceptions of what we do and why is often colored by how important we want to feel and how much we desire others to understand how difficult it is to be us. To those of you who fight to help teams tease out the try complexity from the self-generated complexity… To those of you who struggle to bring clarity to a complex world… Thank you!

As always, I'm open to your comments, suggestions, and questions. Please feel free to post comments.

Best regards,

Charles Gasper

The Evaluation Evangelist

If Your Friends Were Jumping off a Bridge, Would You Do It Too?

Ok, show of hands – how many of you had parents that asked the title of blog or a similar question when you were a teenager? I’m looking forward to a number of years from now when I utter those words out loud. Truth be told, I have had several opportunities in the past years to say something similar when working with organizations around evaluation.

It doesn’t take much to recognize that I’m a strong proponent of evaluation. Would a guy who didn’t think evaluation was important call himself The Evaluation Evangelist?... However, I am also a strong proponent of use of the information gleaned from an evaluation AND very much against wasting resources on creating information that will not be used.

I’m going to ask you to raise your hands again here… How many of you have been asked to design an evaluation and when you ask your client those fateful words – “what would you like to learn?”, you get a response of a blank look, confusion, or something to the effect of, “we don’t know, we were hoping you could tell us”? Trying to get more information, you might follow up with a question like, “why do you want an evaluation done?” and get the response of “the funder wants one”, “we are supposed to do an evaluation”, or the like. More often than not, I find myself on the receiving end of one of these responses.

As consultants, do you find yourself trying to design an evaluation for a client that doesn’t know what they want or why they are hiring you to do the evaluation? As program or organizational leaders, are you finding yourself hiring evaluators without knowing what you plan to get out of the evaluation? My guess is that at least some of you are nodding your heads or at least remembering a time when you might have found these to be true.

So, why is evaluation so popular these days? As people interested in the promotion of evaluation, why should we care as to why evaluation is popular and just enjoy the fact that interest is increasing? As an evangelist, shouldn’t I just be content that people are now asking for evaluation and thus I’m employed to help them understand what evaluation can do for them? To this, I must answer an emphatic NO!

Evaluations done just because a funder requires it or because the leadership has heard or read somewhere that it is a good thing to do (or worse, because it just is something one must do) will end up not being used. At best, the contracted or internally hired evaluator might be able to work with the organization to identify evaluation questions – but in the end, the organization needs to be the one driving the questions.

Metaphorically – think of the joke about the drunk that has lost a quarter and is looking under the streetlight. Along comes a guy who asks the drunk what he is looking for and the drunk tells him about the quarter. The guy asks the drunk where he lost the quarter and the drunk points off in a direction and says, “over there”. When then asked why he is looking under the streetlight, the drunk says, “the light is better over here.” I liken this experience to the organization that is asking for evaluation without guidance. In this case, the drunk (the organization) wants help to find something and the guy (the evaluator) winds up having to ask all sorts of questions that may unpack an issue to address.

But it can be and often is worse… For these organizations often don’t have evaluation questions formulated, it is as if the drunk is searching for something, but doesn’t know what it is. He may actually have never lost the quarter in the first place. As such, the helpful evaluator might find a different quarter, a dime, a stick of gum, and a rusty bolt on that sidewalk as well. All these things might be useful in some ways to the client, but since he doesn’t know what he is missing (if anything), he may not value the findings. As such, the evaluation findings are not used.

Now, some may argue that there are situations where having evaluation questions on the front end isn’t a good thing. Perhaps those situations exist, but even then, I would hope that there is some reason for engagement in evaluation other than just because it is done or others are doing it.

So dear reader, I leave you with a thought for the next time you consider an evaluation (either requesting one or supporting one). Think to yourself, why are you on the bridge and why are you considering taking the leap. Is it because it is in support of thought out evaluation questions or because everyone else is doing it?

As always, I’m open to your comments, suggestions, and questions. Please feel free to post comments.

Best regards,

Charles Gasper

The Evaluation Evangelist

Educating This Generation’s Evaluators

Some of you may know that I am on an incredible learning journey called “Graduate School”. I am nearly a year into this experience and can honestly say that my own thoughts around theory and practice have been influenced this past year. Clearly, the work of Claremont Graduate University as well as Western Michigan and a few other schools are brining a focus and commitment to professional evaluation otherwise not found. They are creating masters and doctoral “prepared” professionals into the world to engage nearly anyone they can in evaluation. If you want to get a sense of how important I think that is, just skim through the titles of my previous postings over the years. I think there is room for more of these institutions around the globe as my own experience of being the ultimate commuter student to pursue my own PhD has taught me. You see, there wasn’t a school nearby with a strong evaluation focused program in which I could expand my own knowledge and expertise. There wasn’t the community of thinkers locally available. So, first with Claremont’s Certificate Program and later my application and acceptance to the Graduate Program – I found my community. However, I think the stars aligned and I was lucky. Claremont and just started the Certificate Program (I was in the first cohort) and if it wasn’t for the vision of the leadership of Claremont’s Psychology Department with Stewart Donaldson at the helm, I would be stuck, wishing.

As you can probably guess, I have an idea… Well, a few anyway.
1) Online programs have a bad reputation in the academic world. There is a viewpoint that they are not as rigorous as residence programs. This viewpoint needs to change. Online participation in residence programs is now possible – my experience is case in point. In fact, there are times that I believe I get a superior experience to the resident in the classroom, having access to a teaching assistant with whom I can discuss thoughts and ideas that occur to me during the class that I wouldn’t want to disrupt the class in their vocalization. Granted, my experience is a bit different than other online experiences – perhaps in the area of requirements. But, with a bit of effort, the technology is currently present to maintain those requirements – even when the student is thousands of miles away from the campus. I suggest that the schools that educate and train professional evaluators examine this idea more closely and experiment.

2) Workshops at conferences, institutes, and the like are good entrées to topics, techniques, and theories of evaluation – but that about covers it. The onus is on the “student” to seek out additional venues of learning, usually books or websites. AEA has done some fantastic things to offer more information to members in the form of AEA365, its Linkedin group, EVALTALK, and others. EVALTALK was my link with the evaluation community, a place to ask questions from time to time and Linkedin as assumed some of that role as well. AEA365 provides great tips and links to useful ideas – but there is still something missing, an organized, progressive training opportunity for evaluation professionals.

On a daily basis, I work with both amateur and professional evaluators. Frankly that differentiation is unfair. I work with folks along a spectrum of evaluation knowledge and skill. I engage academics that have poor evaluation knowledge and skill as well as academics that are highly knowledgeable in this arena. [At some point, I will write more about the differentiation between content experts and evaluation experts – something a good number of nonprofits and funders misunderstand.] I also engage individuals with bachelors and masters degrees in fields not traditionally associated with evaluation or research that are highly knowledgeable and yes there are those with little knowledge in this category as well. Sending all of these people to a workshop to learn aspects of evaluation is not going to go far in improving their abilities. They need more support than that.

My own work in this area is leading me to a coaching model for engaging and training those lower on the evaluation knowledge continuum. In such a model, technical assistance in the more traditional forms of workshops and one-on-one training occurs – but that the “instructor” or “coach” continues to have contact with the “students”, providing continual education as needed for the “student”. Like a player of a coached team, the “student” receives the training and then is allowed to “play” (conduct appropriate evaluation work at their level) with additional mentoring and advice from the coach. Occasionally, the “student” returns for training (again, envision a team practice) for additional skills/knowledge development. We are testing this in a few projects I’m associated with and if you happen to attend this year’s AEA conference in Anaheim (http://www.eval.org/eval2011/default.asp), you are most welcome to catch a presentation sharing our experiences with this in one organization and where our theory of capacity building has evolved.

However… This still leaves a large gap in the education of evaluators – specifically the group I would call semi-professionals. These are the people on the middle of the continuum that have perhaps a master’s degree or even a strong research focused bachelor’s degree. They often have been practicing evaluation for a shorter period of time and if they are lucky, work in an organization with a more experienced and/or better trained evaluator. But often, they are not – and they are looking for additional educational opportunities. They may sign up and attend workshops on topics, but as mentioned earlier, these are just teasers relative to the depth of focus found in a graduate level course on the topic. Oh – and the reason I can speak about this is this was me many years ago and as I mentioned, I eventually got lucky. But until I got lucky and was able to find a program that was a good fit and allowed me to stay in my profession – I did what most of these semi-professional evaluators do. I attended workshops, conferences, and read books, journal articles, and posed my questions on EVALTALK. And honestly, it wasn’t enough. Yet, with exception to a few opportunities, there really is not much out there for the advancement of people falling into this category. Some are early enough in their careers that they can make the move to a direct residence program. In my case, the residence program accommodated me. But, there need to be more opportunities like mine – otherwise, we are leaving the semi-professional evaluators to their own devices with little support.

Do you have ideas to how to build evaluation capacity and knowledge? Please share!


As always, I’m open to your comments, suggestions, and questions. Please feel free to post comments.

Best regards,

Charles Gasper

The Evaluation Evangelist

Language and Evaluation

A great many people have spent a great deal of time thinking about what differentiates evaluation from research. I won’t press too far in this, other than to share that if you Google for the statement “difference between evaluation and research” as of the posting of this Blog, you would get over 5000 hits. Now, I’m sure there is much repetition in the form of quoting others and the like, but still – 5000 pages in which that statement occurs. Well, I’m going to talk about one aspect that affects my life almost on a daily basis – issues of language.

The current state of the art of evaluation suggests that a good evaluator is one that engages his stakeholders. What does that really boil down to? You have to talk to the stakeholders. Now, stakeholder is actually a very large word – and no, I don’t mean the number of letters or the fact that it might be a higher order vocabulary word for the SAT, ACT, or GRE. Rather, the concept of a stakeholder can spread across many different groups of individuals depending upon what sort of approach and philosophy you have about programming and evaluation. I’m not going to go into the various possible combinations, but suffice to say that you can be dealing with people who fund the program, implement the program, participate in the program, are not participating in the program, are in the community where the program is implemented, are not in the community where the program is implemented, and on and on and on. The combinations aren’t so much important to this Blog post as is what consists their background, understand, and vocabulary.

A few years ago, I had a discussion amongst individuals who all work for the same organization. These individuals all held the same position within the organization. This organization used and currently still does use the word – Objective. When asked what the word meant, the broadest definition could be – what the program is trying to achieve. However, that is where things broke down. For some, Objective meant a Programmatic Outcome. For others, Objective equated to a Programmatic Output. Still for others, an Objective was an Organizational Outcome. And for yet another group, it was a change in Organizational Infrastructure. All were focused on “Measureable Objectives”, but no one really agreed on what an Objective was. After a year’s worth of discussion and negotiation, we came the agreement that and Objective would be a Programmatic Outcome or Organizational Outcome. At least we got it to an “Outcome”.

When was the last time you had to have a discussion about the language amongst researchers? Ok, those of you who do language research, put your hands down! You get my point, I hope…

But the point was driven home to me again today. In a meeting with folks that I met with, along with another three evaluators, we discussed an evaluation project we are designing. During this meeting, I uttered another word, that I thought we all understood – “Comparison Group”. And was shocked to discover that their impression of what the term meant and my own impression and that of the other evaluators diverged. When they heard “Comparison Group”, they translated that to “Control Group”. They had a decent definition of a Control Group and we all know that engaging a Control Group for a study can require significantly more resources than engaging a Comparison Group, especially when the Comparison Group is not individually matched.

[Pausing for a moment here, because my own language may differ from your own… Control groups are usually associated with random control trials (RCT) and the costs of engaging in a RCT in community based programming and evaluation are very high. Control groups are a subset of comparison groups, which are just a group with whom you compare the outcomes of the group that experienced your program.]

The meeting around this study was rapidly devolving and the design was in jeopardy until I figured out that this was a language issue and not a design problem. The team had agreed to the design. They were under the impression that I was forcing a more rigorous study that would be costly across several domains. I was under the impression that they were stepping back away from the design and wanting something significantly less rigorous. Conflict was brewing. Fortunately, the issue of language was identified before things spun out of control.

I’ve presented the idea before and I’ll present it again. We need better-informed consumers of evaluation. Too often, I find myself and other evaluators changing language and/or dropping evaluation vocabulary out of discussions to attempt to avoid misunderstandings. I’m starting to wonder whether we are doing our clients and ourselves a disservice for this. In our own desire to make things easier for everyone in the short-term, we might be causing issues for the next evaluator. Worse, like the discussion around the term Objective, our looseness of language might cause more confusion. I’m considering short study for myself – to keep the evaluation language in and attempt to be more precise in my definitions with my clients – to see if I can reduce confusion. Anyone else want to give this a try? I would also like to hear your thoughts on the idea.

As always, I’m open to your comments, suggestions, and questions. Please feel free to post comments.

Best regards,

Charles Gasper

The Evaluation Evangelist

The Man in the Middle

I started writing this post about two days ago and discovered rather quickly that I was writing more than should fit in one Blog post – so… Instead I’m going to subject you to a series of posts discussing the rationalization of the wants/needs of the funder around evaluation with the wants/needs of small to medium sized nonprofits with whom I’m familiar. Tossed in will be some reflections on some reading I’ve been doing for school and of course, you get my own opinions and thoughts – full force!

To begin, I would suggest you take a look at my Blog post in January (2011 if you are reading this years from now – hello future people!). Go ahead – I’ll still be here when you get back…

So, you read my comments about my frustration with the lack of outcomes information coming to me from organizations soliciting me for donations. Well, those winds of change came quick and a partner in funding is looking for outcomes as is my own Board. My CEO, who gave me the name of the “Evaluation Evangelist” has pointed out to me a few times – “a prophet is rarely heard in his own land” and my previous warnings about nonprofits and foundations needing to attend to outcomes (versus outputs and other processes) was unheeded. And as with all crises, I think we are at the beginning of a change.

Before I go further, I should tell you that while I believe that we should always consider outcomes of programs, projects, advocacy, and whatnot – there is a time and place for evaluating said outcomes. This is tied to the questions the stakeholders have for the evaluation and what is possible to measure, given the theory of change of the program. Today, the “top” funding stakeholders are asking for outcomes and unfortunately, because of their attention, nonprofits are going need to react. Why do I say, “unfortunately”? - Because the interest in programmatic outcomes didn’t originate in the nonprofits delivering the program.

Granted, I have access to a small number of nonprofits, but in their study of nonprofits – Reed and Morariu (www.innonet.org) found that more than 75% of the nonprofits spent less than 5% of their budget evaluating their programs – 1 in 8 spent no money. Additionally, funders were characterized as the “highest priority audience for evaluation” and surpise – outcomes and impact evaluation were rated as the highest priority. So, my experience with nonprofits, while small, does seem to echo the broader population of nonprofits.

So, if this has been as it always has been – otherwise we wouldn’t have the results of the Innovation Network’s State of Evaluation 2010, why would I be concerned? Sure, I have been an advocate for evaluation use and just because I’ve been advocating for it (bigger names than mine have for a lot longer), that shouldn’t affect change. In fact, one could argue that I should be pleased – interest in evaluation is increasing in the funding community. Except, there is little education for the funding community around evaluation. There is little use by the funding community around evaluation. And the expectations that are coming out of the funding community are the equivalent of taking an older car that has never gone faster than 20 miles per hour and slamming on the accelerator to go 80 miles per hour (for those of use that use metric, you can substitute KPH and still see the analogy). Nonprofits that had at best conducted some pre-test/post-test analyses of knowledge change in participants in their program (more likely did a satisfaction survey and counted participants) are now being required to engage in significantly more sophisticated evaluations (ranging from interrupted series designs to random control trials). The level of knowledge required to conduct these types of studies with the implied rigor associated with them (I say implied if only because I can find a comparison group for anything – it just might not be appropriate) simply does not reside in most nonprofits. They haven’t been trained and they certainly don’t have the experience.

The funding community’s response is to offer and in some cases require an external contractor to support the evaluation. This could lead me to talk about difficulties in finding qualified evaluators, but we won’t talk about that in this post. It is an issue. However, what occurs with the involvement of an external evaluator? They do the work to support the funder’s objectives and after the funding for the project ends – they tend to leave too. There is also an issue around funding the evaluation at the level of rigor required – that too will come in another post. But, the message I want to leave you with here is that engagement of an external evaluator does little to increase the buy-in, much as less, capacity for the organization to engage in internal evaluation. The “firewall” preventing bias of an internal evaluator (e.g. organizational pressure to make the organization look good), while certainly improving the perception of the funder that the evaluation is more rigorous, does little to help the nonprofit other than to aid them in maintaining the current cash flow. [Incidentally, I’ll address the internal versus external evaluator conflict in a later post as well. I think this is something we can all use to explore.]

So – what am I advocating for? Let’s not take that older car on the highway just yet. Let’s listen a bit more closely to evaluation thought leaders like David Fetterman and consider what we can do to improve the capacity for organizations to do their own evaluations. Let’s show them how attending to outcomes might help them improve their organization and the services they provide to their participants. Perhaps we should think about evaluation from the standpoint of use and then apply the rigor that is reasonable and possible for the organization. Bringing in an external evaluator that is applying techniques and methods beyond the reach of the organization results in something mostly for the funder, not for the nonprofit. At best, the nonprofit does learn something about the one program, but beyond that – nothing. To them, it could almost be considered a research study versus an evaluation. Let’s partner with the nonprofits, get them up to the speed we want to get them, with careful consideration and deliberation versus just slamming on the accelerator.

As always, I look forward to any comments or questions you might have. Please feel free to post comments.

Best regards,

Charles Gasper

The Evaluation Evangelist

Answers To Nothing

Yes, once again I am going to subject you to a musical reference – this time, it is Midge Ure’s song, ANSWERS TO NOTHING. The lyrics and a YouTube video can be found here - http://www.lyricsmode.com/lyrics/m/midge_ure/answers_to_nothing.html

The underlying story of the song is that of a disillusioned and now cynical individual who has heard attention grabbing stories from respected individuals who have told him they have the answers to his most important questions, but has found those answers lacking in substance.

As with many of my Blog posts – I find that my daily life “excites” the evaluator in me. For those of us in the United States of America, the last month or two of the year marks a time where we receive many communications in the form of email, phone calls, and postal mail – all soliciting for donations to support any number of causes. The reason for the timing and intensity has to do with the tendency for many of us to wait till the end of the year to make donations, inspired by our relative wealth and remembering that these donations can be tax deductible. In the Evangelist household, we find ourselves making our decisions based upon the causes that most interest us and as such, many of these solicitations go unheeded.

However… You knew there was going to be a however here, did you? What do these solicitations have to do with the topic of this Blog?

Perhaps a few of you have already sifted through your memories of what you received this year and more importantly, the content there of – and know where this is going. I’ll assume there is someone out there who either doesn’t receive many of these or has not paid much attention to them.

The majority of these solicitations talk about what the agency or organization does – in other words, they tell me how they are spending their money. A smaller percentage will tell me a story of one of the people their programming touches. But few if any will talk about the impact they are making on that person’s life or the outcomes of their programming. They share stories about need. They share stories about what they do. They don’t talk about change in lives.

The evaluator in me applauds that they can talk about need – they have clearly done some form of a Needs Assessment. The evaluator in me is even happier when they can clearly describe their program and the number of people touched by the program. However, the funder in me and to some degree the evaluator, is disappointed that they can’t tell me what sort of change they are affecting.

In my daily life reviewing evaluations of programs, I find a continuum of depth. On one end are the evaluations that focus on describing the process of the program (e.g. number of people attending a training session, number of fliers distributed) and on the other end – far far away, I occasionally see evaluations that include a description of the process of the program, but also speak to measures of change in important outcomes and that relative difference found in the same measures for individuals that didn’t participate in the program. I honestly get excited when I see an evaluation design that is simply a measure of change for participants of a program (without a comparison group). It is rare enough compared to the description only evaluations that I often see. As a funder, these are nice, but they really don’t satisfy – they are often Answers To Nothing as they aren’t the question the funders most often ask.

Peter York of the TCC Group had an interesting take on the interests of individuals engaging in charitable giving. He posits that the mindset for donations has been on buying units of activity, not on impact. The impact has always been assumed. As a result, the solicitations often contain information about how low the overhead is of the organization asking for money and focus on what you are buying. Up until recent years with increased scrutiny being directed towards larger funders and an interest by the public in seeing results, government and funding organizations were also interested in what they bought. However, like the disillusioned young man of the song, they are becoming less interested in the story and more interested in getting answers – answers tied to measurable change (outcomes and impact).

ANSWERS TO NOTHING was published back in 1988, the lyrics were clearly not targeted for nonprofit leadership ears, but as a funder interested in outcomes and impact, and as an evaluator that is interested in helping organizations improve their programs as well as get support I leave you with the refrain from the song:

Oh, oh, oh, lied for the last time

Oh, oh, oh, died for the last time

Oh, oh, oh, cried for the last time, this time

Oh, oh, oh, believed for the last time

Oh, oh, oh, deceived for the last time

Oh, oh, oh, believed for the last time, this time

As a funder, I’ve grown cynical – while I might not go so far as to believe the solicitations contain deception and while my heart cries at the needs – I no longer just want to know what you are doing, I want to know what change you make and I’m not alone. Other individual and larger funders share my position. As evaluators, we need to work with organizations to not only improve their programs, but to help them tell the story so that their Answers are Meaningful.

As always, I’m open to your comments, suggestions, and questions. Please feel free to post comments.

Best regards,

Charles Gasper

The Evaluation Evangelist

You Give Evaluation a Bad Name

Ok, for a moment, I was going to go with the theme again of linking a song to my blog. I’ll spare you the Bon Jovi references -- for those of you who aren’t 80’s music types – the song is You Give Love a Bad Name by Bon Jovi.

However, the title has import in this case as Evaluation has had been given a “bad name” several times over the years. However, there is a new “threat” to evaluation – specifically, the idea that evaluation somehow causes foundations to become risk adverse or to put it more plainly – the idea that if you evaluate, you are less likely to fund higher risk programming. I won’t name names here or the like, but I’ve had opportunity to attend plenary sessions at national conferences where the individuals were not merely implying, they were saying that funders that engage in evaluation are stifling their ability to take risk.

I’ll let that sink in for a moment… No worries, I’ve already written this, so take as much time as you like….

People are paying to hear that evaluation stifles risk and thus innovation. No, I’m not putting words in anyone’s mouth here, there are witnesses that will agree with me and further, unfortunately, agree that evaluation does in fact “prevent” funders from trying risky things.

Well – since I’m writing about this topic, you know I’ve got something to say…

If by risk, you mean reckless funding of programming with no clue what will happen or why or to whom or for what purpose…. Yeah, I guess evaluation stifles that. If you mean blindfolding yourself, plugging your ears, and starting down a black diamond ski slope (nasty things black diamonds, difficult to navigate unless you are an expert) on a snowboard with loose bindings and hope for the best is the type of risk foundations should be taking, then by all means, evaluation is going to hamper it.

However, there is reckless risk and there is informed risk. I actually work with programs that engage in higher risk (as in non-evidence based) programming, but who also have evaluations in place to record the experience of the program as it negotiates the difficult slope of staffing, appropriate development and application of programming, and yes even impact on the participants. Evaluation is the eyes, ears, and to some extent, bindings that can make that risky decent down the slippery, bumpy, slope of programming a bit safer.

Let’s be honest for a moment, funding anything is sort of a gamble. The more risky the gamble, the more informed the gambler wants to be. Let’s take the game of roulette for a moment. It is one of the worst sucker bets you can place in a casino. As a gambler, your only control is where you place your chip. If the wheel is a bit unbalanced, you might catch a pattern – otherwise, it really is random chance. You don’t see a large crowd around the wheel in a casino – why? Because it is nearly impossible to get good information as to what number or color might come up next. Now take a gaze at the poker tables – there’s the crowd. Why? There is a lot of information available to the gambler to enable them to make wiser bets. Granted – there is still the element of random chance, but by using the data available to them at the moment (evaluation!), they are able to make more informed decisions.

So – why are folks clamoring for foundations to head to the roulette table versus the poker table. Why are they implying that it would be a good idea for foundations to go to the top of that dangerous hill and just toss ourselves off without our senses? It is because in fact, they are not suggesting that at all. To them, evaluation is about demonstrating success. High risk investments often fail to pay out. Yet, without those high risk investments, programming will never reach certain groups of people, nor will new innovations occur. Thomas Edison is often quoted as saying – “I have not failed, not once. I have discovered thousands of ways that don’t work.” Yet, after those many thousands of discoveries, he hit upon the electric filament that still is the primary light source for many of us. But it was the learning from those discoveries that led him to greatness.

And so – as always, it is late in the blog that I get to the point…

The people arguing that evaluation is a barrier for innovation are only seeing evaluation as being a way to determine if a program is successful and as such, the reason for a program being cut by a foundation – which is looking for success. They do not know or realize that evaluation can be used to monitor the program internally – as in by the folks implementing the program – to “discover the ways that don’t work” and change the course of the program towards things that seem to be working. As such, their reaction is to blame the evaluation (which, because it is designed to only look to see if there was success or failure at some point) for the eventual cut of funding to the program.

Let me share an alternative viewpoint. Instead of conducting an evaluation to only look for impact of the end of a program. Foundations should support (and yes fund) evaluations of high risk programs that focus on learning about the processes of the program and short term outcomes that can inform the staff of the program as to whether they are moving in the right direction. Innovation rarely occur by accident – rather it is often a stepwise process that incorporates a great deal of trial and error. The unit of these tests should be small enough that they can be conducted within the program’s timeframe, not after several years of work. Thus enabling the gambler or skier or funder the opportunity to see the patterns, terrain ahead, and take informed risks. Evaluation isn’t the bane of high risk programming, it can and does support it – enabling the funder and program implementer an opportunity to learn as they go, fostering great innovation.

As always, I’m open to your comments, suggestions, and questions. Please feel free to post comments.

Best regards,

Charles Gasper

The Evaluation Evangelist

This Is My Beautiful House!

Twas the night before Thanksgiving and all through the house, the Evangelist Family was toiling, even the mouse! We were preparing to have family over for a Thanksgiving Feast and the house needed to be spruced up, floors needed vacuuming, and most importantly – food was being prepared for the next day. About 5 minutes into laying out the necessary ingredients for the stuffing, I discovered that we were a half-pound short of mushrooms. So off to the store I went and this latest stream of consciousness was born. I have Sirius Radio and was listening the 80s on 8. Now, perhaps showing my age here, I attended high school in the 1980s and my formative driving years were during that time. In the region I grew up in, winter can be marked by heavy fog – a fog very similar to the one I was currently experiencing as I listened to the music of my youth. For a brief time, driving from my house to the grocery store, I experienced a vivid sense of déjà vu (ok, it really wasn’t déjà vu, but you get the idea) mixed with happy memories. On the way back from the grocery store, still in that same fog, I found myself reflecting on my journey to where I now live and the profession I now call my own. Children of the 80’s might have already caught the title of today’s Blog – a reference to a Talking Head’s song – Once in a Lifetime. If you had talked to that 16-year-old kid tooling around the neighborhood in the fog in the 1968 Buick Skylark (yes, I knew how to roll back then), he would have scoffed at the idea that over 25 years later he would be Director of Evaluation for a Health Foundation and that he would be so passionate about evaluation that his own CEO would brand him the Evaluation Evangelist. Back then I wanted to fly airplanes. Dreams of tooling around above the fog and clouds filled that kid’s head back then versus today’s dreams of tooling around various clusters of data and information to pull organizations out of the fog of programmatic and organizational complexity.

So, to borrow another line from the Talking Heads – Well… How did I get here?

I think my story is not all that unique in the world of evaluation. While there is a current generation now waking up in college and deciding that they want to pursue masters and doctoral degrees in evaluation, back then my generation was waking up and choosing to study social sciences and education. For me, it was “worse”, I still wanted to fly when I went to college. It was a series of events in my life, all tied to the desire to eat and pay for my education that eventually led me to evaluation. My junior year of college or as my family refers to it as the 3rd year of my undergraduate career, I found myself with out funding for school and a recognition that flying wasn’t in my future. For the next 3 years, I experimented with different majors and held different jobs. My first flirtation with psychology, the social science that would eventually claim the dubious honor of being my bachelor’s degree came as a result of watching two kids interact with each other when I was a childcare director. I felt this need to solve the puzzle of their behavior. So, I was drawn to a class, which led to many more classes and the degree. Flying had been replaced by puzzle solving (with people’s behavior in mind) – something that I enjoyed most of my life. But it all clicked months after my advisor, perhaps seeing how gaunt I had become living off raman noodles and whatever was on sale at the supermarket asked me to work with someone on an evaluation for the state of Missouri. I apprenticed those two years, bringing my knowledge of research methods and statistics to the table and learned about politics, working in the world versus a lab, dissemination of information, and that my work could make a positive difference in people’s lives. That project and the one following it hooked me. Here was an opportunity to solve really complex puzzles and make the world a better place. It was much more fun than flying or just trying to resolve human behavior.

With my return to graduate school this year – oh, did I forget to tell you, I’m back in school again? I’m working towards a doctorate in evaluation and research methods at Claremont Graduate University – it is a great program – look it up if you are interested or email me, I’ll happily tell you about my experience. Anyway… These past few months have had me thinking about what describes a good evaluator and whether I’m really a good evaluator or not. What I have learned is that it is a good thing I’m back in school. There are holes in my education. There have been advances in technique and statistics since I took those courses in the early 1990s. I’m being challenged to think beyond the ruts I formed as a practitioner. But most importantly, I’ve the opportunity to talk to others about my ideas and hear their own thoughts. I’ve started to surround myself with individuals that share some common values.

• They are interested not only in not harming people with their work, but actually improving their lives.
• The seem to be puzzle solvers like myself – although some have “interesting” and different approaches to the solutions of their puzzles.
• They are honest with each other and while sometimes brutal in their observations, intentional with their desire to help one another.
• They are in a program to improve their competence as evaluators.

In other words, they embody many of the values imbedded in the American Evaluation Association’s Guiding Principles. It is not the adherence to these principles that make for a good evaluator. Rather, they describe the outward signs of the internalization of good evaluation values.

Letting the days go by

Back to my journey into the world of evaluation, much like others, I found it be accident. It wasn’t something that I grew up wanting to do, but I would argue that it was something I was born to do. [Woah! Did I really say that?] I’ve been a member of AEA since the late 90’s, perhaps not as long as others, but I can tell you that each conference I’ve attended has felt like a home coming. I knew that evaluation was something I wanted to do as I worked on my first evaluation. The few opportunities I had to meet and talk with other professional evaluators were always more comfortable than with any other group of people. There has always been that sense of a good fit. Returning to graduate school to study and explore my own thoughts about evaluation has been a certain homecoming.

This is my beautiful house!

Now that you’ve read my little affirmation as to why I’m happy to be where I am, I’ll tell you that I still haven’t landed on the notion that I’m a good evaluator. I think I embody the values. I think I do good work. But I also know that there is more to learn.

Why have I shared all this with you? Well… Have you thought about becoming an evaluator? Clearly you are reading this for some reason. What draws you towards evaluation? Are you one of us puzzle solvers? Do you want to help others?

If you are an evaluator, perhaps my tour down memory lane will remind you how you got into the profession and why you stayed.

For both groups, I would be interested in hearing your story. What interests you and draws you to evaluation. And – well… How did you get here?

As always, I’m open to your comments, suggestions, and questions. Please feel free to post comments.

Best regards,

Charles Gasper

The Evaluation Evangelist

My Return

I think it has been over a year since my last post here. Let’s just say I’m not a prolific blogger and leave it at that ok? What I have learned is that I blog when I think of something important that I would like to discuss with you dear reader.

So, what got me to return here after 16 months of just twitter comments? - The American Evaluation Association’s (AEA) annual meeting/conference/party. It is there that you will find the majority of the great thinkers in the field, their students, and the practitioners that attempt to make sense of the world. There is a level of intellectual discourse that differs from any other conference I’ve attended. It is a homecoming of sorts for most of us, who struggle and contemplate how best to evaluate and inform the various communities and organizations we serve.

It was there – last week, that I had a moment of crisis. Oh it was coming – it had been building like a slowly evolving migraine. And on Wednesday, in a session I was attending, it exploded across my poor brain. Evaluators proposing the notion that evaluation should inform decision making in as near real-time as possible. At once I knew that I was in trouble…

You see dear reader, my roots are in applied social psychology – empiricism reigned supreme in my thoughts around anything having to do with research methods. I’m like many evaluators, finding the profession through another and my past clearly colors my future. However, the shade of tint has been changing over time. As you have read my previous blog posts, you probably know that I’ve also been colored by an experience in quality management and a need to help programming succeed. That flavor has affected how I go about my practice as an evaluator as well.

The two views competed with each other a bit and one could argue I was a “mixed method” evaluator in that I craved the “certainty” (yes, I know it really isn’t certain) of empiricism and the impact on program and organizational improvement that more interactive forms of evaluation can provide. I would flip back and forth and to be honest, I still oscillate between the two like a strange quark, vibrating between these “approaches”. But, it wasn’t until my moment of panic today that I noticed how quickly I quivered.

And so dear reader, I come to you. In confusion and admittedly some fear. You see, in my role as Director of Evaluation for a foundation, I want it all. I’m sure my fellow staff members want it all. My Board wants it all. And I think my grantees want it all too. We want to know what “conclusively” works so that we can generalize these learnings to other programs and projects we fund. We want the evaluation “results” (outcomes) to inform our future grantmaking. We want good programmatic ideas to spread. The empiricist in me argues that the evaluator needs to be the critical friend that watched the program get dressed, go out on its date, and succeed or fail without providing any advice.

But, our desire to see the programs we fund succeed, we also want to be that critical friend that after seeing your outfit, suggests that you change it before going out and observes how the date is going and provides ideas of different topics of “small talk” or notices that the place doesn’t work for the person you are with and suggests alternate places to go for the date. We want that date to succeed. We want that program to succeed. But we also want to know at the end of the date whether the whole package works.

Peter York of TCC Group made an interesting observation in a session at AEA. It was in reference to a different issue, but somewhat related. I am curious to hear more from him on his thoughts, but it got me thinking. What if we broke the program or date into smaller parts instead of evaluating the whole thing? The solution allows for more interventional evaluation (preventing you from continuing to talk about your last significant other and suggesting other topics to discuss – like the weather) and maintains some of the possible strengths of empirical rigor. By chunking the process of the program into smaller parts, there is a more rapid cycle of reporting and an opportunity to improve the program.

This only gets us so far. We have to have evaluation questions that are only focused on the components, which have to be time-specific. This might actually be good from a generalizability standpoint as few programs are into copied lock, stock, and barrel. Rather, based upon the context of the environment and resources available, components of the program are implemented.

There is another issue as well - specifically, the “intervention” of the evaluation (assisting with identifying issues with the program and providing suggestions for changes). One great argument against this is that the program has been “tainted” by the process of evaluation and is no longer pure. Here’s where I’ve landed on this topic this morning:

• Programs change over time with or without formal evaluation. They change for various reasons – one being because someone has made an observation that things aren’t working as they would expect. Why is it so wrong that someone not be better informed by a system that has more rigor?

• As I mentioned above, the programs change over time. This is a problem faced by longer-term empirical designs and frankly is ignored often in these discussions. Live programs are not like the labs where much of social science is conducted – things happen.


Huey Chen made an interesting observation in a presentation this past week at AEA. At the time, he was discussing the idea that random control trials (RCT), while appropriate at certain points in evaluation practice, are better conducted when issues of efficacy and validity are addressed in previous evaluations. Taking his comments further (and of course without his permission at this point), I would argue that evaluation focused on program generalizability should only be conducted after a meta-analysis (in the broadest form, not the statistical method) indicates that in fact, the whole program might work across multiple domains and contexts.

So – where does this all leave me in my crisis? I should tell you that I’m feeling better – much better. You see, it really comes down to the evaluation question and whether that question is appropriate. The appropriateness of the question is more tied to timing and results of previous evaluations. If we are talking about a new program, it is important to conduct interventional evaluation – we are collaborating in building the best program possible. In more mature designs that have been conducted in a few places, assessment of the programmatic model now makes more sense and a more empirical model of evaluation would be more appropriate. It is all about timing and maturity.

Funders still want it all and so do I. How do we allow a funder that only is interested in starting new programs the opportunity to say that their program idea should be replicated, yet allow for interventional evaluation as well? I’ve three criteria here:

• Fund multiple programs (and no, not just 5).

• Fund internal interventional evaluations for each program.

• Fund a separate initiative level evaluation that can look across all the programs, their contexts, and the organizational interventions. (Including the interventions of the internal evaluations).

In this case, there is a different focus/viewpoint of programming. For as long as I’ve been an evaluator, there has been this constant complaint that organizations do not internalize evaluation – that they do not integrate evaluation into programming. Here is the opportunity to build that framework. Evaluators complain that evaluation is viewed as separate from programming – yet the whole empiricist view of evaluation would place evaluation outside the program – an observer looking through the glass and watching to later remark on what happened and perhaps why. “Evaluation” is being conducted by amateurs on a daily basis to run their programs – why don’t we empower them to do it right? Empower them to integrate better evaluative practices into their programming? And then recognize that evaluation is an integral part of programming, seeing it as an operational activity that affects programming in similar manners as staffing, networks of support, and environment – which we already consider appropriate evaluands?

Michael Scriven talks about it being time for a revolution in evaluation. Perhaps it is time to drive the spike in to connect the rails that he and David Fetterman have been building. Perhaps it is time to agree that interventional evaluation and the more empirical forms of evaluation can coexist much like we have found that qualitative and quantitative methods can coexist and in fact enhance one another through mixed methods approaches to evaluation.

As always, I’m open to your comments, suggestions, and questions. Please feel free to post comments.

Best regards,
Charles Gasper
The Evaluation Evangelist

Evaluation, Standards of the Profession

Well, it certainly has been some time since I posted here. As I mentioned earlier, my goal is to post twice a month. Clearly, I’m not meeting my milestones. After a review of my process, I have modified my plan and expect you will see more posts in the future, meeting that goal.

Today’s topic focuses on the professional side of evaluation. Up till now, I’ve been presenting the rationale for evaluation and its importance in organizational and program development and maintenance. In the past few months, I ran into an issue that has made me think more about the profession of evaluation and standards. Funders and nonprofits are often faced with the decision of hiring outside their organization for evaluation support. How does one pick an evaluator and how do you know if an evaluator is good?

These are great questions and something that I am still struggling with. I would say that the contractors I work with are fantastic. Yet, the only commonality that they share is that they are university based. Otherwise, they have different backgrounds, different skill sets, and different views on how evaluation should be conducted. Last year, I brought them together to share their work with each other and it is something that our Foundation will continue to do. The differences amongst these contractors were rather striking with some noted below:

· Quantitative versus qualitative methods

· Different focus on aspects of program (e.g. sustainability, quality improvement)

· Different background training and focus (e.g. public health, public policy, education)

However, there was a common factor that all shared. They had training in and followed “research” methodologies tied to their backgrounds. While there are some language differences, individuals trained in public health, psychology, social work, and sociology all have taken basic levels of social science research methodologies. As all of the evaluators are university based, they are required to conform to human subjects rules and pass their designs through an Internal Review Board (IRB). That is a very large commonality and it constrains the work that they do. Further, it develops a form of minimum standard for these evaluators.

But evaluators aren’t just based at universities. There are many independent contractors that provide evaluation services. These contractors can come from similar backgrounds to the ones I listed above, but can also have other backgrounds that vary in education (type and level), philosophy, and technique. Those without social science backgrounds may have different standards of “research” that they have learned. Finally, most of these contractors are not subject to some form of IRB. As a result, there is the possibility of greater variation. The purpose of these thoughts are not to speak to the idea of variation, for I believe that it can be both good and bad, depending on the situation, needs of the stakeholders, etc. Rather, I want to look at this issue from a concept of minimum standards.

So, to identify a minimum standard, we need to all agree on what is evaluation. Again, we can argue this as different cultures have different views on this. Instead, let us assume that you and I have are own definitions with the common idea that at the end of the day, we will have the information we want to have. So, I would argue that the first standard of evaluation is really driven by the needs and wants of the primary and associated stakeholders. In my framework, that means the development of a theory-based logic model of some type that links what the program or project or whatnot is doing with outputs and outcomes that we are trying to affect which will in turn inform my team as to what they might want to know. Additionally, there are other strategic needs that can inform the evaluation design and minimum standard for review (e.g. organizational strategic focus, environmental assessment needs)

Once this first standard of informational need is identified, we now have the minimum standard of what we want to know. The next step is to identify the how and what will be done or some sort of methodological standard. This is where things get a big complicated, but can be teased out/cleaned up.

To begin, there is the basic rule of human subject rules that borrows a bit from the Hippocratic oath – “do no harm”. If some harm must come to the participants, then the benefits of what are learned must outweigh the cost and reasonable efforts must be taken to ensure that the damage is addressed. Incidentally, I would propose that the organizations engaged should also be viewed in this manner. The evaluation should not damage the organization and reasonable efforts should be taken to ensure that any damage is addressed. Unfortunately, I have had an experience in which tenets of this rule were not applied at the organizational level (the aspect of informed consent to the evaluation) and some damage was done and worse, ignored. So, the second standard of professional evaluation should be not to harm the individuals, programs, or organizations engaged in the process.

I should clarify, that the manner in which the individuals and organizations go about applying their evaluation derived information can and should be covered under this as well. It is the evaluator’s responsibility to ensure that the organization that receives the information is given every opportunity to correctly interpret the information. However, beyond ensuring that the information is interpreted appropriately, I don’t bind the evaluator.

The third standard would be acceptance and willingness of the evaluator to be bound by the guiding principles of the American Evaluation Association. - http://www.eval.org/Publications/GuidingPrinciples.asp. In essence, the guiding principles cover the first two standards listed above, but I feel them so important as to separate them out. However, the guiding principles also address in general the concepts of systematic inquiry, including education of the stakeholders on methodology and limitations, evaluator competence, integrity, respect for people, and responsibilities for general and public welfare. While membership in the American Evaluation Association does not indicate that the evaluator considers themselves “bound” by these principles, they have been made aware of them in various forms including –the Association’s website and the American Journal of Evaluation.

Earlier this decade, members of the American Evaluation Association discussed the development of constraints to better define an evaluator. Ideas floated included an exam and some sort of certification. Even this year, membership still struggles with identification and development of a tighter, more distinguishing definition of an evaluator. Again, one can find calls for an Association related certification, but given the breadth of what defines evaluation, a single test or even series has been rejected by other membership. Many universities provide training in evaluation and/or evaluation linked skills - http://www.eval.org/Training/university_programs.asp as well as other organizations that provide professional training and in some cases certification. This patchwork of diplomas, certifications, and training provide something in the area of constraint. One will have a better sense of the skills and training of a graduate of the Claremont Graduate University or Western Michigan’s programs, but it requires the person hiring said evaluator to be familiar with the programs. That means that I, as Director of Evaluation for my Foundation, must be familiar with these and other programs. Fortunately, I’ve been a member of the Association for several years and have had a goodly amount of contact with faculty and graduates of these programs. I have not had much contact with faculty and graduates of American University or California State University, Los Angeles. I have known people to attend the Evaluator’s Institute - http://www.tei.gwu.edu/, but am unfamiliar with their work and know little other than the brochures that lap against my inbox on a yearly bases about that training. So, what is a Director of Evaluation for a foundation to do, or for that matter, a Director of a nonprofit, when reviewing a proposal from a potential contractor?

First, know what it is that you want out of an evaluation. What is the information you want to know about the program(s)/project(s) and document it. It has been my experience that when presented with a vacuum, evaluators will build what they can into the evaluation’s structure. While some serendipitous information that could be of value can be discovered, it is far better to give the contractors a sense of what you and organization wish to learn. This information should be incorporated into the request for proposals (RFP). Second, the RFP should also include a requirement that the contractor agree to and adhere to the American Evaluation Association’s Guiding Principles. Finally, request to see previous work from the contractor, to get a general sense of philosophy and style of evaluation.

In review of these documents, think about your organization and the stakeholders of the evaluation. Do the stakeholders value one methodology for garnering information over another? Will the evaluation provide you with what you want to know? Really, the question is - is the contractor and their evaluation design a good fit for you? That fit, agreement in philosophy, focus, intent, and concern is critical. Without that fit, the most rigorous evaluation design that develops all sorts of potentially useful information will lay fallow for a lack of investment of the stakeholders.

Incidentally, I struggle with selection of contractors for our evaluations, much as others do. I value the diversity that makes up the Association and the profession of evaluation, so I oppose stricter constraints on the use of the title of evaluator. However, the above is the “best methodology” I’ve developed to select contractors.

As always, I'm open to your comments, suggestions, and questions. Please feel free to post comments.

Best regards,

Charles Gasper

The Evaluation Evangelist

Evaluation & Dissemination Can Save a Life

Warning, the following post contains images and discussion tied to the 2004 Indonesian Tsunami.

At 00:58:53 UTC, December 26, 2004, a subduction created 9.1 magnitude earthquake struck off the West Coast of Sumatra. The event was recorded on various seismographs throughout the world and the Pacific Tsunami Warning Center issued an bulletin at 01:14 UTC, December 26, 2004 - http://www.prh.noaa.gov/ptwc/messages/pacific/2004/pacific.2004.12.26.011419.txt. The bulletin was followed at 02:04 UTC, December 26, 2004 - http://www.prh.noaa.gov/ptwc/messages/pacific/2004/pacific.2004.12.26.020428.txt upgrading the magnitude estimate of the earthquake from 8.0 to 8.5. Later, the same center issued subsequent bulletins (December 27, 2004) for the Pacific Basin updating shore areas as to upwards of a half meter's difference in sea level from crest to trough.

The Indonesian Tsunami event resulted in an estimated 230,000 casualties and is an example where a form of evaluation, in this case observational assessment, could have saved lives.

In 1949, the Pacific Tsunami Warning Center was established as a reaction to the 1946 Aleutian Island Tsunami that was tied to 165 deaths between Alaska and Hawaii. The system is based initially on seismic data, which was all that was available to the center at the time of the 2004 Tsunami. No similar center existed in the Indian Ocean with the closest center being in Japan. However, the initial magnitude of the quake did indicate to some individuals that a significant event would occur in the Indian Ocean.

At this point this blog will move from fact to a bit of conjecture as I am unable to find actual evidence that the following is true, but certainly seems plausible.

Apparently, NOAA (National Oceanographic and Atmospheric Administration), a United States Department of Commerce agency attempted to contact governmental officials for countries which border the Indian Ocean to provide warning. Further, as the Tsunami made its way across the ocean, the warning was passed from government to government. Unfortunately, a method to effectively and efficiently disseminate this critical information was not in place and hundreds of thousands of people were caught unaware, with the first warning either being the serious reduction in sea level due to the preceding trough or the froth of the crest on the horizon.

Be that as it may, there are two major facts of this tragedy:
1) Incomplete evaluation systems were available to assess the issue (NOAA knew there was a possibility for a Tsunami, but had no idea as to intensity or direction).
2) There was no dissemination plan for the emergency.

The United Nations has taken steps to ensure that the "evaluation" of sea level be implemented consistently. There are plans for similar systems to be set up in the Mediterranean Sea, North Eastern Atlantic, and the Caribbean Sea. The expectation is that major communities can be given warnings that will allow for timely evacuations. The issue still is dissemination and it remains to be seen how effective the program will be.

There is an even more striking "measure" that wasn't discovered till recently, that jarred my thoughts around evaluation. The Centre for Remote Imaging, Sensing and Processing recently made available images that they unearthed when reviewing satellite imagery on the date of the Tsunami. As can be seen in the attached images (http://www.crisp.nus.edu.sg/tsunami/waves_20041226/waves_zoom_in_b.jpg , http://www.crisp.nus.edu.sg/tsunami/waves_20041226/waves_zoom_in_c.jpg), the waves from the Tsunami were clearly visible - giving a sense of intensity and direction of propagation.





The system, which had the trip wire measure of the seismograph could have enhanced information from the satellite images. But, again the "evaluation" would have been useless as it would not be presented to the individuals who would have most benefited from it. Instead, the world who was connected, watched helplessly as the waves crossed the Indian Ocean and inundated the coastal communities, killing hundreds of thousands of people.

Being an evaluator, I take home two critical thoughts from this:
1) Evaluation of systems is very important, in some cases, it just doesn't describe a program, but can in fact save lives.
2) Evaluation is only good if it is placed in the hands of the people who need the information to inform their decision making.

Evaluation can serve as a warning of impending doom for a program.

Granted, the earthquake and subsequent tsunami was a very low risk for the region, even given the fact that the region rests in a very tectonically violent area. The UN has "learned its lesson" and is implementing a system in the region, deploying evaluation after the program of safety for the population in the area failed (reached a very negative outcome). Today, nonprofits as well as for profit businesses, funders, and the like (really all of us) have received tremors that might indicate a larger earthquake that will crush us further. Even given the fragility of economies, knowing where the program is going and what is going well (and conversely, what is not) is critical. In most cases, the programming isn't going to harm individuals and loss of it will not have the same widespread impact as a tsunami, but the metaphor remains. A few key sensors connected to the right stakeholders can forestal major disaster.

As always I'm open to your comments, suggestions, and questions. Please feel free to post comments.

Best regards,

Charles Gasper

The Evaluation Evangelist

Evaluation, "Best Practices", and Fears

Those of you following my blog and facebook updates/tweets know that I traveled out to Baltimore a few weeks ago to speak about how our Foundation uses the information we collect as part of evaluation to inform our strategic planning. Recently, I have been in the throws of data analysis of an evaluation we are doing of an initiative that funds organizations with whom our Foundation normally does not partner. These organizations are not traditionally associated with health care delivery, but were viewed by our Board as a possible link to populations normally not reached by our more traditional partners. The evaluation and analysis have gone well and I have some interesting things to share with our staff, Board, and the grantees we serve – but the experience has given me food for thought.

First and foremost, there is a common issue of the organizations collecting data, but not having the resources to aggregate in a meaningful manner. The solution is pretty simple and inexpensive relative to other interventions, for those organizations without it, buy them a spreadsheet program like Excel and/or some data management software like Access. Of course the other key part is to teach them how to use the software. It is very frustrating to an evaluator to discover that the data is all there, ready to be used, but can not be accessed by the people who need to see it, because they just are unable to take the next step and put it in some sort of system that renders it available for analysis.

However, this is not the issue I’m interested in exploring in this post. Rather, this issue is a symptom of another problem – namely the requirement of grantees, by funders, to collect data to report. In many cases, if the grantee were using the data to support their internal operations, they would already have mechanisms to analyze it – even if it is tabulation on a piece of paper. Instead, funders are asking for information that the grantee does not find as valuable to their organization and thus, they collect and collect and then have difficulty in figuring out what to do next.

Part of the underlying issue has to do with the evaluation community that I belong to. Yep, I’m calling us out on this one. If a had a dollar for every time I have heard an evaluation contractor or organization (funders included) try to “sell” a nonprofit on evaluation, the first words out of their mouth is often – “the data you collect will help you solicit funding.” Raise your hand if you have heard or said it. --Incidentally, my hand is raised too. I’m just as guilty.--

We have trained the nonprofit community to believe that evaluation is something that you add on to a program, not something that can be a useful part of a program. But worse, while many funders require an evaluation, they often do not fund it or provide additional capacity building for their grantees. I’m finding that few foundations actually provide workshops, much as less more in depth training for evaluation to their grantees.

But it gets worse – much worse - and from my recent experience, somewhat scary for the future of funding.

There is a growing trend for funders to focus on funding “Best Practices”. At the conference in Baltimore, I often heard that statement tossed around. The staff at my own organization specifically ask for “Best Practice” programming when soliciting proposals for certain initiatives and priority areas. But, here’s the dilemma. If no one is supporting evaluation, how are “Best Practices” identified?

Recently, one of our Board members shared with me a program that is very interesting and judging from the evaluative work they have done so far, very promising. The Board member came to me, looking to see if I knew anyone that funded evaluation. Our Foundation funds programs and the evaluations are in support of those programs. In this case, they were looking to establish the evidence that the program works well to promote it nationally. Beyond a smaller group of funders, you would be hard pressed to find someone who would fund an evaluation on the scale that they are requesting – and that would only be for the programming they are funding. So, the question to me was, “where do we find funding just for evaluation?” And the answer?... I had no clue. Well, I did and I’m forwarding it on to a colleague who will be nameless, as I want to surprise her with it. But, even in this case, her organization funds research, not evaluation, so it probably is a stretch.

The point I’m making is that funders and nonprofits rely upon evaluation, even if they aren’t aware that they are doing so. The “Best Practices” evolve from the results of evaluative work. But if the nonprofits are only doing them to appease the requirements of funding or to solicit more funding, then the quality and the focus of the evaluation often moves away from program improvement/analysis to tracking of operational goals (basic outputs) and a focus on outcomes that are selected either to demonstrate success (as opposed to test success) or may even just be determined by the funder who isn’t the expert in the programming compared to those implementing it.

So, back to the grantees and the programs I’ve been evaluating. First of all, given the trend of funding, such innovative funding might be squelched in the future as they programs do not have “Best Practice” status. Their programs aren’t as effective as they can be as they are not fully aware of their programming. And yes, I must admit, their lack of having the knowledge of using Excel or some other spreadsheet program was a blind spot for me. I’m moving to rectify it for my grantees, but it makes me wonder how many more nonprofits are suffering from the lack of skill and resources to take the hard work they have done in collecting data and making it more useful. How many organizations still tabulate on a piece of paper or at best, use their spreadsheet program for only that purpose? I’m afraid of the answer, as I am afraid of the focus on “Best Practices”.

I have an answer that I’m implementing within the sphere of influence I have – namely mostly with my own grantees.

1) Over a year ago, I stopped “selling” evaluation as a way to get more funding and focus the grantee’s (and my own Foundation’s) attention on program improvement.

2) I had started workshops for my grantees and other nonprofits in the region, to teach evaluation methods.

3) I require that all my grantees that receive funding, beyond basic infrastructure, conduct an internal evaluation.

4) My Foundation funds these evaluations along with measurement supporting additional questions we have of the initiative group as a whole.

But, my experience has also taught me that I need to provide more basic training and support for the grantees in the form of funding for software and training to use it.

So, what can you do? If you are a funder or work for a funder, you might want to take a look at what your policies are around evaluation and how you use it, much as less encourage your grantees to do with it. If you work for a nonprofit, I urge you to think about how you use the measures you collect and why you collect them. But, before you run off to do that (because I know this has inspired you to push away from your computer and get busy), I would greatly like to hear your thoughts and reflections.

As always, I look forward to any comments or questions you might have. Please feel free to post comments.

Best regards,

Charles Gasper

The Evaluation Evangelist

Impact and Responsibility

In the past post, I wrote about how I thought that not considering impact of a program is irresponsible. Yesterday, I was struck by a fantastic example where consideration of the impact is critical. I haven't posted much about my family (other than a few tweets that you can see on the side of the blog page), but I have a father that is significantly older than me. He lives in an assisted living community which always strikes me of a college dorm with walkers and rascals. In any case, yesterday, in the early afternoon, he was taking some sun in the atrium (he refers to it as his vitamin D treatment). He got up to go back inside and that's where the story gets fuzzy.

At some point, he was found in the atrium, on the ground. A large, bloody bump resided on his skull (bump seems to trivialize the size of the hematoma). He was found to be disoriented and he was unable to get up off the ground. Along with the nausea and short term memory loss, the staff contacted me and emergency services to get him to the emergency room.

This is the background story for my discussion of outcomes...

In the next 9 hours that my father and I hung out together at the hospital, the thought of outcomes barely entered our minds. I should clarify, before I saw my father, outcomes were very strong in my mind, specifically one in particular. After catching up with him in the emergency room and spending a hour or so with him as various scans were done of his body and blood drawn, the "program" of my father's assessment and care was definitely being assessed through the eyes of process versus outcome. The fear was gone and now we both were focused, as several people spending their time in an emergency department, on the process. How long is this taking? Another x-ray? More blood? Can't they find the vein? I could go on. Fortunately, the hospital staff did keep the outcome of the program in mind and steadily worked towards it.

Incidentally, the program got extended beyond 9 hours in an emergency department to an overnight stay in the hospital - for observation. In this case, the tests done by the hospital staff indicated no impact of the impact on the concrete ("road rash" at the point of impact of his skull indicated that he did his concrete with his head), but being responsible evaluators, they chose in conversation with his primary care physician to keep my dad overnight for observation - to see if anything developed from the accident. Impact.

From an evaluator's perspective, I greatly appreciated the program that my dad experienced. From a son's concerned perspective, I did as well. Now, the hospital could have said that they performed their tests and found nothing wrong with my father and discharged him from the emergency department home last night - much like some evaluations that place the sole post-intervention immediately after the intervention or program is concluded. Further, the folks at the residence could have halted the process of assessment when he regained his faculties at the nurse's station. After all, other than the large bump on his head and a few abrasions, he really was "fine" - back to his "normal" self. Or, the nurses at his residence could have simply assumed that he would be ok if he just went back to his room, which is where he was heading in the first place. No one had to do an assessment of whether he was ok (disoriented or not). The his original plan was just to go back to his room - if no assessment had been done of the impact of his fall, we all wouldn't have been the wiser (other than the nasty bloody thing on his head and some additional stiffness in his movements). Instead, the staff of the residence and the hospital did the responsible thing and continued to assess impact.

Now, I recognize that some programs are either so new as to not have something to measure outcome-wise. I also recognize that there is only so far that you can follow an impact before you either loose contact with your participants or the cost of the evaluation far exceeds the benefits to follow up. That being said, more than lip service must be paid to outcomes. At minimum they need to be considered, even in the early stages of program design and implementation (I would argue that is probably the most critical point of consideration), and reasonable measures should be taken to make sure that the outcomes are represented in the evaluation.

Now, I would not be surprised if you have read the above and said something to the effect of "great, so you want me to consider outcomes in my program and evaluation designs. You want me to measure them, if possible, too. So, how much is good enough?"

The answer that question is to be responsible. Recognize what you need to know to effectively know that the program is doing what you want it to do. In the case of my father's head injury, the head containing one of the more critical organs for his survival and quality of life, the staff of the residence, and later the hospital, thought he required more outcomes assessment. However, along the way, there was also evidence of relationships between each activity of the programs administered by the residence and hospital staff and the outcome of my dad surviving and feeling better. They got him off the ground, into a cool setting (it was 90 degrees Fahrenheit and probably much hotter in the atrium). They helped him with his nausea (mostly just providing him a bucket). And they reassured him. Along the way, they conducted their process and outcomes evaluations. But, each activity had a theoretical link to my dad's health and well being. Each assessment grew off the previous one. Looking at the chain of interventions and assessments conducted by the staff of both places, you can see how each one had a decision point as to whether to add the next intervention or not, whether to add the next assessment or not. These linkages are part of their "theory of care" with each linked to that endpoint outcome. And based upon how my dad was doing, they chose to continue down the theory of care with associated evaluations and assessments of my dad. In the end, they took the assessments to where we are today, with him in the hospital under observation. If he had been younger, with less comorbid conditions, he probably would have been discharged last night from the emergency room. If the bump didn't form or if he had not been disoriented, he probably would have hung out in the nursing station for quite a while yesterday and not gone to the emergency room. The point I'm making here is that the evaluation reflected the need of the staff to garner more information to be responsible to the over all outcome of the interventions - my dad surviving with reduced lasting effects from the fall.

So, I would argue that in this case the evaluation has been done appropriately with correct attention to outcomes. Now, when the hospital discharges my dad (hopefully today), I expect that their evaluation of him will also end. There will be the expected handoff to his primary care physician (more than what some folks will get who are uninsured by the way), but the hospital staff will have satisfied themselves that my dad is fine when they discharge him. They will have counted their interventions a success. Now, in this case, my father doesn't have a chronic disease tied to the program, so no longer term tracking is necessary - but if he did, I would expect that his primary care physician would follow up on that and track the issues going forward.

So, the statement I'm making is that the evaluation of outcomes must be rigorous that the program staff and participants can be truly satisfied that the program does what it purports to do. I often see evaluations of educational programming with a pre-test/post-test methodology. On paper, this sounds pretty good. The evaluation focuses often on change in knowledge and sometimes includes change in attitude. However, often the post-test is conducted immediately following the training. I would find this acceptable if the evaluator or program staff could give me a theory of retention or the like that would give us a sense of what this really would indicate for longer term knowledge, behavioral, and attitudinal change. Instead, the post-test is often presented as if it is the ultimate outcome evaluation. To me, this is irresponsible. It would be the same as if the residence staff took a look at my dad after he sat in the nurses station for a while (both to cool off and get his bearings) and seeing that he was fully oriented again, just let him go back to his room, with little concern that being able to tell them who he was and where he was were poor indicators as to any sub-cranial bleeding which would result in other issues later.

Speaking of responsibility - I alluded to the process of my dad's care in the emergency department earlier in this post. We both were very much focused on the process and the discomforts associated with being in an emergency room for 9 hours with associated tests. One of the common process evaluation techniques and for that matter, sometimes pushed at me as an outcome measure (shame on you) is participant satisfaction. I can tell you that my dad was not satisfied with his care at the moment and I was pretty put out that we had to sit and wait and wait and wait in uncomfortable furniture. That would have been a poor measure of the program. I'm not even sure how they could have made the experience more pleasurable (other than more comfortable surroundings and perhaps a clown that did balloon animals), but to be honest, that isn't their core business. Their core business (the program) is to get people triaged, get them the treatment to get them stabilized, and get them the care they need to move them either into the hospital for more care or back out the door. For that matter, lets say that we do receive an assessment of satisfaction when my dad is discharged. Does the fact that my dad's room is rather nice looking, has a couch that I could sleep on, and has more cable channels than either of us do at home really assess the program? Certainly it might be a predictor of whether we return to that hospital for care (an as such is important to the hospital to collect), but it really doesn't determine whether the interventions my dad received at the hospital have an impact on his health - the core of the program.

So, now that I have gone on a very long example to emphasize a point, I'm going to flog that dead horse just a bit more...

All I ask is that program designers, implementers, and evaluators really think about the outcomes of a program and the core of its intent. I ask that they be honest with themselves as to what is important and really measurable. I ask them to draw those direct connections between their resources (inputs), activities, associated outputs, and short, medium, and long term outcomes. At minimum, consider these and then figure out what can really be measured and where you have to, draw upon theory to make the connections to the unmeasurable.

As always, comments are encouraged.

Best regards,

Charles Gasper