Ok, for a moment, I was going to go with the theme again of linking a song to my blog. I’ll spare you the Bon Jovi references -- for those of you who aren’t 80’s music types – the song is You Give Love a Bad Name by Bon Jovi.
However, the title has import in this case as Evaluation has had been given a “bad name” several times over the years. However, there is a new “threat” to evaluation – specifically, the idea that evaluation somehow causes foundations to become risk adverse or to put it more plainly – the idea that if you evaluate, you are less likely to fund higher risk programming. I won’t name names here or the like, but I’ve had opportunity to attend plenary sessions at national conferences where the individuals were not merely implying, they were saying that funders that engage in evaluation are stifling their ability to take risk.
I’ll let that sink in for a moment… No worries, I’ve already written this, so take as much time as you like….
People are paying to hear that evaluation stifles risk and thus innovation. No, I’m not putting words in anyone’s mouth here, there are witnesses that will agree with me and further, unfortunately, agree that evaluation does in fact “prevent” funders from trying risky things.
Well – since I’m writing about this topic, you know I’ve got something to say…
If by risk, you mean reckless funding of programming with no clue what will happen or why or to whom or for what purpose…. Yeah, I guess evaluation stifles that. If you mean blindfolding yourself, plugging your ears, and starting down a black diamond ski slope (nasty things black diamonds, difficult to navigate unless you are an expert) on a snowboard with loose bindings and hope for the best is the type of risk foundations should be taking, then by all means, evaluation is going to hamper it.
However, there is reckless risk and there is informed risk. I actually work with programs that engage in higher risk (as in non-evidence based) programming, but who also have evaluations in place to record the experience of the program as it negotiates the difficult slope of staffing, appropriate development and application of programming, and yes even impact on the participants. Evaluation is the eyes, ears, and to some extent, bindings that can make that risky decent down the slippery, bumpy, slope of programming a bit safer.
Let’s be honest for a moment, funding anything is sort of a gamble. The more risky the gamble, the more informed the gambler wants to be. Let’s take the game of roulette for a moment. It is one of the worst sucker bets you can place in a casino. As a gambler, your only control is where you place your chip. If the wheel is a bit unbalanced, you might catch a pattern – otherwise, it really is random chance. You don’t see a large crowd around the wheel in a casino – why? Because it is nearly impossible to get good information as to what number or color might come up next. Now take a gaze at the poker tables – there’s the crowd. Why? There is a lot of information available to the gambler to enable them to make wiser bets. Granted – there is still the element of random chance, but by using the data available to them at the moment (evaluation!), they are able to make more informed decisions.
So – why are folks clamoring for foundations to head to the roulette table versus the poker table. Why are they implying that it would be a good idea for foundations to go to the top of that dangerous hill and just toss ourselves off without our senses? It is because in fact, they are not suggesting that at all. To them, evaluation is about demonstrating success. High risk investments often fail to pay out. Yet, without those high risk investments, programming will never reach certain groups of people, nor will new innovations occur. Thomas Edison is often quoted as saying – “I have not failed, not once. I have discovered thousands of ways that don’t work.” Yet, after those many thousands of discoveries, he hit upon the electric filament that still is the primary light source for many of us. But it was the learning from those discoveries that led him to greatness.
And so – as always, it is late in the blog that I get to the point…
The people arguing that evaluation is a barrier for innovation are only seeing evaluation as being a way to determine if a program is successful and as such, the reason for a program being cut by a foundation – which is looking for success. They do not know or realize that evaluation can be used to monitor the program internally – as in by the folks implementing the program – to “discover the ways that don’t work” and change the course of the program towards things that seem to be working. As such, their reaction is to blame the evaluation (which, because it is designed to only look to see if there was success or failure at some point) for the eventual cut of funding to the program.
Let me share an alternative viewpoint. Instead of conducting an evaluation to only look for impact of the end of a program. Foundations should support (and yes fund) evaluations of high risk programs that focus on learning about the processes of the program and short term outcomes that can inform the staff of the program as to whether they are moving in the right direction. Innovation rarely occur by accident – rather it is often a stepwise process that incorporates a great deal of trial and error. The unit of these tests should be small enough that they can be conducted within the program’s timeframe, not after several years of work. Thus enabling the gambler or skier or funder the opportunity to see the patterns, terrain ahead, and take informed risks. Evaluation isn’t the bane of high risk programming, it can and does support it – enabling the funder and program implementer an opportunity to learn as they go, fostering great innovation.
As always, I’m open to your comments, suggestions, and questions. Please feel free to post comments.
Best regards,
Charles Gasper
The Evaluation Evangelist
Sunday, December 26, 2010
Wednesday, December 1, 2010
This Is My Beautiful House!
Twas the night before Thanksgiving and all through the house, the Evangelist Family was toiling, even the mouse! We were preparing to have family over for a Thanksgiving Feast and the house needed to be spruced up, floors needed vacuuming, and most importantly – food was being prepared for the next day. About 5 minutes into laying out the necessary ingredients for the stuffing, I discovered that we were a half-pound short of mushrooms. So off to the store I went and this latest stream of consciousness was born. I have Sirius Radio and was listening the 80s on 8. Now, perhaps showing my age here, I attended high school in the 1980s and my formative driving years were during that time. In the region I grew up in, winter can be marked by heavy fog – a fog very similar to the one I was currently experiencing as I listened to the music of my youth. For a brief time, driving from my house to the grocery store, I experienced a vivid sense of déjà vu (ok, it really wasn’t déjà vu, but you get the idea) mixed with happy memories. On the way back from the grocery store, still in that same fog, I found myself reflecting on my journey to where I now live and the profession I now call my own. Children of the 80’s might have already caught the title of today’s Blog – a reference to a Talking Head’s song – Once in a Lifetime. If you had talked to that 16-year-old kid tooling around the neighborhood in the fog in the 1968 Buick Skylark (yes, I knew how to roll back then), he would have scoffed at the idea that over 25 years later he would be Director of Evaluation for a Health Foundation and that he would be so passionate about evaluation that his own CEO would brand him the Evaluation Evangelist. Back then I wanted to fly airplanes. Dreams of tooling around above the fog and clouds filled that kid’s head back then versus today’s dreams of tooling around various clusters of data and information to pull organizations out of the fog of programmatic and organizational complexity.
So, to borrow another line from the Talking Heads – Well… How did I get here?
I think my story is not all that unique in the world of evaluation. While there is a current generation now waking up in college and deciding that they want to pursue masters and doctoral degrees in evaluation, back then my generation was waking up and choosing to study social sciences and education. For me, it was “worse”, I still wanted to fly when I went to college. It was a series of events in my life, all tied to the desire to eat and pay for my education that eventually led me to evaluation. My junior year of college or as my family refers to it as the 3rd year of my undergraduate career, I found myself with out funding for school and a recognition that flying wasn’t in my future. For the next 3 years, I experimented with different majors and held different jobs. My first flirtation with psychology, the social science that would eventually claim the dubious honor of being my bachelor’s degree came as a result of watching two kids interact with each other when I was a childcare director. I felt this need to solve the puzzle of their behavior. So, I was drawn to a class, which led to many more classes and the degree. Flying had been replaced by puzzle solving (with people’s behavior in mind) – something that I enjoyed most of my life. But it all clicked months after my advisor, perhaps seeing how gaunt I had become living off raman noodles and whatever was on sale at the supermarket asked me to work with someone on an evaluation for the state of Missouri. I apprenticed those two years, bringing my knowledge of research methods and statistics to the table and learned about politics, working in the world versus a lab, dissemination of information, and that my work could make a positive difference in people’s lives. That project and the one following it hooked me. Here was an opportunity to solve really complex puzzles and make the world a better place. It was much more fun than flying or just trying to resolve human behavior.
With my return to graduate school this year – oh, did I forget to tell you, I’m back in school again? I’m working towards a doctorate in evaluation and research methods at Claremont Graduate University – it is a great program – look it up if you are interested or email me, I’ll happily tell you about my experience. Anyway… These past few months have had me thinking about what describes a good evaluator and whether I’m really a good evaluator or not. What I have learned is that it is a good thing I’m back in school. There are holes in my education. There have been advances in technique and statistics since I took those courses in the early 1990s. I’m being challenged to think beyond the ruts I formed as a practitioner. But most importantly, I’ve the opportunity to talk to others about my ideas and hear their own thoughts. I’ve started to surround myself with individuals that share some common values.
• They are interested not only in not harming people with their work, but actually improving their lives.
• The seem to be puzzle solvers like myself – although some have “interesting” and different approaches to the solutions of their puzzles.
• They are honest with each other and while sometimes brutal in their observations, intentional with their desire to help one another.
• They are in a program to improve their competence as evaluators.
In other words, they embody many of the values imbedded in the American Evaluation Association’s Guiding Principles. It is not the adherence to these principles that make for a good evaluator. Rather, they describe the outward signs of the internalization of good evaluation values.
Letting the days go by
Back to my journey into the world of evaluation, much like others, I found it be accident. It wasn’t something that I grew up wanting to do, but I would argue that it was something I was born to do. [Woah! Did I really say that?] I’ve been a member of AEA since the late 90’s, perhaps not as long as others, but I can tell you that each conference I’ve attended has felt like a home coming. I knew that evaluation was something I wanted to do as I worked on my first evaluation. The few opportunities I had to meet and talk with other professional evaluators were always more comfortable than with any other group of people. There has always been that sense of a good fit. Returning to graduate school to study and explore my own thoughts about evaluation has been a certain homecoming.
This is my beautiful house!
Now that you’ve read my little affirmation as to why I’m happy to be where I am, I’ll tell you that I still haven’t landed on the notion that I’m a good evaluator. I think I embody the values. I think I do good work. But I also know that there is more to learn.
Why have I shared all this with you? Well… Have you thought about becoming an evaluator? Clearly you are reading this for some reason. What draws you towards evaluation? Are you one of us puzzle solvers? Do you want to help others?
If you are an evaluator, perhaps my tour down memory lane will remind you how you got into the profession and why you stayed.
For both groups, I would be interested in hearing your story. What interests you and draws you to evaluation. And – well… How did you get here?
As always, I’m open to your comments, suggestions, and questions. Please feel free to post comments.
Best regards,
Charles Gasper
The Evaluation Evangelist
So, to borrow another line from the Talking Heads – Well… How did I get here?
I think my story is not all that unique in the world of evaluation. While there is a current generation now waking up in college and deciding that they want to pursue masters and doctoral degrees in evaluation, back then my generation was waking up and choosing to study social sciences and education. For me, it was “worse”, I still wanted to fly when I went to college. It was a series of events in my life, all tied to the desire to eat and pay for my education that eventually led me to evaluation. My junior year of college or as my family refers to it as the 3rd year of my undergraduate career, I found myself with out funding for school and a recognition that flying wasn’t in my future. For the next 3 years, I experimented with different majors and held different jobs. My first flirtation with psychology, the social science that would eventually claim the dubious honor of being my bachelor’s degree came as a result of watching two kids interact with each other when I was a childcare director. I felt this need to solve the puzzle of their behavior. So, I was drawn to a class, which led to many more classes and the degree. Flying had been replaced by puzzle solving (with people’s behavior in mind) – something that I enjoyed most of my life. But it all clicked months after my advisor, perhaps seeing how gaunt I had become living off raman noodles and whatever was on sale at the supermarket asked me to work with someone on an evaluation for the state of Missouri. I apprenticed those two years, bringing my knowledge of research methods and statistics to the table and learned about politics, working in the world versus a lab, dissemination of information, and that my work could make a positive difference in people’s lives. That project and the one following it hooked me. Here was an opportunity to solve really complex puzzles and make the world a better place. It was much more fun than flying or just trying to resolve human behavior.
With my return to graduate school this year – oh, did I forget to tell you, I’m back in school again? I’m working towards a doctorate in evaluation and research methods at Claremont Graduate University – it is a great program – look it up if you are interested or email me, I’ll happily tell you about my experience. Anyway… These past few months have had me thinking about what describes a good evaluator and whether I’m really a good evaluator or not. What I have learned is that it is a good thing I’m back in school. There are holes in my education. There have been advances in technique and statistics since I took those courses in the early 1990s. I’m being challenged to think beyond the ruts I formed as a practitioner. But most importantly, I’ve the opportunity to talk to others about my ideas and hear their own thoughts. I’ve started to surround myself with individuals that share some common values.
• They are interested not only in not harming people with their work, but actually improving their lives.
• The seem to be puzzle solvers like myself – although some have “interesting” and different approaches to the solutions of their puzzles.
• They are honest with each other and while sometimes brutal in their observations, intentional with their desire to help one another.
• They are in a program to improve their competence as evaluators.
In other words, they embody many of the values imbedded in the American Evaluation Association’s Guiding Principles. It is not the adherence to these principles that make for a good evaluator. Rather, they describe the outward signs of the internalization of good evaluation values.
Letting the days go by
Back to my journey into the world of evaluation, much like others, I found it be accident. It wasn’t something that I grew up wanting to do, but I would argue that it was something I was born to do. [Woah! Did I really say that?] I’ve been a member of AEA since the late 90’s, perhaps not as long as others, but I can tell you that each conference I’ve attended has felt like a home coming. I knew that evaluation was something I wanted to do as I worked on my first evaluation. The few opportunities I had to meet and talk with other professional evaluators were always more comfortable than with any other group of people. There has always been that sense of a good fit. Returning to graduate school to study and explore my own thoughts about evaluation has been a certain homecoming.
This is my beautiful house!
Now that you’ve read my little affirmation as to why I’m happy to be where I am, I’ll tell you that I still haven’t landed on the notion that I’m a good evaluator. I think I embody the values. I think I do good work. But I also know that there is more to learn.
Why have I shared all this with you? Well… Have you thought about becoming an evaluator? Clearly you are reading this for some reason. What draws you towards evaluation? Are you one of us puzzle solvers? Do you want to help others?
If you are an evaluator, perhaps my tour down memory lane will remind you how you got into the profession and why you stayed.
For both groups, I would be interested in hearing your story. What interests you and draws you to evaluation. And – well… How did you get here?
As always, I’m open to your comments, suggestions, and questions. Please feel free to post comments.
Best regards,
Charles Gasper
The Evaluation Evangelist
Subscribe to:
Posts (Atom)
Posts
