March Madness and Evaluation

Let’s just get it out of the way - March in the United States is about basketball, college basketball to be exact.  The colleges have been playing since November, but it is March where the general public (people who normally don’t follow collegiate basketball) suddenly get interested and involved in filling out “brackets”.  If you haven’t seen one of these - here is mine...

 

So what is the relationship between the assignment of schools to the brackets and evaluation?  Well, that isn’t such a simple explanation.  What you see are my selections, but the story isn’t about what I wound up selecting, but rather the process I took in selecting them and the manner in which others make their own choices.  Let’s be honest, one of the major reasons we engage in evaluative work is to predict how something/someone will perform in the future.  In the case of selecting which schools are going to advance through the tournament, there is a good amount of evaluative data on the page for you to see.

The schools are ranked going into the tournament.

Their individual win/loss records are presented.

The location the games are being played are also presented.

Finally, who they are scheduled to play are part of the brackets.

While the analysts and bookies that spend significantly more time on this than me bring additional evaluative data to bear, we do have some interesting data which speaks to how we use evaluative data to make decisions.  Let’s break this down by the manner in which I looked at information and came to the startling conclusion that UCLA will win this year’s NCAA tournament.

One of things you will note, if you scour over my selections is that the higher ranked teams tend to be the ones I picked.  They also tended to have a higher win/loss ratio.  This basal information explains some of my decision process.  However, context of the game also plays a factor.  Much is made of the impact crowds have over games.  While a different sport, the Seattle Seahawks in professional American football and Texas A&M’s collegiate football team promote their in-stadium fans as the “twelfth man” - recognizing their “contribution” to the game.  So, the context of the game and a guestimate of the ratio of fans in the arena played a factor.

There is also missing data that went into the decision process…  Wait!  Did I say, missing data?

It isn’t missing, rather it is evaluative data that I used that wasn’t included on the paper - from other sources of “truth”.  To share a bit more about me, I’m an athlete and a coach.  I have participated in and coached multiple sports at a competitive level and recognize that in addition to the crowd’s influence in the outcome of the game, there are other factors that can affect an athlete’s performance.  This leads me to explain why I picked UCLA to win, much as less why other teams advance over others.  Namely, I’m familiar with the history of UCLA basketball and the fact that the coaches and the athletes can “call upon” that history to give them a bit of extra boost in their games.

This all leads me to explain more around why I’m using the tournament as a metaphor for a program and the information in the brackets as evaluative data.  There are a few lessons to learn here.

1. I’m taking into account both the context of the game (program) and incorporating my sources of truth.  Good evaluation practices should incorporate what the stakeholders revere for information and data and the context of the the program implementation.  Better use of evaluation data for decision making should also take these factors into account.
2. I’m clearly using flawed data for my decision making.  Just looking at my final game - UCLA versus Michigan.  I know UCLA’s history, I don’t know Michigan’s.  It may be that Michigan has an incredible history as well, and to be honest, there is a nagging part of my brain that is saying that there is something there as well, something big.  However, I made decision to ignore that part of my brain when making my selection.
3. Speaking of ignoring information, let’s look at my decision to pick Stanford over Kansas.  And this highlights decisions that can be disguised as informed by evaluation, but in fact are made “from the heart”.  Living in Missouri, there is a bias against Kansas - don’t ask me where it comes from, but there has been at minimum a rivalry for years between the states’ schools.  Add that I attended a Pac-12 school for a period of time and my “allegiances” and thus decision making becomes clear. The lesson here is that while we would like to say that our programmatic decisions are driven by evaluative data, our own biases do creep in.  As leaders who use evaluative data to make decisions, we need to recognize our biases and be honest with ourselves and our teams.

It is these lessons that will serve us all well in both working as evaluators, but also as consumers of evaluative information and decision makers.  Paying attention to the context of the program, the agreed upon sources of truth, the fact that some information may not be found in the official structures of the evaluation, helps improve our understanding of the program and inform decision making going forward. And even if we have all the data we need, the complexity of the program may result in a different outcome than we expect - as history will certainly prove with my bracket results.

As always, I’m open to your comments, suggestions, and questions.  Please feel free to post comments.

Best regards,

Charles Gasper

The Evaluation Evangelist

Educating This Generation’s Evaluators

Some of you may know that I am on an incredible learning journey called “Graduate School”. I am nearly a year into this experience and can honestly say that my own thoughts around theory and practice have been influenced this past year. Clearly, the work of Claremont Graduate University as well as Western Michigan and a few other schools are brining a focus and commitment to professional evaluation otherwise not found. They are creating masters and doctoral “prepared” professionals into the world to engage nearly anyone they can in evaluation. If you want to get a sense of how important I think that is, just skim through the titles of my previous postings over the years. I think there is room for more of these institutions around the globe as my own experience of being the ultimate commuter student to pursue my own PhD has taught me. You see, there wasn’t a school nearby with a strong evaluation focused program in which I could expand my own knowledge and expertise. There wasn’t the community of thinkers locally available. So, first with Claremont’s Certificate Program and later my application and acceptance to the Graduate Program – I found my community. However, I think the stars aligned and I was lucky. Claremont and just started the Certificate Program (I was in the first cohort) and if it wasn’t for the vision of the leadership of Claremont’s Psychology Department with Stewart Donaldson at the helm, I would be stuck, wishing.

As you can probably guess, I have an idea… Well, a few anyway.
1) Online programs have a bad reputation in the academic world. There is a viewpoint that they are not as rigorous as residence programs. This viewpoint needs to change. Online participation in residence programs is now possible – my experience is case in point. In fact, there are times that I believe I get a superior experience to the resident in the classroom, having access to a teaching assistant with whom I can discuss thoughts and ideas that occur to me during the class that I wouldn’t want to disrupt the class in their vocalization. Granted, my experience is a bit different than other online experiences – perhaps in the area of requirements. But, with a bit of effort, the technology is currently present to maintain those requirements – even when the student is thousands of miles away from the campus. I suggest that the schools that educate and train professional evaluators examine this idea more closely and experiment.

2) Workshops at conferences, institutes, and the like are good entrées to topics, techniques, and theories of evaluation – but that about covers it. The onus is on the “student” to seek out additional venues of learning, usually books or websites. AEA has done some fantastic things to offer more information to members in the form of AEA365, its Linkedin group, EVALTALK, and others. EVALTALK was my link with the evaluation community, a place to ask questions from time to time and Linkedin as assumed some of that role as well. AEA365 provides great tips and links to useful ideas – but there is still something missing, an organized, progressive training opportunity for evaluation professionals.

On a daily basis, I work with both amateur and professional evaluators. Frankly that differentiation is unfair. I work with folks along a spectrum of evaluation knowledge and skill. I engage academics that have poor evaluation knowledge and skill as well as academics that are highly knowledgeable in this arena. [At some point, I will write more about the differentiation between content experts and evaluation experts – something a good number of nonprofits and funders misunderstand.] I also engage individuals with bachelors and masters degrees in fields not traditionally associated with evaluation or research that are highly knowledgeable and yes there are those with little knowledge in this category as well. Sending all of these people to a workshop to learn aspects of evaluation is not going to go far in improving their abilities. They need more support than that.

My own work in this area is leading me to a coaching model for engaging and training those lower on the evaluation knowledge continuum. In such a model, technical assistance in the more traditional forms of workshops and one-on-one training occurs – but that the “instructor” or “coach” continues to have contact with the “students”, providing continual education as needed for the “student”. Like a player of a coached team, the “student” receives the training and then is allowed to “play” (conduct appropriate evaluation work at their level) with additional mentoring and advice from the coach. Occasionally, the “student” returns for training (again, envision a team practice) for additional skills/knowledge development. We are testing this in a few projects I’m associated with and if you happen to attend this year’s AEA conference in Anaheim (http://www.eval.org/eval2011/default.asp), you are most welcome to catch a presentation sharing our experiences with this in one organization and where our theory of capacity building has evolved.

However… This still leaves a large gap in the education of evaluators – specifically the group I would call semi-professionals. These are the people on the middle of the continuum that have perhaps a master’s degree or even a strong research focused bachelor’s degree. They often have been practicing evaluation for a shorter period of time and if they are lucky, work in an organization with a more experienced and/or better trained evaluator. But often, they are not – and they are looking for additional educational opportunities. They may sign up and attend workshops on topics, but as mentioned earlier, these are just teasers relative to the depth of focus found in a graduate level course on the topic. Oh – and the reason I can speak about this is this was me many years ago and as I mentioned, I eventually got lucky. But until I got lucky and was able to find a program that was a good fit and allowed me to stay in my profession – I did what most of these semi-professional evaluators do. I attended workshops, conferences, and read books, journal articles, and posed my questions on EVALTALK. And honestly, it wasn’t enough. Yet, with exception to a few opportunities, there really is not much out there for the advancement of people falling into this category. Some are early enough in their careers that they can make the move to a direct residence program. In my case, the residence program accommodated me. But, there need to be more opportunities like mine – otherwise, we are leaving the semi-professional evaluators to their own devices with little support.

Do you have ideas to how to build evaluation capacity and knowledge? Please share!


As always, I’m open to your comments, suggestions, and questions. Please feel free to post comments.

Best regards,

Charles Gasper

The Evaluation Evangelist