I recently had a divisional meeting on assessment and it went about the way virtually every other assessment meeting has gone in my career. There was moderate enthusiasm for the topic at the beginning. As we got into specifics, whatever energy was in room evaporated like water on asphalt in August. There is no topic with the ability to make a group of student affairs professionals feel inadequate and crappy quite like assessment. And it has only become worse as assessment of learning outcomes have been added to the mix.
I can’t possibly be the only person who has experienced this.
About an hour into the meeting I had an epiphany. I realized—We are going about this all wrong! The messages my staffs perceive regarding assessment are putting them in a situation where assessment comes to be perceived as the enemy, something to be avoided. I realized that I must find a way to turn their minds around; otherwise, they will continue to flounder and HATE the topic of assessment.
So, what follows are pairs of statements. The first in the pair is what I term Conventional Wisdom (CW) as perceived and accepted by my staff (and other staffs I have experienced in the past). The second is the New Reality (NR) I want to take hold in my division.
CW #1 – You aren’t assessing, if you are not formally collecting data via surveys or focus groups.
This was the shared belief among the dozen professionals at this particular meeting. So they perceived themselves to be rarely and only sporadically assessing their work, since they rarely distributed surveys or conducted focus groups.
NR #1 – All effective professionals continually assess their work, projects, and programs through personal reflection, informal debriefing conversations, and professional dialogue.
In fact, these professionals need to understand that they are non-stop assessing machines. Counselors discuss each other’s clients. Staff talk at the end of a program and review what went right, what went wrong, and what they would do differently. My direct reports submit weekly reports answering questions, such as what successes have you had? What challenges have you faced? What have you learned? This is assessment!
Once professionals already see themselves as people who do assessment, it becomes a different conversation about what they can add to their assessment toolkit or how they can enhance their assessment practice.
CW #2 – Assessment is important but it’s extra work.
Assessment = More work. That’s the mantra. It means more time, more resources, and more energy. This belief is an assessment—and motivation—killer.
NR #2 – Formal assessment needs to be accounted for and incorporated in our work.
All the informal assessment mentioned in NR #1 is already incorporated into our work. In fact, it IS our work. We don’t consider it an add-on. We consider it part of the process of being a professional. Adding formal elements of assessment (e.g., surveys, focus groups, data analysis) needs to be accounted for. We don’t consider the planning part of program development to be an add-on; we account for it as part of the overall process and add the time necessary to accomplish it. Assessment needs to be viewed the same way.
Example – Let’s say I have a two-hour staff meeting where I have five agenda items, yet don’t talk about assessment in any of them. When I realize I need to incorporate assessment into the conversation, I do not add the assessment conversation and extend the staff meeting an additional half hour. No, if necessary, I reduce the number of agenda items to account for the assessment conversation, so I might only do four agenda items at that meeting.
What I am arguing is that meetings, planning processes, and such need to account for and incorporate assessment, and if it means we do a little less overall that is perfectly appropriate.
CW #3 – If you can’t measure it, you can’t assess it.
Staff have had drummed into their heads that the focus of assessment must be measurable or you will not know if you have improved it. This is the M in SMART goals. This is a focus on metrics, counting, and precision that persists from the positivist paradigm.
NR #3 – Assessment is not just about measurement; it is about understanding and improvement, both of which can be accomplished in many circumstances without measurement.
One danger of focusing only on the quantifiable is that we eliminate any focus on important, yet impossible to measure, aspects of our work. How do you precisely measure joy? Motivation? Love? Passion? Need? Wisdom? Additionally, we often create “measurements” that have little or no basis in reality. For example, we “measure” self-reported satisfaction and act as if we have actually measured something. We obtain an average score of 4.3 on a 5-point Likert scale for satisfaction and impute meaning into that number. In reality, we actually have no idea what we measured.
Parenting, managing relationships, and supervising staff are all vital practices in our society that people improve upon through on-going assessment that involves virtually no measurement.
When I started my new job I detected some potentially ineffective elements of the overall organizational culture through repeated observations and conversations in multiple contexts—I assessed the organizational culture. I didn’t collect numbers or measurable data in my assessment. When I shared my observations, most people agreed with both those observations and how elements of the culture were detrimental to organizational success. Many also saw how these cultural patterns held them back. There were multiple “aha” moments in these conversations! I am now resolved to improve the organizational culture by encouraging, reinforcing, challenging, discerning, and setting an example. None of which involves measurement.
CW #4 – Quantitative results that fail to meet the appropriate level of statistical significance or surveys that have low response rates are useless.
Staff report that when they submit reports on the results of their formal assessment activities (i.e., surveys), they are often criticized by Institutional Research offices or methodologically trained staff people for poor response rates. The message they receive is that the results are useless, so they wonder why they bothered to participate in this futile activity.
NR #4 – Assessment differs from research in that we are not seeking generalizable truth, but trustworthy, useful information to direct our efforts to bring about results or improvements.
As I have often said to staff, I do not need 51% of the passengers of an airplane to tell me the door is still open before I take action. I only need one.
As all students do not need to be surveyed to understand what problems are plaguing the student population, all attendees at a program do not need to be surveyed to help us understand what they got out of it or obtain ideas for how to improve it.
There are so many good resources in our field on assessment. We are learning more and more how to effectively assess student learning outcomes, but I fear much of this is not reaching many of the staff people who should be incorporating these lessons into their work because of strongly held belief systems about assessment. We must examine and dismantle these mythic belief systems and substitute new realities if we are to see the kind of advancements we all hope for in this area.
Comments and alternative perspectives are welcomed and desired!