Why use criteria scoring?

Popularized by Daniel Kahneman and often used in hiring decisions, this method ensures that decision feedback is relative to what is important about the outcome. For example, without criteria we may make a hiring decision based on whether we personally like the person over whether or not they ‘align to company values’ or ‘demonstrated an ability to execute’. This is a powerful tool that has application beyond hiring decisions

Bias impact

Distinction Bias: When we weigh options separately without distinct criteria to measure against, we dilute what’s important. Defining what’s important up front helps make the differences between options (that actually matter) more pronounced.

Attention Bias: Simply put, attention is a finite cognitive resource. Because of this, defining what’s important ensures the majority of collaboration is focused on the important criteria, not irrelevant rabbit holes.

Take-the-best Heuristic: Left to our own devices, we tend to make decisions based on a single criteria and ignoring others. If we aren’t forced, evaluating multiple criteria demands higher cognitive load (even though it produces better results), so we tend to avoid it.

Less-is-better Effect: When options are presented in isolation, we tend to choose the worse option. When we bring context to the decision and put options on the same comparative plane, it allows us to make more objective decisions.

Defining your criteria

These criteria ultimately create a decision matrix where we rate options against a specific criteria and then discuss. A few helpful tips to keep in mind when defining criteria:

• Criteria guide the conversation towards and outcome-driven focus and align to core principles. Ask “If we make the right decision, what change would we expect to see?” and work backwards
• Less in more. As a rule of thumb, it’s best to have at least three criteria and though there are exceptions, it’s best to stay under five
• Think about anti-criteria — meaning you may have criteria to surface options that could have an unwanted negative impact.

Explain how you score

We’re relational creatures, so everything exists on a scale with upper and lower bounds. Now that you have the criteria, explain what a 5 means and what a 1 means. The more descriptive these definitions are, the better collaborators will be able to scale.

While explaining the scoring, if a specific criteria feels vague, break it out into more descriptive criteria. For example, a criteria like ‘Customer Impact’ is vague and would likely be difficult to produce a single definition for what is high customer impact vs low customer impact.

Let collaborators and stakeholders identify possible options

At this point the decision has been effectively framed and there are clear roles and responsibilities. Depending on the decision, you may want to only source options from the contributors or have it completely open, but keep these tips in mind:

• Keep a timeframe on options. Set a date and let contributors know when the window closes.
• Encourage contributors to define the options based on how they compare to the criteria
• As an added bonus, it’s a powerful exercise to have contributors argue against themselves and provide counter-points when defining the options

Score the options

Anonymity is very power when giving feedback on options. Ideally, whoever is giving feedback through scoring should not know who submitted the options. They also should not be able to see other feedback before they give their own to avoid the halo effect or anchoring bias.

It’s best to walk through this process async or give time at the beginning of a meeting for everyone to give their feedback first and then discuss.

It’s important to capture qualitative feedback with the scores. These become valuable succinct insights that also help frame a contributors thinking when scoring options

Watch out for these traps

Weighted Scoring

You may have criteria that should be considered more important than others. Utilizing a simple weighted score can help give a more relative output, but keep in mind that decision making is still art as much as it is science. There is a very real human element.

Weighted scoring works well when we are trying to prioritize similar options with fine margins, but use sparingly when it comes to more nuanced decision making.

Pro/Con Lists

outlining pros and cons in an unstructured way is dangerous and fraught with bias. Simply the visual impact of seeing red exes and green checks can have a profoundly skewed impact on the process. Studies show this approach causes more damage than any good.

“a pros-and-cons list generally becomes something that supports your decision as opposed to helping you arrive at a more objective one. Like your gut, it gets you to the decision that you want to make rather than the decision that is objectively better” — Annie Duke, author of ‘How to Decide’

‍