Opinions And Ramblings By Adam Kmiec On All Things

The Soft Edges of Evaluating Success

From YouTube.com

Have you ever watched competitive figure skating? What people can do on skates is nothing short of amazing. In watching, have you ever shook your head at a low score and yelled at the TV, “how could that only be a 4.7???”

Well, did you know, there’s a very complicated scoring system – broken into 2 parts? The first part is simple, straight-forward, and mathematically sound. It’s called the “Technical Score.”

Each element of the program is assigned a base value, which gives skaters credit for every element they perform. Some elements, such as spins and step sequences, have levels of difficulty on which the base values are established.

Judges grade the quality of each element using a grade of execution score within a range of -5 to +5, which is added to or deducted from the base value. GOEs are proportional to the base value of each element.

The highest and lowest scores for each element are thrown out, and the remaining scores are averaged to determine the final GOE for each element. The GOE is then added to or subtracted from the base value for each element, and the sum of the scores for all elements forms the technical score.

Like nearly all rule books, it took 3 paragraphs to basically say, a skater earns points for the actual technical skills they execute. Each skill move has a definition. Those moves have specific point values attached to them. Here’s a great example of how this plays out.

Each row is a skill move that’s been executed and you can clearly see the points earned. Simple, or as simple as it’s going to get.

In business language, let’s think of each row as a type of ad you’re running, and each column as a metric with the “Base Value” column being the industry benchmark/average for that ad unit. For example, the industry benchmark for an ad’s click-thru rate might be 0.05%.

If this was the only evaluation criteria, it would be incredibly easy to know who performed well and who didn’t. It would also be very difficult to cheat.

But, alas, like life and the workplace, figure skating doesn’t stop at clear, well-defined, and consistent evaluative metrics. The second part of the scoring system is called “Program Components.” Here’s how it works according to the U.S. Figure Skating Association:

The judges will award points on a scale from 0.25 to 10 (in increments of 0.25) for five program components to grade overall presentation. As with GOEs, the highest and lowest scores for each component are thrown out, and the remaining scores are averaged. The final program components scores are then multiplied by a set factor to ensure the technical score and program components score are balanced.

The five program components are:

Skating Skills – Overall skating quality, including edge control and flow over the ice surface (edges, steps, turns, speed, etc.), clarity of technique and use of effortless power to accelerate and vary speed.

Transitions – The varied and/or intricate footwork, positions, movement and holds that link all elements.

Performance – The involvement of the skater physically, emotionally and intellectually in translating the music and choreography.

Composition – An intentional, developed and/or original arrangement of all types of movements according to the principles of proportion, unity, space, pattern, structure and phrasing.

Interpretation of Music – The personal and creative translation of the music to the movement on the ice.

That’s a lot. Wow. Basically, rather than just let objective, clearly-defined, and measurable scoring criteria be the only method for scoring a performance, they added in 5 elements that basically equal subjectivity.

In the business world, this would mean rather than look at clicks, downloads, and views – things that can be quantified – we would treat with equal weighting, things like look, feel, tone, and voice.

By every piece of objective logic, this is insanity. And yet, time and again, we see bad decisions made for these bad reasons. Marketers and Communicators hate “ugly” creative…even when it performs. Think about it; there’s a reason that despite the relative inability of Super Bowl ads to drive sales, companies still spend $5M+ for a 30-second spot. These decisions are often made on “feel”, not data.

The simple truth is, content that lacks the tone, voice, personality, look, or feel that we prefer, may actually perform better. The box office is a great example of how this plays out routinely. No one would ever credibly argue that Armageddon was a better movie than The Shawshank Redemption. Acclaimed movie critic Roger Ebert had this to say about Shawshank, after giving it 4 out of 4 stars:

It is deeper than most films; about continuity in a lifetime, based on friendship and hope.

As for Armageddon, a movie he awarded only 1 star, he had this offer:

The movie is an assault on the eyes, the ears, the brain, common sense and the human desire to be entertained.

Ouch. Got it. Shawshank is a better movie from a taste perspective. But, by every measure, Armageddon was a far more successful movie – especially where it matters the most. Movies are a business. Their creation is funded to create revenue. The Shawshank Redemption made $16M during its initial theatrical run, against a budget of $25M. You don’t need a calculator to see that’s a $9M loss. But, Armageddon, while costing $140M to make, generated $553M+. That’s a 4X return.

The last time I checked, I’d rather have a 4X return on my investment than a 36% loss.

The Shawshank Redemption is a better movie than Armageddon, but let’s be clear – it’s not more successful. If your organization favors the soft edges of success, you might have content that looks prettier and words that seem like poetry, but is it actually more successful than ugly content?