Getting What You Measure: Why Connor McDavid May Simply Suck at Defence

As best I can tell, Connor McDavid sucks defensively. 

On the surfece, the Edmonton Oilers have been getting shelled with him on the ice for quite some time now. Then when we dig deeper and move to statistics that try and account for context, we can see our best public estimates (Shout out Hockeyviz's Isolated Threat and Evolving Hockey's RAPM) suggest Connor McDavid is at least partially responsible for the onslaught of shots, chances and goals against when he is on the ice.

Now if you're trying to sound intelligent on twitter and maximize your likes to time spent ratio, you see Micah or the Twins estimates and have 3 options:

1) Dive into their methodology and figure out a flaw in the model.
2) Dig into the film and watch to see how McDavid plays in the defensive zone.
3) Tweet something like “Now I think hockey analytics have a tone of value, but this shows public models clearly have not solved quality of competition yet"

Of course, everyone who has spent some time on twitter dot com knows to choose number 3. This response is sure to make you sound smart and should rake in plenty of likes and agreements, but the more I've been thinking about it, the more stupid I think this retort is (without further proof at least). To imagine why let's think about another kind of job.

Instead of hockey players, let's think about CEO's of billion-dollar companies. To simplify things, we'll say these CEO's are tasked with 2 jobs:

1) Maximize profits, and
2) Maximize environmental quality

Unfortunately, we know we don't have perfect data, but we can easily measure the first goal. So even though both are important, we tie CEO pay directly to maximizing profits because we can measure it easier and better. Now, this world is a messy one, so unfortunately, the CEO's will sometimes be presented with opportunities where they can increase profits at the cost of environmental quality. Would it surprise literally anyone on earth if, at least some of the CEO's in this simple world were willing to sacrifice something that is a part of their job to increase the thing that gets them paid? Of course it wouldn't! We take it as a given at least some people are more than willing to trade off the aspect of their job which does not get them paid to be better at the part of their job that does.

(Taking it one step further, I hope people appreciate how nuts someone would sound in the CEO scenario if they set their sort of H0 or null hypothesis such that they simply assumed the CEO's who were strong at maximizing profits must also be good at maximizing environmental quality. Worse yet, they held this belief so strongly that when presented with solid statistical evidence showing a CEO who was great at maximizing profits actually did way more terrible things to the environment than most, they assumed it must be a problem with the statistics. You would obviously find this ridiculous)

Now back to hockey. We know how much hockey players get paid is mostly driven by 2 things.
1) Their counting stats (which measure offence) and
2)  Time on ice (which is much more highly correlated with our offensive statistics than our defensive ones).

​As for defense, it's obviously important, but as the Twins have pointed out, defensive measurements essentially have no impact on driving player compensation. Given this information, I don't know why it blows everyone's minds each time a model suggest some star player is great offensively, (the thing we tend to measure, and chose how much skaters are payed and played by) but sucks defensively, (the thing that most people don't measure, and has little to no influence over their compensation or playing time).

Am I saying public models have quality of competition and other contextual factors entirely solved? No, I honestly have no idea. But what I am saying is in different contexts, not only would the offensive juggernaut/defensive liability archetype not surprise anyone, we would expect it. So I think dumping on play driving metrics because they often show superstar players are defensive liabilities, might just be a shitty argument, not some profound proof we currently don't know how to solve for quality of competition.