Turnover Adjusted EPA Per Play

Evaluating NFL Quarterbacks is incredibly hard. There are a few ways this can be done, each with its own strengths and weaknesses. Right now, public analysts usually use Expected Points Added (EPA) per play for quarterbacks. This post will show I adjusted for fumbles in my adjusted EPA per play to better understand a quarterback's isolated impact on his EPA per play.

Why Adjust EPA Per Play?
While I cannot speak for everyone, QBs EPA metrics are usually used to compare Quarterbacks, not offences. So, if we want to use this statistic to evaluate QBs to the best of our abilities using public statistics, I do not believe we should punish QBs for things they cannot control. This brings me to the first turnover adjustment, fumbles. There is already a method to begin adjusting for fumbles. The NFLfastR dataset calls this QB EPA. This metric does not penalize passers if the receiver catches the ball and goes on to fumble at some point after the catch. Because such fumbles are not the QB's fault, we should not penalize them.

Further Fumble Adjustment
But what about QB fumbles? Once a QB has fumbled, the outcome is almost entirely out of their hands. Sure they can recover it themselves. This does happen a small percentage of the time, but that is really getting into the weeds. Compounding this problem, turnovers make a massive difference in EPA. Especially in small samples. Here are the distributions of QB EPA on fumbles over the past few seasons.

We can see by the fumbles distribution that the range of outcomes is enormous. Say the quarterback fumbles. If his running back recovers, he can still gain expected points on that play. On the contrary, if someone on the other team is a step ahead, it can cost them 7+ expected points because of field position and the other team's potential to score.
​
The difference between a -7 EPA or a breakeven 0 on a given play will take about 23 plays at Aaron Rodgers's average efficiency this year to make up for. Then for more average QBs, it can take games to make up for these losses. All because the other team's corner beat his team's running back to the loose ball and returned it for 6? That does not seem like something we should include in QB analysis. There is no reason to pin these massive swings on QBs unless we can confirm some QBs are better/worse at fumbling.

Is Fumble Quality Skill?
So is fumble quality a skill? We know from above that QBs rarely recover their own fumbles. This suggests outside of the act itself, QBs do not impact fumble EPA. But, QBs may be able to influence fumble EPA by mitigating fumbles in high leverage situations. If QBs are good or bad at this, we should see some repeatability in the EPA lost per fumble from year to year.

This has not been the case. QBs who fumble in high expected points situations have been no more or less likely to do the same next year. It seems clear that fumble quality is not a skill among QBs. So, EPA analysis can probably be improved by treating all fumbles as equal, rather than swinging wildly based on an outcome that the QB has little to no control over.

An Interception Adjustment?
​The other type of turnover worth looking into in this article is interceptions. Earlier this season some people were defending Jamies Winston on Twitter. Their reasoning? Sure, Jamies throws a tone of interceptions but using EPA they were able to show he tended to throw them in low leverage situations.

As a result, the number of expected points he cost his team on a per interception basis was much lower than any starting QB. This lead me down a rabbit hole. It turns out the opposite was true in 2019.

Such a wild swing made me question if there was any skill here. Outside of 1 case study, are some QBs better or worse at throwing interceptions in high/low leverage situations? If so, interception numbers won't tell the whole story. But, if not, is having EPA weigh certain interceptions wildly differently just adding noise to our QB evaluations for no real reason? So is it a skill? Well, again we generally expect skills to repeat themselves year over year.

There is a small relationship here. The year over year R squared of EPA lost per pick relative to league average is 0.01. Meaning about 1% of the variance in EPA lost above average on interceptions carries from year to year. This is an incredibly small number, so we can be reasonably certain interception quality is not a skill because it is almost entirely random year over year, and skills will tend to repeat themselves over time.

Turnover Adjusted EPA
If the amount of EPA lost fumbling and throwing interceptions is not something any QB is good or bad at, it makes sense not to include it in our metric to evaluate players. After all, we have such a small amount of data to begin with, there is no need to inject massive swings into our QB analysis if we are not picking up any skill there. A straightforward solution to this problem? Treat all interceptions and QB fumbles the same. Each equal to their average value and compare QBs then. Here are the two variables plotted against each other.

We can see here that while the values are similar, there are some meaningful differences for a small amount of work. Joe Burrow was hammered on turnovers this year. He lost about 14 expected points more than average on his turnovers. Once we have adjusted, his 2021 campaign looks even more impressive, vaulting Burrow into the top 3 in terms of EPA per play. A similar story happens with Justin Herbert. Turnover variance is the difference between him ranking 6th in EPA to inside the top 5 (4th).

The flip side includes QBs like Zach Wilson. Wilson's horrible 2021 season looks worse if all his turnovers are treated as average. Here is a table showing the effects for the 25 QBs with the most drop backs in 2021.

This gives us a better way to view players' performance with some of the noise stripped out.

Conclusion
The analysis above shows it is highly unlikely that mitigating points lost on a per fumble and interception basis is skill. So, including different EPA values for their fumbles and interceptions is not adding any signal about QB skill to our analysis. So, QB analysis can be improved by treating all interceptions and QB fumbles with the same value. This is especially true in smaller samples. While the changes aren't huge, this simple trick can strip some noise out of public QB evaluations with a few lines of code.