Out Of The Blue Clear Sky: SIERA

Before we resume discussing advanced pitching metrics, David Pinto posted that ESPN will being a lot more with advanced statistics on their programming this season.  It will be interesting to see how they integrate it: in a separate, brief segment, or in their typical shove-it-down-your-throat fashion.  It will also be interesting to see, since they contract with writers from both Baseball Prospectus and Fangraphs, which advanced metrics they use for pitchers, batters and fielders.

SIERA

Skill-Interactive Earned Run Average, or SIERA for short, was introduced about a year ago by Baseball Prospectus (the link takes you to their glossary page where there are further links to the five-part introduction).  It's a lengthy introduction, but the authors, Matt Swartz and Eric Seidman, note that SIERA is a successor to Nate Silver's QERA, which has a "simple" formula: QERA =(2.69+K%*(-3.4)+BB%*3.88+GB%*(-0.66))^2. 

You can click over to BP for the SIERA formula, but essentially the authors attempt to unfold all of the individual components of QERA, apply an appropriate multiplier, and correct for the fact that GB% is actually a percentage of balls in play instead of a percentage of plate appearances (such as K% or BB%).  For ground balls, SIERA uses (GB-(FB+PU))/PA, which puts less weight on them.  SIERA is way more arithmetically complicated than the simple examination that follows, but hopefully this will be a helpful introduction (or re-introduction) to the metric.

What does that accomplish?

Taking a step back, let's recall the video referenced in the FIP and xFIP overview.  In that, FIP is said to track the pitcher, umpire, stadium and luck.  The use of HR in FIP's formula is largely responsible for it tracking the stadium and luck (xFIP reduces the presence of stadium and luck by using expected HRs).  SIERA, which at its core relies on Ks, BBs, and GBs, claims to go a step further by eliminating the effects of the stadium and luck.  It accomplishes this partially by treating GBs as a pitcher skill (as an Ubaldo lover, an idea I greatly appreciate) instead of considering HRs.*  Removing HR from the equation essentially eliminates the key park and luck factor, but HRs are still accounted for by the notion that high Ks and lots of GBs should result in fewer HR.  In particular, high Ks matter in limiting the possibility of HRs and XBH.  The umpire, by way of controlling the strike zone, is still accounted for in SIERA. 

*As stated in their introduction, HR/FB is highly variable from year to year and dependent on luck.   Also, as noted in BP's Part 1, there is an inherent unfairness in treating all HRs the same. 

On its face, SIERA does eliminate the effects of the stadium.  As far as I know, there are no park factors listed for ground balls.  However, there are park factors for walks.  I'm not a believer, but there's no denying that 1) less foul territory could allow for more opportunities to draw a walk (or other outcomes) by decreasing the opportunity for foul pop outs, and 2) the optics of a park, or perception as a difficult place to hit HR, could change a hitter's approach.  Yet, neither of those are necessarily borne out when you sort the park factors by BBs.  There are lots of favorable parks for HR in the top ten for walks.  In sum, SIERA goes further than FIP or xFIP in removing park effects, but might not entirely eliminate them.

It's a bit tough tough to say that SIERA completely eliminates defense and/or luck when ground balls are one of the three core factors.  In a vacuum, ground balls are always better than fly balls because they can't become fly balls.  There's (probably) less variation in trajectory and fewer potential landing spots (the ball has to touch the infield to qualify, and there's less space by volume in the infield than the outfield).  It's a little like the Woody Hayes saying in football, "Three things can happen when you pass the ball, and two of them are bad."  Three things can happen on a fly ball - a home run, batter reaches, and an out - and two of them are bad.  Only two things can happen on a ground ball: batter reaches or an out.

However, the key reason that ground balls are considered better than fly balls is that they preclude the greatest danger (HRs) and are more likely to be converted to outs by the defense.   In that loose sense, SIERA still tracks a bit of defense.

We will go into how pitchers end up ranking (versus other metrics), but I want to discuss one more formulaic aspect of SIERA (and other advanced pitching metrics).  Follow over the jump, and then share your own thoughts, likes and displeasures with this crazy little metric...

SIERA = same philosophy, better outcome?

This was touched on in the re-introduction to FIP and xFIP, but these advanced metrics aren't necessarily designed to replace ERA.  They are intended to estimate the runs a pitcher should have allowed based on things solely within his control (with differences of opinion on what is/is not within a pitcher's control).  SIERA aims to be a better predictor of how many runs a pitcher should allow than other advanced metrics.  To that end, in Part 4 of their introduction, the BP authors demonstrated that SIERA serves as a better indicator of next season park-adjusted ERA than any other metric, and is a better indicator of same-season park adjusted ERA than any metric that treats home runs as luck and not a pitcher skill (QERA and xFIP).

Why is this significant?  In Swartz's followup on SIERA this year, he reiterates that it is the best estimator of pitchers' skill levels (in this case, future ERA).  The ideology makes sense: ground balls are better, so eliminate explicit references in a formula to HR.  Swartz elaborates on SIERA's success as an estimator:

1. Ground balls matter more for pitchers who get more walks and fewer strikeouts because they allow more runners to reach first base.
2. Ground-ball pitchers allow fewer hits and fewer extra-base hits on ground balls than non-ground-ball pitchers, and SIERA acknowledges this effect due to its negative coefficient on ground-ball rate squared.
3. Pitchers with higher ground-ball rates (but not too high) allow the highest BABIPs and SIERA picks up on this reversing effect of ground balls on BABIP due to their correlation.
4. Pitchers with higher strikeout rates allow lower BABIPs and lower HR/FB rates, and SIERA picks up on this correlation. This is why the coefficient on strikeout rate in SIERA is so negative--because pitchers with high strikeout rates not only prevent runs by getting outs, but because they also allow fewer hits on balls in play and fewer home runs on fly balls.
5. Pitchers with higher strikeout rates get more ground balls in double-play situations.
6. Pitchers with lower walk rates issue more of their walks strategically, and thus the average damage of a walk from a high walk pitcher is higher, another effect which SIERA picks up.

From the FIP/xFIP re-intro, we understand that an "earned run" does not actually mean the pitcher is responsible for the run.  Instead, pitchers should be judged on what could loosely be called "expected earned runs" (this is not meant to be a defined term).  Metrics that attempt to determine expected earned runs are built from peripherals over which the pitcher has a high degree of control.  SIERA is the best at using peripheral statistics (Ks, BBs, GBs) to predict a pitchers' ability to limit runs. 

Who looks good?

With all of that introductory stuff in mind, you would think that SIERA would treat Ubaldo pretty well.  Instead, SIERA saw 2010 Ubaldo (3.57 SIERA, 21st) about the same as 2009 Ubaldo (3.60 SIERA, 20th).  On June 17, 2010, right before Ubaldo entered his only prolonged skid of the season, Swartz wrote an article articulating that Ubaldo was in line for some serious regression.  On that day, prior to the start versus the Twins that spawned the infamous article in which Jack Moore yawned at Ubaldo's first half, Ubaldo's SIERA was 3.43.  As Swartz noted, that's pretty impressive and lends itself well to having a low ERA, but not as low as Ubaldo's to that point (1.16).  The main reason identified by Swartz was Ubaldo's unsustainably low BABIP, particular his BABIP on line drives. 

It's not a new story around Purple Row that Ubaldo pitched about the same in each half of 2010, and in some ways he was a better pitcher in the second half (+1.6 K/9).  Unfortunately, he walked more, his BABIP returned to regular levels, and his LOB% regressed to normal.  Ubaldo can improve on the walks in 2011, but the incredibly high LOB% and low BABIP weren't sustainable over a whole season, and, while they could occur again at times in 2011, shouldn't be expected over an entire season. 

In many senses, SIERA is fair to Ubaldo.  It basically tells us that he is a really good pitcher who offsets a relatively high walk rate (among the peers in his talent group) with a good ground ball rate and lots of strikeouts.

Let's take a look at how SIERA stacks up against the ERA, FIP and xFIP for pitchers with 150 IP+ from 2008-2010.  I've listed the top three pitchers by SIERA, plus one pitcher immediately above and below Ubaldo (overall SIERA ranking in parentheses).  Click on the year to see the full table at Baseball Prospectus.

2008:

2009:

2010:

One temptation is to dismiss SIERA based on Javier Vazquez topping the chart in 2009.  But don't forget that he had a 9.77 K/9 and 1.81 BB/9 (5.41 K/BB), with a 41.7% GB rate.  Considering Timmy only pitched 6 more innings than Javy, It says a lot about the weights in SIERA's formula that he was able to beat out Tim Lincecum's 10.42 K/9, 2.72 BB/9 and 48.9% GB rate.  And, while James Shields' ERA looks terrible, especially next to Ubaldo's, he walked nearly 1.5 batters less per 9 innings than Ubaldo (while only striking out .5 batters less).

While I think Ubaldo's a better pitcher than a lot of the people ranked ahead of him (and probably would be ranked higher if you threw un-regressed HR/FB into this mix), nobody ahead of him on the list had a higher walk rate in 2008, 09, or 10.  The walks will always hurt his advanced metric score and ranking because of the increased potential for runs to score. 

What's it worth?

SIERA is an interesting metric.  The premise, that Ks, BBs, and ground balls are a good recipe to measure run prevention, is sound.  The formula is a little cumbersome to grasp, but the reasoning behind it is similar to FIP and xFIP: Ks and BBs matter a ton in a pitcher's ability to prevent runs, but ground balls are better than any other batted ball in a variety of ways and circumstances.  They're more likely to help a pitcher out of a jam with less than two outs, and they're high up on the expected out values without allowing for the possible negative outcome of a HR. 

However, it's a big step to eliminate HRs altogether (as opposed to reducing the weight given to them in FIP, or providing expected HRs as in xFIP).  While it may be a step forward in reducing the presence of luck in the evaluation of a pitcher's true skill, luck is certainly part of the game, particularly for a pitcher who constantly flirts with bad luck in the form of a fly ball rate well above the league average (walking the line and crossing the line are hard to distinguish at different points in time).  Few pitchers, without off-setting it with a tremendous amount of Ks, are going to be able to flirt with that line without crossing it in a season or two. 

SIERA is also a good reminder that advanced metrics are, in part or whole, meant to be better predictors of future ERA than ERA.  When that is the definition of true talent level, people are going to have different ideas about how to get there.  If assimilating and properly weighting the things a pitcher can control is the measure of true talent level, and then adding a constant to make it look like ERA, other metrics may be better.