Isolating Player Movement by Eliminating Camera Motion: An Ongoing Project

Note: This post departs from the general format used on this blog. That's because the post is not about a specific analysis I've done but rather a demonstration of a tool I've been developing to make more detailed studies possible.

For those not counting at home, this is the 18th post on this blog. In the last 17 entries I've investigated a wide variety of topics, from whether or not it's a good idea to start rookie QBs, to home field advantage's dependence on time zone shifts, to crowd noise's impact on penalties. All of these analyses were performed using the standard statistics that the NFL keeps, chiefly play-by-play data from Armchair Analysis. These stats are very rich in content, and so it's not too surprising that I (among many, many others) have been so successful in exploiting them.

At this point, however, I believe that the returns from this kind of data are rapidly diminishing, and fewer and fewer cool new results will be forthcoming. At this point the community has explored a significant fraction of the power of the existing data; most of the really interesting questions that can be satisfyingly answered with play-by-play data already have been addressed. That's not to say this resource is fully exhausted, but I strongly believe that smart people have been using these statistics for long enough now that new work will be increasingly more difficult to find and perform and will require even greater care to ensure its validity.

That doesn't mean that there's nothing more to be done with advanced football analytics. In fact, I would argue that everything done up to now has only scratched the surface of what could be possible with better data.

I'm talking, of course, about player tracking systems.

It's taken many years of innovation, but technology has finally progressed enough to allow the positions of athletes to be monitored with high accuracy during a game or match. This capability opens up an entire new world of possibilities for improving our understanding of sports as it eliminates the reliance on simple statistics that can be easily tracked by humans. (Of course, if you like looking at the old statistics because they're easy to understand, don't worry: those will stick around, at least for the foreseeable future. In fact, they'll likely be improved, as player tracking systems can be used to automatically reconstruct all of the common statistics and thus eliminate transcription mistakes.)

While the NFL has generally taken a rather dim view on modern technology, in the last couple of years things have started to change – slowly. Here's an article from the league's website breathlessly describing how teams have finally started keeping their playbooks on computers – from just two years ago. This year the NFL has finally relaxed its rules prohibiting new tech on the sidelines, only to force teams to use old Surface tablets which have been modified to prohibit anything except viewing photos of plays from the prior drive.

Given this spotty track record, I was pleasantly surprised (stunned, really) to read the news that the NFL is jumping on-board the player tracking train in a big way, outfitting 17 stadiums with the technology to read RFID chips mounted in the shoulder pads of the players. (For a less pleased – but quite amusing – take on the announcement, see here.) Once statisticians and commentators get used to the new data at their disposal I'm sure we'll see a bevy of interesting new stats during games and in write-ups.

I want to be clear that I see the NFL's unexpected embrace of player tracking as an undeniable good: obviously the stats viewers and fans are provided will only get better once the system is up and running. However, given the limited nature of the stats the league provides on their website I would be shocked if the raw data from this system was provided to fans. I imagine that instead we'll be drip-fed highly aggregated statistics of minimal use for deep analysis, similar to the exceedingly limited results provided to the public by the NBA from their similar system.

I'm not sure why major sports leagues are so stingy with their statistics. It's possible they just don't see the demand to justify adding that capability to their websites, although the cynic in me thinks they're worried that with free access to the data the general public will start putting their own analysts and pundits to shame.

Regardless of why this information tends to be rationed, it's pretty clear (to me, at least) that despite the NFL's new commitment to this technology there is still value in developing an open system for player tracking. It's also clear that the broadcast footage is woefully inadequate for this task, as it's far too zoomed-in to see what players are doing away from the ball.

Fortunately, despite some ridiculous objections, the NFL has recently decided to make All-22 game footage available to the public. For those of you who don't follow my hyperlinks, the All-22 footage refers to two very wide camera angles, designed to show all 22 (hence the name) players on the field rather than focusing on the football. One camera shoots from the sideline at midfield while the other one films from one of the endzones, both from a very high vantage point. If you want to do any kind of player tracking based on game video, this is the footage you want to use.

Thanks to the excellent nflvid module, I was able to download the All-22 footage for the 2013 Jets-Falcons game to use as a guinea pig. My first discovery was that, despite the name, the All-22 film does not show every player for the duration of each play. Rather, the All-22 cameras function much like the regular cameras you see during the broadcast, except from a wider angle. For running plays, it's not a terrible issue, as the play is fairly compact and action away from the ball is of marginal importance:

But for passing plays, especially longer ones, the camera tends to lose the other receivers (as well as the linemen and QB) after the catch, when it zooms in to follow the ball:

(Note that the quality in the actual video is significantly better than in these GIFs, as I didn't want this post to take hours to load so I compressed them by a fair amount.)

A proper system would cover the whole field at once, allowing the viewer to watch every player for the full duration of each play. Unfortunately this is clearly not the case for the All-22 footage (the endzone view is actually worse in this regard, for reference), which makes identifying and following players between video frames much more challenging.

So, rather than being able to use the raw All-22 footage for player tracking, first we need to remove the camera motion. This is necessary in order to both get absolute position shifts for players as well as to keep track of which player is which between frames. The gist of the technique is simple: Find easily identifiable regions in each frame, and then compare their locations between frames to compute how the camera has moved (the homography, if you want to sound smart). Once you know the motion of the camera, you can correct the frame for it, effectively removing all camera motion.

Football is actually really well-suited for this process, as there are so many lines and markers painted on the field (soccer, for instance, would be significantly more challenging). The program I wrote to do this is relatively straightforward, and can be found on github. You can compare the original footage with the motion corrected video for a run:

and a pass:

I personally just prefer watching with the camera stationary – it's much more like actually being at the stadium, where your attention isn't controlled by the camera operator. Of course, it's not perfect; the NFL shield now moves, and you can tell from the pass play that the re-projection isn't exactly accurate on a few of the frames. The program also tends to have trouble when the ball is near the sidelines as well as at the end of most plays, when the camera has zoomed in really far. It isn't very fast (it takes about 10 minutes or so to process most plays) and can require several gigabytes of RAM since it currently stores all the motion-removed frames in memory as it processes the play.

Overall, however, it generally works fairly well, especially for a first attempt. From this point you can begin testing player identification and tracking algorithms for a variety of different plays, while continuing to iterate on the robustness of the camera motion program. Everything is open source, so if you're interested in contributing, or just want to try out the code for yourself, feel free to grab it and go nuts!