Erik Ulken has posted a must-read top 10 list of lessons learned while setting up the data desk in the LA Times newsroom.
The data desk's job is to take detailed information that's dreary to read in text or table form and make it useful by presenting it in compelling and interactive formats. A well-known example is the LA Time's Homicide Map which allows readers to filter through a database of crime statistics and see them represented visually in graphics and on a map.
The 10 lessons Erik posted are all good so I'm pasting them all in here but it's still worth checking out his post to see what else he has to say.
Erik's list echoes some of the things learned at the Telegraph when it was reinvigorating its website and beginning to integrate its web and print operations. I think these lessons would apply in all sorts of development and change management scenarios.
- Find the believers: You'll likely discover enthusiasts and experts in places you didn't expect. In our case, teaming up with the Times' computer-assisted reporting staff, led by Doug Smith, was a no-brainer. Doug was publishing data to the web before the website had anybody devoted to interactive projects. But besides Doug's group, we found eager partners on the paper's graphics staff, where, for example, GIS expert Tom Lauder had already been playing with Flash and web-based mapping tools for a while. A number of reporters were collecting data for their stories and wondering what else could be done with it. We also found people on the tech side with a good news sense who intuitively understood what we were trying to do.
- Get buy-in from above: For small projects, you might be able to collaborate informally with your fellow believers, but for big initiatives, you need the commitment of top editors who control the newsroom departments whose resources you'll draw on. At the Times, a series of meetings among senior editors to chart a strategic vision for the paper gave us an opportunity to float the data desk idea. This led to plans to devote some reporting resources to gathering data and to move members of the data team into a shared space near the editorial library (see #8).
- Set some priorities: Your group may come from a variety of departments, but if their priorities are in alignment, disparate reporting structures might not be such a big issue. We engaged in "priority alignment" by inviting stakeholders from all the relevant departments (and their bosses) to a series of meetings with the goal of drafting a data strategy memo and setting some project priorities. (We arrived at these projects democratically by taping a big list on the wall and letting people vote by checkmark; ideas with the most checks made the cut.) Priorities will change, of course, but having some concrete goals to guide you will help.
- Go off the reservation: No matter how good your IT department is, their priorities are unlikely to be in sync with yours. They're thinking big-picture product roadmaps with lots of moving pieces. Good luck fitting your database of dog names (oh yes, we did one of those) into their pipeline. Early on, database producer Ben Welsh set up a Django box at projects.latimes.com, where many of the Times' interactive projects live. There are other great solutions besides Django, including Ruby on Rails (the framework that powers the Times' articles and topics pages and many of the great data projects produced by The New York Times) and PHP (an inline scripting language so simple even I managed to learn it). Some people (including the L.A. Times, occasionally) are using Caspio to create and host data apps, sans programming. I am not a fan, for reasons Derek Willis sums up much better than I could, but if you have no other options, it's better than sitting on your hands.
- Templatize: Don't build it unless you can reuse it. The goal of all this is to be able to roll out projects rapidly (see #6), so you need templates, code snippets, Flash components, widgets, etc., that you can get at, customize and turn around quickly. Interactive graphics producer Sean Connelley was able to use the same county-level California map umpteen times as the basis for various election visualizations in Flash.
- Do breaking news: Your priority list may be full of long-term projects like school profiles and test scores, but often it's the quick-turnaround stuff that has the biggest immediate effect. This is where a close relationship with your newsgathering staff is crucial. At the Times, assistant metro editor Megan Garvey has been overseeing the metro staff's contributions to data projects for a few months now. When a Metrolink commuter train collided with a freight train on Sept. 12, Megan began mobilizing reporters to collect key information on the victims while Ben adapted an earlier Django project (templatizing in action!) to create a database of fatalities, complete with reader comments. Metro staffers updated the database via Django's easy-to-use admin interface. (We've also used Google Spreadsheets for drama-free collaborative data entry.) ... Update 11/29/2008: I was remiss in not pointing out Ben's earlier post on this topic.
- Develop new skills: Disclaimer: I know neither Django nor Flash, so I'm kind of a hypocrite here. I'm a lucky hypocrite, though, because I got to work with guys who dream in ActionScript and Python. If you don't have access to a Sean or a Ben — and I realize few newsrooms have the budget to hire tech gurus right now — then train and nurture your enthusiasts. IRE runs occasional Django boot camps, and there are a number of good online tutorials, including Jeff Croft's explanation of Django for non-programmers. Here's a nice primer on data visualization with Flash.
- Cohabitate (but marriage is optional): This may be less of an issue in smaller newsrooms, but in large organizations, collaboration can suffer when teams are split among several floors (or cities). The constituent parts of the Times' Data Desk — print and web graphics, the computer-assisted reporting team and the interactive projects team — have only been in the same place for a couple months, but the benefits to innovation and efficiency are already clear. For one thing, being in brainstorming distance of all the people you might want to bounce ideas off of is ideal, especially in breaking news situations. Also, once we had everybody in the same place, our onetime goal of unifying the reporting structure became less important. The interactive folks still report to latimes.com managing editor Daniel Gaines, and the computer-assisted reporting people continue to report to metro editor David Lauter. The graphics folks still report to their respective bosses. Yes, there are the occasional communication breakdowns and mixed messages. But there is broad agreement on the major priorities and regular conversation on needs and goals.
- Integrate: Don't let your projects dangle out there with a big ugly search box as their only point of entry. Weave them into the fabric of your site. We were inspired by the efforts of a number of newspapers — in particular the Indianapolis Star and its Gannett siblings — to make data projects a central goal of their newsgathering operations. But we wanted to do more than publish data for data's sake. We wanted it to have context and depth, and we didn't want to relegate data projects to a "Data Central"-type page, something Matt Waite (of Politifact fame) memorably dubbed the "data ghetto." (I would link to Waite's thoughtful post, but his site unfortunately reports that it "took a dirt nap recently.") I should note that the Times recently did fashion a data projects index of its own, but only as a secondary way in. The most important routes into data projects are still through related Times content and search engines.
- Give back: Understand that database and visualization projects demand substantial resources at a time when they're in very short supply. Not everyone in your newsroom will see the benefit. Make clear the value your work brings to the organization by looking for ways to pipe the best parts (interesting slices of data, say, or novel visualizations) into your print or broadcast product. For example, some of the election visualizations the data team produced were adapted for print use, and another was used on the air by a partner TV station.
