The Evolving Newsroom

Erik Ulken has posted a must-read top 10 list of lessons learned while setting up the data desk in the LA Times newsroom.

The data desk's job is to take detailed information that's dreary to read in text or table form and make it useful by presenting it in compelling and interactive formats. A well-known example is the LA Time's Homicide Map which allows readers to filter through a database of crime statistics and see them represented visually in graphics and on a map.



















The 10 lessons Erik posted are all good so I'm pasting them all in here but it's still worth checking out his post to see what else he has to say.

Erik's list echoes some of the things learned at the Telegraph when it was reinvigorating its website and beginning to integrate its web and print operations. I think these lessons would apply in all sorts of development and change management scenarios.

1. Find the believers: You'll likely discover enthusiasts and experts in places you didn't expect. In our case, teaming up with the Times' computer-assisted reporting staff, led by Doug Smith, was a no-brainer. Doug was publishing data to the web before the website had anybody devoted to interactive projects. But besides Doug's group, we found eager partners on the paper's graphics staff, where, for example, GIS expert Tom Lauder had already been playing with Flash and web-based mapping tools for a while. A number of reporters were collecting data for their stories and wondering what else could be done with it. We also found people on the tech side with a good news sense who intuitively understood what we were trying to do.
2. Get buy-in from above: For small projects, you might be able to collaborate informally with your fellow believers, but for big initiatives, you need the commitment of top editors who control the newsroom departments whose resources you'll draw on. At the Times, a series of meetings among senior editors to chart a strategic vision for the paper gave us an opportunity to float the data desk idea. This led to plans to devote some reporting resources to gathering data and to move members of the data team into a shared space near the editorial library (see #8).
3. Set some priorities: Your group may come from a variety of departments, but if their priorities are in alignment, disparate reporting structures might not be such a big issue. We engaged in "priority alignment" by inviting stakeholders from all the relevant departments (and their bosses) to a series of meetings with the goal of drafting a data strategy memo and setting some project priorities. (We arrived at these projects democratically by taping a big list on the wall and letting people vote by checkmark; ideas with the most checks made the cut.) Priorities will change, of course, but having some concrete goals to guide you will help.
4. Go off the reservation: No matter how good your IT department is, their priorities are unlikely to be in sync with yours. They're thinking big-picture product roadmaps with lots of moving pieces. Good luck fitting your database of dog names (oh yes, we did one of those) into their pipeline. Early on, database producer Ben Welsh set up a Django box at projects.latimes.com, where many of the Times' interactive projects live. There are other great solutions besides Django, including Ruby on Rails (the framework that powers the Times' articles and topics pages and many of the great data projects produced by The New York Times) and PHP (an inline scripting language so simple even I managed to learn it). Some people (including the L.A. Times, occasionally) are using Caspio to create and host data apps, sans programming. I am not a fan, for reasons Derek Willis sums up much better than I could, but if you have no other options, it's better than sitting on your hands.
5. Templatize: Don't build it unless you can reuse it. The goal of all this is to be able to roll out projects rapidly (see #6), so you need templates, code snippets, Flash components, widgets, etc., that you can get at, customize and turn around quickly. Interactive graphics producer Sean Connelley was able to use the same county-level California map umpteen times as the basis for various election visualizations in Flash.
6. Do breaking news: Your priority list may be full of long-term projects like school profiles and test scores, but often it's the quick-turnaround stuff that has the biggest immediate effect. This is where a close relationship with your newsgathering staff is crucial. At the Times, assistant metro editor Megan Garvey has been overseeing the metro staff's contributions to data projects for a few months now. When a Metrolink commuter train collided with a freight train on Sept. 12, Megan began mobilizing reporters to collect key information on the victims while Ben adapted an earlier Django project (templatizing in action!) to create a database of fatalities, complete with reader comments. Metro staffers updated the database via Django's easy-to-use admin interface. (We've also used Google Spreadsheets for drama-free collaborative data entry.) ... Update 11/29/2008: I was remiss in not pointing out Ben's earlier post on this topic.
7. Develop new skills: Disclaimer: I know neither Django nor Flash, so I'm kind of a hypocrite here. I'm a lucky hypocrite, though, because I got to work with guys who dream in ActionScript and Python. If you don't have access to a Sean or a Ben — and I realize few newsrooms have the budget to hire tech gurus right now — then train and nurture your enthusiasts. IRE runs occasional Django boot camps, and there are a number of good online tutorials, including Jeff Croft's explanation of Django for non-programmers. Here's a nice primer on data visualization with Flash.
8. Cohabitate (but marriage is optional): This may be less of an issue in smaller newsrooms, but in large organizations, collaboration can suffer when teams are split among several floors (or cities). The constituent parts of the Times' Data Desk — print and web graphics, the computer-assisted reporting team and the interactive projects team — have only been in the same place for a couple months, but the benefits to innovation and efficiency are already clear. For one thing, being in brainstorming distance of all the people you might want to bounce ideas off of is ideal, especially in breaking news situations. Also, once we had everybody in the same place, our onetime goal of unifying the reporting structure became less important. The interactive folks still report to latimes.com managing editor Daniel Gaines, and the computer-assisted reporting people continue to report to metro editor David Lauter. The graphics folks still report to their respective bosses. Yes, there are the occasional communication breakdowns and mixed messages. But there is broad agreement on the major priorities and regular conversation on needs and goals.
9. Integrate: Don't let your projects dangle out there with a big ugly search box as their only point of entry. Weave them into the fabric of your site. We were inspired by the efforts of a number of newspapers — in particular the Indianapolis Star and its Gannett siblings — to make data projects a central goal of their newsgathering operations. But we wanted to do more than publish data for data's sake. We wanted it to have context and depth, and we didn't want to relegate data projects to a "Data Central"-type page, something Matt Waite (of Politifact fame) memorably dubbed the "data ghetto." (I would link to Waite's thoughtful post, but his site unfortunately reports that it "took a dirt nap recently.") I should note that the Times recently did fashion a data projects index of its own, but only as a secondary way in. The most important routes into data projects are still through related Times content and search engines.
10. Give back: Understand that database and visualization projects demand substantial resources at a time when they're in very short supply. Not everyone in your newsroom will see the benefit. Make clear the value your work brings to the organization by looking for ways to pipe the best parts (interesting slices of data, say, or novel visualizations) into your print or broadcast product. For example, some of the election visualizations the data team produced were adapted for print use, and another was used on the air by a partner TV station.

Another piece of nostalgia from the NZ Herald Manual of Journalism 1967.

Pneumatic tubes as a story delivery system within newsrooms were before my time but what a shame, they look cracking.

Can't help but be impressed by the scope of this school newsroom at the City University of New York.

Newsroom

Reportr.net caught up with my former colleague Chris Lloyd, assistant managing editor of Telegraph Media Group, at the Online Journalism Symposium in Austin, Texas. He talked about the challenges the group faced in integrating its newsroom, including structural and cultural changes. Worth a listen if you haven't heard much about the process of integration before.

I've had a few interesting conversations this week about quality control in newsrooms, in part sparked by the diagram I posted last week showing how many pairs of hands various kinds of news stories go through before being published.

One thing that came up was how irritated people are by the number of typos regularly seen on news websites - including nzherald.co.nz and stuff.co.nz. As Nathan said in a recent comment, "I generally expect better of professionals."

I haven't worked at either of those sites so I can't speak for what goes on there. But in my experience the problem is twofold: inadequate systems (Content Management Systems (CMSs) that require users to undertake multiple steps for simple tasks - increasing the likelihood of mistakes); and poor workflows that were designed decades ago for print and haven't been revised to account for the website.

Some typos simply come from typing too fast - you'll find plenty of these in blogs and comments (including mine) where people type quickly and give only a cursory check before hitting 'send'.

More prevalent in news publishing are cut and paste errors - where a sentence has been edited or moved and during the process a word/punctuation mark/sentence has been inadvertently cut out, cut in half, transposed or reconstituted without due care for syntax or grammar. (That's why you see double words in text and missing full-stops).

How does it happen? Easy. Take a breaking news story. They're often bits of wire stories spliced together with a few paragraphs filed by an inhouse reporter. Very often those paragraphs will have been filed by email and copied and pasted into the web CMS. Just as often, the wire story will have been copied and pasted into the web CMS.

Take all that copying and pasting and re-working of sentences, throw in some time pressure and you've got mistakes waiting to happen. Speed is of the essence so the story goes live without anyone else reading it. Even with less urgent stories there's often only one duty web editor anyway, so there's no one else around to have a look at it before it goes online.

So why doesn't someone read it after it's gone live? Good question. Some structured peer review could go a long way here. And as I keep saying, readers would gladly help if you made it easy for them.

Most stories, though, are prepared for the newspaper first before finally being loaded online late at night by a tired person. On the way, they'll go through nine or more pairs of hands, typically, and two CMSs.

Most likely both CMSs will make hard work of cutting, pasting, categorising and styling text - increasing the likelihood of mistakes being made along the way. Cyber, a system widely used for print in NZ, would fall into this category I suspect, based on my brief encounter with it. If you have a different view, I'd love to hear from you.

What's the answer? One, news organisations need better CMSs which have been designed for neutral input and multi-platform output - no more cutting and pasting, easier editing and mark-up, and the fewer steps the better.



Two, they need to take a critical look at their workflows and redistribute resources: essentially take a couple of those people in the right-hand columns and move them to the left.

On a similar tack to yesterday's post about when a story is 'finished' enough to go online, I was reminded while planning a sub-editing lecture recently of how lopsided today's newsrooms are when it comes to handling stories for web and print.

The image below is a rough snapshot of how many pairs of hands various kinds of stories might pass through in your average newsroom - from reporter through chief reporter, news editor, web editor, chief sub-editor, page layout, sub-editor, check sub, proofreader, editor, edition controller and so on.

Hard to imagine anyone allocating resources like that if they were designing a newsroom today, isn't it.

On the upside: there's lots of pairs of eyes on those newspaper and shovelware stories to catch mistakes before they're published.

On the downside: there's lots of pairs of hands on those newspaper and shovelware stories to introduce mistakes before they're published.

Not to mention the extraordinary length of time it takes to get a story online using the shovelware method.

Time for a rethink, surely. A rethink, a critical analysis of newsroom processes, a redesign, re-training and roll-out.

"We were being asked to do things that we just couldn't bend the system any further to do."

Sound familiar?

That's PA's IT development manager Paul Berman talking to Silicon.com about the company's decision to use an open source platform, Nuxeo, to build a better content management system.

Like everyone else on the planet PA is handling an increasing volume of video, audio and slide shows as well as text and they need a system that can manage that simply and efficiently.

Anyone who works in a newsroom will know that the legacy systems we have simply aren't up to the job. Older web CMSs weren't built for multimedia in any volume and no matter how many scripts the good people in IT write to make the ageing print CMS talk nicely to the ageing web CMS, it's not a smooth process. In most newsrooms reporters still write into the print CMS.

Invariably getting something online involves multiple steps, things to remember, repetition and miles of mouse clicks and keystrokes.

That makes it difficult to distribute the burden of marking up and loading web copy among authors and editors, who are the natural candidates for the role in a web-first operation.

If it's a really complicated process, they're not going to want to do it, might not understand it well and cut corners, and will be distracted from the business at hand of researching and producing good stories.

"We have some really very specific requirements for our editorial applications in which we want to deliver a very effective tool for our journalists to do things with as few keystrokes as possible, efficiently and quickly," Paul Berman told Silicon.com.

On the reason for choosing open source, he said: "It's a mixture of the flexibility, the cost and the potential to scale it and make it really adaptable for our environment."

Interesting to see a major news group choose open source given the customary preference in such organisations for proprietary 'best of breed' systems.

Still on the theme of newsroom technology I wanted to point to this post from Rich Gordon on Mediashift Idea Lab about a Knight Foundation initiative to grant journalism scholarships to computer programmers. He notes that Georgia Tech is already running a computational journalism course.

Every now and then I get the feeling another piece of the puzzle is fitting into place. This, I think, is another piece of the puzzle.

It's not uncommon to find the odd programmer in a newsroom - they're the folk who fix the broken widgets and figure out how to translate the editor's bright ideas into code. But there's often only one, they're often not on the editorial payroll and sometimes they're not even on the editorial floor but tucked away in an IT department cubicle.

But imagine if you had a programmer at the morning news conference or weekly planning session. Someone pitches a story, it gets tossed around the table - what can we do for the paper, what can we do for the web, what's the community potential? The picture ed pulls up some images on the screen, the reporter mentions some data he came across that's relevant and a news editor recalls previous stories that dovetail nicely and an old series of pics that would be perfect. Round the table you go, asking: Can we do this? Would that work better? Are we telling this story in the most compelling way?

Instead of having to scurry off and find someone to ask what's possible or, sigh, put in a request form, the programmer can say right then and there - no, that's going to take too long and I can't be sure it'll work properly, but I have a better idea and I can do it if you give me x, y and z.

Now the editors can allocate staff and tech resources accordingly. The editors know what they're getting for the web and the paper, the reporter knows what words and data are required by when, the designers can get to work and the pic editor can get researchers or the duty snapper on the case.

Ah, it's a lovely world in my head.

Rich Gordon sees 'journalism programmers' as creators of a new generation of newsroom tools. Here's his take on why that's so important:

"Many journalists just aren't comfortable with technology. And even if they learn to use technology tools successfully in their work, few want to delve deeply into the process of developing new technology. And most media organizations don't seem to value their programming staffs or involve them in the journalism process. Instead, their work supports back-end systems like payroll and billing.

"But there's also clearly a need to educate computer scientists about journalism, which is why what Georgia Tech is doing is so important. When computer scientists think about journalism, it seems they often are most interested in trying to create software that replicates what journalists do - or makes them unnecessary. Think Google News - or the Northwestern InfoLab's News at Seven, an automated system that generates TV news 'shows' by harvesting information from the Web, translating it into human speech and delivering it through avatars.

"Don't get me wrong - if an algorithm can truly replace what a journalist does, I'm happy to let a computer do the work. But I'm convinced that the most indispensable things that journalists do - reporting, interviewing, analyzing, writing and editing - need to be done by humans.

"I'm also convinced that most technology professionals just don't understand how journalists do their jobs, what makes them essential to a democratic society, or how technology is helping destroy the business model that has supported the creation of original journalism. I'd like to see computer science scholars and professionals thinking more deeply about how technology can help journalists do a better job, not just put them out of a job."

I like this post from Scott Karp on Publishing 2.0 about the need for simplicity in newsroom systems. He quotes a newspaper exec who wanted any new system to be "so seamless, so transparent, so idiot-proof that I can do it without training — and so that it doesn’t add more than a heartbeat to my day."

I know the feeling. I remember the horror at being asked to teach print journalists how to use an overly complicated legacy web CMS. We didn't do it in the end, which is just as well because it would have been a disaster. Making even relatively minor workflow changes in the overly complicated print CMS was hard enough, and for good reason.

Most people don't give a fig how systems work and few want to expend any energy on learning how to do things quicker or work smarter or with more flexibility. 'Just tell me what button to push' is about the size of it. As Kathy Sierra said at Webstock: people don't want to be tool experts, but experts at what they're using the tool for.

This took me a while to accept, to be honest, I guess because I always like learning about a system so I can work more efficiently and flexibly. I assumed others would be happier once they'd learned a few more tricks. I was wrong. Most just wanted one simple way of working and as little variation as possible. You live and learn.

Scott's post goes on to cite a number of successful ventures based on really simple user interfaces - Google, Twitter, YouTube. It's good to be reminded now and then how important it really is to keep it simple, stupid.

Following on from looking at Twitter as a news delivery mechanism, I've been reading Paul Bradshaw's thoughts on using it as a communication tool in newsrooms.

He suggests signing up editors and reporters to Twitter accounts with mobile access and having them follow one another. That way they can give and take direction, file updates on where they are and what they're working on, and ask each other for tips, contacts, information.

This is not an entirely new idea. The topline instant message systems built into older print and radio CMSs have long given reporters a way to talk to each other instantly, directly and succinctly.

Twitter's different only in that it's web-based and mobile so you can take it with you wherever you go (give or take a few connectivity black spots). And it's group-based so you can have as few or as many people in your network as you want.

Obvious drawbacks raised by one commenter were that some people would see it as 'being watched' and resent having to participate; others would file updates only to look busy.

Fair point. It can sometimes be better to introduce tools rather than impose them - ie make a range of tools available and see which ones stick. That way there's more ownership and less friction: 'design for how people really are.' Early adopters have a way of catching others up in their enthusiasm and once enough people find the tool useful it will gain traction and can then become officially adopted.

Another current drawback is that you can't have multiple Twitter profiles on a single email account, so if a journalist wanted to keep work and private life separate they'd have to have two Twitter identities using different email accounts, which is a real faff.

The final hazard would be the IT department getting wind of it: they'd probably put it on the banned list.

Perhaps an in-house version might be better for newsroom communication, with Twitter best used as a tool for individual journalists to tell people what they're working on and ask for ideas and help not just from colleagues but their Twitter buddies and the blogosphere as well.

Irish Times has begun integrating its print newsroom with website Ireland.com. It is aiming for a 24-hour newsroom.

Link: Editor & Publisher

Knowing that I worked on a major newsroom integration project at the Telegraph in London, it will come as no surprise that I think integrating web and print is a good idea.

I was reading a Jeff Jarvis post about it and liked the way he put it:

"At the end of the day...we know this: Every journalist needs every tool to gather and tell every story how best it should be told. Every reader/listener/viewer/user should be able to get the news however, whenever, and wherever he or she wants. News operations won’t be able to afford the inefficiency of separate staffs all putting out the same news. And that’s why I think consolidation is inevitable."

I agree. More than that, news companies need to see themselves as news providers, not as newspapers or news websites. They need to get their content out to as many audiences as they can, in as many formats as they can, if they are to remain relevant. If you are publishing to mutiple formats, the logical way to go about it is to create the content in a neutral format and then modify it as required for each platform. Write into a neutral text editor, then mark it up for mobile, web and print as required. Writing for one format, then having to parse it to another is invariably a fag and terribly inefficient.

Jeff raises another good point in his post:

"One could consolidate too much and, in the words of one of my students, turn every journalist into an eight-armed monster — and do a halfassed job in any medium."

Yep, that's a danger, in the medium term anyway. But it needn't be so. The following points are worth making again and again. Yes, you must modernise your newsroom. Yes, you must tell stories in different formats where appropriate. Yes, you must move faster than you used to if you are to give your mobile and online audiences what they want. But no, it does not mean that every journalist has to file audio, video, text and timelines for every story every day. It means they need to talk to their editors about the best formats for telling this story on this day. It means the editors must make decisions about the best use of newsroom resources (reporters' time, equipment, studio time etc), and about the stories they think are relevant and interesting to their readers.

News judgement and resource management don't disappear in an integrated newsroom. They are central to it.

Anecdotally, I have been hearing more about reporters putting up their hands to go over to the digital desk, which is heartening, and not a bit surprising given the doom hanging over the future of print and the fact that online is so much fun.

Also starting to see more posts along the lines of this one from Melissa Worden, in which she notes the online desk can get a bit cliquey in some newsrooms and hard for others to break into.

It's a good point for newsroom change agents to be aware of: make sure that once you've explained how great online is, and given people a taste of it, that they can actually do some of it when they walk out of the training room and go back on the floor.

I like Melissa's point about sharing the toys. She's absolutely right. Ultimately, we want to get useful kit into as many people's hands at possible. Yes, some people will be better at some things than others - maybe audio, maybe visuals, maybe graphics, maybe Flash. That's all good. If you don't give people a chance, you'll never realise the talent you're sitting on.

While you're at it, why not give everyone a point and shoot video/camera/writing tool like an N95 or JasJam or HTC Tytn and show them how to upload, tag, assign categories, write abstracts etc. Give them a bit of time once a week to play and work on stories that need a bit of technical stretching. Make sure they know how to use the CMS and the appropriate workflow - who signs off on what. Then sit back and watch morning news conferences get more interesting.

Martin Newlands, a former Daily Telegraph editor, talks to Sky about the layout of the integrated newsroom he's building for a new English-language paper in Abu Dhabi. "I am, frankly, ripping off the design from The Daily Telegraph," he says.

That means a hub and spoke model with a central circular table where editors meet, and desks radiating out in spokes populated by various departments - arts, pictures, news etc - which are responsible for both web and print.

Gratifying to see the model showing up in all corners of the world.