Dig into data journalism with AP

The Associated Press will be sharing expertise and learning from other data journalists from around the country at a computer-assisted reporting conference put on by NICAR and Investigative Reporters & Editors (IRE). The annual event runs today through March 8 in Atlanta.

AP Editor for Interactive Technology Troy Thibodeaux (AP Photo)

AP Editor for Interactive Technology Troy Thibodeaux (AP Photo)

“Every year we pick up tools and techniques that become essential to our data journalism toolkit. And every year, I personally steal a few teaching ideas that help me bring the material back to the broader AP,” said AP Editor for Interactive Technology Troy Thibodeaux. “All this, and we get to compare notes and ask questions of the people in other news organizations whose work most inspires and challenges us.”

Here’s some of the hands-on training AP is providing:

THURSDAY, MARCH 5

FRIDAY, MARCH 6

SATURDAY, MARCH 7

  • Getting started with SQLite
    This workshop led by Thibodeaux will provide an introduction to the world of SQL (Structured Query Language), “the lingua franca of relational databases.”

Automated earnings stories multiply

The Associated Press, working with Automated Insights and Zacks Investment Research, is now automatically generating more than 3,000 stories about U.S. corporate earnings each quarter, a tenfold increase over what AP reporters and editors created previously. Here, Assistant Business Editor Philana Patterson, who has been overseeing the rollout of this process in the newsroom, gives an update on AP’s automation efforts that began last summer.

Assistant Business Editor Philana Patterson (AP Photo).

Assistant Business Editor Philana Patterson (AP Photo).

What changes has AP made to the automation process?
Since automation began in July, AP has added a number of enhancements to the stories. Descriptions of businesses have been added and the stories now include forward-looking guidance provided by the companies. We are running smoothly, and always looking for opportunities, along with Zacks and AI, to improve what we are producing with automation.

What has the reaction been?
There has been a great deal of interest about how automation works from both members and readers, and overall the reaction has been incredibly positive. AP members are getting more stories about companies in their markets than ever before. We want this process to be as transparent as possible so we have added an explanation of how earnings automation works. It can be found on Automated Insights’ landing page: http://www.automatedinsights.com/ap/.

That link, and one from Zacks, is provided in the tagline of each story. We’ve also encouraged our members and subscribers to make these links available to readers when using the stories, especially online.

Internally, the reaction has been positive from staff, largely because automation has freed up valuable reporting time and reduced the amount of data-processing type work they had been doing.

How does AP ensure quality control?
Quality control was critical from the outset. We worked with Zacks and AI to make sure that every step of the process would produce stories without errors. When we launched last summer, a fair number of errors were discovered in the testing process. We then worked with Zacks and AI on solutions to ensure they wouldn’t happen again. Today, mistakes are rare. Pretty much the only time we will now have an error is if a number is entered incorrectly into the system at the beginning. Once you set up automation, and go through a rigorous testing process, you reduce the prospect of errors. In fact, we have far fewer errors than we did when we were writing earnings reports manually.

Has automation allowed staff to focus more on reporting?
Absolutely. Like all media, we are working with limited resources and it’s critical that we maximize the time reporters have to do journalism and break news. We estimate the automation of earnings reports has freed up about 20 percent of the time that we had spread throughout the staff in producing earnings reports each quarter. It is enabling us to reconfigure our business breaking news operations to be more in sync with social media and user-generated content, and focus more reporters on higher-end enterprise stories that break news that no one else has. Our goals are to break more business news than our competitors, aim higher on investigative and explanatory journalism and focus more of our work on the general consumer. We’ve got some big projects in the works. Automation is helping us free up resources to do all of these things.

What’s next?
This quarter, we are testing the automation of earnings from Canadian and European companies. We expect to add further enhancements and more companies in future quarters. My colleagues in the sports department are also exploring small-audience sports for automation in order to free staff to report news that fans and consumers do not get on the field or a broadcast. We expect to be talking about automation through the year, including at this year’s SXSW Interactive in Austin, Texas.

New AP tool helps journalists manage data

troy_thibodeaux_headshot

Troy Thibodeaux (AP Photo)

The Associated Press’ data team and developers from civic technology company DataMade have created a new tool to make it easier for journalists to add context to a data set. Here, AP Editor for Interactive Technology Troy Thibodeaux, who conceived and launched the tool called Geomancer, explains its potential:

Why did AP create Geomancer?
AP has been a pioneer in data journalism and is committed to helping journalists use data more efficiently to find and tell important stories. We won a grant last year from the Knight Prototype Fund to build an open-source tool to help journalists make sense of data by mashing it up with other data sets about the same geographic location.

For reporters who work with data, it’s a common and laborious task to look up population or demographic data about the counties or ZIP codes represented in a given data set. Geomancer puts this data just a few clicks away. Our goal is to remove the drudgery from data so reporters can focus on finding the story.

How does it work?
Currently, the Geomancer prototype includes two data sources: the U.S. Census Bureau’s 2013 American Community Survey (via CensusReporter.org) and federal contracts from the U.S. Office of Management and Budget’s USASpending.org. It allows users to find data based on 10 geographic levels, including state, county and congressional district. There has already been some community interest in adding new data sets and geographical types.

The Geomancer team has created a working demo (geomancer.io) to show the tool’s potential and simple instructions that any newsroom can follow to install its own Geomancer and build its own warehouse of geography-based data sources. The Geomancer blog at geomancer.ap.org includes links to example data sets for getting started.

What’s next?
We’ll be working to help journalists inside and outside of the AP use Geomancer, and we are excited to see what stories it will help them produce. We’re looking forward to getting feedback from users and will be exploring ways to refine the tool, which is in a beta version.

Q&A: The changing market for video news

The Associated Press today released a report looking at the news market in the Middle East and North Africa and suggesting ways it needs to evolve, particularly when it comes to video. The report is the latest in a series of Deloitte studies for AP into video news consumption globally. (The first covered Europe and the second covered Asia.)

Here, Sue Brooks, director of international products and platforms for AP, explains why the market for video news has never been stronger.

What have been the most striking findings of the reports?

The big “ah-ha” moment for me was the realization that news junkies see video as an essential part of their daily news fix. Although there are a lot of variations in the data across markets, consumers were consistent in their demand for more high-quality online video content – and this is especially true of consumers who are interested in the news, generally.

Sue Brooks

Sue Brooks

The research shows that this group is more likely to access a story if it has an accompanying video, and that video consumers have a higher dwell time on news content each day. When we asked why, people told us it was because video helps bring a story to life and improve their understanding of it. For example, in the Middle East, a massive 83 percent of consumers find this to be the case.

This overwhelming demand for video presents a number of opportunities for us and our customers. It also highlights how critical it is for the industry to adapt. In Europe, more than a quarter of respondents said they’d go elsewhere if video wasn’t available at their preferred news source.

How and why has demand for video news changed?

Video news stopped being the sole preserve of terrestrial and satellite broadcasters quite some time ago and online and mobile video news are now the norm; in fact many of our video customers are now newspapers.

It’s clear that the need for video has continued to grow and has achieved ever-greater importance. We expect this will continue with the spread of smartphones and strong growth in tablets, as well as steadily increasing broadband speeds via fixed and mobile connections.

How is AP helping its customers evolve to satisfy this demand?

The primary goal of the research is to help our customers understand the changes in consumer demand, but it has also given us insight into what we need to do to help our customers meet the challenges facing them.

We are at the forefront of change and, of course, our customers need us to keep our products and services relevant. That’s why in 2012 we launched AP Video Hub. We needed to address the increase in demand from online publishers for video news with a service that was compelling and easy to use. These customers saw video as another critical element of their storytelling tool box, but before 2012 it was difficult for non-broadcasters to access and use AP video easily.

Since the launch of AP Video Hub, the platform has gone from strength to strength and we recently announced our Content Partner Offer, which allows third-party content to be sold via the platform. The first partner to go live was Newsflare, an online video news community for user-generated video, which adds a new dimension to the site and meets an increasing demand for this type of content.

We also launched a new video service in the Middle East earlier this year to meet the insatiable demand for news in the region, offering customers more unique video content centered on the news that matters most to consumers there. Our Deloitte research showed that, perhaps more than anywhere else in the world, Middle East consumers value trusted news sources – particularly when it comes to video. We want to ensure that our customers are in a position to provide their own customers exactly what they need.

Q&A: How AP stays ahead in the mobile space

The Associated Press has released a new version of AP Mobile, its award-winning news app, to offer full support for Apple’s new operating system, iOS 8, and the hotly anticipated iPhone 6 and iPhone 6 Plus. Here, Michael Boord, director of mobile products, explains how AP is staying ahead in the mobile space:

Director of Mobile Products Michael Boord (AP Photo).

Director of Mobile Products Michael Boord (AP Photo).

How is AP Mobile optimized for iOS 8 and the new iPhone?
From a user perspective, the app won’t look much different, but our developers have been working for months making backend changes that should make the app perform better than ever. We’ve also made some enhancements to the sharing features, so users can more easily post stories on social media. Users will notice a new “Big Story” carousel on their home screen to more prominently surface the major developing stories AP is covering around the world, from the Ebola outbreak to the entertainment awards season.

What content can users find on AP Mobile?
The app features the best of AP’s journalism in every format, including breaking news alerts, hourly radio updates, stunning images, interactive graphics and video reports. It’s customizable, so users can choose what news categories they want, from sports and entertainment to politics or local news.

MOBILEHow does AP Mobile showcase local content?
Users pick which local publications they want to see from our more than 1,100 contributing partners. We’re also working closely with AP members to showcase their work on the app’s tiled-based home screen. For example, we recently featured reporting from the St. Louis Post-Dispatch on the unrest in Ferguson, Missouri, and content from the Richmond Times-Dispatch on the corruption trial of former Virginia Gov. Bob McDonnell. These were stories of national interest and we helped our users discover content they might not otherwise see.

Who’s reading AP Mobile?
Since its launch in 2008, AP Mobile has been downloaded over 14 million times and is consistently among the highest-rated news apps in the app stores. We tend to see spikes in the number of downloads when breaking news happens, and we find it gratifying that people turn to AP for fast, accurate information. Even the White House press secretary told CNN he uses the app to stay up to date.

What’s next?
Because we’ve already optimized for iO8, we are well-positioned for substantial updates in the future. We’re hoping to roll out some new content features in November and are continuing to look for unique ways to surface compelling, authoritative local content.

Download AP Mobile from the Apple iTunes or Google Play app stores.

A leap forward in quarterly earnings stories

The Associated Press announced in an advisory to customers today that the majority of U.S. corporate earnings stories for our business news report will eventually be produced using automation technology.

Here, Lou Ferrara, the AP managing editor who oversees business news, explains how this leap forward takes advantage of new technologies to free journalists to spend more time on things like beat reporting and source development while increasing, by a factor of more than 10, the volume of earnings reports for customers.

Lou Ferrara, vice president and managing editor

Lou Ferrara, vice president and managing editor

Why is the AP doing this?

Like all media companies, AP is constantly reviewing what content it needs to provide to customers and the best use of its reporting resources. At the same time, we analyze the value of the content we produce in the marketplace.

For many years, we have been spending a lot of time crunching numbers and rewriting information from companies to publish approximately 300 earnings reports each quarter. We discovered that automation technology, from a company called Automated Insights, paired with data from Zacks Investment Research, would allow us to automate short stories – 150 to 300 words — about the earnings of companies in roughly the same time that it took our reporters.

And instead of providing 300 stories manually, we can provide up to 4,400 automatically for companies throughout the United States each quarter.

We believe technological automation will be a part of many businesses, including those in media. As part of its business relationship with Automated Insights, AP participated in the company’s latest round of investment financing with other strategic partners.

Does it mean we are no longer providing editorial coverage of earnings reports?

No. If anything, we are doubling down on the journalism we will do around earnings reports and business coverage.

We are going to use our brains and time in more enterprising ways during earnings season. Rather than spending a great deal of time focusing on the release of earnings and hammering out a quick story recapping each one, we are going to automate that process for all U.S. companies in the 4,400. (We are exploring whether we can automate earnings from companies outside the United States.)

Instead, our journalists will focus on reporting and writing stories about what the numbers mean and what gets said in earnings calls on the day of the release, identifying trends and finding exclusive stories we can publish at the time of the earnings reports.

AP’s staff breaks a lot of business news and obtains numerous exclusives throughout the year from many of the top companies in the world. We know that is what our customers want and we are going to deliver more of it through this process.

Are we eliminating jobs to do this?

No. This is about using technology to free journalists to do more journalism and less data processing, not about eliminating jobs. In fact, most of the staff has been receptive to the effort and involved for the past few months of discussion.

How does it work?

Zacks maintains the data when the earnings reports are issued. Automated Insights has algorithms that ping that data and then in seconds output a story. The structure for the earnings reports stories was crafted by AP with Automated Insights. All conform to AP Style, the standard of journalistic style.

The stories will be labeled as being produced automatically with material from Zacks.

As we begin using automation technology in July, we will check each automatically generated report and then publish to the AP wire. As we work out any problems, we hope to move to a model of more fully automating the reports and spot-checking the feed for quality control.

Will you be automating other parts of the AP report?

Interestingly, we already have been automating a good chunk of AP’s sports agate report for several years. Data comes from STATS, the sports statistics company, and is automated and formatted into our systems for distribution. A majority of our agate is produced this way.

By comparison, though, the earnings reports are produced into stories – not just data feeds. And we are looking at whether there are other things we should be automating in this way. Last football season, we introduced an automated NFL player ranking on the website for pro football that we host for newspapers. That ranking included automated text descriptions of player performances each week, which were produced by Automated Insights. We also are examining the potential for automating results stories for lower-audience sports.

When will the automated earnings reports be available?

We are planning to go live in July, and we will be paying close attention to all of the reports as we adapt to this new process. We will address any concerns or bugs, and then keep moving ahead.

Our hope is that customers will begin to see the benefits almost immediately through more breaking business news and an increased volume of earnings reports. Many customers will receive info for companies in their markets that they never received from AP before.

Visit AP at ONA in Atlanta

Kent_ONA

AP Standards Editor Tom Kent helps lead a break out session for an ethics panel at ONA. (Photo by Fergus Bell)

The Associated Press is joining digital journalists from around the country at the 2013 Online News Association conference in Atlanta, which runs today through Oct. 19. Here’s a rundown of where you’ll find AP:

October 17:

  • Standards Editor Tom Kent (@tjrkent) and International Social Media Editor Fergus Bell (@fergb) will participate in a discussion about an online code of ethics from 2:45 p.m. – 3:45 p.m. ET.

October 18:

  • Minkoff teams up with ProPublica to give a workshop from 10:30 a.m. – 11:30 a.m. ET on how reporters can find hidden structured data online to both drive and supplement stories, using non-programmatic tools.
  • Stop by the AP table from 2 p.m. – 4 p.m. ET for a sneak peak at new AP Stylebook (@APStylebook) products that are coming soon.

And looking ahead to 2014, you can also stop by the AP table to learn more AP’s Election services.

Paul Cheung named director of interactive and digital news production

Image

Paul Cheung

The following promotion was announced today by AP Vice President and Managing Editor Lou Ferrara in an internal memo:

I’m pleased to announce that Paul Cheung is now AP’s director of interactive and digital news production.

Paul joined the AP newsroom in 2010 after roles at the Miami Herald and the Wall Street Journal. Most recently, he’s been serving as the president of the Asian American Journalists Association, where has received rave reviews from colleagues throughout the industry for his leadership and approach in building the organization.

Paul has been in the interim director role at the AP for the past several months, leading the team responsible for some of the company’s key products and innovation areas: interactives, data analysis and visualization, video explainers, mapping, GraphicsBank, news research, AP Overview and print graphics. The team — which includes staff at several AP offices around the world — also has been instrumental in the launch of the online products known as the digital news experiences, as well as parts of AP Mobile.

In this new role, Paul will oversee more of the production of those digital products, which had been part of the Nerve Center’s evolution the past few years. The digital news experiences, AP Mobile and a few other products related to specific customers will now roll up into the interactive and digital news production department. As part of this move, Jaime Holguin, who as the news development manager at the Nerve Center has played a pivotal role in the execution of many of these products, will report into Paul and the new department. Jaime, as he has been, will work with the business operations on products and be the point person between the rebooted, news-focused Nerve Center and the new department.

In the weeks ahead, Jaime and Paul will be working with Tamer Fakahany, the deputy managing editor overseeing the Nerve Center, on workflows and the location of products within the headquarters newsroom.

Please join me in offering congratulations, as these moves will allow us to continue to fine-tune our products and grow while refocusing the Nerve Center on the coordination of AP’s news report every day.