brian abelson

BarCamp News Innovation, Philadelphia, April 27th

2013-04-10T00:00:00-04:00

A little more than two weeks from now, Erika Owens, Greg Linch, and I will be holding an “unconference” at BarCamp News Innovation to talk about the theory, methods, and implications of media impact measurement. We’re hoping to have a pretty focused discussion, centered on implementation. In particular, I’ll be talking about tangible experiments I’ve been carrying out with the New York Times as my laboratory.

In preparation for the discussion, I’m hoping to get your thoughts and concerns about impact measurement (read up here) – what have we done right so far and what’s missing from the conversation? If you’re interested in these questions, make sure to join Greg and I’s google group and follow along. See you on April 27th!

An impact reading list

2013-04-02T00:00:00-04:00

EDIT THIS DOCUMENT

What’s in here?

This bibliography contains a smattering of pieces, long and short, that are relevant to the study of media impact. From blog posts and case studies to fully-developed frameworks for analysis, reading selections from this list should quickly get you up to speed with the field.

Where do I start?

There are so many good pieces in here, but these are three of my favorites:

Tofel, Richard J., “Non-profit journalism – Issues around impact: A white paper from ProPublica,” February, 2013. http://s3.amazonaws.com/propublica/assets/about/LFA_ProPublica-white-paper_2.1.pdf
Stray, Jonathan, “Metrics, metrics everywhere: How do we measure the impact of journalism?,” Nieman Journalism Lab, August 17, 2012. http://www.niemanlab.org/2012/08/metrics-metrics-everywhere-how-do-we-measure-the-impact-of-journalism/
Linch, Greg, “Quantifying impact: A better metric for measuring journalism”, January 14, 2012. http://www.greglinch.com/blog/2012/01/14/quantifying-impact-a-better-metric-for-measuring-journalism/

Can I add something?

Sure! Just go here and edit the document using Draft – a great new tool for versioned writing. Once you add something, I’ll update this website to reflect those changes.

This format is ugly!

I know, I’m horrible at design. Feel free to take all these links and make something pretty and more useful!

Impact Reading List

Abelson, Brian, “HI Score: Towards a new metric of influence,” June 27, 2012. http://harmony-institute.org/therippleeffect/2012/06/27/hi-score-towards-a-new-metric-of-influence/

Ad Council, “Overview of Ad Council research & evaluation procedures,” Ad Council, Date Unknown. http://www.adcouncil.org/Impact/Research/Overview-of-Ad-Council-Research-Evaluation-Procedures

Anderson, C.W. et al, “Post-Industrial Journalism: Adapting to the Present,” Tow Center, Fall 2012. http://towcenter.org/wp-content/uploads/2012/11/TOWCenter-Post_Industrial_Journalism.pdf

Barrett, Diana and Leddy, Sheila, “Assessing Creative Media’s Social Impact”, The Fledgling Fund, December 2008. http://www.thefledglingfund.org/wp-content/uploads/2012/11/Impact-Paper.pdf

Blakely, Johanna (Norman Lear Center), “Research study finds that a film can have a measurable impact on audience behavior”, February 22, 2012 http://www.learcenter.org/pdf/FoodInc.pdf

Blakely, Johanna, “TedX Phoenix – Movies for a change”, February 12, 2012. http://youtu.be/Pb0FZPzzWuk

Bornstein, David, “Why we need solutions journalism,” Skoll World Forum, 2012. http://skollworldforum.org/debate-post/why-we-need-solutions-journalism/

Brock, Andrea, et al./ Center for Effective Philanthropy, “Room for Improvement: Foundations’ Support of Nonprofit Performance Assessment,” 2012 http://www.effectivephilanthropy.org/assets/pdfs/Room%20for%20Improvement.pdf

Caulkin, Simon, “The rule is simple: Be careful what you measure”, February 9, 2008. http://www.guardian.co.uk/business/2008/feb/10/businesscomment1

Chinn, Dana et al., “Measuring the online impact of your information project”, Knight Foundation / FSG Social Impact Advisors, May 31, 2011. http://www.knightfoundation.org/media/uploads/publication_pdfs/Measuring-the-Online-Impact-of-Information-Projects-092910-FINAL_1.pdf

Clark, Jessica, “Find needs and five tools for measuring media impact,” May 11, 2010. http://www.pbs.org/mediashift/2010/05/5-needs-and-5-tools-for-measuring-media-impact131.html

Clark, Jessica and Van Slyke, Tracy, “Investing in Impact”, The Center for Social Media, May 12, 2010. http://www.centerforsocialmedia.org/sites/default/files/documents/pages/Investing_in_Impact.pdf

Clark, Jessica and Abrash, Barbara, “Social justice documentary: Designing for impact,” Center for Social Media, September 2011. http://www.centerforsocialmedia.org/sites/default/files/documents/pages/designing_for_impact.pdf

Community Wealth Ventures, “How nonprofit news ventures seek sustainability”, Knight Foundation, October 2011. http://www.knightfoundation.org/media/uploads/publication_pdfs/13664_KF_NPNews_Overview_10-17-2.pdf

Channel 4 BritDoc Foundation, “Evaluation”, http://britdoc.org/real_good/evaluation

Duros, Sally, “Impact counts for hyperlocal news, but how to count it?”, August 6, 2012. http://www.blockbyblock.us/2012/08/06/impact-and-what-it-is-for-community-and-hyperlocal-news/

Fisch, Shaolm M. and Truglio, Rosemarie T., “G is for growing: Thirty years of research on children and Sesame Street,” Children’s Television Workshop, 2000. http://www.amazon.com/Growing-Thirty-Research-Children-Communications/dp/0805833951

Fox, Steve, “Why are we spending so much time ‘Measuring the Impact of Journalism?’” March 30, 2012. http://umassjournalismprofs.wordpress.com/2012/03/30/why-are-we-spending-so-much-time-measuring-the-impact-of-journalism/

FSG Social Impact Advisors/ John S. and James L. Knight Foundation, “Measuring the Online Impact of Your Information Project: A Primer for Practitioners and Funders,” 2010. http://www.fsg.org/tabid/191/ArticleId/428/Default.aspx?srpush=true

FSG Social Impact Advisors/ John S. and James L. Knight Foundation, “IMPACT: A practical guide to evaluating community information projects,” February 2011. http://www.knightfoundation.org/media/uploads/publication_pdfs/Impact-a-guide-to-Evaluating_Community_Info_Projects.pdf

Gates, Bill, “My Plan to Fix The World’s Biggest Problems”, Wall Street Journal, January 25, 2013. http://online.wsj.com/article/SB10001424127887323539804578261780648285770.html

Gates Foundation, “A Guide to Actionable Measurement,” 2010. http://www.gatesfoundation.org/learning/Documents/guide-to-actionable-measurement.pdf

Gigli, Susan, “What Is ‘Disruptive Metrics’“, March 20, 2013. http://www.intermedia.org/2013/03/20/what-is-disruptive-metrics/

Gill, Kathy E., “Carnival of Journalism: How To Measure What Matters?”, April 4, 2012. http://wiredpen.com/2012/04/04/carnival-of-journalism-how-to-measure-what-matters/

Gordon, John, “See, say, feel, do: Social media metrics that matter,” Fenton, Date Unknown. http://www.fenton.com/resources/see-say-feel-do/

Graves, Lucas, “Traffic Jam: We’ll never agree about online audience size,” September 7, 2010. http://www.cjr.org/reports/traffic_jam.php?page=all

Graves, Lucas et al., “Confusion online: Faulty metrics and the future of digital journalism,” Tow Center, September, 2010. http://brianabelson.com/assets/impact_papers/faulty_metrics.pdf

Green, Daniel, “Eyeballs and Impact: Are we measuring the right things if we care about social progress?,” Skoll World Forum, 2012. http://skollworldforum.org/debate-post/eyeballs-and-impact-are-we-measuring-the-right-things-if-we-care-about-social-progress/

Harmony Institute, “Waiting for Superman: Entertainment evaluation highlights”, Harmony Institute, May 2011. http://harmony-institute.org/wp-content/uploads/2011/07/WFS_Highlights_20110701.pdf

Heedy, Lucy and Keen, Sara, “SROI for funders,” New Philanthropy Capital, September 2010. http://www.thinknpc.org/?attachment_id=815&post-parent=4924

Hickman, Blair et al., “Best of MuckReads 2012”, ProPublica, December 31, 2012. http://www.propublica.org/article/best-of-muckreads-2012

International Center For Journalists, “An evaluation of the Knight International Journalism Fellowships”, ICFJ, Date Unknown. http://www.knightfoundation.org/media/uploads/publication_pdfs/Evaluation_of_Knight_ICFJ_Fellowships_final.pdf

International Center For Journalists, “Evaluation field manual and tools for the Knight International Journalism Fellowships”, ICFJ, January 2011. http://issuu.com/kijf/docs/icfj_knight_international_evaluation_manual

Johnson, John, “A New Approach to Making Films That Matter,” January 11, 2013. http://www.good.is/posts/a-new-approach-to-making-films-that-matter

Karlan, Dean et al., “More than good intentions: How a new economics is helping to solve global poverty”, Dutton, 2011. http://www.amazon.com/More-Than-Good-Intentions-Economics/dp/052595189X

KETC, “Facing the mortgage crisis: People, connections, resources”, Spring 2008. https://mediaengage.adobeconnect.com/_a938034862/p90449908/

Kohavi, Ron, et al., “Trustworthy online controlled experiments: Five puzzling outcomes explained”, Microsoft, 2012. http://www.exp-platform.com/Documents/puzzlingOutcomesInControlledExperiments.pdf

Kramer, Mark and Kania, John, “Collective Impact”, Stanford Social Innovation Review, Winter 2011. http://www.fsg.org/tabid/191/ArticleId/211/Default.aspx?srpush=true

Kristof, Nicoholas D., “Getting smart on aid,” May 18, 2011. http://www.nytimes.com/2011/05/19/opinion/19kristof.html?_r=1&partner=rssnyt&emc=rss

King, Gary et al., “Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference,” Political Analysis 15:199–236, 2007. http://gking.harvard.edu/files/gking/files/matchp.pdf

Linch, Greg, “Quantifying impact: A better metric for measuring journalism”, January 14, 2012. http://www.greglinch.com/blog/2012/01/14/quantifying-impact-a-better-metric-for-measuring-journalism/

LFA Group: Learning for Action / Bill & Melinda Gates Foundation / John S. and James L. Knight Foundation, “Deepening Engagement for Lasting Impact: Measuring Media Performance and Results,” Februrary, 2013. (Forthcoming)

Los Angeles Times Data Desk, “Complete guide to the LAFD data controversy”, Los Angeles Times, April 12, 2012 (Ongoing). http://timelines.latimes.com/lafd-data-controversy/

Mayer, Joy and Stern, Ruben, “A resource for newsrooms: Identifying and measuring audience engagement efforts” June 3, 2011, http://www.rjionline.org/sites/default/files/theengagementmetric-fullreport-spring2011.pdf

McKinsey & Company, “Learning for Social Impact”, http://lsi.mckinsey.com/

National Center for Media Engagement (NCME), “Measuring Public Media’s Impact: Challenges and Opportunities”, March 2013. http://mediaengage.org/CommunicateImpact/measure3.cfm

Ní Ógáin, Eibhlín et al., “Making an impact: Impact measurement among charities and social enterprises in the UK”, New Philanthropy Capital, October 2012. http://www.thinknpc.org/publications/making-an-impact/making-an-impact/

Peters, Jeremy W., “Some Newspapers, Tracking Readers Online, Shift Coverage”, The New York Times, September 5, 2010. http://www.nytimes.com/2010/09/06/business/media/06track.html

Peters, Jeremy W., “A Newspaper, and a Legacy, Reordered”, February 11, 2012 http://www.nytimes.com/2012/02/12/business/media/the-washington-post-recast-for-a-digital-future.html?pagewanted=all&_r=0

Pilhofer, Aron, “Find the right metric for news”, July 25, 2012. http://aronpilhofer.com/post/27993980039/the-right-metric-for-news

Priem, Jason et al., “The Altmetrics Collection”, Public Library of Science, 2012. http://www.ploscollections.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0048753

Ramsay, Clay et al., “Misinformation and the 2010 Election: A Study of the US Electorate,” December 10, 2010. http://www.worldpublicopinion.org/pipa/pdf/dec10/Misinformation_Dec10_rpt.pdf

Reisman, Jane et al., “A handbook of data collection tools: Companion to ‘A guide to measuring advocacy and policy’“, Organizational Research Services, 2007. http://www.organizationalresearch.com/publicationsandresources/a_handbook_of_data_collection_tools.pdf

Richardson, Breeze, “Measuring community engagement: A case study from Chicago Public Media,” Reynolds Journalism Institute, December 1, 2011. http://www.rjionline.org/blog/measuring-community-engagement-case-study-chicago-public-media

Rosenblum, Michael, “How to Quantify the Impact of Journalism,” March 30, 2012. http://www.nyvs.com/blog/user/michael/How-To-Quantify-The-Impact-of-Journalism

Salagnik, Matthew J. et al., “Experimental study of inequality and unpredictability in an artificial cultural market” Science 311, 854, 2006.http://brianabelson.com/assets/impact_papers/SalagnikWatts2006.pdf

Search, Jess, “Beyond the box office: New docuemntary valuations,” Channel 4 BritDoc Foundation, May 2011. http://www.documentary.org/images/news/2011/AnInconvenientTruth_BeyondTheBoxOffice.pdf

Sparkwise: http://sparkwi.se/

Spittle, Andrew, “Defining new metrics for journalism,” April 28, 2012. http://andrewspittle.net/2012/04/28/new-metrics/

Stray, Jonathan, “By the numbers, American journalism failed to inform voters,” December 29, 2010. http://jonathanstray.com/american-journalism-failed-to-inform-voters

Stray, Jonathan, “Designing journalism to be used,” September 26, 2010. http://jonathanstray.com/designing-journalism-to-be-used

Stray, Jonathan, “Does Journalism Work?,” December 14, 2010. http://jonathanstray.com/does-journalism-work

Stray, Jonathan, “Metrics, metrics everywhere: How do we measure the impact of journalism?,” Nieman Journalism Lab, August 17, 2012. http://www.niemanlab.org/2012/08/metrics-metrics-everywhere-how-do-we-measure-the-impact-of-journalism/

Stuart, E.A., “Matching methods for causal inference: A review and a look forward,” Statistical Science 25(1): 1-21, 2010. http://biostat.jhsph.edu/~estuart/Stuart10.StatSci.pdf

Tofel, Richard J., “Non-profit journalism – Issues around impact: A white paper from ProPublica,” February, 2013. http://s3.amazonaws.com/propublica/assets/about/LFA_ProPublica-white-paper_2.1.pdf

TRASI: Tools and Resources for Assessing Social Impact: http://trasicommunity.ning.com/

Ward, Nicholas/ Infomart, “Un-juking the stats: Measuring journalism’s impact on society,” October 17, 2012. http://www.infomart.com/2012/10/17/un-juking-the-stats-measuring-journalisms-impact-on-society/

Ward, Nicholas/ Infomart, “We become what we measure: Developing impact metrics for journalism,” October 3, 2012. http://www.infomart.com/2012/10/03/we-become-what-we-measure-developing-impact-metrics-for-journalism/

White, John Myles, “Bandit Algorithms for Website Optimization: Developing, Deploying, and Debugging”, O’Reilly Media, 2012. http://oreilly.com/shop/product/0636920027393.html?bB=g

WITNESS, “WITNESS performance dashboard,” WITNESS, December 2009. http://www.witness.org/sites/default/files/downloads/witness-dashboard-evaluation-2010.pdf

Zuckerman, Ethan, “Metrics for civic impacts of journalism,” June 30, 2011. http://www.ethanzuckerman.com/blog/2011/06/30/metrics-for-civic-impacts-of-journalism/

Creating a metric for news apps

2013-03-18T00:00:00-04:00

More and more, newsrooms large and small are adopting digital and data-driven techniques to engage readers with their work. News applications, or “large web-based interactive databases that tell a journalistic story using software instead of words and pictures”, have emerged as the standard medium through which journalists present the outcomes of data-driven projects. From Dollars for Docs to Dogs of NYC, news apps present readers with interactive experiences that offer them opportunities to engage more deeply with a story than traditional article-based formats. Given the common structure of news apps (discussed below) and the existence of well-developed tools for tracking user-level events on a website, I propose a simple framework for measuring engagement with news applications.

What makes a news app “successful”?

In a recent InsideThunderdome live chat, Scott Klein, editor of News Applications at ProPublica, wrote, “what I like to see in a news application is a way to see the “far” view (in other words, the big national picture) and the “near” view (how the big national phenomenon relates to me personally).” Put another way, the far view “lets you know why you should care, what the story is, what the national phenomenon is and how places compare to each other.” The near view, on the other hand, walks you “through levels of abstraction down to the very specific – your town, your street, your school. So if we’re successful now you know the national picture, why you should care, and what it’s got to do with you.”

Echoing this philosophy, Michael Keller, Senior Data Reporter at Daily Beast says, “we think of interactives as tools to help people to see how a story might matter to them. Here, ‘success’ means drawing people into the story and, if we’re lucky, also building in ways to hear how it affected them.”

Other indications of what makes a “successful” news app come through job ads. For instance, PBS’s News Hour is looking for someone to “develop news applications that allow users to explore the stories behind data.”

From these examples, a pattern emerges: a “successful” news application is one that presents a reader with a story and offers them an opportunity to dig deeper. Given this common structure of news applications, it should be possible to create a simple metric that captures the degree to which readers use an app to move from the far view to the near view. Such a metric might help newsrooms begin to assess readers’ level of engagement; defined here as the intersection of what readers want and what newsrooms want readers to do.

How might we measure this?

Over the past couple of years, journalists, editors, and news room developers have begun to openly criticize the poverty of traditional metrics like page views and average time spent on page. Most of their frustration lies in the inability of these metrics to capture deeper ideas like attitudinal, behavioral, or legislative change - in a word: impact. But while automated metrics may never fully capture these important considerations, they could help us get at some idea of overall reader engagement with a news application.

Event tracking

One step past page-views and time spent on page are more granular web metrics that capture individual events. From Google Analytics, Omniture, to WebTrends, many free and paid-for analytics services offer access to raw event-level data, or individual actions taken on a given page. “Events” can be created with small tags in your site’s JavaScript (see how to do this with Google Analytics). By tagging elements of a project associated with clicking through a site, entering a search term, watching a video, or sharing on social media, event tracking can be used to track a users path from the ‘far’ to the ‘near’ of news apps.

Identifying users

The difficult (and perhaps troubling) aspect of event tracking is the necessity of identifying individual users of a site. The New York Times is lucky enough to have substantial traffic from registered users. In these cases, collecting user-level event data over time and across multiple devices is simplified through logins. However, IP addresses or IDs extracted from cookies, small bits of information that sites use to track their users, can serve as proxies. While these can be deactivated on most modern browsers, or through services like Tor, in most cases they can represent an “individual user.” In turn, reconciling this information allows us to study the scope of each user’s interaction with an app.

A case study: The Red Carpet Project

In late January, the New York Times interactive team released an app for exploring 15 years of outfits on the Red Carpet. The app represents a new take on the slide show, enabling filtering, searching, and sharing of certain slices of slides. Built into the app is an implicit logic of engagement: while the images are on showcase, readers are encouraged to explore the dresses and suits curated by editors and filter the images by time, style, and color. If all goes well, the readers share their own selection of outfits on social media.

Tagging news apps

In The Red Carpet Project, the far view are the images themselves - what the app would have been had it been merely a slide show. Readers move closer to the near view as they filter the outfits, explore the editors’ selections, and share the site on social media. By tagging each of these events with unique ids, we can isolate users whose behavior on the site was most indicative of “engagement.”

Exploring the data

Given the data for these events, we can now measure the most-used elements of the app. In the figure above, we group events into simpler categories. Note that “faves” connote instances where users selected their favorite outfits without necessarily sharing them on social media. We find that over 95% of the events captured were associated with viewing an image - the “far view.” However, without knowing the degree to which each reader used all possible features, we cannot approximate levels of engagement.

By counting the number of unique features each reader used, we get a better idea of the degree to which users explored the app’s near view. We immediately see that readers rarely engaged with more than one or two of the apps features. Taken together these two graphs beg the question: what was the difference between readers who only viewed slides and those who used additional features?

In this view, we see that more than half of the users viewed only slides and 5% clicked every slide without engaging other features. Readers that only clicked on slides also spent less time on the page than readers that explored more of the site. While such readers may have used the app extensively, they didn’t reach its “near view.” In creating a metric for news apps, we should account for these cases, assigning penalties for behaviors that suggest a lack of engagement.

Creating a metric of engagement

With these insights, we begin to get a sense of the things we can measure to approximate a user’s overall engagement with the app. While we’d like to give weight to the raw number of events – after all, rapidly clicking through 500 images without filtering is its own form of engagement – we also want to assign influence to features that capture whether a user reached the app’s near view. A metric of engagement for this app might looking something like this:

While I’ve simplified the math here – the variables are also transformed to decrease the influence of outliers and re-scaled to increase interpretability – the abstracted equation represents a simplified measurement which captures the logic of engagement embedded in the site. In this case, the corresponding density plot reveals a bifurcated distribution of user experiences, with one half barely engaging with the site (1-20), and the other half dispersed over a long-tail (20-100). However, we might imagine alternate situations where more/less users explored the deeper intricacies of the app. In these cases we would expect the distribution of engagement to look much different.

What might we do with this?

Metrics, while abstractions, are useful in that they provide a single number one can use to track progress. As Bill Gates wrote in a recent Wall Street Journal article (while referencing William Rosen’s The Most Powerful Idea in the World), “Without feedback from precise measurement … invention is ‘doomed to be rare and erratic.’ With it, invention becomes ‘commonplace.’” Without a metric for news apps, our only tools for gauging success will be conjecture and anecdote - a great irony given the meticulousness with which data is cleaned and analyzed to construct these projects. With a standard, interpretable metric, editors and journalists can begin to judge whether their intuitions match up with their readers behavior; analysts can compare apps over time and across news organizations; developers will come to value simplicity of presentation and ease of use over technical complexity; and, perhaps most importantly, news organizations will begin designing their digital offerings with users in mind. While the measurements proposed here are just a rough sketch, I hope they will lead to an open discussion on the development of meaningul metrics for journalism.

Using spam bots for live reporting

2013-01-29T00:00:00-05:00

For an upcoming project for NewsBeastLabs on the gun debate, I’ve been monitoring statements representatives have made on the topic. As President Obama prepared to unveil his proposal for gun control last Wednesday, Michael Keller and I were curious to see the reactions of representatives to the highly publicized announcement. Given the degree to which breaking news is now reported (and responded to) on social media, we thought it would be useful to build a bot to log officials’ comments on certain issues and present them in real-time. Such a tool could be used by news rooms to engage their readers on a continuous basis by aggregating and serving content from members of particular communities or who serve on different committees.

@RepsGunTweets was born.

We were inspired by the work of 2013 Mozilla-Knight OpenNews fellows who recently built a prototpe for an app called “if (this) then news,” a news-oriented take on IFTTT – a site for linking triggers from gmail, twitter, dropbox, and other services to actions on the web. Applying this logic to news coverage, the fellows created the shell for a tool that would monitor live data streams, detect important events, and issue notifications. As Vice President Biden took the mic, we started furiously coding up a bot that would follow the twitter accounts of US Representatives and retweet any comment that included “gun”, “assault weapon”, “firearm”, or other relvant keywords. After a couple hours of missteps and headaches, we eventually got @RepsGunTweets up and running. In the last ten days, the bot has logged 307 tweets; two-thirds of which came in the first three days. We’re still analyzing the conversation but one interesting observation is representatives who are not in favor of gun control tend to link to longer explanations of their position on their website instead of tweet a comment.

Under the hood

At its core a retweet bot is a pretty simple tool: Follow a feed, find what matters, and serve it back up under a single account. The harder part is figuring out how to accurately communicate with Twitter’s API. Using tweepy for python we were able to easily access twitter’s numerous methods. All we needed to provide it with were the the consumer key, consumer secret, access token, and access token secret for an application generated on http://dev.twitter.com/apps). The bot follows CSPAN’s member of congress list and applies a regular expression for the desired keywords and retweets any matches.For even more technical info, including an easy way to create your own retweet bot, check out this Github page.