For the most part, we've been happy with the conversation you all have created; we've learned a lot in the past couple of months, from how a community polices itself to more ordinary things like useful links and amazingly human tales about the toll of war.

Terrell Spencer's comment on Ivan Watson's story about a tumultuous marriage between an Iraqi woman an American serviceman was especially poignant.

He wrote:

I'm an Iraq War vet, and I've recently come out of PTSD. Fallujah was a hell hole. You can't live/fight there and it not mess you up. I'm a loving husband and father, I consider it my duty to sacrifice for my family. I love and respect critters, but 8 months ago I snapped at the world's loyalest dog for not coming. I beat her, pummeling her with my fists, screaming while choking her, then threw her off the porch. I was completely out of control. I never hit my wife, but I shamefully created a home where she and my son lived on edge. I'm better now, I've dealt with what happened over there. Why am I spilling all this? Because these people need help. They're hurt and messed up. These people are ashamed, and hurting. They should be rebuilt - not abandoned and condemned.

The comment was left amidst a hostile conversation. The gist of it is that, after struggling economically, the Iraqi woman in the story had turned to stripping to support her family in the United States. A lot of the comments were disconcerting in their judgment.

One of the mild ones came from Jan Shields, who wrote, "Where is the dignity and discipline that we associate with our veterans of war? Shameful. shameful, shameful!"

A couple of producers asked that the comment thread be closed for the story and we considered that seriously, but, then, out of the steam of the conversation emerged Spencer's earnest plea.

Part of the reason we launched community tools on the site was to open NPR to the outside but another big reason was that we thought the wisdom of the many would better inform the stories on NPR.

To see so little empathy given to such a human, flawed family was, to be honest, disheartening. Part of my greatest hope for a community like this is that we go back and forth civilly on a diversity of opinions and come to find some understanding.

But I guess the lesson learned with Spencer's comment is that sometimes to come to that understanding, we need a little tousling, that sometimes out of dissonance emerges harmony.

Next Time: A lighter fare: We look at the NPR Community's top favorites.
The Time After Next: We consider two new discussion rules.

-- Eyder Peralta

I will start up again by talking about my recent presentation at Mashery's API Conference last week. The conference itself was primarily focused on the business of APIs. In my presentation, I mainly discussed NPR's goals for opening up an API along with some of the challenges we faced leading up to the launch.

As NPR reviewed the landscape of content syndication, we found that there were quite a few APIs already in the marketplace. Most of them, however, belong to content aggregators (eg. Google, Yahoo!, etc.), user-generated content sites (eg. Flickr, Wikipedia, etc.), and some e-commerce sites (eg. eBay, Amazon, etc.). There were surprisingly few comprehensive APIs from major media organizations. Some organizations, like DayLife, CBS and BBC, offered APIs, but these limited in a variety of ways.

Mostly, these major media organizations were syndicating their content through RSS or extended RSS, such as Podcasts or MediaRSS. This approach has been surprisingly effective - what I call "Really Successful Syndication". It is successful because RSS is simple, widely adopted in the marketplace, and succeeds in driving traffic back to the site. The major problems with RSS are the same things that make it really successful. That is, in the current marketplace, RSS now stands for "Really Stingy Syndication" because it does not contain very much real content. Instead, it provides enough content to drive traffic back to the source, embracing the "lock-down" model of content.

The marketplace is changing dramatically, though, and people have destinations to which they are attached. They go to Facebook, MySpace, etc. and expect to find content there. Content providers will have to put their content on these sites through widgets and other means of distribution. If the users of Facebook, for example, find the content they want on Facebook, then they are less likely to leave Facebook to get more content (unless the user has a keen interest in a specific content provider). As a result, the richer the content is on Facebook, the more likely the user identifies your brand as a trusted news source. So, RSS is ok only if no other providers offer richer content. But it is only a matter of time before the richer content is there...

Because of these changes in the marketplace, NPR decided to release a comprehensive API of all of our content that we have rights to redistribute. If our content is truly open, it will enable users to mash it up, keep it relevant to them, and share it with new audiences in places where those people are. Although NPR.org is still critical to our strategy, we can no longer rely exclusively on the site as a way to reach people.

There were two other major factors in our decision. First, it is critically important for NPR to provide content and services to our Member stations. The API will enable stations to get NPR content on their sites. We also plan to offer local station content through the API, which will provide a local/national view of content to the users. The second major influence in our decision was NPR's Mission to "create a more informed public". By offering both local and national content in our API, enabling users to mash it up and use it in ways that we have not thought of or don't have the resources to execute, we hope to reach and inform new audiences.

Once we decided to release an API, there were several questions that we needed to answer. First and foremost, we needed to establish what our target audiences for the API would be. They are as follows:

• End-users and other web developers (These users can post content to blogs as well as create innovative ways of using NPR content)
• NPR's Digital Media team (NPR Product and Project Managers can improve their products using the API without a lot of effort from NPR Developers)
• NPR Member Stations
• Content aggregators and NPR's business partners

Serving each of these audiences through the API enables us to seamlessly integrate with them in such a way that it requires very little involvement from NPR's development staff.

In the slides (attached below) from the conference, I have provided some examples of how these audiences are using the API.

We will be discussing more of our challenges in later posts.
-- Daniel Jacobson

But what if you want to hear more...?

Well if you missed us present at OSCON 08 there will be other opportunities to hear us first hand discuss what we have done, and where we are going with the API.

Here are several of the upcoming events we plan to be at:

Today (11/03) at 5:15pm PST Daniel Jacobson will be discussing our efforts on the API at The Business of APIs Conference. If you are attending please stop by.

For those in the Public Broadcasting family, we will be at IMA Public Media 09 in Atlanta Feb 19-21. This is definitely a must attend for those in public broadcasting who see their future world meshing traditional and new media experiences.

We are also very excited to be a finalist for the We Media Game changer award. Out of 150 Nominees we are one of 35 finalist. Additionally we could be chosen as keynote speaker based on community votes.

And, finally we recently got the word from the folks at O'Reilly that we have been invited to present at the Web 2.0 Expo Mar 31st-Apr. 3rd.

Hope to see you soon.

-- Zach Brand

The biggest thing we've done is added a new community-building tool called NPR Groups. We now have the ability to create individual communities on the site that feature their own discussion boards, a group blog, event listings, and galleries for user-generated photos and video. They're not unlike the groups you see available on Facebook and other social networking sites.

With today's release, we've set up groups capability for almost 300 NPR member stations and station networks. You can browse or search the list of stations in our new station group directory. Initially, most stations won't have the new tools activated for their group pages, but you can still friend them by going to their group page and clicking the "join" button on the right side of the page. Stations with group pages each get to decide for themselves whether they'll use the new community tools or not, so not all of your favorite stations will have the full functionalities set up. One example of a station that has just activated the community tools on the site is WDAV Classical Public Radio in Davidson, NC. If you've already listed any favorite stations by editing your account on NPR.org, you'll automatically be added to those groups; they'll also appear on your user profile as well.

The new groups tools aren't just for stations. We're also making them available to NPR shows and journalists, so we can roll out new community spaces for a variety of topics. This will happen over the course of the coming weeks and months; I'll post updates about new groups on the blog.

Meanwhile, today's release addresses several bugs and other fixes, including some that were suggested by blog readers.

• Added text to the NPR.org registration page to clarify that user's full names are displayed in their profiles and comments

• Fixed the bug that prevented users with apostrophes, dashes and other characters in their names can register successfully

• Fixed our blog software so blog posts are displayed properly in various parts of the site in relation to our social networking tools

• Comments written with multiple paragraph no longer appear as one long paragraph

Like I said, I'll post updates as new groups roll out. In the meantime, please feel free to let me know if you have any questions or comments.

-- Andy Carvin

Meanwhile, another experiment occurred to me: can you use Twitter as a form of distributed dial-testing during the debate? If you've watched CNN during the previous debates, you may have noticed the dial-test data they display on the screen. A group of people are sitting in a room with a device that has a dial on it. As they hear stuff from the candidates that they like or dislike, they turn the dial to reflect how they feel about it. CNN then averages the dial test results and maps them on the screen.

Imagine if we used Twitter to do the same thing on a mass scale. The simplest way to do it would be to ask users to post tweets with a 1-10 numerical score whenever they have a reaction to a candidate's statement, then tag it with the keyword #dialtest. You could then follow the search results using Twitter's search engine and get a feel for how Twitter users are reacting to the candidates.

Of course, it'd be great if you could do it in a more sophisticated way. For example, I'd love to see some sort of application that could observe the search results for #dialtest and average the numbers included all of the tweets in rapid succession - like every 10-15 seconds - and then retweet the average through another Twitter account. Ideally, you'd want the app to be smart enough to ignore numbers submitted outside of the 1-10 range, and maybe limit the number of tweets from an individual user to a few times a minute so a user can't skew the average. Similarly, I'd love to see the app let users register themselves as supporting a particular candidate or as undecided, so you could follow dial test averages for each category of user type, since the Twitter community probably skews towards Obama supporters.

For tonight, maybe we could just have users tweet something simple, like #dialtest Obama 7.5 if they wanted to give Obama a 7.5 out of 10 for a particular remark, then monitor the search results. I think that's the easiest way to get started. Meanwhile, some of you could also try the new Twitter plotting tool Plodt, which tracks tweets that reference a keyword and assign it a numerical score, placed between asterisks. For example, if you wanted to give McCain an 8.0 for a comment of his, you'd tweet *McCain 8*, with the asterisks included. If you want to try this method, be sure to follow @plodt on Twitter first so they know to track your tweets.

Update: Okay, here's how it's gonna work.

Step 1: Follow @plodt on Twitter.
Step 2: each time you want to rate a candidate's statement, format your tweet like these examples:

#dialtest *McCain 7.5* Good answer on Iran

or

#dialtest *Obama 7.0* Like what he said re: bailout

By including #dialtest in your tweets, everyone will able to follow along using this Twitter search page. And for those of you who are more visual, the tweets will be plotted on a graph using Plodt.com. The graph will only accept your tweet if you follow @plodt on Twitter and surround your ratings with asterisks, like the examples above.

You should now be able to access the Plodt Web site. And like I said, you'll need to follow @plodt on Twitter for your tweets to be processed, though.

Anyway, this is all just a nutty little experiment, so please take the results with a grain of salt. In the meantime, I'm reserving @dialtest on Twitter, just in case.

-- Andy Carvin, aka acarvin on Twitter

Starting now, it will be easier for you to talk to us, for us to talk to you and for you all to talk to each other. We are making it possible for anyone who registers with us to comment on a story and to create a profile page where many interesting things can happen. We are providing a forum for infinite conversations on NPR.org. Our hopes are high. We hope the conversations will be smart and generous of spirit. We hope the adventure is exciting, fun, helpful and informative. This is important for the NPR community.

That last phrase -- "important for the NPR community" -- is not phony baloney corporate rhetoric, I promise.

The NPR community is a real thing; it is made up of the people who work here, the people who work at member stations, the people who listen to NPR on the radio, the people who use NPR.org and the people who support NPR. And many in that community think of ourselves as "NPR people." Few other American news organizations inspire such allegiance, have a real community and have "people." NPR does and it is vitally important to our health and growth to be able to talk to each other more and more openly.

NPR is late to this game, to be blunt. Many big news operations have had open comments and other "social media" functions for quite awhile. Some of you are grizzled veterans of Facebook, MySpace, YouTube and online news commenting; for some this will be new. NPR has been cautious because we want to do it right; we want the comments and the conversations to be useful, friendly and civil; we want NPR employees to participate and talk about their work. We needed the right tools and the right philosophy to come together. Now it has.

NPR is a non-profit. We are not launching the project to get more "hits" that will make more money. We are doing it because it is the respectful thing to do for the NPR community. We expect to get story ideas, tips, insights into the world we cover, tough criticism and even the occasional compliment. We want to share more of the news we gather and the stories we tell with you. And we want to do all this in the NPR style -- with both dignity and self-deprecating lightness.

We won't hit the social media ball out of the park on the first swing. But we encourage you to create a profile and let us know what you like and don't like. We apologize in advance for any bugs you encounter. Also be sure to take a look at some of the more specific rules of the road. And if you don't like to do this stuff in public, here's my e-mail: editorial.director@npr.org.

--Dick Meyer

Beginning the first week of October, we'll start rolling out a number of new features on the Web site:

User profiles. Visitors to the NPR site will be able to create a profile page for themselves. A profile page will let you upload an avatar, post a short bio and share your interests. This will also allow other users with similar interests to find each other. For example, if you say you're a fan of a particular band, you'll be able to click a link and see all the other people on NPR.org who like them as well. We'll also show you recent stories from the site related to that topic. Users will also be able to "friend" each other and post comments on each other's profile wall. We're also going to encourage NPR journalists and other staff to create their own profiles, so you can interact with them as well.

Discussion threads for all stories. Currently, discussion threads take place only on blogs, but we'll now have the ability to incorporate them in news stories. Each time a comment is posted, you'll be able to see the person's avatar, access their profile page, recommend their comment or even report it if you think it breaks the site's discussion rules. You can also sort the comments for each story by newest, oldest or most recommended. Meanwhile, site editors will follow the discussions to see what's taking place, so we can feature interesting comments either in the story itself, or even on the NPR homepage.

Story recommendations. Along with recommending comments in a given story, users will be able to recommend the story itself, not unlike the way you digg a story on Digg.com. This will let you explore stories on the site based on how often they've been recommended by the community.

Not too long after we roll out these features, we're also planning to launch a set of community tools not unlike groups on Facebook. We'll be able to set up community pages for shows and other NPR activities where users can start conversations in a discussion forum, upload photos and video, post event listings and the like.

We're really excited about rolling out these new features on the Web site. It's the latest step we've taken to open up the ways we interact with the public. NPR community members have always been eager to engage each other - just go to Facebook and take a look at the number of groups that have been created by members of the public. And we're eager to reach out to the public as well, using the Internet to foster new relationships between our journalists and the public. By creating new ways for that interaction to take place, we hope it'll impact the quality and diversity of our journalism in a positive way.

-- Andy Carvin

Click image to enlarge

There are several reasons for this architectural approach:

1. PERFORMANCE : Requests will first go through the Memcache and file cache layers, which will always be the most efficient. If the requested document is not in Memcache, we have PHP render the output using the XML files. If the XML file cannot be obtained, PHP will access the database for the data. If PHP hits the database, however, a version of the request will be stored back in Memcache to speed up the delivery of the next request. This ultimately takes strain off of the database, which is the most expensive operation in serving documents.

2. ABSTRACTION : Creating a separate layer between the various presentations and the actual database allows the presentation layers to be agnostic with respect to the data repository. Currently, our database is Oracle, but if want to move to MySQL, then the presentation layers don't really care because they are served primarily off of the XML repository (although the final fail-over to the database would require changes).

3. SIMPLIFICATION : The database itself is a complicated relational system. The schema is largely normalized for scalability and efficiency in our write operations. Building pages, as a result, requires expensive table joins across very tall tables. These queries, although tuned, add up when you consider how many queries there are throughout a story page, for example. Executing these queries once and storing the data in a flatter file system enables the pages to be built more efficiently (both because of the flatter model as well as not having to access the database).

4. SCALABILITY : Because of the rendering framework, we are able to easily add new transformation and presentation layers without having to write a lot of extra code or customized database queries. The rendering engine knows how to handle the XML files in a cohesive way because they are relatively flat, so the transformation layers really aren't that different from each other. The framework also allows for reuse of code in the presentation layers because most of the presentations are dealing with the same content and are displaying that content in similar ways. New presentations for NPR.org are the hardest because of all of the design nuances, but adding Atom and MediaRSS are pretty quick and painless. The difficult part is figuring out how to map our fields to those structures, not in the coding of it.

So, the system was largely in place almost a year ago, alleviating many of the technical hurdles in building an API. We knew that if we wanted to open the API up to the world we would still have some technical challenges left, including filtering engines, the registration engine, the query generator, etc. Before getting to those tasks, however, we needed to determine if the public API fits with the overall NPR strategy.

-- Daniel Jacobson

* Output formats
* OpenID
* Query generator
* Caching layer and performance
* Number of requests per user per day
* Audio stream vs. Download
* Amount and type of content offered
* Terms of use
* Rights
* Metrics
* Station content
* The archive and the deep NPR archive

I am sure that during the course of this series other topics will be added, but these capture some of the more prominent issues that were discussed. As you can see, these topics involve technical issues as well as legal and business ones.

Before we can get to any of the above topics, though, we have to address the single most important decision that we made: Should we open up the API?. That will be the first post in this series.

The purpose of this series is to continue to be as transparent as we can be and to be an active, engaging part of the technical community. We hope that some of these decisions that we dealt with will help others successfully pursue creating APIs as well. We also hope that this blog will act as a forum to continue the discussion and will help us continue to better deliver useful tools and services.

I am looking forward to the discussion!
--Daniel Jacobson

My favorite way to consume the API is using JSON. With just a few lines of code, I get a data object that I can use with JavaScript--no messy parsing of XML or the DOM necessary. The structure of this JSON data object strongly resembles the structure of the NPRML XML output document. In fact, to create the JSON output, we first generate the NPRML document, and then do some transformations to create the JSON output.

However, XML does not map to JSON seamlessly. The XML in NPRML has element nodes that contain either other element nodes or textual content. The element nodes may also have attributes. It is common practice to map element nodes to objects in JSON, with each sub-element becoming a nested object. However, we had to decide on how to treat textual content and attributes.

It makes sense to make the textual content be a property of the object that contains it, but we need a name for that property. We looked at other APIs for a standard naming convention, but there doesn't appear to be one at this time. For example, Google Data APIs puts textual content in a property named $t. The Flickr API uses a property named _content. In the NPR API, we use a property named $text.

Some APIs take a different approach, treating text nodes as string properties of the object, which means the name of the property is the element node name. Yahoo! Shopping Web Services take this approach. This makes the JSON more readable and simpler, but it doesn't work if nodes with textual content also have attributes.

We map element attributes to object properties. This approach is used by many APIs, although some (such as Yahoo! Shopping) create a specially named nested object to hold all of the attribute values. With our approach, this NPRML fragment:

<show>

    <program id="2" code="ATC">All Things Considered</program>

        <showDate>Fri, 22 Aug 2008 16:00:00 -0400</showDate>

    <segNum>12</segNum>

</show>

gets mapped to this JSON:

"show": [{

    "program": {

        "id": "2",

        "code": "ATC",

        "$text": "All Things Considered"

    },

    "showDate": {

        "$text": "Fri, 22 Aug 2008 16:00:00 -0400"

    },

    "segNum": {

        "$text": "12"

    }

}]

Note that the show property contains an array. It is possible that a story was used in multiple shows. We use arrays for properties that could have more than one value. This is done even when a given story has only one value for the property.

We are interested on hearing what you think is the best approach to JSON. Have you seen other approaches that work better? Is JSON important to you? Let us know in comments.

--Harold Neal

Sharing this presentation in this forum is the first step to making our process, architecture and decisions around the API more transparent and open to our users. There will be other documents and blog posts to follow with more information. Let us know if you have specific questions about our process so we can try to address them in these future posts.

Click here to view the presentation ( (requires Adobe Acrobat)

Several folks have asked me what the relationship will be between the NPR Digital Media team and the team at PI. As happens frequently in public media, our two groups have have collaborated on a number of projects over the years. For example, NPR and PI are currently partners (along with several other organizations) in a collaboration to improve online election coverage across public media. My hope is that with a closer connection between the two groups we will find many more opportunities where it's beneficial to work together.

However, while there are synergies, each group really has a distinct focus. NPR Digital Media's main mission is to create and distribute NPR content to users, stations and other partners on a variety of digital platforms like NPR.org, NPR Podcasts and NPR Mobile. Public Interactive is really focused on directly serving the digital needs of stations by providing tools and services. PI also distributes content to stations, but they work with multiple content providers including Reuters, BBC, PRI and the New York Times Syndicate (as well as NPR). We believe that it's important for PI to maintain this kind of independence to ensure that they can best meet stations' needs. To reflect these two important but distinct roles, NPR has decided that they will be managed as separate units, though there will likely also be many areas where we can collaborate.

I'm looking forward to getting to know the PI team better over the coming weeks and to see how we can work together to best serve the overall mission of public media.

What do you think about this announcement? Do you have any advice for NPR and Public Interactive as we go forward in working together?

--Darren Mauro