Curated by: Luigi Canali De Rossi
 


Thursday, February 19, 2004

The Birth Of The NewsMaster: The Network Starts To Organize Itself

NewsMasters: RSS And The Opportunity For Sustainable Filtering And Aggregation Of Online Content Into Niche Websites And Dedicated Information Channels

An independent publisher opportunity and a socially valuable way of filtering and organizing news information and the Web at large.

eyeinthenews.jpg
Photo credit: Emsago from Stock Exchange

Introduction

Yesterday, at the latest DEMO 2004 Conference in Scottsdale Arizona, Feedster

''announced a new tool called FeedPaper, which combines syndicated Internet content (culled from the 500,000 newsfeeds continuously searched by Feedster), such as The New York Times, Washington Post, USA Today, Amazon.com, many IT publications, PRNewswire, and hundreds of thousands of blogs. A Feedpaper is a web-based and RSS-enabled micropublication on any topic of its creators choosing. Creating a Feedpaper enables its publisher to blend, track, and share information on any topic.

Online publishers and e-commerce sites no longer need to see search engines simply as sources of traffic and advertising networks.

For the first time, publishers can use a search engine to filter and deliver relevant content to their readers within their own pages, including customized topic search and syndication,''

Scott Rafer, Feedster's CEO said.
(http://www.internetnews.com/dev-news/article.php/3313991)

Indeed the key problem we are confronted with is this: how to navigate, filter and reduce to humanly manageable dimensions the gigantic flow of information coming at us in increasing amounts.

The entry of the RSS publishing and distribution format into the fray of media and technologies available for communication has only added to the tidal wave of news and information that has gradually been submerging us.

The proliferation of many alternative online independent news sources and the epidemic expansion of the bloggers community have created a sharp increase in information channels available to us outside the mainstream, established sources.

Further, the Web keeps growing at a steady pace, and the amount of content being made available on a daily basis is simply too much to be managed by our everyday Googles and RSS aggregators.

We need something of an entirely new order to manage this.

An army of newsmasters.

(And a special Toolkit to start working at it right now.)



What Is The Discovery?

The discovery is the unlimited and yet untapped power we now have to search, filter, aggregate and create focussed news/information channels with the only support of our know-how, culture, experience and a little unknown free technology called: RSS.



Where Is The Real Value?

The real value is in the fact that the effective widespread application of this filtering and redistribution process would be a highly valuable social endeavour and one that not only can enrich our rapidly evolving global culture but one which can also give sustainability to those who have the will and skill to ride it.

Summary: Critical Issues/ Problems At Hand


  • Too much information coming in
  • Too many sources too scan
  • General categorization of sources too limited. By author or publishing house is too limiting. I can't scan 100 authors a day to see if there is some interesting for me to read or write about
  • Too many new sources coming online daily. Which ones do I pick?
  • Difficulty in separating relevant news from press releases, personal rants and job vacancy announcements
  • Dubious quality of many news sources
  • Our search for relevant content and news is an individual activity, in general with no direct benefit for the community at large
  • Searching for information is mostly carried out as separated, uncontextual, just-in-time quick personal search.



The Filtering Mechanisms Available To Us

As Stephen Downes mentions in his beautiful post about networks in his last issue of OL Daily, the network of information needs to organize itself automatically, and we must help it in doing so.

We must learn to perceive, to use, the internet in exactly the same way, as a self-organizing network.

Up until now we have organized and searched for information mostly by:

a) Typical major search engines (Google, Alltheweb, Teoma, Wisenut, etc.)

b) Directories (ODP, Yahoo, etc.)

c) Meta search tools (Copernic Meta, Vivissimo, Surfwax, Ixquick)

d) Data Visualization tools (Grokker, Kartoo, TouchGraph, Musicplasma)

We have also built internal directories inside our companies and organizations, databases and knowledgebases of all kinds. But of these last ones each, most of them remain like isolated islands of which few search tools can report the size, content and updated map.

More recently, the army of individual reporters, writers, journalists, academics and researchers who has adopted independent publishing systems like CMSs, weblogs, and other database-based content management solutions, has been proactively weaving, organizing and filtering, at an initial level of refinement the news flow and content coming at us daily.

Bloggers foremost, have taken a natural role of filtering and aggregating mechanisms of the newsflows running through the Internet.

Bloggers have been first to scout new independent sources, alternative voices, and who have pointed links to new ways of looking at any issue. They, more than any other group, have brought, among much useless noise, the true emergence of effective meta news sources that originate, filter and aggregate valuable content online.

So, what it appears to the many superficial onlookers as a universe of mindless writings (blogs) is nothing less than the initial phase of a complex and orderly process whereby humanity at large takes control of filtering, gathering and re-organizing his own know-how an discoveries.

The world becomes a global village and as such it starts taking care of itself in a global, dispersed, self-sufficient, synchronized fashion.

A Google query is an individualistic just-in-time search with no benefit for society.

A Google query is a one-time shot.

You type the query, you get the results and it is over.

You probably don't re-use that search anymore. You will retype it some day when you will need it again.

Nobody else outside of you benefits from that search.

Again, it is a one-time shot.



A Learning Voyage Into RSS

To better understand what this new RSS-based universe really entails and may likely involve, I have dived, soul and body, into it, to understand in first-person how much the new information publishing and distribution protocol could be effectively revolutionizing our immediate future.

The process has taken quite a few months, but the let me tell you that the view derived from this open training has been pretty amazing, to say the least.

In short order and for the sceptics out there, here is what I have done to "understand" RSS and to develop a good feel for where it is headed:

I have converted all of my online sites into content management systems that would be able to generate RSS feeds. I have organized content into categories and created individual RSS feeds for each one of them.

I have started using RSS newsreaders and aggregators and have reported extensively about my findings.

I have tested tools to convert RSS feeds into email, and to convert emails into RSS feeds.

I have started analysing sites and services syndicating other RSS feed content.

I have extensively delved into using all of the RSS monitoring tools like Technorati, DayPop, Blogdex, Popdex, Feedster, Blogstreet and many other ones.

Inspired by the work of Ari Paparo http://www.aripaparo.com/archive/000632.html and by the growing number of valuable directories and search engines aggregating and indexing valuable RSS-based content I went on to preparing the publicly accessible RSSTop55, a best and most recommended list of top RSS-based submission directories.

Early in this discovery voyage I got interested and fascinated by tools like MyRSS who would allow you to theoretically create an RSS feed out of anyone Web site page updating its content out there.

A love for "scraping" has followed right after. Scraping is the art of deciphering the underlying content structure of a Web page content, in order to be able to create a framework or container that can "extract" the content of that site at will in an organized fashion. Scraping a Web site allows anyone to create an RSS feed for any Web page. The feed is fuelled by the new content appearing on that page, and therefore not all pages lend themselves well to be scraped and to be sources for useful feeds.

I have then started syndicating RSS feeds generated by my different sites across the sites themselves and then later I have started creating valuable and unique news channels by aggregating RSS news feeds available out there.

Two good examples of this are: Kolabora, where you can see the home page populated by news coming from different RSS news feeds I have selected and MasterViews where once again the news page is completely generated by carefully filtering and aggregating selected RSS-based news feeds.



Possible Solutions: RSS Aggregators?

Many have touted the advent of RSS aggregators has one possible solution to the issue of information overflow, without yet realizing the intrinsic limitations of this approach when applied literally.

You can subscribe easily to hundreds of RSS feeds in no time at all. And there are a great number of people I like to read about.

And what I really want to read is what these people say about the things that interest me.

Jon Udell is a great guy. Of his posts though, only one in thirty is something that matches my level of interest and competence. The rest is too "geeky" for me. So, it is impossible for me to feel motivated to look at Jon's feed, though I know, that from time to time he will bring me some wonderful unique gems.

So, I give in, and I tend to scan only those sources that are more directly in tune with me. And by accepting this compromise I often loose all of the opportunity to discover Jon's magic realizations when they do really happen.

I have seen and heard of people subscribing to hundreds if not to thousands of feeds inside their RSS aggregators.

Is that manageable?
Do these people get better and more information than everyone else?

It is not.
They don't.

They don't because it is just not possible and not enjoyable at all to go through so many posts, in which, the greatest majority is not even of relevance to their interest.

It is just like doing a one keyword search on Google and then going through all of the results pages that come up. Yes, you can come up with some interesting discoveries but, would you really do that?

Like for Google, where the enormous amount of relevant results obscures the view of most of what could be of interest to me, we do need some new more effective way of filtering the river of news coming at us.

Visualization is one solution, and you will see many new fascinating applications of interface and information design to search mechanisms in the near future. Visualization is the quantum leap needed to perceive the forest from the trees before wanting to dive-in deeper into it.



How Do I Want To Find Information Relevant To Me

Overloaded and disappointed I have spent some thinking time testing many RSS tools and aggregators only to gradually increase my particular appreciation for Bloglines. Though Bloglines features are not completely unique and are available in parts across several other resources online, this is the service that allowed me to come closest to the idea of news and content filtering that I was starting to build up in my head.

Bloglines in fact, not only allows you to add as many feeds as you like inside a neat set of organizable content folders, but it also allows you to subscribe to specific search queries you create. That is, Bloglines allows you to record any search you do across the RSS-based content it has indexed and to automatically rerun that search for you each time you check for new content.

As simple as it sounds this facility provides for a terrific resource of fresh news on any topic you may select!

The first time you experience the use of this you will feel some exhilarating vibe. The crafting of your own little search query becomes a key to generate valuable content on most any topic you can think of.

Unfortunately Bloglines indexes only less than one hundred thousands blogs (!), but think if you could do the same search query on Technorati where over one million blogs are tracked daily. Would that be interesting?

With the help of Bloglines' "subscribed searches" feature I am now a happy scout reading the "topics" I am interested in and free from NOT having to read as a robot the individual feeds of one hundred and more authors and sites I am interested in.

That brings by itself another important corollary. When creating a saved search, you are in fact creating a very custom and personal news feed, tailored to your own specific request.

If you could only save that Blogline feed as an actual RSS feed, you would start appreciating more directly the feeling of becoming an independent newsmaster yourself. (While Bloglines doesn't yet offer this option, other online services, Blogdigger, have started to do so.)



Emerging Technologies

The whole idea I am presenting in this essay is made possible by the rapid emergence of a new group of tools and technologies that allow scraping, search, filtering, aggregation and syndication of RSS-based content at costs and with ease of use that make them accessible to the layman.

My ideal future RSS aggregator gives me the possibility of not only subscribing to as many selected feeds as I like, but it also allows me to pick out my preferred content by filtering out through sophisticated queries and inclusion/exclusion features what is not relevant to my interests.

If you can select sources, formulate advance queries, create search feeds, and mix this combination in newly created information/news feeds you have just visualized the primitive profile of the professional role I have been visualizing.



A New Digital Professional Role: The NewsMaster

Figure now the ability to concert, orchestrate, edit, and refine quality search formulas that tap into the whole RSS universe and beyond, and that filter out relevant content based on selected keywords, sources, type of content, ranking and many other possible criteria.

Moreover, iSyndicate (today Yellowbrix) and others have done this before, and many sites still benefit from this simple automated mechanism.

But imagine now this mechanism being made available to each one who wants to use it.

Imagine having the power to select, aggregate, filter content while creating a new RSS feed that is as unique as the person who has built it.

And this set of "magic search formulas" that anyone can now build is not a one-time shot anymore.

This is a "hose" from which useful information around a tightly defined topic can be extracted every day or as soon as there is something of value out there.

It is a hose from which other people can quench their knowledge thirst.

My unique personal filtering effort becomes useful for everyone.

I have a new profession for information seekers, digital librarians, and knowledge management evangelists: the NewsMaster.

You may probably find a better term for this but the guy I am envisioning is a new type of webmaster who specializes in crafting uniquely powerful magic search formulas generating continuos RSS feed on narrowly selected topics by:

a) selecting and aggregating valuable resources (like a normal aggregator does)

b) creating advanced search queries in the blogosphere ( a la Bloglines)

c) creating advanced search queries on the traditional Web content at large

d) creating advanced search queries on the overall pot of content derived at points a), b) and c) and generating new highly filtered RSS feeds matching specific content and quality criteria.

Searching and identifying key valuable sources and complex filtering formulas will be the outstanding job of the new information gatherers and publishers online: the NewsMasters.

Their ability and craft will be in identifying the query formulas that when cleverly combined and refined by trial and error will allow for powerful automatic news gathering formulas. These formulas need not only to identify the keywords/keyphrases that generate the best and most relevant set of results but need to grow through the intelligent use of multiple filtering criteria and precisely targeted inclusion/exclusion mechanisms.

The creation of dedicated information channels, originated by independent publishers and not by vested commercial interests or mainstream media conglomerates may create the opportunity for a true renaissance of culture, learning and to a multiplication of our abilities to manage large amounts of rapidly changing information.

Search specialist and librarians who will craft with time investments appropriate queries to get at the information they were looking for, will be generously rewarded with a eternal fountains of relevant info for the time to come.

The act of search evolves from a mere set one-time shots at finding something just-in-time, to the start of a collective refined meta-filtering process that generates better information for everyone.



Are Newsmasters Sustainable?

In the wake of all this I am sure some of you have seen also the opportunity to create niche Web site and information channels on very specific topics and with a possible significant rewarding economy. The opportunities are indeed in abundance and one need only to explore some of the first rough ideas I have listed here below, to get some directions on where to start.

Here is an initial list of possible revenue streams that could be ethically generated around the NewsMasters activity.

1. Excerpt only RSS feeds bring readers back online to check for interesting stories and allowing for online only contextual advertising (Google AdSense, Blogads)

2. Low subscription rate to receive premium or "full content" feeds.

3. Editorial promotion inside RSS feeds. Ethical review and inclusion of commercial reviews by independent publishers.

4. RSSAds and similar initiatives.

5. Sponsorships! - This is one of the best ways to make specialized RSS newsfeeds pay back. Have the feed description and reference image carry a valuable sponsorship.

6. Donations

7. Micropayments

8. More traffic to Web sites where other products/services are offered.

9. Custom newsfeed creation for third parties.

10. Provision of high quality custom feeds to major news syndication companies and organizations.

11. Syndication of content on commercial sites.

12. Premium subscription of feeds from a specific author or group of associates.

13. More (please suggest some in the bottom comments area)



Conclusions

RSS puts a large quantity of content de facto in the public domain and it offers an easy way to reuse, syndicate and aggregate such content in an unlimited number of ways.

If someone puts an RSS feed on the Internet it is most likely that she is doing this for others to read, and possibly further syndicate. So it is in the DNA of RSS to be wanting to be free, to be further reused, personalized and syndicated.

The above described ability in the above essay to have many individual users work as Newsmasters in organizing the very own content we are generating provides us all with a new horizon of access to news in ways and fashion that we had not even conceived of until yesterday.

This is not Google News at the nth power. This is much, much more.
Specialized, focused information niches generated by the dedicated work of individual technology-enabled NewsMasters is something totally new for us to consider and reflect upon.

The power of many people at work seems to me irresistible and capable of creating profound changes in the way we operate and inform oursleves.

With this new search and filtering approach we can all start generating better and more refined information channels, as we become enabled to aggregate, select and filter the most relevant content according to extremely sophisticated criteria and long nurtured selected source lists.

The simplification of these new gathering and filtering tools, of which FeedPaper may be the first actual working example, is just around the corner. There is no way of avoiding this. It is just coming.

Tools like FeedPaper will allow capable thinkers to dedicate themselves to this new highly valuable socio-alchemical effort and to create fountains of information hopefully useful for much time to come.

With these technologies and with our applied intelligence many alternative new sources of info can be effectively created. This will naturally filter out the blog noise from the valuable blog/RSS info. True diversity will bloom as unlimited number of views and perspectives can be made available in such a possible information system.

So, it does look to me as a renaissance of sorts is truly taking place. And if can draw any conclusion from the first hand feelings I have had, I must say that this is going to be a fascinating, revolutionary turn.



Special Note from the author:

While we are only seeing the birth of the first primitive group of such tools, I expect many to become available soon and with an array of features and controls that will make the independent news reporters work a truly fascinating one.

Feedpaper from Feedster maybe the first one to become available.

For now, if you wish to pioneer this new exciting vastland, I have prepared a special toolkit just for this purpose.

It is called the The NewsMasters Toolkit and it contains all of the tools and resources you can start using now to build and syndicate your own selected news information channels.



Appendix:

The NewsMaster dream tool


(advanced aggregation/filtering tool - specification) Feb.17 2004

What I would want to have in the ideal tool I yet have to meet.

Characteristics:

  • Search according to multiple criteria

  • Ability to group multiple queries together

  • Full Boolean expression control integration

  • Ability to filter out selected keywords/ keyphrases

  • Ability to pre-select predetermined sources

  • Ability to ban selected news sources

  • Ability to filter news sources based on several criteria:

    o Technorati Cosmos indicators
    o Google PageRank
    o Alexa absolute ranking
    o Google linkbacks (or other equivalent)
    o More to be defined

  • Ability to filter news items based on: article length

  • Ability to filter news items based on link exclusion and inclusion rules

  • Control of frequency of updates

  • Generation of RSS feed

  • Generation Of OPML of news sources utilized

  • Query formula export

  • Ability include search results for specific queries on major search engines

  • Export to multiple RSS formats



    Would be NewsMasters here is your toolkit:

    If you want to learn more about the advanced tools that allow you to start experimenting now with becoming a novel NewsMaster please see my NewsMasters Toolkit for 32 nifty resources.

    newsmaster_cover_350px.gif

    Click here to find out more.

  •  
     
     
    Readers' Comments    
    2006-11-24 12:31:01

    Robin Good

    Shadilac, I think those are great examples of grassroots collective aggregation of contents and that is also a great way to help find what is valuable from within this ocean of information.

    Digg, delicious, Reddit, Magnolia, as well as NewsVine and similar grassroots contributed news aggregators are all early models of how we are going about organizing the information flow.



    2006-11-24 10:51:20

    shadilac

    What is your opinion of the aggregation of user filtered content (ala digg, reddit) and how that fits in to this? The big players today may not be very good at categorizing the articles but newsvine is headed in that direction with structured categories, and feedbite is headed in that direction with user created bundles of your own topic.



    2004-03-01 04:27:17

    Anonymous

    Robin,

    Your piece really helped cement my thoughts with regard to where I want to go with our news aggregation and delivery techniques.

    My business partner has been less than enthusiastic about this project so I sent him a link to 'The Birth Of The NewsMaster: The Network Starts To Organize Itself' to let him know I'm not the only person out there thinking about this stuff!



    2004-03-01 04:24:10

    Tom Young

    A group I'm working with has done something similar (I think) to what you are suggesting.

    While we are still in beta, we have developed a system that organizes legal news (primarily plaintiff oriented issues) into approximately 75,000 categories. Creating thousands of RSS feeds might be the next step for us. Are we some of the first NewsMasters? Maybe.

    Search for a plaintiff legal topic here:

    http://firms.injuryboard.com/index.jsp

    The topics are organized hierarchically, from general to specific. For instance, this page carries auto accident news in general:

    http://www.injuryboard.com/articles.cfm/Topic=31/Type=News

    While this page carries news on the more specific category of auto accidents, front impact collisions:

    http://www.injuryboard.com/articles.cfm/Topic=71/Type=News

    Drilling down even further, this page deals specifically with head on collisions where the front of one vehicle collides with the rear of another:

    http://www.injuryboard.com/articles.cfm/Topic=472/Type=News

    The pages are updated automatically each morning from approximately 3,500 news sources. We have spent almost one year formulating queries and training the system to return relevant information. We publish links to between 1,000 and 5,000 new legal articles each morning through this system. Hopefully, we'll have a bunch of RSS feeds soon.



    2004-02-26 13:19:21

    Seb

    Thought-provoking indeed! Small note: your links to RSS2Email and Email2RSS are identical.



    2004-02-23 08:50:14

    Jay Cross

    You asked for additional uses.
    Business intelligence. Competitive analysis. Professional updates. Trend spotting.

    I foresee the rise of editorial brands. Feeds will vie for popularity instead of individual blogs. People will pay a premium if they trust they're being directed to the right stuff.



    2004-02-22 17:11:27

    l.m.orchard

    If you could only save that Blogline feed as an actual RSS feed...

    Not sure if this is what you meant, but I'm subscribed to a dozen or to searches-in-RSS from Feedster.com, since the results of every search there is available in RSS among many other flavors.

    As for gleaning information from 100's of feeds... I'm subbed to around 500 now and don't have a lot of problems scanning through them. I likely miss some things, but my aggregator collapses items into their titles. Quick scanning with my eyes and the page-down key are my friends. I think it has a lot to do with information architecture and UI. But even with the greatest UI, you'll still reach human limits and need the services of NewsMasters and good searching to get further



    2004-02-22 17:10:18

    Bill French

    Excellent resource - thought-provoking.

    --bf



     
    posted by Robin Good on Thursday, February 19 2004, updated on Tuesday, January 15 2008


    Search this site for more with 

  •  

     

     

     

     

    1253




     




    Curated by


    Publisher

    MasterNewMedia.org
    New media explorer
    Communication designer

     

    POP Newsletter

    Robin Good's Newsletter for Professional Online Publishers  

    Name:
    Email:

     

     
    Real Time Web Analytics