Beyond NewsMastering: Yahoo! Pipes Is The Internet RSS Remixer - Overview And Reports

Sponsored Links

Yahoo! Pipes, is essentially a very powerful RSS feed remixer, which goes well and beyond the original newsmastering concept I described a few years ago. Potentially, Yahoo! Pipes is a highly disruptive visual programming environment that puts in the hands of many people the ability to create web mashups and web-based applications that combine data from different sources with much greater ease and effectiveness.

Photo credit: Jack Parry

Put simply, Yahoo! Pipes is a way of visually manipulating data feeds from around the web, and mashing them together into new interactive creations.

Yahoo! Pipes lets you drag and drop different information feeds - for example, the latest news items from your favourite news sources, or search queries from the online shop of your choice - and combine them through a series of filters into so-called "pipes".

The term "pipes" comes from the Unix operating system terminology and refers to the ability to connect sources of data to filters and utilities.

"A pipe is a way of constructing ad-hoc workflows composed of any number of inputs, filters, and manipulation tools. And the beauty of the whole system is that they all use a very simple input and output method, so there's a nearly infinite set of ways you can combine and recombine them."

Wikipedia says: "In Unix-like computer operating systems, a pipeline is the original software pipeline: a set of processes chained by their standard streams, so that the output of each process feeds directly as input of the next one. Filter programs are often used in this configuration. The concept was invented by Douglas McIlroy for Unix shells and it was named by analogy to a physical pipeline."

While Yahoo! describes its new service as "an interactive feed aggregator and manipulator" that allows you to "create feeds that are more powerful, useful and relevant" the real potential of Yahoo! Pipes sits all in the promise of "turning the web into a programmable environment for everyone."

Pipes examples currently in circulation include a New York Times Thru Flickr application that matches images from photo sharing website Flickr with news items from the New York Times, and an aggregated news alert pulling in the latest headlines from Yahoo, Google, MSN, Findory, Bloglines and Technorati.

In essence here is a service that promises to let everyday people (perhaps with a slightly geeky streak) grab different data from all over the web and manipulate it to do their bidding, simply by dragging and dropping elements around a visual interface.

But how does it work?

Who is likely to use it?

What do experts think about it?

In this mini-guide to Yahoo! Pipes, me and Michael Pick have summarized, organized and commented the key facts, issues, and thoughts that have emerged from the Web in the few hours that have passed since Yahoo! Pipes launch.

Read on to find out:

• What it is - what exactly this tool is capable of
• Usability - the latest feedback on how to go about using Yahoo Pipes!
• Examples - how the tool has been put to use so far, and for what
• Positive buzz - the positive feedback bouncing around the blogosphere about Yahoo! Pipes
• Concerns and caveats - the potential issues users might face
• Related tools - the rollcall of other similar services and tools out there already

Photo credit: Mateusz Zagorski

What are Yahoo Pipes?

Mashups are applications that merge data from different sources and bring them together to serve a new purpose. If you've ever run into an online map that displays photos of key sights, or clicked on a person's name to make an instant Skype call, you might well have come into contact with mashups.

The problem with mashups is that you need to be a code genius programmer to have a hope of making your own. This is where web widgets come in handy, given that they give you the chance to plug this mashup content right into your website as easily as you might embed a YouTube video. But still, as much as you can fiddle with widgets and bend what they do slightly to your needs, the level of control you have is limited to say the least.

Photo credit: Richard MacManus

Enter Yahoo! Pipes, which, while certainly not as easy to get to grips with as widgets, certainly take the pain out of mashing up your online data sources. If this sounds a little geeky, the long and the short of it is this - it is. While I would trust my grandmother to throw together a widget or two, Yahoo! Pipes would probably present her with a few more problems.

So what we have in Yahoo! Pipes is a way of programming without really programming. Instead of typing in code, you drag and drop tools and filters around a visual interface, filling in the details as you go. As Yahoo! Pipes is based on a social network paradigm, there are lots of prefabricated elements that you can grab from other users and change slightly to suit your needs.

So while a lot of the more complicated mashups being put together with Yahoo! Pipes are likely to be from the geek fraternity, less tech-savvy users will be able to go in, copy and tweak the work already done for them by the caring, sharing members of the community. Let's say that someone had already made a news aggregator that pulls in the latest headlines from generic sources like Yahoo News, CNN and the BBC. It wouldn't be too difficult to go on in there, use this mashup as a starting point, and swap the news feeds for something more suited to your needs - Tech news sites, for instance.

Usability

Obviously the chief appeal of Yahoo Pipes! is in the fact that it takes coding out of the equation of creating mashups, and instead brings a user-friendly visual interface into play. But just how user friendly is it, and who is likely to be able to make use of the service in its early beta stage?

Photo credit: Brady Forrest

Reports vary as to the ease of use of the service, with some suggesting that it is a piece of cake to get stuck in and working with mashups, others suggest that it is not quite ready for primetime, and is more likely to be of use to the tech-savvy. Brady Forrest notes that:

''The Pipes editor/creator is a really amazing piece of software. You are presented with a white, graphpaper-esque canvas to build your pipe on. A toolbox with a slew of potential modules is on the left, tabs for working with multiples pipes are across the top, and a debugger on the bottom. To build the Pipe you drag modules onto the canvas, enter the relevant data and connect them together via "wires".''

Brady Forrest, O'Reilly Radar

His enthusiasm is mirrored by Nik Cubrilovic, who emphasizes not only the interface design, but the speed and simplicity with which it can be put to use:

''The beauty of the application is with its simplicity - a user can take any sources, user input requests or the above mentioned module and drag+drop them into place and then connect the pipes. Within minutes I had built an application (also known as a pipe, they should probably change the name as not everything can be a pipe) that would search for 'Techcrunch' in a variety of feeds, bring that data together, sort it and filter it for unique results.''

Nik Cubrilovic - TechCrunch

Richard MacManus is slightly more reserved about this ease-of-use, and while he seems enthusiastic about the service, brings in the first hint that it might not be entirely suitable for Joe Public to make use of:

''The UI seems a little geeky and kind of reminds me of Ning (not sure if that's a compliment or not, as Ning never took off). But I've long thought that RSS remix feeds are the future of RSS - and certainly one way to try and filter information overload. So this is a great move by Yahoo to release an RSS remix service to the early adopter crowd.''

Richard MacManus, Read / Write Web

This praise mixed with a gentle warning about Yahoo! Pipes' complexity also finds its way into Anil Dash's coverage of the service. The message seems clear - this is not going to have the same market penetration as MySpace or YouTube:

''Pipes combines a remarkably sophisticated development environment with some core social features such as the ability to clone or share the web services you produce. The service is fairly approachable, but somewhat complex once you get just under the surface, and should be moderately successful while radically raising the bar for other tools in its category.''

Anil Dash, Dashes.com

Even Tim O'Reilly, who authored a veritable love sonnet to Yahoo! Pipes stresses this same point:

''It's not quite as easy as drag and drop. I have to understand the query syntax of the sites I want to search, and modify the URL-builder modules to use that syntax rather than the syntax of the sites I'm replacing. But it's relatively easy once you play around a bit.

Tim O'Reilly, O'Reilly Radar

In short then Yahoo! Pipes would seem to be something less than rocket science, but certainly taxing enough to make several tech luminaries issue caveats about its complexity and slight inaccessibility to the technologically uninitiated.

Examples

The following list includes some of the Yahoo! Pipes currently in circulation:

• Apartment Near Something - allows you to input what you would like to be near (a mountain, a shopping mall), which city you would like to be in or close to, and how far you would be willing to travel, before spitting out geographical data to fit your query.
• A Yahoo blog feed aggregator
• A hot deal search that searches numerous shopping search engines to get you the best price on the item of your choice
• Aggregated news feeds from the top online news sites
• Browse the other Pipes available at Yahoo! Pipes

Positive buzz

There will be more than a few champagne corks popped at Yahoo! HQ tomorrow, given that the response to Yahoo! Pipes has been resoundingly positive, and as widespread as any PR agent could dream of. Perhaps the least restrained, and glowing of the responses comes from Tim O'Reilly, who sees the service as a new step forward in the evolution of the web. He writes:

''Yahoo!'s new Pipes service is a milestone in the history of the internet. It's a service that generalizes the idea of the mashup, providing a drag and drop editor that allows you to connect internet data sources, process them, and redirect the output. Yahoo! describes it as "an interactive feed aggregator and manipulator" that allows you to "create feeds that are more powerful, useful and relevant." While it's still a bit rough around the edges, it has enormous promise in turning the web into a programmable environment for everyone.''

Tim O'Reilly - O'Reilly Radar

Photo credit: Tim O'Reilly

Pete Cashmore is less grandiose, but nevertheless sees the service as a step in the right direction. He notes that:

''Pipes is still a little geeky, admittedly, but it's a great first step in creating a mashup tool for the masses.''

Pete Cashmore, Mashable

Overall, however, the feedback on the service falls somewhere between these two examples, making Yahoo! Pipes' reception an incredibly positive affair. It is easy to see how, while the service might not be a killer app in terms of market penetration, it is certainly leading the way and hinting at the shape of things to come for the evolving social, malleable web.

Concerns and caveats

While the mood is a largely positive one out in the tech blogosphere, a couple of concerns and caveats have risen from the jollity. Chief among them is the previously mentioned issue of usability:

''Now, while I say Pipes opens up mashup programming to the non-programmer, it's not entirely for the faint of heart. At minimum, you need to be able to look at a URL line and parse out the parameters (so, for example, you can use Pipes' "URL builder" module to construct input to a site's query function), understand variables and loops, and so on. But you don't really need to know these things to get started''

Tim O'Reilly, O'Reilly Radar

Photo credit: Niall Kennedy

The other possible issue raised is from a content publishers perspective. While consumers might welcome the ability to weed out advertising content from blog posts, for instance, this could potentially threaten the livelihood of those pro-bloggers that rely on this important revenue stream:

''Yahoo! Pipes makes it easy to remove advertising from feeds or otherwise reformat your content. I already know a few publishers who hold back the publishing the full content of their posts for fear of easy resyndication and brand dilution, and if Pipes becomes popular publishers might hold back a bit further or ban Yahoo! Pipes outright. A Yahoo! Mail user searching for a new feed subscription will likely choose an identical feed labeled "No Ads!!!" associated with their favorite brands.''

Niall Kennedy - Niall Kennedy.com

On the one hand the usability issue threatens the extent to which Yahoo! Pipes will be taken up by everyday web users, and on the other, its powerful ability to filter and refine the content that passes through it has content producers a little concerned about how it might facilitate both ad-dodging and unchecked wholesale copying of content by unscrupulous sploggers.

Related tools

Several predecessors and alternatives to Yahoo! Pipes have been discussed in it's ongoing coverage. Anil Dash suggests:

''Plagger: an open-source, installable feed routing system created by Tatsuhiko Miyagawa which performs much of the core functionality of Pipes and is customizable, but lacks the user interface and integrated development environment (IDE) which distinguish Pipes.

Ning: Perhaps the archetypal social application platform for the web. Headed by Gina Bianchini, Ning has thus far defined the feature set for end-user creation of web applications, though the focus has not been on creating web services.''

Jamie Pitts suggests that Yahoo! Pipes leans heavily on the interface design of Apple Quartz Composer and Propellerhead's Reason applications.

Ivan Pope throws a recommendation in the direction of Dappit, a data mapping web app that serves a similar function to Yahoo! Pipes.

Conclusions

Photo credit: Pete Cashmore

"Pipes is a hosted service that lets you remix feeds and create new data mashups in a visual programming environment. The name of the service pays tribute to Unix pipes, which let programmers do astonishingly clever things by making it easy to chain simple utilities together on the command line."

Here is a tool with the potential to change the way non-programmers interact with data on the web, allowing for the relatively easy mixing, matching and filtering of data streams into new information services and products which we have started to invent only in recent times.

Nevertheless, this is not yet, a tool quite ready to be unleashed on the general public.

While Web widgets bring mash ups to the masses, anyone that wants to get really stuck into the possibilities offered up by Yahoo! Pipes is going to have to know at least a little bit about the way web applications and protocols work behind the scenes.

"Now, while I say Pipes opens up mashup programming to the non-programmer, it's not entirely for the faint of heart. At minimum, you need to be able to look at a URL line and parse out the parameters (so, for example, you can use Pipes' "URL builder" module to construct input to a site's query function), understand variables and loops, and so on. But you don't really need to know these things to get started.

What's really lovely about this is that, like the Unix shell, Pipes provides a gradual introduction to web programming. You start out by modifying someone else's pipe just a bit, then branch out into something more adventurous."
(Source: Tim O'Reilly)

That said, given that the site is emphasizing its social network aspect, and that many users are already freely sharing their mashups, it should not prove too difficult to modify and tweak the groundwork laid by others in remixing RSS feeds for your own needs and interests. What that means is that nonetheless some technical prowess is required to make the best out of Yahoo! Pipes, by simply cloning and refining on the Pipes work done by others, many more will have the opportunity to more easily learn and familiarize themselves with "piping".

Yahoo! Pipes is the first tangible tool that will allow many of you to customize, re-arrange and engineer new data views of the web as well as compelling new services that mix complementary and isolated data components available out there.

Yahoo! Pipes is indeed, at least in historical terms, a milestone technology for the Web. It opens up opportunities that are orders of magnitude larger than what is typically possible today and it gives also to the less-technically equipped the means to mashup and remix content and information sources in ways and fashions not possible until now.

Additional resources

If you are hungry to learn more about Yahoo! Pipes, you might want to check out the following websites:

• Yahoo! Pipes website
• Tim O'Reilly's resounding praise for Yahoo! Pipes
• Nik Cubrilovic's Tech Crunch feature on the service
• Niall Kennedy's coverage of Pipes
• Pete Cashmore's Mashable post on Pipes
• Richard MacManus' Read/Write Web coverage of Pipes
• Anil Dash's review of the service
• Brady Forrrest's Modules for building pipes and Deconstructing a pipe
  tutorials for the service, over at O'Reilly Radar
• Rok Hrastnik marketing insight into Yahoo! Pipes

Yahoo! Pipes News Radar
by Marjolein Hoekstra

Beyond NewsMastering: Yahoo! Pipes Is The Internet RSS Remixer - Overview And Reports - Originally published by Robin Good on MasterNewMedia.org

Robin Good and Michael Pick -