STFW, fix Pipes, and repair Mac disks; voila!

Network World - The next time someone asks you a question and you know they haven't bothered to do the obvious, to wit, look it up on Google, send them a link to Let Me Google That For You. For example, if someone were to write to me and ask if there's an archive of Gearhead columns, I might reply with "Try this link: http://tinyurl.com/yfg33ty". Voila!

Anyway, last week I was discussing my trials and tribulations attempting to get a list of numbers randomized (exactly why I wanted to do this is too complex to go into).

My solution was to use Yahoo Pipes to grab the random sequence from the service random.org using an HTTP proxy on my own server.

The reason for this labyrinthine methodology was that Pipes, being the good citizen it is, will not access remote content blocked by a robots.txt access control file -- exactly what random.org has. My proxy hasn't got a robots.txt file so when Pipes looks for the robots.txt file it won't see one: Problem solved.

Now here's an interesting thing: When I finally got my pipe working it tested fine. In the development interface I could monitor the RSS feed output and whenever I refreshed the monitor I could see a new sequence being returned. Foolishly I concluded that it was working correctly.

It turns out that Pipes has a caching algorithm that is only used when you request the pipe be run from the Internet; requests made from within the Yahoo Pipes site are always honored. It took me a goodly time to figure this out because, of course, I assumed that the problem lay with everything other than Yahoo Pipes.

Thus, should you externally request the output of a pipe more than once during an unspecified and undocumented time period set by Yahoo, only the result of the first request is returned. There have been quite a few support questions about this raised in both the Pipes forum and other places, but apparently Yahoo doesn't care to explain what's going on.

For all of you (countless thousands I suspect) struggling with this issue, there's a simple solution: When you define the arguments for the pipe's URL, add another argument that the pipe will do nothing with.

For example, let’s name this additional argument "fudge". When you call the pipe, just make the value of this additional argument something such as the time; thus: "fudge=10:33:45".

The format of the value doesn't matter as long you change it for each request. Now, when Yahoo Pipes gets a request for your pipe it will always assume that it is a new request and therefore run the pipe rather than pull the previous results from cache. Voila encore!

Reprinted with permission from

For more information about enterprise networking, go to NetworkWorld.com
Story copyright 2009 Network World, Inc. All rights reserved.