Got a call last week from Matt Hicks, a senior writer at eWeek.com. Matt has since released an executive-summary story about the immanent meltdown of the Internet under the strain of RSS and wanted to talk about my experiences -- unaware of the Wired story, Matt's interest was sparked by the Big Corporate angle exposed by news of Microsoft clamping their RSS to teaser excerpts in an attempt to stem the bit flood due to alledgedly over-zealous aggregators.

The story out of Redmond is on their providing a full-text composite feed for all the MSDN blog postings, which most likely means the average consumer of the feed was really reading only a tiny fraction of the total. They'd done as I'd done and trimmed the RSS, but in doing so incurred the wrath of those who don't want just an alert to changed content, but who want that integrated infospace of woven webservices with the full items all under one roof, their roof.

Microsoft rescinded and upped the limit to only trim multi-K posts, but the issue remained, what are we going to do about RSS now that the covered wagons have come? Microsoft technical evangelist Robert Scoble decided the problem was with the polling frequencies ...

"RSS is broken. It's not scalable when tens of thousands of people start subscribing to thousands of separate RSS feeds and start pulling down those feeds every few minutes...Clearly, RSS is losing some of its advantages. More and more sites are not providing full-text feeds."

His solution is a co-ordinated en masse adoption of on-demand polling that only hits your list of 10,000 subscribed feeds when you open your aggregator -- and I wish him good luck with that one considering we can't even get developer consensus on a sane implementation of RFC 2616.

Mind you, breaking my Conditional GET implementation to conform to the broken masses did help, but it was not a solution. The majority readers don't even try to store Last-Modified between launches and their owners only launch when they have time to read blogs, and I still had a situation where adding just one short paragraph in a brief post would flip the dirty-bit and the next swarm of fans would be pulling nearly as much from my site as they would if they were just grabbing my real page. Yeah, sure, I could, if I was rich enough to own my own server, run with mod_gzip (that I couldn't get working anyway) to gain another order of magnitude's headroom, but I can't, and so I went back to Plan A anyway, trimming the feed, stripping the HTML, teledyn-lite as merely your remote-access notice that I'd said something new.

And therein the latest Great Idea in webservices. As Yoda said, there is another hope, and the key to this new GI is what I just said there, that my adding one small part invalidates the whole mass of it ...h3. Just the diffs, ma'am

Hey, wouldn't it be nice if your aggregator only fetched the part that had changed? Enter RFC 3229 - Delta encoding in HTTP put forth by J. Mogul et al ..

Common sense suggests (and traces confirm), however, that even when a Web resource does change, the new instance is often substantially similar to the old one. If the difference, or "delta", between the two instances could be sent to the client instead of the entire new instance, a client holding a cached copy of the old instance could apply the delta to construct the new version. In a world of finite bandwidth, the reduction in response size and delay could be significant.

This still leaves us with the considerable coordination problem of asking developers who couldn't grok 2616 to not only grok but also be aware of yet another HTTP extension, but there are precidents here that make sense to developers. In the way we archive our source code using CVS to transport only 'patches', the way we synchronize server contents using rsync to, where possible, move only the delta, only the difference between the version you have and the new and improved latest thing.

And RSS (or Atom) is XML, and XML is text, beautifully amiable to robotic splicing.

This is also, the argument goes and in a simplified way of looking at it, the way MPEG achieves sufficient bandwidth reduction to send tunes over basic broadband or send full-frame movies across your satellite feed. You pay the full price for the first checkpoint frame, and from there, it's just updates to this square here, that pixel over there -- there are problems if a storm glitch loses an update frame throwing you out of sync, but there are solutions for that ... even if ExpressVu doesn't implement them.

So ... is this the near future? mod_rsync? mod_gzip_rsync and content truncation?

Compression (and of which chopping out bits is the most extreme) is a salve, not a solution. It's not a distributed distribution channel, it is still that old familiar ego-centric star topology that only buys us time by lowering K in the B = nK relation of popularity to bits served. Sure, making K arbitrarily small might buy us enough time until we all run petabyte quantum hard-drives with paired-photon teleportation googol-BAUD modems ... but what we want is a B that varies with some fractional power of n ... regardless of K.

RSS is just the canary in the coal mine. A forewarning of a pending collision with a fundamental law of network services. For all our webservices, the only sustainable growth path is where server load is mediated by the network size, and that, as we all know, is the domain of the unspeakable technology, P2P.

hitting the wall of techno-politics

A footnote, an idea, a thought, probably just a pipedream but I think it's funny how we'd focussed our aggregate design wits to the problem of pipe-constrained shipping of porn video and other mega-byte media files, and yet, it's not the hippos but the insects who pose the real and present danger.

While 3229 might buy us some time, we're back to square one, ie the my square lines of reasoning that say what's on my server is mine and not to ever be ever confused with what's on TeledoN or TeledeN or anyone you might confuse with me, which includes, actually, AT&T, but nonetheless, it's a political thing disguised as a business thing. All your eyeballs are belong to us and so says the design deciders, we must please funding egos first, and only then proceed, within that space, to seek solutions.

I love to watch the narrow tunelvision of some tech thinking. No one dare speak the above, so all the solutions simply assume the constraint and charge full-steam ahead into brilliant solutions that only seek to reduce the effective size of the RSS file, or it's frequency, or it's contents. The problem, they assume, is in the consumer endpoint. No one suggests the problem might be elsewhere, in a thousand scattered boardrooms where even angels fear to tread.

Because the real solution is, to my naive P2P anarchist personal information space cadet mindset, really quite simple, affordable, doable and easy: Every new copy of Firefox would contain a tiny HTTP server, every ISP would encourage uplink and multicast, and every time your aggregator wondered if this site had any new golden gems of tech wisdom, you wouldn't ask some far off distant unique point in the starry sky, you would ask your network and get a nearest answer. It may not be absolutely to the letter of the moment perfect, but certainly close enough for jazz and all it takes is that paradigm shift that says, The more we get together
together, together
the more we get together
the happier we'll be
'cause your friends are my friends
and my friends are your friends
the more we get together ... ... only, if you do plan to present such cross-firewall inter-cooperative network thinking to your design bored directors, do me a favour, patch me in on a teleconf because I want to hear the peels of laughter.

I'll also want to know who they are so that, 5 years from now when all the top services work this way, I can check up on them and see how many jumped out the window over the pot of gold their tunnelvision passed up.

but wait a sec. Five years? Need it be so long? After all, quasi-immediate wide distribution of microcontent updates ... 'scuse me for asking but isn't that how DNS works?