Pipe Dream

In which we speculate about software reuse, serverless architecture, and find out it is all a pipe dream in the end.

Let us begin in a different business: Plumbing. I can go to an online store and buy pipes and fittings, taps, heaters, and a lot of other stuff. Components can be bought for electricians and electronics in a similar fashion. That is, if I want, I can buy components and put them together as I see fit - luckily you have to be certified to do create installations in your home - at least in Denmark. Software on the other hand is almost always bought in complete installations.

While it did not make any sense to buy a host of single components in the old days, there is basically no obstacles to do so today. Amazon (and other *aaS vendors) sell by the second or even millisecond, which means you could make micropayments for micro - or even nano - services. It seems it should be possible to have a McIlroy (bash) pipe connection of components across the internet with differentiated payment schemes. Perhaps a Mac Automator for the web.

The step up from “basically no obstacles”

Communication - both the format and the pipes are obstacles. In plumbing, you cannot seamlessly fit a 1 inch to a 2 inch pipe, which is visibly different. If water runs through the system, then water runs through every part. In IT the common denominator is text. Sometimes this seems too simple, but it works quite well for bash programming.

The other element, the pipes or rather the network. Networks are slow compared to local disk access, yet n-distributed parallel execution is extremely fast, and with a forwarding address on the output, i.e. redirection of STDOUT in bash, the data would not have to return to the origin for every segment.

This then suggests that there should be a protocol along the lines:

setup: parameters, forward to, payment key
execute: stream of data

Having such a setup would allow you to actually define the schema of the data - if it conforms to a schema. This in turn would allow SQL like filtering, but that really depends upon the service.

Why would I want this?

Well, first off, real re-use. It seems silly that we’re re-inventing the wheel every time a new Rails/Django/Drupal/… is installed. It seems unnecessary that a non-programming customer should have to setup and maintain a webservice to do something which is already possible to do by using existing components possibly in a different way. Even if the customer does this, everyone else with the same ideas would have to do the same for themselves.

Naturally I want it because I’ve run into some missing functionality in one product, that I could construct myself or use from another vendor, but without the pipe and the linking it is not possible.

Whether it is a poor search functionality in a webshop, an e-mail filtering/handling service, or news feeds I’d like to filter - I have to do it myself, and I have to pull the information.

Why it is all a pipe dream after all

Spinning up the services can take a lot of time. Sending massive amounts of data across the web will congest an already congested infrastructure. Hoping for every service provider to provide a standardized service interface is in itself a pipe dream.

Pipe dream - a hope, wish, or dream that is impossible to achieve or not practical (Merriam-Webster)

Comments are closed.