The plan echoes a similarly ambitious project launched by the then Commission President Jacques Delors that I remember discussing in detail at the time and that led in 1994 to the first commitments to develop a “Trans European Network”(TEN), announced at the Essen European Council summit of December that year.

Although the lion’s share of proposed investment this time around – with the proposed “Connecting Europe Facility” – is also on high-speed rail networks, there are also considerable chunks of money for the development of a European “Smart Grid” (playing catch-up with the USA) and support for “high speed digital networks”.

On this last point, much media emphasis has been on the support proposed for high-speed broadband Internet access but it is another aspect that I want to highlight: the development of infrastructure for the roll-out of many cross-border digital services, including digital identification, electronic public procurement, health care, justice and customs services.

This is significant because EU Member States have been very wary of sharing any responsibility for services that are considered the bedrock of national and local public administrations and into which ïnterference”by Brussels has been strongly resisted – and this despite the obvious needs for a high degree of interoperability across national borders between disparate eGovernment services.

A number of pilot projects have been under way for a few years, originating in EU eGovernment strategy work in which I participated when working for the eGovernment Unit of the Austrian Federal Government in the mid 2000′s.

The problem with the pilot projects has been precisely that – they are projects, with a start and an end, rather than sustained or sustainable infrastructure. They aimed to identify the major barriers to interoperability and propose open standards-based solutions but they stop short of being operational solutions.

The real leap forward therefore for the Commission is the proposal to create a coordinated EU-level infrastructure that would manage the current and possible future cross-border eServices. Talk in the Commission services last week was about the likely shape and governance model for this future infrastructure, likely to be in the form of a new “Executive Agency” under the operational supervision of the Commission – rather than beinga coordinated governance by the EU Member States.

Two issues in particular concern me: the choice of technologies needed to build and sustain the proposed services; and the governance model.

The increased roll-out of and reliance upon cloud-based eGovernment services should push to the back burner the often fraught hand-wringing over technology choice: the choice of open source solutions has often been promoted as some “neutral” way forward that avoids the Commission being accused of favouring particular technology platforms. The emphasis on the other hand of cloud-based solutions is rather on core service requirements and clear, agreed (hopefully standards-based) interfaces rather than on the technologies that actually deliver the solution and capability. Hopefully, the Commission will keep these considerations uppermost when it comes to filling out the details.

The governance model will be interesting too: many of the pilot projects over the years have been heavily influenced and domintaed by representatives of the EU Member States’ administrations. This was understandable in the circumstances in which national administrations were reluctant to cede any responsibility for cross-boder services to the European level and resisted any perceived attempt by the European Commission to arrogate such rights to itself. Increased tensions have emerged however as the 27 member states have been faced with a choice of trying to stitch together a large number of bilateral agreements between themselves in order to address EU legislative requirements; or recognise, however reluctantly, that a “hub and spoke” approach – with the Commission at the hub – would be a more efficient model to pursue. The choice of an Executive Agency to drive this forward will put the Commission in the driving seat – but will require intelligence and diplomacy in establishing a governance model that strikes the right balance between the interests of the member states and the need for an efficient hub.

What I want to argue is that the expectation to interact with devices in increasingly sophisticated ways is rapidly becoming a generalised meme rather than the unique selling point of any particular device, however smoothly the Apple marketing machine want to pitch it.

I think Tom’s work is extraordinary and I saw his pre-release demo of Siri to a hushed Web3.0 event last year and remember thinking how groundbreaking this would be. So I don’t want to take anything from today’s announcement but rather underline something else – that in fact Tom did very elegantly at the time.

He called it “The Perfect Storm” – he argued that we are approaching a moment when we can leverage four important and inter-related trends:

• The near-infinite scalability of cloud-computing, providing off-device computing, application and service power and content storage;
• The ever-increasing bandwidth available to devices (increasingly, mobile devices) connecting to the Internet;
• The growth of ecosystems of users providing rich crowdsourcing and powerful communities of interest;
• Increasingly “sensitive” mobile devices – sensitive (and responsive) to touch, sound, movement, light, in ways that allow them to act as proxies to our own senses in realtime

I’ll be covering the first of these in more detail next week when I open the International Cloud Symposium in England. On the second, I’ll restrain myself from too much comment about how we allow ourselves to be milked by mobile operators for our insatiable appetite to be “always on”. Many greater minfs have applied themselves to the third issue and Tom himself elegantly exposes the fourth.

However, taken as a whole, we are already in The Perfect Storm, or at the very least we all feel the leaves starting to rustle around us as the storm picks up.

I was struck by this forcefully only yesterday. My Windows Phone pinged me with the much-hyped and long-awaited news that the “Mango” update was ready to install. It didn’t disappoint. Being just a little bit nerdy, I pulled up my “wish list” and bug-tracking notes from a year ago when the Windows Phone was first released, and started to check off how many of my “must have”s and “nice to have”s are now covered – all so far. Nice. But the real surprises were those of the “oh so simple, why has no-one thought of this before” variety and really underlined Tom’s point in more general terms. A couple of examples:

• Press the ‘Search’ button and then the new ‘Vision’ icon: simply point the now-activated camera at a barcode (on a book, a DVD, any item on a shop’s or your own shelves) or QR code and the software will recognise and scan the object, check it against information in Bing services and send me a page summary of information or a web site. All this without having to take a photo, press the shutter, import to a scanning app, copy the data to a search query and await the result.
• A text message arrives while I’m driving – the phone offers to read the message aloud to me and, when done, offers me the options to dictate a reply or call the other party – all without taking a hand off the wheel or my eye off the road – something which I would welcome widespread adoption of here in Los Angeles where stupid in-car behaviour is elevated to a unique art-form (The other day, I watched from the passenger seat in horror as a driver – “only” doing 60mph on the motorway – was casually reading his New York Times on an iPad while talking on his hand-held phone).

It is computing power where and when I want it – it leverages the power of the local (multi-sense) device and couples it with the power of cloud-based services: each playing to their respective strong suits. The experience for the user is natural and intuitive. The mobile phone becomes the telechiric device of the information age.

I wish Tom and the whole Siri crew the best for continued success from their partnership with Apple but it’s already, in less than a year, no longer as unique an idea as it seemed – it’s rapidly becoming an expectation that mobile devices are natural and intuitive. I’ll leave you with the image of a father and his three-year old daughter on a recent long-haul flight, the daughter frustrated that the pages of the eBook she’s reading on her dad’s reader wouldn’t come “unstuck” if she held it upside down and shook it. Made me think how much technology has contributed to such cognitive dissonance and, on the positive side, how much it can also make our virtual worlds resonate with the real world around us.

Update (12 Oct 2011): It seems that Siri will not appear in the latest update of the iPhone.

The debate about standards and interoperability, when engaged, all too often (and all too soon) descends into details about which technical standards should be used in order to address particular problems rather than expending more effort on fully understanding the initial problem! It could be seen as laziness or lack of intellectual rigour that public officials (myself included, in my time) quickly grasp for off-the-shelf standards or objective criteria in order to justify a particular decision, whether it concerns a choice of platform or software; or a longer term strategic choice. A more charitable interpretation is that busy officials are concerned that their decisions and recommendations have objective foundations that can be justified as part of broader public sector strategy.

Whatever might be the reality in different administrations, our discussion yesterday focussed on the need for greater attention to policy and less to technical implementation concerns. Remember the sage advice: “be wary of answers before you have properly understood the question”. Whilst it is tempting, in response to a particular policy consideration, to immediately declare “there’s a standard for that!” and assume the problem thus solved, it is a temptation that ought to be resisted.

Let’s take an example: the often heated debate about document formats. Many public administrations are rightly concerned about maintaining vendor independence and ensuring long-term preservation of their documents. The key concerns should therefore be about what high-level policy decisions need to be taken before (and possibly instead of) more technology, solution oriented decisions. For example, storing any document in a proprietary, binary format, is largely accepted as “probably a very bad idea”. A clear policy commitment could therefore be that any technology used should ensure that this trap is avoided.

But to jump from that to mandating that content be stored in a specificformat, however it might be, seems less obvious. Technologies that manage, in this example, documents and information content are far more mature today and the process of importing and exporting content between different open standards has become trivial and a boon for the end user. Import a Word document to OpenOffice? No problem; Export from Word to .odf? No problem; Generate a .pdf from either? No problem. The core technologies – whether proprietary, open source, or a mix of both, have largely solved the technical interoperability problems of yesteryear. Policy makers need only to keep sight of their original high-level objective – avoid lock-in to binary, proprietary, formats – and move further up the value chain to address interoperability issues where they really still need to be addressed – which tend to be at the organisational, legal and “political” levels.

UKOLN’s approach, while taking a view on some core criteria for assessing open standards, emphasises this with its “Risk and Opportunities Framework” – it allows policy makers to move away from the heat of battle along ideological (often sterile) lines and about binary yes/no choices, and towards the light of more considered reflection of what is fit for purpose and value for money. “Use RAND”; “Use Open Source”; “RF only”; whatever the statement, it allows only one approach, one view and assumes there is a single, ideal truth or way, a binary bind. (There is also an implication when setting the debate up in this manner that there is a default ‘right’ choice – but I’ll return to that debate in another post).

Imagine instead a policy statement along the lines of: “Favour solutions that are not encumbered by licensing or patent restrictions”. These are not weasel words nor is the statement a cop-out of responsibility, it is a far more powerful and flexible policy statement and actually forces a debate about what “encumbered” means. It is not a binary choice, it forces a debate about how appropriate a particular IP model is to a particular situation. This is surely a good thing. It is a recognition that the domain is very complex and that policy makers need to take responsibility for assessing what is indeed fit for purpose and value for money rather than pushing it off to a curt reference to a standard, at least not before a considered examination of what specific needs are being addressed.

Working with standards and interoperability, as with any domain of expertise, requires a certain skill set. I believe that using approaches such as proposed by UKOLN, approach will create a climate of more mature, evidence-based, decision-making around technology choices and far less, ultimately vacuous, flag waving.

The obvious answer is “of course” – even if the “we respect your privacy” statements of many service providers is tantamount to saying “of course your data is safe in our hands – we’re not going to give it to anyone else. But, hey, this is a service provided free to you so you’ll just have to trust that we are using your data to provide you with an awesome service…”

Your average user, consciously or not, enters the Faustian pact in exchange for the free service. The service can offer you something for free because they make they money from investors and speculators who see that a big – very big – database of information on users is something that has marketable value. Whether it is providing profiling information for advertisers or selling on your data to third parties, there is money to be had. At a major CRM conference a few years back, one senior analyst had run some figures and estimated that each data field about a person has a market price of at least $150. Whilst I would dispute the figures, it is clear that the potential for increased market value of personal data will increase exponentially with the size of the user base: holding rich data on a few hundred people isn’t going to impress anyone – once in the realm of the millions, then people start taking notice.

When it comes to putting a company’s balance sheet together, calculating the value of intangible assets is something of an art, and even the best guidelines in the business are prefaced with comments such as “it’s worth whatever someone is willing to pay for it”. There is no evidence that the same isn’t true when it comes to assessing the value of all the personal data that some online services accumulate. Their whole business model is predicated on it. Sometimes the value only becomes apparent when you lose the data and end up being sued – the cost of the pay-out and your total legal bill will give you a fair assessment of what the market value is seen as for the personal data lost or compromised.

So, for the sake of argument, imagine turning the tables a moment. Instead of a company’s base of user records and data being considered as a pure intangible asset, imagine it being considered on the liability side of the comapny balance sheet – and that the proportionate liability increased with the number of users and complexity of data being held. Such companies would then need to put in place detailed plans for the protection of their user data, explain clearly how, where and when the data is (re-)used before accountants would be able to reduce the liability.

With data breaches becoming more common and more spectacular, investors should seriously ask themselves whether they are in the business that they thought they were – for social networks read: data harvesting and mining; and if they are, whether they feel sufficient safeguards are in place to protect valued intangible assets from being a major liability.

Another knock-on effect of this would that other business models, that truly place a premium on the protection and control by users of personal data and demonstrate this through their online practices, ought to be evaluated more favourably: a service that gives users total control of their data would indeed be managing valuable assets and ought to be recognised as such.

I ask only that you bear with me and keep in mind that the Civil War was fought at least in part about the powers of individual states compared with a ‘greater’, federal authority – and that point is important.

I was a little disturbed to read the somewhat gleefully written piece by Andy Updegrove about the British Government’s recent advisory note on the use of standards in public procurement. Aside from the slant given on the actual facts, it read as tantamount to a call to arms to EU nation states to cede from the impositions of the ‘federal troops’ of the European Commission in Brussels on matters of standards policy.

(Update 14-04-2011: A detailed and elegant rebuttal of Updegrove’s article has been published by Mondaq apparently following an outcry about the imbalance over the original piece)

The European Commission updated in December last its guidelines on standards in the new version of the so-called “European Interoperability Framework”. This bold step forward was widely welcomed as both pragmatic (the ‘law of unforeseen consequences’ had hurt the Commission with the first version of this work, as many open and legitimate standards organisations – including OASIS, the W3C and the European standards organisations themselves – risked falling outside of a very narrow and poorly scoped definition of ‘open’) and aimed at taking standards to the next level – beyond the important but down in the entrails stuff of data interoperability standards and to the ‘higher ground’ of semantic, organisational and legal interoperability.

I have been concerned for many years about how standards are used as a central element to public sector ICT policy and specifically how they help deliver better, citizen and business focussed solutions and services.

Our priority, as thought leaders in the standards community should be to identify a value chain for the delivery of services and provide tractable frameworks for policy makers, senior analysts, and business leaders to achieve that for the next generation of services.

But to take that step requires some maturity and willingness to avoid the rat holes that still dominate – or at the very least overshadow – many discussions around ‘open standards’.

Let me be clear – this is not a call to abandon the ‘moral high ground’ of wanting to see open standards developed, defined, promoted and used. There will always be those who will argue that they are special, are innovative, or so unique as to be beyond the perview of these considerations (Apple, Google, Facebook – are you listening? You and others are great innovators but not such good team players when it comes to discussing and agreeing common, open, solutions to important standardization issues), so everyone in the standards world needs to remain vigilant.

But to be equally clear – sterile turf-wars about who is more open than whom; attempts to claim one definition of ‘open’ as the definition and then denounce anyone else with an alternative take as “having an agenda” – all this simply underlines the fact that those who shout loudest have a clear agenda – and it often isn’t pretty and rarely has anything to do with moving forward with the adoption of open standards.

[For what it's worth - and some of the more belligerent stakeholders in this domain always want to pin others down (whilst themselves conveniently sidestepping similar requests) - I will state my position clearly. I actively support 'open standards' as defined and used by one of the key global open standards consortia, OASIS - on whose Board of Directors I sit as a member elected by the OASIS membership. What I particularly like about OASIS is that its standards development process is so open - one of my key criteria for judging the openness of a standard is assessing who is involved, what the membership structure is, who can participate and at what cost, how a proposal gets started, advanced and adopted, and - importantly - what intellectual property rules apply. Yes, you read that right - open standards can and do contain intellectual property, which is licensed for use by anyone implementing the standard according to the rules in play. Take the popularly cited Open Document Format, developed as an OASIS Standard - to claim that it should be considered as the document format of preference because it 'has no IP' and it thus 'truly open' is not only untrue (see the IPR statement from SUN Microsystems, don't take my word for it), it is a red herring. No open standards body defines open standards as solely being 'IP free' - even if many, including OASIS, have that as an optional path taken for the development of some of their standards work. What is important is to strike a balance between solving real world problems with innovation and conforming to open standards. Some put a premium on pure innovation and standards are only used or developed if they help the bottom line; others put a premium on being totally 'open' (often, unfortunately, with more than a whiff of smug superiority!) whilst totally failing to innovate (I have to agree with Jason Lanier who notes that "Linux is a superbly polished copy of an antique - shinier than the original, perhaps, but still defined by it."....But I digress – and that’s the danger in this debate...]

As discussions about standards start to look higher up the stack of service provision – not just data standards concerned with the interoperability of bits and bytes between arbitrary and heterogeneous systems but the way that government agencies define key principles, performance indicators, delivery mechanisms and measure value for their customers – they need to focus minds on the best ways to achieve these objectives through greater use of standards and not be side-tracked by the sterile debates of the previous decade.

The danger, as the European Union is about to debate the future of the European standardisation system, is that heavy lobbying will not only turn the clock back and have EU member states secede from a forward-looking approach to standards and interoperability that is so desperately needed to face the challenges of the next decade at a European level – cross-border services, procurement and invoicing to name but a few – but also see a perfectly valid and workable definition of ‘open standards’ reduced to meaninglessness – with all the consequences that flow from that.

The file is a set of philosophical ideas made into eternal flesh. The ideas expressed by the file include the notion that human expression comes in severable chunks that can be organized…and need to be matched to compatible applications.

What do files mean to the future of human expression?

What indeed. We treat files as an inherent part of our daily computing reality. But a proper understand of metaphors shows us that the notion of ‘files’ is nothing but a useful construct – there is nothing ‘real’ about them. So the question is: is it time to get rid of the file as the principal – and limiting – building block of computer-based human expression?

Aside from the wider debates that Lanier raises, this issue alone warrants some attention and response and there is the seed of a response further in the book where he states that “Instead of collections of bits being offered as a product, they would be rendered as a service”. After last Friday’s Semantic Link podcast recording, I re-found that quote – it was obviously haunting my sub-conscious when I stated, about OpenData initiatives, that we should worry less about datasets being offered up as a product and concentrate more effort on providing public information as a service.

This is certainly the attraction of the Open Data and Open Gov movements – public information is ‘trapped’ in files, where data is difficult to access and reposition, analyse and reuse, and providing access directly to the source will liberate data from that straightjacket. Or so the story goes.

The file can certainly seem like a straightjacket and, with the advent of versioning, backups, synchronisation between different devices and other functions that lead to the plethora of different versions of any piece of work (see my post on the whole idea of versions as a social construct), files can be increasingly unmanageable and even inappropriate to the way in which we, as humans, collect our thoughts and ideas and present them for sharing with others.

Files do, however represent a very explicit, user defined context – a flow of words, images and ideas are put together in a particular way for a particular reason and at a particular time.

Some, less responsible elements within the OpenData movement, would seem to be arguing that it is both desirable and possible to simply unpackage all the boxes and strew the content on the floor, leaving anyone to pick up arbitrary pieces and put them back together as they see fit. Most would sensibly argue that this is a step in the wrong direction but if the tools and the exposed content allow it, who is to stop you? What guarantees are there that the data is always going to be understood in some context, whether as intended or implied initially or as explicitly claimed by a future masher-uper?

It would seem that this remains the single biggest challenge for the OpenData movement: to guarantee the persistence of relevant context when data is re-purposed, including the data’s provenance, reliability, relationship to a particular event, geographical scope, etc.

There is growing support for public and untrammelled access to vast datasets held and curated by public authorities but less concern about the role that those same agencies have for putting the data in context – bluntly put: before or when releasing such data sets to the public, is there a public service imperative to deliver an information service as well as just a data product? Where should we draw the line between letting anyone mash-up, remix, re-analyse and re-position any public data – admitting the possibility (nay, given the global nature of the Internet, the likelihood) that it will be misused, abused and deliberately taken out of context to serve any one person’s ends – and ensuring that public data sets at least carry a Government Health Warning that “The Tolstoy Syndrome is extremely addictive. Selective use of public data will reinforce the habit.”

You will find a high correlation between references to ‘open data’ and ‘disintermediation’ – ‘cutting out the middle man’ – as the movement is driven precisely by a desire to ‘get at the facts and the truth’. But facts don’t speak for themselves. Today’s conversation with Evan Sandhaus from the New York Times underlined my conviction: technology tools are great but they have their greatest impact when in the hands of skilled craftspeople –rNews is of limited value if journalists simply cream off the top and choose not to apply the scientific rigour that we have come to expect from first class investigative journalism. In the words of Carnegie Corporation in an article back in 2006, “it’s fair to ask whether the news organisations of today – and tomorrow – are up to the task of sustaining the informed citizenry on which democracy depends”. Institutions such as the New York Times and The Guardian probably are, and have shown how capable they are through continual reinvention and investment in technology. Your average open data enthusiast, working alone with little or no training or access to informed context, probably isn’t. But then I never bought into the “wisdom of crowds” and noosphere either.

The Committee will start work now on a formal standard – probably using the ‘patterns language’ paradigm discussed in committee, and that will provide a suitable stepping stone between a policy document – useable and readable by senior policy and decision-makers – and a formally cast and fully machine-readable document in RDF or other expressive syntax.

The paper is available for download from my company web site. All comments welcome.