Surveillance, Britain's Secret Agencies, and Drowning in Data

I was speaking to someone today about this, and it occurred to me that it is a piece of information that is not widely known but should be.

Every year in Britain, there is an official report that comes out detailing the activities of the UK's spy agencies — MI5, MI6 and GCHQ. It is authored by a group of politicians who function as a kind of oversight authority, under the name the Intelligence and Security Committee.

In this year's report, published in July, I noticed a section of particular interest in light of new proposals for more surveillance powers in the UK. The second paragraph is what is important here — it is a comment made by Jonathan Evans, chief of domestic security agency MI5.

The Security Service is undertaking a number of major projects covering estates, business continuity, core IT systems and improving its digital investigative capabilities. A notable success during the reporting period was the completion of the Digital Intelligence (DIGINT) programme, which aimed to improve systems for the collection and analysis of intelligence material gathered electronically. The Director General explained:

"One of the things that really drove us on the investment of DIGINT was a discussion where the relevant directors explained that actually, of all the material that we’ve caught, over half was not being processed. Now, as an intelligence organisation, that’s a nightmare. I mean, quite frankly, I would rather not have the intelligence at all and miss something than have the intelligence and not actually having processed it… We have made real progress on that, and I’m very proud on DIGINT." (Emphasis added)

What this comment suggests, for the sake of clarity, is that the UK's spy agencies in recent years have been mining and storing quantities of electronic data — or "digital intelligence" — so large that they have not been able to analyse it. The data, most of it I would expect is mined from the internet, has probably been gathered and then left to sit and gather digital dust in a secret storeroom somewhere. The claim from Evans in the above quote is that MI5 has worked to address the problem as part of a new programme, which presumably involves a great deal of automated analysis. But the statement also illustrates how new surveillance powers currently being proposed in the UK could pose problems for the UK if the security services are already near a point where they are drowning in data.

There tends to be two main schools of thought within the intelligence community. Some believe that targeted surveillance of specific individuals and groups is the best method, because it provides information that can be dissected and acted upon fairly quickly by human analysts. The other school of thought, and the one which seems to be prevailing, is that a kind of dragnet surveillance is superior. What this entails is gathering huge quantities of data based on key words, locations, phrases, and then mining through it to find anything useful. From a rights and civil liberties perspective, targeted surveillance is clearly more attractive because it is likely to involve much less intrusion of innocent individuals' communications. But rights and civil liberties do not appear to be high on the agenda at our secret agencies, and so what we get is something closer to the dragnet option.

I should add that surveillance in the UK is not without regulation. To intercept domestic communications, police and security services require ministerial authorisation, and must show that any interception is in the interests of national security, safeguarding economic well being, or to prevent and detect serious crime. That said, these justifications are fairly broad, and there were 2,911 interception warrants granted in 2011 — but any one warrant can cover countless individuals, so we actually have little idea how many people had their communications snooped on. (Also, to monitor content posted on social networks and other "open source" websites, there are no laws or restrictions at all. So websites like Facebook, Twitter and Foursquare are all fair game for the likes of MI5's "DIGINT" team to gather data from.)