Big Data, the NSA, Phone Records, and You

August 2013

Susie has a mild case of the flu. And, being young, naturally she solicits sympathy by posting to Twitter.

Her post, of course, is a matter of public record. If you go to Twitter and search on "flu," you'll find her post someplace in the welter of posts. And, it turns out, her post is of interest to epidemiologists.

It's not just Susie, but a lot of other people, too, are also posting their misery to Twitter. A number years ago someone got the idea to download Twitter data, write a computer program to sift through it looking for mentions of the flu, and then see if that could be useful in tracking outbreaks.

As a matter of fact, Twitter and other sources of large sets of data are being found to be quite useful in epidemiology. There's even a name for it: big epidemiology. Google actually has something called Google Flu: a computer program that analyzes searches for phrases such as "flu symptoms" and identifies where outbreaks are starting to occur. Not only has it been found to be as accurate as the standard tools of epidemiology for tracking outbreaks, but it's capable of identifying these trends a full two weeks before the standard approaches, which are based on tallying doctors' reports nationwide.

Note, though, that epidemiologists aren't interested in Susie's particular situation; she's just a data point in a vast trove of data. They likely don't even know her name.

This has come to be called "big data": mining huge volumes of information for patterns and trends. This technique of gaining insights from large data sets has become quite prominent in the past couple years, and is being used not only for epidemiology but also in everything from marketing to stock trading,

So is the National Security Agency listening in on your phone calls? No. As I write this, the NSA has been in the news for weeks, ever since it was first reported that phone carriers are being required to turn over records of phone calls.

The NSA is using these phone records as big data, sifting through them to identify patterns and trends. They aren't necessarily looking for something specific. That is, they don't likely have a suspect in mind and are looking to see whom he's calling. At least initially. What they're doing is unleashing a computer program on these unimaginally vast data sets to see if it notices anything suspicious, such as a sudden inordinate number of phone calls from a neighborhood in Chicago to an area of Yemen known to be friendly to Al Qaida.

Our government says this snooping has foiled quite a number of terrorist attacks.

Here's how today's situation differs from before. In the past, access to phone records and permission for wiretapping typically came AFTER a suspect had been identified. There was cause for concern, and the ability to snoop was granted.

Today, however, there's no suspect, and instead of requesting information or access related to specific individuals, the government is asking for the whole data set. The goal, as with flu tracking, is to more accurately identify patterns much sooner than otherwise possible. Identify the criminal before the crime can be committed.

Frankly, the NSA isn't likely interested in you, but it is interested in having the entire data set, which includes your information — a speck in an ocean of data.

I'm not that interested in the related privacy issues, so I won't go there. What interests me is an emergent phenomenon: a collective intelligence today that wasn't possible in the past.

Computer networks have created a sort of global brain, with each individual like a neuron contributing to a collective intelligence. Like it or not, a record of much of what you do is being kept: your purchases, your Internet searches, your Facebook and Twitter posts, your posts in online discussions, your phone calls, your email.

Some of this information can be traced back to your identity, and some can't. But all of it becomes a part of the fabric of big data — and a global brain. An individual neuron in the brain is just one among a hundred billion, but together they act in concert to store memories, to generate ideas, and to create your world.

Susie and others get the flu, and the global brain reacts, almost like a worldwide immune system. You and others search the Internet or make purchases online, and these propensities become like ideas propagating through the global brain. What's different now is that big data tools are, in a sense, bringing these phenomena to conscious awareness.

It's a different world.

This Month's Hot Tip

Find great Amazon shopping here.

© 2013 by Jim Karpen, Ph.D.

E-mail Jim Karpen