What does a top secret stealth submarine have in common with Edward Snowden's revelations about Internet snooping? How do you "snoop" on the Internet anyway, and what does it look like to the snooper? These are questions you'll struggle to find answers to in the mainstream media, but fear not - I'll cover them and more in today's super soaraway post!

[Picture credit: CC-BY-SA Flickr user ooocha. The photo shows USS Jimmy Carter being degaussed to reduce its magnetic signature]


An Internet Primer

So, we all know that the Internet is a series of tubes, right? Here's a nice video that explains in lay person's terms what really happens when two computers talk over the Internet, and takes in a few key physical Internet landmarks for good measure:



You might be surprised to discover that there are a relatively small number of Internet Exchanges that the majority of wide area Internet traffic passes through. What's that all about? Principally it's about creating opportunities for network and service providers to peer with each other.

For example, if I'm the BBC, then I know that by putting in a connection to the London Internet Exchange from my network I can reach customers of the major consumer broadband providers in the UK by the most direct route possible. It will save me money and give them a better experience.

As we'll see, snooping on the data passing through the Internet Exchanges is something that could be done entirely without the knowledge of the operators...

Demystifying Snooping

There are two kinds of snooping: The first is a high level peek at the conversation's metadata, which tells us the IP addresses of the communicating computers, and will give us a (not wholly reliable) insight into the service being accessed (website, email, file sharing, VOIP service, instant messaging etc). The second is a deep dive into the actual contents of that conversation - e.g. the text of email messages or the contents of files being shared.

Most network admins routinely use the metadata, known as flow data, to track the performance of their network and help debug problems such as compromised PCs. Here's a nice video promoting one of the leading products in this space, from a firm called Solarwinds:

All the major vendors of networking equipment provide a flow export capability in their routers, and there is a healthy market in terms of open source and commercial software to process flows looking for anomalies and trends. Flow based stats gathering is a key part of the network admin's toolkit, particularly for large enterprise scale networks with hundreds or thousands of switches and routers.


A Deeper Dive

To explore the content of a conversation (our second scenario) you would need a full copy of the data passing between the two computers. This can be done via physical interventions such as passive optical splitters or fibre splices, through to logical interventions such as port mirroring, where the traffiic from one network connection is copied to a "packet sniffer" machine plugged into another socket on the networking hardware. Another interesting wrinkle is to trick the target computer into sending its communications through the sniffer machine using a technique like ARP spoofing.

Here's a video showing the sort of thing you can do if you have access to the content of the conversation, and how this would look to a network admin (or security analyst...):



I should note that it's now fairly unusual for passwords to be transmitted unencrypted, although many websites are vulnerable to a cookie hijack attack, particularly over unencrypted (coffee shop style) wireless LANs, as in the Firesheep attack. However, passwords are routinely transmitted in the clear in large web server farms due to the widespread use of SSL Accelerator appliances and load balancers.

The software being demonstrated in the video is an open source package called Wireshark. You might find it interesting to download Wireshark and monitor your own machine to see who it is talking to on the Internet. Another piece of related software called tcpdump is installed on most Linux and Mac systems by default - be aware that you will need to run it from the command line (OS X Terminal) with admin privileges ("sudo bash"). Here's a 4 second tcpdump snippit of my Mac when it was sitting "idle". Lots of chatter, isn't there?

bash-3.2# tcpdump -i en0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on en0, link-type EN10MB (Ethernet), capture size 65535 bytes
23:07:07.328236 IP6 phoenix.local.59377 > ff02::c.ssdp: UDP, length 146
23:07:07.736625 ARP, Request who-has 192.168.1.108 tell 192.168.1.108, length 50
23:07:07.772846 IP 192.168.1.109.58316 > cache1.service.virginmedia.net.domain: 38702+ PTR? 108.1.168.192.in-addr.arpa. (44)
23:07:07.783254 IP cache1.service.virginmedia.net.domain > 192.168.1.109.58316: 38702 NXDomain 0/0/0 (44)
23:07:08.355570 IP lhr14s20-in-f4.1e100.net.https > 192.168.1.109.51908: Flags [P.], seq 802494782:802494863, ack 1527048200, win 1002, options [nop,nop,TS val 1790662823 ecr 1026470350], length 81
23:07:08.355703 IP 192.168.1.109.51908 > lhr14s20-in-f4.1e100.net.https: Flags [.], ack 81, win 8186, options [nop,nop,TS val 1026483881 ecr 1790662823], length 0
23:07:09.789475 IP lhr14s20-in-f4.1e100.net.https > 192.168.1.109.51908: Flags [P.], seq 81:163, ack 1, win 1002, options [nop,nop,TS val 1790664214 ecr 1026483881], length 82
23:07:09.789621 IP 192.168.1.109.51908 > lhr14s20-in-f4.1e100.net.https: Flags [.], ack 163, win 8186, options [nop,nop,TS val 1026485310 ecr 1790664214], length 0
23:07:10.432720 IP 192.168.1.109.62368 > cache1.service.virginmedia.net.domain: 8446+ A? e3191.c.akamaiedge.net. (40)
23:07:10.441583 IP cache1.service.virginmedia.net.domain > 192.168.1.109.62368: 8446 1/0/0 A 2.18.189.15 (56)
23:07:10.605345 IP 192.168.1.109.55930 > 192.168.1.1.osu-nms: UDP, length 4
23:07:10.606785 IP 192.168.1.1 > 192.168.1.109: ICMP 192.168.1.1 udp port osu-nms unreachable, length 40
23:07:10.613025 IP 192.168.1.109.51925 > lhr08s02-in-f2.1e100.net.http: Flags [.], seq 1789762529:1789763947, ack 450234823, win 8192, options [nop,nop,TS val 1026486128 ecr 935088968], length 1418
23:07:10.613027 IP 192.168.1.109.51925 > lhr08s02-in-f2.1e100.net.http: Flags [P.], seq 1418:2144, ack 1, win 8192, options [nop,nop,TS val 1026486128 ecr 935088968], length 726
23:07:10.636522 IP lhr08s02-in-f2.1e100.net.http > 192.168.1.109.51925: Flags [.], ack 1418, win 1002, options [nop,nop,TS val 935097857 ecr 1026486128], length 0
23:07:10.639042 IP lhr08s02-in-f2.1e100.net.http > 192.168.1.109.51925: Flags [.], ack 2144, win 1002, options [nop,nop,TS val 935097860 ecr 1026486128], length 0
23:07:10.726238 IP lhr08s02-in-f2.1e100.net.http > 192.168.1.109.51925: Flags [P.], seq 1:347, ack 2144, win 1002, options [nop,nop,TS val 935097950 ecr 1026486128], length 346
23:07:10.726358 IP 192.168.1.109.51925 > lhr08s02-in-f2.1e100.net.http: Flags [.], ack 347, win 8170, options [nop,nop,TS val 1026486239 ecr 935097950], length 0
23:07:11.109670 IP 192.168.1.109.50216 > 192.168.1.1.osu-nms: UDP, length 4
23:07:11.116171 IP 192.168.1.1 > 192.168.1.109: ICMP 192.168.1.1 udp port osu-nms unreachable, length 40
23:07:11.528622 IP 192.168.1.1.34430 > 239.255.255.250.ssdp: UDP, length 325
23:07:11.630967 IP 192.168.1.1.34430 > 239.255.255.250.ssdp: UDP, length 334
23:07:11.733248 IP 192.168.1.1.34430 > 239.255.255.250.ssdp: UDP, length 334
^C
23 packets captured
36 packets received by filter
0 packets dropped by kernel

Back Doors

Like most organizations nowadays we run a mixture of in house and cloud services. We have heard a lot about top secret back doors into cloud services, but what might those look like? Our Google Apps domains support the Email Audit API, which lets us do a range of interesting things such as set up a "monitor" to silently copy a person's email to another account. Here's a video of a third party product called FlashPanel, that builds upon this and other Google APIs to make your domains easier to manage:



Scary, huh? And that's not the half of it - you can also request a report of all of times and dates and IP addresses for a user's sessions, and secretly download an encrypted copy of their mailbox.

And there's more - in the regular Google Apps control panel there is a report that lets the administrator search through the logs of email passing through their domain, as shown below.

Before your paranoia level reaches fever pitch I should mention here that these are all things that we could and occasionally did do with our old in-house email system, with content itself only being accessed in exceptional cases. This would typically be when we were aiding a police investigation, and on receipt of the appropriate paperwork from our designated point of contact in the local county constabulary. Yes, in the UK there is a form for everything...


The Dual Use Dilemma

Now, here's the ironic thing - a couple of years ago I remember sitting down in the fancy deckchair area in Google's London offices with the product manager who led the Email Audit API work. I was bending his ear about the need for reporting on access stats and message deliveries, which we had often found invaluable in our old in-house system when dealing with misconfigured systems, disputes, compromised accounts and (yes) police inquiries. Back in those days we had to log a support call to ask someone to trawl through the logs Google-side when we needed this info.

So now these features exist, and would it be fanciful to think that there is a GPG encryption key that permits the NSA to use the Email Audit API against any Google account? Google have said no - disclosures have to be specifically requested and are handled manually. The nuance being that at present they are not permitted to disclose Foreign Intelligence Surveillance Act (FISA) notices, which they would like to do in future.

What this doesn't change is the dual use nature of the technologies I have described above - on the one hand they are needed for the correct running of computer networks, email systems and other network services. On the other, they are inherently open to abuse by unscrupulous individuals or systematic intervention by government agencies and organised crime. But this is a post about the technology, so I will skip the social commentary.

If you're reading this from years into the future, and wondering what all the fuss was about, let me leave you with this rather excellent video exposition from Taiwan's NMA World News:



Postscript: The Stealthy Submarine

Now the link to the submarine can be revealed: It has been asserted that the USS Jimmy Carter has been doctoring a number of trans-oceanic Internet cables to insert optical splitters for surveillance purposes. However, once the circuit has been split you would still need to plug the monitoring connection into something in order to do your snooping. That could be a bit tricky at the sea bottom, several miles into international waters... (but not impossible if you have access to all the resources of a major government!)