<p>Fourteen years ago, a pasty Irish teenager with a flair for inventions arrived at Edinburgh University to study artificial intelligence and computer science. For his thesis project, Ian Clarke created “a Distributed, Decentralised Information Storage and Retrieval System”, or a revolutionary new way for people to use the Internet without detection. By downloading Clarke’s software, which he intended to distribute for free, anyone could chat online, or read or set up a website, or share files, with almost complete anonymity.<br /><br />“It seemed so obvious that that was what the net was supposed to be about – freedom to communicate,” Clarke says. <br />His tutors were not bowled over. “I would say the response was a bit lukewarm. They gave me a B. They thought the project was a bit wacky … they said, ‘You didn’t cite enough prior work.”<br /><br />In 2000, Clarke publicly released Freenet. “At least 2m copies have been downloaded from the website, primarily in Europe and the US. The website is blocked in countries like China, so there, people tend to get Freenet from friends.” Last year Clarke produced an improved version: it hides not only the identities of Freenet users but also the fact that someone is using Freenet at all.<br /><br />Installing the software takes barely a couple of minutes and requires minimal computer skills. You find the Freenet website, read a few terse instructions, and answer a few questions. Then you enter a previously hidden online world. In utilitarian type and bald capsule descriptions, an official Freenet index lists the hundreds of “freesites” available. There is material written in Russian, Spanish, Dutch, Polish and Italian. There is English-language material from America and Thailand, from Argentina and Japan. There is all the teeming life of the everyday Internet, but rendered a little stranger and more intense. One of the Freenet bloggers sums up the difference: “If you’re reading this now, then you’re on the darkweb.”<br /><br />The modern Internet is often thought of as a miracle of openness. “Many users think when they Google they’re getting all the web pages,” says Anand Rajaraman, co-founder of Kosmix, one of a new generation of post-Google search engine companies. He adds: “I think it’s a very small fraction of the deep web which search engines are bringing to the surface. No one has a really good estimate of how big the deep web is.” <br /><br />Unfathomable and mysterious<br />The “darkweb”, “the deep web”, “beneath the surface web” – the metaphors alone make the Internet feel suddenly more unfathomable and mysterious. Other terms are “darknet”, “invisible web”, “dark address space”, “murky address space” and “dirty address space”. While a darknet is an online network such as Freenet that is concealed from non-users, with all the potential for transgressive behaviour that implies, much of “the deep web” consists of unremarkable consumer and research data that is beyond the reach of search engines. “Dark address space” often refers to Internet addresses that have simply stopped working.<br /><br />Beyond the confines of most people’s online lives, there is a vast other Internet out there, used by millions. How was it created? What exactly happens in it? <br />Michael K Bergman, an American academic and entrepreneur, is one of the foremost authorities on this other Internet. In the late 90s he undertook research to try to gauge its scale. In 2001 he published a paper on the deep web. “The deep web is currently 400 to 550 times larger than the commonly defined world wide web,” he wrote. <br /><br />In the eight years since, the use of Internet has been utterly transformed in many ways, but improvements in search technology by Google, Kosmix and others have only begun to plumb the deep web. “A hidden web (search) engine that’s going to have everything – that’s not quite practical,” says Professor Juliana Freire of the University of Utah, who is leading a deep web search project called Deep Peep. <br />But sheer scale is not the only problem. “When we’ve crawled (searched) several sites, we’ve gotten blocked,” says Freire. <br /><br />“There’s a well-known crime syndicate called the Russian Business Network (RBN),” says Craig Labovitz, chief scientist at Arbor Networks, a leading online security firm, “and they’re always jumping around the Internet, grabbing bits of (disused) address space, sending out millions of spam emails from there, and then quickly disconnecting.”<br /><br />The RBN also rents temporary websites to other criminals for online identity theft, child pornography and releasing computer viruses. What had been less understood until recently was how the increasingly complex geography of the Internet had aided them. “In 2000, dark and murky address space was a bit of a novelty,” says Labovitz. <br /><br />Defunct online companies; technical errors and failures; disputes between internet service providers; abandoned addresses once used by the US military in the earliest days of the internet – all these have left the online landscape scattered with derelict or forgotten properties, perfect for illicit exploitation, sometimes for only a few seconds before they are returned to disuse. How easy is it to take over a dark address? Says Labovitz: “But it just takes a PC and a connection. The Internet has been largely built on trust.”<br /><br />Open or closed?<br />In fact, the Internet has always been driven as much by a desire for secrecy as a desire for transparency. The network was the joint creation of the US defence department and the American counterculture – the WELL, one of the first and most influential online communities, was a spinoff from hippy bible the Whole Earth Catalog – and both groups had reasons to build hidden or semi-hidden online environments as well as open ones. “Strong encryption (code-writing) developed in parallel with the Internet,” says Danny O’Brien, an activist with the Electronic Frontier Foundation, a long-established pressure group for online privacy.<br /><br />The Onion Router, or Tor, is an American volunteer-run project that offers free software to those seeking anonymous online communication, like a more respectable version of Freenet. <br /><br />Tor’s users, according to its website, include US secret service “field agents” and “law enforcement officers . . . Tor allows officials to surf questionable websites and services without leaving tell-tale tracks,” but also “activists and whistleblowers”, for example “environmental groups that are increasingly falling under surveillance in the US under laws meant to protect against terrorism”. Tor, in short, is used both by the American state and by some of its fiercest opponents.<br />The hollow legs of Sealand<br />In 1975, only half a dozen years after the internet was created, the science-fiction author John Brunner wrote of “so many worms and counter-worms loose in the data-net” in his novel ‘The Shockwave Rider’. By the 80s “data havens”, at first physical then online locations where sensitive computerised information could be concealed, were established in discreet jurisdictions such as Caribbean tax havens. In 2000 an American internet startup called HavenCo set up a much more provocative data haven, in a former Second World War sea fort just outside the British territorial waters off the Suffolk coast, which since the 60s had housed an eccentric independent “principality” called Sealand. <br /><br />HavenCo announced it would store any data unless it concerned terrorism or child pornography, on servers built into the hollow legs of Sealand as they extended beneath the waves. <br /><br />In 2007 the highly successful Swedish filesharing website The Pirate Bay – the downloading of music and films for free being another booming darknet enterprise – announced its intention to buy Sealand. Last year it was reported that HavenCo had ceased operation. <br /><br />Services such as Tor and Freenet perform the same function electronically; and in a sense, even the “open” Internet has increasingly become a place for concealment: people posting and blogging under pseudonyms, people walling off their online lives from prying eyes on social networking websites.<br /><br />“The more people do everything online, the more there’s going to be bits of your life that you don’t want to be part of your public online persona,” says O’Brien. <br />To libertarians such as O’Brien, the hidden internet is constantly under threat from restrictive governments and corporations. “Child pornography does exist on Freenet,” says Clarke. “But it exists all over the web, in the post . . . At Freenet we could establish a virus to destroy any child pornography on Freenet – we could implement that technically. But then whoever has the key (to that filtering software) becomes a target. Suddenly we’d start getting served copyright notices; anything suspect on Freenet, we’d get pressure to shut it down. To modify Freenet would be the end of Freenet.”<br /><br />Always recorded<br />According to the police, for criminal users of services such as Freenet, the end is coming anyway. <br />The Internet, for all its anarchy, is becoming steadily more commercialised; as Internet service providers, for example, become larger and more profit-driven, the spokesman suggests, it is increasingly in their interests to accept a degree of policing.<br /><br />Search engine companies are restlessly looking for paths into the deep web and other sections of the Internet currently denied to them. Says Anand Rajaraman: “Tonnes and tonnes of stuff out there on the deep web has what I call security through obscurity. But security through obscurity is actually a false security. You (the average internet user) can’t find something, but the bad guys can find it if they try hard enough.”<br /><br />As Kosmix and other search engines improve, he says, they will make the internet truly transparent. The internet as a sort of electronic panopticon, everything on it unforgivingly visible and retrievable, suddenly its current murky depths seem in some ways preferable.<br />Ten years ago Tim Berners-Lee, the British computer scientist credited with inventing the web, wrote: “I have a dream for the web in which computers become capable of analysing all the data on the web – the content, links, and transactions between people … A Semantic Web’, which should make this possible, has yet to emerge, but when it does, the day-to-day mechanisms of trade, bureaucracy and our daily lives will be handled by machines talking to machines.” <br />Says Bergman: “It’s really been the holy grail for 30 years.” One obstacle, he continues, is that the Internet continues to expand in unpredictable and messy surges. “The boundaries of what the web is have become much more blurred. Is Twitter part of the web or part of something else? Now the web, in a sense, is just everything. <br /><br />Gold rush<br />It seems likely that the internet will remain in its gold rush phase for some time yet. And in the crevices and corners of its slightly thrown-together structures, darknets and other private online environments will continue to flourish.<br />They can be inspiring places to spend time in, full of dissidents and eccentrics and the internet's original freewheeling spirit. But a darknet is not always somewhere for the squeamish.<br /><br />On Freenet, there is a currently a “freesite”, which makes allegations against supposed paedophiles, complete with names, photographs, extensive details of their lives online, and partial home addresses. <br />In much smaller type underneath runs the disclaimer: “The material contained in this freesite is hearsay . . . It is not admissable in court proceedings and would certainly not reach the burden of proof requirement of a criminal trial. For the time being, when I’m wandering around online, I may stick to Google.</p>
<p>Fourteen years ago, a pasty Irish teenager with a flair for inventions arrived at Edinburgh University to study artificial intelligence and computer science. For his thesis project, Ian Clarke created “a Distributed, Decentralised Information Storage and Retrieval System”, or a revolutionary new way for people to use the Internet without detection. By downloading Clarke’s software, which he intended to distribute for free, anyone could chat online, or read or set up a website, or share files, with almost complete anonymity.<br /><br />“It seemed so obvious that that was what the net was supposed to be about – freedom to communicate,” Clarke says. <br />His tutors were not bowled over. “I would say the response was a bit lukewarm. They gave me a B. They thought the project was a bit wacky … they said, ‘You didn’t cite enough prior work.”<br /><br />In 2000, Clarke publicly released Freenet. “At least 2m copies have been downloaded from the website, primarily in Europe and the US. The website is blocked in countries like China, so there, people tend to get Freenet from friends.” Last year Clarke produced an improved version: it hides not only the identities of Freenet users but also the fact that someone is using Freenet at all.<br /><br />Installing the software takes barely a couple of minutes and requires minimal computer skills. You find the Freenet website, read a few terse instructions, and answer a few questions. Then you enter a previously hidden online world. In utilitarian type and bald capsule descriptions, an official Freenet index lists the hundreds of “freesites” available. There is material written in Russian, Spanish, Dutch, Polish and Italian. There is English-language material from America and Thailand, from Argentina and Japan. There is all the teeming life of the everyday Internet, but rendered a little stranger and more intense. One of the Freenet bloggers sums up the difference: “If you’re reading this now, then you’re on the darkweb.”<br /><br />The modern Internet is often thought of as a miracle of openness. “Many users think when they Google they’re getting all the web pages,” says Anand Rajaraman, co-founder of Kosmix, one of a new generation of post-Google search engine companies. He adds: “I think it’s a very small fraction of the deep web which search engines are bringing to the surface. No one has a really good estimate of how big the deep web is.” <br /><br />Unfathomable and mysterious<br />The “darkweb”, “the deep web”, “beneath the surface web” – the metaphors alone make the Internet feel suddenly more unfathomable and mysterious. Other terms are “darknet”, “invisible web”, “dark address space”, “murky address space” and “dirty address space”. While a darknet is an online network such as Freenet that is concealed from non-users, with all the potential for transgressive behaviour that implies, much of “the deep web” consists of unremarkable consumer and research data that is beyond the reach of search engines. “Dark address space” often refers to Internet addresses that have simply stopped working.<br /><br />Beyond the confines of most people’s online lives, there is a vast other Internet out there, used by millions. How was it created? What exactly happens in it? <br />Michael K Bergman, an American academic and entrepreneur, is one of the foremost authorities on this other Internet. In the late 90s he undertook research to try to gauge its scale. In 2001 he published a paper on the deep web. “The deep web is currently 400 to 550 times larger than the commonly defined world wide web,” he wrote. <br /><br />In the eight years since, the use of Internet has been utterly transformed in many ways, but improvements in search technology by Google, Kosmix and others have only begun to plumb the deep web. “A hidden web (search) engine that’s going to have everything – that’s not quite practical,” says Professor Juliana Freire of the University of Utah, who is leading a deep web search project called Deep Peep. <br />But sheer scale is not the only problem. “When we’ve crawled (searched) several sites, we’ve gotten blocked,” says Freire. <br /><br />“There’s a well-known crime syndicate called the Russian Business Network (RBN),” says Craig Labovitz, chief scientist at Arbor Networks, a leading online security firm, “and they’re always jumping around the Internet, grabbing bits of (disused) address space, sending out millions of spam emails from there, and then quickly disconnecting.”<br /><br />The RBN also rents temporary websites to other criminals for online identity theft, child pornography and releasing computer viruses. What had been less understood until recently was how the increasingly complex geography of the Internet had aided them. “In 2000, dark and murky address space was a bit of a novelty,” says Labovitz. <br /><br />Defunct online companies; technical errors and failures; disputes between internet service providers; abandoned addresses once used by the US military in the earliest days of the internet – all these have left the online landscape scattered with derelict or forgotten properties, perfect for illicit exploitation, sometimes for only a few seconds before they are returned to disuse. How easy is it to take over a dark address? Says Labovitz: “But it just takes a PC and a connection. The Internet has been largely built on trust.”<br /><br />Open or closed?<br />In fact, the Internet has always been driven as much by a desire for secrecy as a desire for transparency. The network was the joint creation of the US defence department and the American counterculture – the WELL, one of the first and most influential online communities, was a spinoff from hippy bible the Whole Earth Catalog – and both groups had reasons to build hidden or semi-hidden online environments as well as open ones. “Strong encryption (code-writing) developed in parallel with the Internet,” says Danny O’Brien, an activist with the Electronic Frontier Foundation, a long-established pressure group for online privacy.<br /><br />The Onion Router, or Tor, is an American volunteer-run project that offers free software to those seeking anonymous online communication, like a more respectable version of Freenet. <br /><br />Tor’s users, according to its website, include US secret service “field agents” and “law enforcement officers . . . Tor allows officials to surf questionable websites and services without leaving tell-tale tracks,” but also “activists and whistleblowers”, for example “environmental groups that are increasingly falling under surveillance in the US under laws meant to protect against terrorism”. Tor, in short, is used both by the American state and by some of its fiercest opponents.<br />The hollow legs of Sealand<br />In 1975, only half a dozen years after the internet was created, the science-fiction author John Brunner wrote of “so many worms and counter-worms loose in the data-net” in his novel ‘The Shockwave Rider’. By the 80s “data havens”, at first physical then online locations where sensitive computerised information could be concealed, were established in discreet jurisdictions such as Caribbean tax havens. In 2000 an American internet startup called HavenCo set up a much more provocative data haven, in a former Second World War sea fort just outside the British territorial waters off the Suffolk coast, which since the 60s had housed an eccentric independent “principality” called Sealand. <br /><br />HavenCo announced it would store any data unless it concerned terrorism or child pornography, on servers built into the hollow legs of Sealand as they extended beneath the waves. <br /><br />In 2007 the highly successful Swedish filesharing website The Pirate Bay – the downloading of music and films for free being another booming darknet enterprise – announced its intention to buy Sealand. Last year it was reported that HavenCo had ceased operation. <br /><br />Services such as Tor and Freenet perform the same function electronically; and in a sense, even the “open” Internet has increasingly become a place for concealment: people posting and blogging under pseudonyms, people walling off their online lives from prying eyes on social networking websites.<br /><br />“The more people do everything online, the more there’s going to be bits of your life that you don’t want to be part of your public online persona,” says O’Brien. <br />To libertarians such as O’Brien, the hidden internet is constantly under threat from restrictive governments and corporations. “Child pornography does exist on Freenet,” says Clarke. “But it exists all over the web, in the post . . . At Freenet we could establish a virus to destroy any child pornography on Freenet – we could implement that technically. But then whoever has the key (to that filtering software) becomes a target. Suddenly we’d start getting served copyright notices; anything suspect on Freenet, we’d get pressure to shut it down. To modify Freenet would be the end of Freenet.”<br /><br />Always recorded<br />According to the police, for criminal users of services such as Freenet, the end is coming anyway. <br />The Internet, for all its anarchy, is becoming steadily more commercialised; as Internet service providers, for example, become larger and more profit-driven, the spokesman suggests, it is increasingly in their interests to accept a degree of policing.<br /><br />Search engine companies are restlessly looking for paths into the deep web and other sections of the Internet currently denied to them. Says Anand Rajaraman: “Tonnes and tonnes of stuff out there on the deep web has what I call security through obscurity. But security through obscurity is actually a false security. You (the average internet user) can’t find something, but the bad guys can find it if they try hard enough.”<br /><br />As Kosmix and other search engines improve, he says, they will make the internet truly transparent. The internet as a sort of electronic panopticon, everything on it unforgivingly visible and retrievable, suddenly its current murky depths seem in some ways preferable.<br />Ten years ago Tim Berners-Lee, the British computer scientist credited with inventing the web, wrote: “I have a dream for the web in which computers become capable of analysing all the data on the web – the content, links, and transactions between people … A Semantic Web’, which should make this possible, has yet to emerge, but when it does, the day-to-day mechanisms of trade, bureaucracy and our daily lives will be handled by machines talking to machines.” <br />Says Bergman: “It’s really been the holy grail for 30 years.” One obstacle, he continues, is that the Internet continues to expand in unpredictable and messy surges. “The boundaries of what the web is have become much more blurred. Is Twitter part of the web or part of something else? Now the web, in a sense, is just everything. <br /><br />Gold rush<br />It seems likely that the internet will remain in its gold rush phase for some time yet. And in the crevices and corners of its slightly thrown-together structures, darknets and other private online environments will continue to flourish.<br />They can be inspiring places to spend time in, full of dissidents and eccentrics and the internet's original freewheeling spirit. But a darknet is not always somewhere for the squeamish.<br /><br />On Freenet, there is a currently a “freesite”, which makes allegations against supposed paedophiles, complete with names, photographs, extensive details of their lives online, and partial home addresses. <br />In much smaller type underneath runs the disclaimer: “The material contained in this freesite is hearsay . . . It is not admissable in court proceedings and would certainly not reach the burden of proof requirement of a criminal trial. For the time being, when I’m wandering around online, I may stick to Google.</p>
