Mickey, Barrett and Ivan

How a bookmaker and a whiz kid took on an extortionist, and won^[1]

Saturday, Nov. 22, 2003, 7:57 a.m. Origins
The first e-mail began:

Your site is under attack

It gave Mickey Richardson two choices:

You can send us $40K by Western Union [and] your site will be protected not just this weekend but for the next 12 months [or] If you choose not to pay...you will be under attack each weekend for the next 20 weeks, or until you close your doors.

Richardson runs BetCris.com, an online wagering site, one of hundreds of sites ensconced in Costa Rica that take bets from Americans (and others around the world) without concern for U.S. bookmaking laws. Richardson received the e-mail just as he and his competitors were preparing for the year's busiest wagering season. With pro and college football, pro and college basketball and other sports in full swing, and with Thanksgiving and Christmas about to create plenty of free time, BetCris and the others stood to rake in millions over the holidays. Richardson was even planning an advertising blitz for the season to drive new traffic to his site.

If BetCris went down, he knew his customers would find another online bookmaker, "which will cost you tens of thousands of dollars in lost wagers and customers," the extortionists reminded him.

Despite all that, the e-mail didn't have the fearsome effect on Richardson that the extortionists hoped it would. He just asked his network administrator, Glenn Lebumfacil, if they should be concerned. "I said—God, in hindsight, what an idiot—I said, 'We should be safe. I think our network is nice and tight,'" recalls Lebumfacil.

As a precaution, Richardson alerted his ISP, but essentially, he says, "We kind of fluffed it off." The veteran bookmaker didn't panic because, in fact, he had dealt with online extortionists before. Two years earlier, hackers crashed BetCris.com with a denial-of-service (DoS) attack, and then demanded by e-mail a $500 protection fee in eGold (an online form of trading bullion). Richardson paid without a second thought. Compared to downtime, $500 was trivial.

I said—God, in hindsight, what an idiot—I said, We should be safe. I think our network is nice and tight.

That first attack got his attention, though. Richardson consulted another industry veteran who confessed to having a similar problem, and who told Richardson to call a consultant named Barrett Lyon in Sacramento, Calif. Lyon didn't come to BetCris's offices—he had no interest in baby-sitting infrastructure in Costa Rica—but he did recommend some new off-the-shelf products developed specifically to fight DoS attacks. Lyon thought (actually he hoped) that he'd never hear from BetCris again. Richardson and Lebumfacil were confident they had protected themselves.

When the attack finally came on that Saturday in November, sometime after that first e-mail but before 11:30 a.m., BetCris crashed hard. The off-the-shelf products Lyon had recommended survived less than 10 minutes. BetCris's ISP crashed, and then the ISP for BetCris's ISP crashed. Richardson ran to the IT department, where Lebumfacil was watching the biggest DDoS attack he'd ever seen. He remembers feeling sick to his stomach.

At 1:03 p.m., another e-mail arrived.

I guess you have decided to fight instead of making a deal. We thought you were smart.... You have 1 hour to make a deal today or it will cost you $50K to make a deal on Sunday.

Then they knocked BetCris.com offline again.

The Extortion Problem
We know this about online extortion: It happens. Evidence of its prevalence or damage is speculative and anecdotal but useful nonetheless. Anecdotally, experts from law enforcement and information security consultants believe that perhaps one in 10 companies has been threatened with online extortion; in one survey by Carnegie Mellon University researchers, 17 out of 100 small and midsize businesses reported being targeted. Interviews with security consultants and industry players suggest that as many as three out of four cases of online extortion are never reported. Maybe a third or more of targeted companies pay extortion fees, drawing the money from disaster funds, acceptable loss budgets or insurance. Consultants like to tell stories of being called for help after companies pay protection money twice.

It would be easy to view online extortion as indigenous to gambling sites, the karmic price one pays for choosing that line of work. It would also be wrong. True, the Thanksgiving-week attack on BetCris fronted a wave of extortion against gaming sites, but that wave has since ebbed (in part, we'll see, due to BetCris) while the online extortion phenomenon has not.

In fact, that wave of attacks against gaming sites, starting in late 2003 and going through mid-2004, appears to have been a training ground for extortionists. Now they've moved on, applying what they learned, along with more sophisticated technical tools, to attack far less prepared and more mainstream targets such as online payment services, foreign currency exchanges and financial services companies. Here is a good rule of thumb: Anyone who could lose money by being offline is a potential online extortion target. And the more one stands to lose, the bigger the bull's-eye.

Yet you probably haven't thought much about online extortion unless you've been targeted. As with fraud, a certain shame attaches itself to victims, especially those who choose to pay protection fees. Even antiextortion consultants participate in a code of silence. One such company contacted for this story declined to comment "because we feel it brings attention to the crime."

That's why we're telling this story—to bring attention to the crime. To help readers to learn from a real-world case what worked in an extortion crisis and what didn't. To sort out the choices one has before the choices one has are dictated by an e-mail.

Saturday, Nov. 22, 2003: Pleas
Richardson and Lebumfacil decided to reply to the extortionists' e-mail. They stalled. Lebumfacil, the network admin, recalls the pleading tone of their missives. (They sent several.) They'd say that they would lose their jobs if they didn't get more time. Richardson reluctantly admits that he feigned a family emergency and begged the extortionists to give him time until he could return from that to set up a payment.

Meanwhile, Lebumfacil and the IT team tried in vain to stop the attacks and get BetCris back online. The equation was simple: Downtime equals lost revenue. Richardson says the company stood to lose $1.16 every second, as much as $100,000 per day.

He tracked down Barrett Lyon, who was in Phoenix helping another company fight off a DDoS attack. Lyon told Richardson to call the off-the-shelf equipment vendor. (He did. No help.) Call the ISP. (It couldn't help, either.)

Lyon says he sensed desperation, and he was right. Lebumfacil, who had a 5-month-old daughter at home, says, "I thought about losing my job. I thought about the company going out of business. There was a lot of money on the line. It was a constant state of panic." That night he tried in vain to sleep and says he even entertained the fantasy that "everything could be OK in the morning."

But it wasn't OK in the morning. At 10:01 a.m. on Sunday, Richardson got another e-mail. This one sounded less like a threat and more like the start of negotiations.

Dear Mickey, The attacks have been stopped 2 hours prior to the last e-mail. Your site is back up for most and should be up for all shortly...P.S. We will e-mail you Monday.

Still, Richardson wasn't encouraged. The site wasn't up at all; it only came to life sporadically and for short periods of time. No one knows for sure, but the extortionists might have stopped their attack. At some point, the downtime was the result of BetCris's ISP deciding to null-route the site's traffic. Null-routing means the ISP collects all of the traffic going to a site and drives it into the ground. This frees up the ISP's pipes when a site it hosts is receiving massive amounts of DDoS attack traffic; even if the extortionists stopped attacking, the site would stay down.

Confusion and stress reigned. Richardson called Lyon again. This time, Lyon agreed to help. "I was thinking this would be a big mess for me," he says. "But they had no one to turn to. I knew by Sunday I couldn't pass them off any longer." Lyon flew back to Sacramento and started working on the problem. He, too, had dealt with online extortionists before.

Sunday, July 21, 2002: Flashback
From a low-slung building off of Flamingo Drive in Las Vegas, a company called Don Best delivers the ever-fluctuating odds on sporting events to most of the glitzy sports books on the Strip. All of this is done by computers, and late in the evening on July 21, files started moving around one of those computers by themselves. An employee working late called Don Best's general manager, Rick Allec, and asked him what to do. Allec told him to turn off the server. The employee couldn't, so he literally pulled the plug out of the wall.

Allec rushed to the office, and soon he was holding the printout of an extortion e-mail demanding $200,000. He replied—and stalled—just as Richardson would a year later.

The next day, a security consultant told Allec to call Barrett Lyon for help. "When Barrett showed up," Allec recalls, "I remember thinking, There's no way he can help us."

Lyon was 23 and looked at least that young. His blond hair offset a tan, handsome face. Allec says Lyon looked like he had given up a day of surfing to swing by and help out.

Lyon had never taken a computer science class. His degree from California State University, Sacramento, was in philosophy, applied law and ethics. And yet he was cocky about computers. Once, he bet some friends he could map the entire Internet in a day. They scoffed. He launched Opte.org and mapped the entire Internet in a day. (Sort of. The open-source project is ongoing.) "People have never worried about my background," Lyon says, "because when they ask questions, I can answer them."

He had to win over Allec quickly, since Allec's customers were irate. A sports book forced to turn away wagers is like a bank turning down deposits. "We were down for three hours at one point, which was absolutely unheard of in our business," says Allec. "But Barrett made me comfortable. He would say, 'They're going to do this next, and we'll fight it this way.' And every time, he was exactly right. It was almost eerie."

When Barrett showed up I remember thinking, There's no way he can help us.

At the time, off-the-shelf anti-DDoS hardware wasn't readily available. Lyon's solution for Don Best was not to turn back the attack, but to scale Don Best's infrastructure of Web servers, load balancers and other hardware so that it was bigger than the volume of attack traffic coming in. "We basically built a humongous Web farm in, like, four days," Lyon says.

It proved to be enough to fend off the extortionists, who were sloppy. They attacked during the slowest gambling season, when the mark had less impetus to capitulate under pressure. They also asked for so much money that Allec didn't immediately determine that paying would be his smartest option.

Within a week, it was over. Except—and this impressed Allec the most—"a couple of weeks later I get a call from Barrett, and he says, 'I know who attacked your site.'"

Lyon says, "I could have left it alone, but I had gotten attached, and I started investigating. I came up with some interesting techniques to trace back the attacks." He turned over his work to several law enforcement agencies, but he never heard about it again.

It was Allec who recommended Lyon to Mickey Richardson at BetCris after Richardson's $500 eGold incident. "During that time when all those sites were getting extorted, you only stopped it one of two ways," Allec says. "You either paid them off, or you called Barrett."

Monday, Nov. 24, 2003: Defense
Lyon's plan for BetCris was to build a system that would absorb huge DoS attacks, and he had an idea how, technically, he might do that. But he had little idea how he would convince a tier-one hosting facility (essentially an ISP's ISP), to host his system—to voluntarily accept massive DDoS attacks to see if his little project could thwart them.

Through his Opte.org project, Lyon knew of an ISP called PureGig in Phoenix with a 10Gbps pipe, plenty of bandwidth to host his system without disturbing PureGig's other customers. Lyon called Matt Wilson at PureGig. He begged.

A heated internal debate took place at PureGig. The company was ready to say no, Wilson told Lyon. Lyon begged harder.

Lyon believes what tipped PureGig to support his cause was some bit of altruism. "They told me they don't like to back down from challenges," he recalls. But it probably had as much to do with generating business. For, if Lyon and PureGig did figure out how to stop DDoS attacks, they would have something that their competitors didn't.

"There was a great deal of skepticism here; it was not a popular idea," recalls Wilson. "My thinking was that normally the ISP's solution for DDoS attacks is to shut off the customer," he says. (In other words, null-route them like the ISP did in shutting down BetCris.) Wilson adds, "In our minds, that wasn't a good long-term solution. Revenue issues aside, we thought maybe we could learn how to fix the problem. But still, it was a huge risk."

With PureGig committed, Lyon worked for the next three days without sleep, designing, building, testing, rebuilding and retesting his system. "I used all the methodologies I knew, all the code I knew, plus some new ideas."

Lyon kept in constant contact with PureGig and with Lebumfacil in Costa Rica. Lebumfacil deferred to Lyon. "I was part of it, I stayed up all night with him on the line," Lebumfacil says. "I was never allowed to touch any of the boxes. I would make suggestions, and he'd take some of it and not take some of it.

"Barrett had his idea. There was so much uncertainty. Many times I thought, I hope he knows what he's doing. But Barrett had this calm confidence. You want to freak out, and he just works. He's so focused."

By Wednesday, Lyon had something. A patchwork of original code stitched together with commercial products, he described it as "a highly fortified data center with proxy and security software and some monitoring, and more bandwidth than the bad guys."

DDoS, Deconstructed
Denial-of-service attacks are an old and crass way to disrupt a network, and yet still are immensely effective. DoS attacks overload the pipes that connect computers to the Internet with massive amounts of legitimate but useless data. DoS attacks create epic traffic jams. The cars in this analogy would be requests for service that hackers send to the target website. Each time the target site gets a request, it check it out, decide it's not useful and then deny it. But because the hacker sends massive numbers of requests, the target must use nearly all of its time and resources just to deny these requests for service, effectively blocking access to anyone with a legitimate request.

To generate a sufficient volume of useless traffic, a hacker must create a network of computers big enough to overwhelm the target. They don't buy these computers, they commandeer them. They plant software scripts on systems distributed throughout the world (hence, distributed denial of service, or DDoS). These compromised computers are called zombies, or bots, because they generate attack traffic automatically and continuously, without the owners' knowledge.

He connected an unsecured computer to the Internet; it was recruited as a zombie within three minutes.

Hackers create zombies by scanning for exposed systems that they can manipulate remotely. Often these are home and office broadband users. They can also insert their attack code through phishing, spyware, viruses and social engineering. Universities have long been popular spots for creating zombies because of the number of easily accessible, unsecured public computers.

With a zombie network in place, the only issue left is scale. The more zombies on a network, and the more aggregate upstream bandwidth they have, the swifter and more severe havoc they can wreak. Several hundred computers could generate 100MB of traffic, enough to knock a small network offline. A 10,000-computer bot network could deliver a 1Gb attack, enough to knock anyone offline who hasn't installed some rudimentary anti-DDoS infrastructure.

Bots don't just attack though. They can be assigned any number of tasks. Lately, existing bot networks have been found scanning for more computers to turn into bots when they're not launching attacks of their own—akin to an army recruiting its soldiers in peacetime. One security consultant said he connected an unsecured computer to the Internet to see what would happen, and it was recruited within three minutes.

Some experts believe that right now different sets of hackers are engaged in an arms race to see who can build the biggest zombie network. Not for bragging rights, but for renting out the networks to anyone who wants to launch an attack, the raw capitalist idea being that the biggest network will generate the best rental business.

Tuesday, Nov. 25, 2003: Escalation
The extortionists' e-mail that arrived on this morning demonstrated that they were losing whatever patience they had: [all typos sic]:

I told you that if you try and fuck with us that your site will be down forever.... The excuse that you were in the hospital does not matter to me. So here are your choices: 1) You have until 4pm est today to send us our $40K. 2) You have until 4pm est Wednesday to send us $50K if you can not send the $40K today. 3) You do not pay and your site will be down for 4 days starting Thursday and it will cost you $75K to come back up Monday. 4) You do nothing and do not respond to this email within an hour and we will make sure you are down forever....

Richardson was panicked. He can't remember precisely when—the entire week has blurred in his memory—but by this time, he had reported the crime to the National Hi-Tech Crime Unit (NHTCU) in Scotland Yard. According to an NHTCU spokeswoman, the unit had already opened a similar investigation with a British gaming site called CanBet.

According to Richardson and Lyon, the NHTCU encouraged Richardson to wire two extortion payments of a few thousand dollars each to separate Western Union offices in Eastern Europe. The NHTCU wanted to nab anyone who showed up to take the cash. (NHTCU won't confirm this; the spokeswoman said the unit does not discuss investigative tactics.) Richardson agreed, but for a different reason: He wanted his site back up. "I knew another person [in the industry] who was successful getting back online by sending three or four small payments like this," Richardson says, "and those guys didn't even have a solution to the problem when they paid. I knew Barrett was getting closer and closer to a solution. So I sent the payments, thinking maybe I can get a good week out of this."

But no one took the bait. After about two weeks, Richardson pulled the money back.

Wednesday, Nov. 26, 2003: Counter
From Sacramento, Lyon instructed the PureGig engineers who would turn on his system 630 miles southeast, in Phoenix. Another 2,400 miles southeast from Phoenix, everyone at BetCris waited impatiently.

Lyon's system intercepted traffic headed for BetCris's servers in Costa Rica, diverted it to his creation in Phoenix, scrubbed off the attack traffic and delivered legitimate traffic back to Costa Rica. It was designed to bar DDoS traffic from touching BetCris. If the system failed, it couldn't defend BetCris, and it wouldn't be able to send legitimate traffic to Costa Rica. But BetCris itself wasn't getting attacked. The system did a lot of other stuff too: monitoring, capacity planning, logging and analysis.

It wasn't perfect. After it was installed, Lyon had to tweak routers on the network, install new versions of software and add capacity to his system. The extortionists kept changing attack vectors, and Lyon and his team kept tweaking. It was a constant battle, but Lyon was confident that the system would enable BetCris.com to stay online. Wilson at PureGig called Lyon's system "ingenious" not because it was unique—it was monitoring and filtering at a proxy location—but because Lyon's monitoring and filtering seemed to stop attacks better than any other effort he'd seen.

But when it was first turned on, the extortionists stuffed too much traffic down its throat. Wilson recalls the math: "We had 100MB links to the DNS servers. We went from handling under 2MB per link to, all of a sudden, 600MB." That's six times a full load. Imagine Fenway Park, which holds about 35,000 people. Now imagine 200,000 people trying to get inside Fenway Park at one time.

The DNS servers were overloaded, and Phoenix got tense.

Costa Rica had been tense for nearly a week (half a million dollars in lost revenue), but now BetCris was bordering on despair. Mickey Richardson lacked sleep, and he struggled to make decisions and lead. His IT staff was fracturing, feeling impotent as they watched the attacks and waited for Lyon. BetCris's small call center staff was getting abused around the clock by customers calling in to vent frustration and demand to know what the heck was going on. The simple task of creating a smart message about what was happening eluded Richardson. "You can't just have your call center staff tell people you were hacked," Richardson says, because it creates more questions than answers.

At the same time, his decision not to pay the extortionists was affecting other wagering sites that shared the same ISP and were experiencing network problems. "I'm getting calls from friendly competitors saying, 'Look, Mickey, we paid. Just pay. We're going down because of you.'"

He was running out of time and energy. Richardson remembers around this time having to update his staff—275 or so people who weren't entirely sure they'd have a job soon—and he couldn't even find words. He thought, "I wish they could read my mind because I'm too exhausted to explain it anymore. I don't have any answers."

In hindsight, Richardson says, he would have spent more time preparing for these human issues attached to the crisis—decision making under pressure, keeping the staff together—and less time worrying about technical defenses. Yes, create those technical defenses and make sure you have a crisis response plan. But also focus more on issues like exhaustion and emotional distress, and how they can be handled.

Richardson would learn they were not bluffing. You do not make a deal and then fuck around with us.

It was in this context that Richardson received an e-mail, at 11:12 a.m. It caused him to feel, for the first time, "blind fear."

I would like to thank you for not keeping your end of the deal and making this upcoming weekend an enjoyable one for me.

The extortionists demanded $75,000, but then seemed to disregard the money.

I do not care how long I have to destroy your business and I will. You will learn the hard way that you do not make a deal and then fuck around with us.... Let the games begin.

Richardson would soon learn they were not bluffing. They could destroy his business, and they were going to try. For BetCris to survive, Lyon's slapdash system in Phoenix, which was just starting to find its purchase, would have to stand up to the biggest DDoS attack any of them had ever seen.

The DNS servers that had overloaded in Phoenix were brought back online in a couple of hours, after Lyon and Wilson adapted some filtering scripts and increased the size of their network pipes.

Lyon then spent Thanksgiving and Friday eating leftover turkey his girlfriend delivered and tweaking his system to absorb bigger DDoS attacks. On Friday, he believed it could handle a 1Gb attack, and he felt good about that. He assured a frayed Richardson that he'd never see an attack that big. It would take tens of thousands of zombie computers.

Which is exactly what happened. It turns out the extortionists had more than 20,000 zombies. PureGig's data center suffered badly, which affected several of its ISP customers. PureGig decided to take Lyon's system offline to fix it.

"The attack went to 1.5Gb, with bursts up to 3Gb. It wasn't targeted at one thing. It was going to routers, DNS servers, mail servers, websites. It was like a battlefield, where there's an explosion over here, then over there, then it's quiet, then another explosion somewhere else," says Lyon. "They threw everything they had at us. I was just in shock."

Richardson recalls the attack: "So I have Barrett on the line, who I think is the second coming, and he says, 'Let me think about this. Give me some time.' And I say, 'OK, I don't want to pressure you. I have faith. But if you don't fix it, I'm out of business.'"

Why Online Extortion Works
It was never supposed to have gotten to this point; Richardson was supposed to have paid long ago. The extortionists expertly optimized the chances of it.

To ensure a quick, quiet transaction, the extortionists did what all extortionists (in the physical or online world) do: They exploited the problem of the commons. An ecological principle, the problem of the commons states that people will act in self-interest if it profits them in the short term, even if that act will hurt everyone, including themselves, in the long term. Every act, every threat, every negotiation tactic, every single move extortionists make is designed to make paying the protection fee not only appealing, but in fact, the smartest business decision you can make in the short term, even if you know in the long run that you haven't stopped the problem at all.

Thus, extortionists attack when it hurts the target the most; they ask for $10,000 to $100,000 (generally considered the sweet spot of extortionist profitability versus victim willingness to pay, depending on the size of the victim company).

In BetCris's case, the extortionists revealed they were Eastern European, which would make them hard to find, never mind prosecute. Online crime laws are weaker in Eastern Europe than in the United States and the desire to enforce them weaker still (and the FBI wouldn't get involved with offshore gaming sites being extorted from overseas).

The online version of extortion provides unique advantages (relative anonymity, low probability of prosecution, lots of easy targets, diminished chance of physical violence) that have made it a highly lucrative business alternative for bad guys.

BetCris was just another easy target. What the extortionists didn't count on was the unlikely confluence of Richardson's resolve, Lyon's ingenuity and an ISP that would provide them a place to fight back.

Friday, Dec. 12, 2003: Attrition
The extortionists must have screamed "Hooy na ny!" or some other Russian expletive after their blitzkrieg, when Lyon "got the chemistry down" and managed to absorb the massive amounts of attack traffic and get PureGig and BetCris back up and running. Lyon assumed the bad guys would come back with something bigger, as hard as that was to imagine, so he set out to scale up his system "for whatever was next, a 6Gb attack or something."

But for the next week, the attack stayed steady at around 1Gb. BetCris, Lyon and PureGig had entered a war of attrition. The extortionists would find a way to kick Lyon's system, Lyon and Lebumfacil would tweak it and get back up. Cat and mouse. "Attack, counterattack, back and forth," Lebumfacil says. "It was 24-by-7 monitoring for two weeks." Wilson and PureGig stopped noticing any of this because the attacks had been segregated from PureGig's other traffic.

And then, suddenly, the attacks stopped.

At 8:46 a.m. on Friday, Dec. 12, two weeks after the assault that nearly put him out of business and three weeks after he first read the words "Your site is under attack," Richardson received an e-mail:

Dear Mickey, I tried getting to your site today and I could not. I thought with all the money you spent you would not have these problems anymore. I guess you wasted your money instead of keeping your word. Good luck. P.S. I bet you feel real stupid that you did not keep your word. I figure by now you have lost 5 times what we asked and by the end of the year your decision will cost you more than 20 times what we asked.

Richardson knew this was an admission of defeat, even if it was disguised as braggadocio. His site was up. The extortionists couldn't get to it because they were blocked. He hadn't paid them a dime. They made no more threats. They couldn't because they couldn't back them up with action. The extortionists had lost.

And yet, the e-mail was not far off. Richardson figures it cost him a million dollars in lost revenue and IT investments to win this war. "It was worth it," he says. "I just didn't know it would take a couple years off my life."

"It was amazing we made that system work against that attack," Lyon says. "It was a wake-up call on how good the bad guys had gotten."

And Lyon knows the bad guys have gotten even better since. They've built zombie networks of 35,000 machines, capable of delivering a steady stream of 3Gb traffic. Peter Rendell, CEO of Top Layer Networks, which makes intrusion prevention and anti-DDoS hardware, says he expects botnets to pass 50,000 machines (and 4Gb to 5Gb) by the end of this year. It's an arms race, as defenses scale, then offenses scale, though Lyon is convinced the defenses have far outpaced what extortionists can throw at them.

It cost him a million dollars. It was worth it. I just didn't know it would take a couple years off my life.

But the bad guys have a response. Extortionists have encrypted DDoS attack scripts and have put them on peer-to-peer networks, making criminals who use them nearly impossible to track or contain. They're registering domains and then attacking those domains, only those domains are redirected to other targets. "The only way to stop that is to delete the domain," Lyon says, "and that's not something you can just do." Lyon stopped an attack but certainly didn't stop the problem.

Still, he wouldn't learn of all this until later, after he decided to start a business and, as he did with Don Best, track down the BetCris extortionists. At that moment, though, after the extortionists admitted defeat, he was ready to relax. He booked a vacation in San Jose, Costa Rica, for New Year's. Finally, he'd meet the people he saved and celebrate with them.

New Year's, 2004: Top-notch
Costa Rica is about the size of West Virginia, bookended by Nicaragua to the northwest and Panama to the southeast on the Central American isthmus. With coastlines on both the Pacific Ocean and Caribbean Sea, and mountainous terrain inland, Costa Rica sits along the Ring of Fire, so volcanoes and earthquakes are native. Political strife is not. The CIA calls Costa Rica a "Central American success story."

Lured by its stability, BetCris located there in 1993. Richardson joined as a "utility man" in 1996. Back then, the business wasn't online, it was a call center. BetCris's call center once employed more than 500 operators at peak hours, but the number dwindled as the business moved online. Today, maybe 30 operators will man a call center at peak hours, or during an extortion crisis.

As the Internet took off, so did San Jose as an offshore gaming mecca, for several reasons. The government encouraged the industry to expand its economy. (BetCris supports an industry group to lobby local politicians.) Also, the people are educated, with an excellent work ethic, Richardson says. Costa Rica has a 96 percent literacy rate. Most high-level employees at gaming companies are Costa Rican, including all of BetCris's accounting staff and 90 percent of its managers.

The other reason gaming companies swarmed here is, of course, because it's not the United States, where gambling laws are difficult to negotiate. Today, hundreds of offshore gaming companies, most of them online ventures, operate from San Jose. In the seven-story building where BetCris is headquartered, Richardson says, there are 10 such enterprises, two software companies and a telecom company—pretty much offering everything you need to get started in the online gambling business in one building. The competition is mostly friendly. Richardson says it's not unusual to bump into competitors at a restaurant and join them for dinner.

The valley that makes up the San Jose metropolitan area holds almost half the country's 4 million people. Richardson says the valley gets blistering hot, and downtown San Jose is "undesirable." But BetCris, and most of the gaming and tourism industries, are above all that, nestled in the higher elevations of the valley's surrounding mountains, where Richardson compares the weather—and the lifestyle—favorably to San Diego.

When Lyon arrived here, he felt a sense of pride for helping. He saw "this beautiful building with this top-notch data center," he recalls. "And I met all the people who work there, and I kept thinking, I protected all of this. Me and my keyboard helped all these people keep their jobs. It was so neat to see how good a thing it was that we did."

Richardson and Lyon bonded immediately. There was a party with professional-grade fireworks launched from Richardson's front lawn. They went to dinner, talked about life and the attacks. Lyon had developed antipathy to the extortionists; he wanted to nail them. He told Richardson and Lebumfacil he was going to start a business, a service whereby people could subscribe to his anti-DDoS attack infrastructure. Lyon recruited Lebumfacil to help him start DigiDefense. BetCris was his first customer. Richardson gave them office space to start.

That business talk, though, was in the background. Lyon relaxed, went deep-sea fishing and zip-lining through the rain forest.

Jan. 12, 2004, Phoenix: Revamp
On Jan. 12, Lyon met Lebumfacil in Phoenix. They drove to PureGig to rip out and replace the system that saved BetCris. Lyon knew it was already a relic. He had to build something that could support 10, 20, 50 customers or more without one customer's traffic interfering with another's, and without his customers affecting the rest of PureGig's customers too. He also planned to hone his traffic logging and analysis. His new system would not include commercial products.

The Super Bowl—a significant moment for betting sites that extortionists would exploit—was just weeks away. Some gaming sites had heard about Lyon's exploits with BetCris and wanted to sign up. Lyon had customers before he had a product.

Lyon and Lebumfacil "went on a rampage" of building and testing, and three days later, Lyon says most of the system was online. Over the coming months, as more customers signed on, Lyon flew to Phoenix more than 20 times to build up the infrastructure. A routine developed. Dozens of hardware boxes would arrive. Wilson from PureGig would sign for the equipment and store it until Lyon showed up to get it. "He'd live here for a couple of days installing everything," Wilson remembers. Once, Lyon slept in the data center.

But even as Lyon's business grew, the extortionists' business did too. That fall, after CanBet and a site called eHorse were attacked, BetCris was attacked, and then the extortionists hit other sites across the industry: BoDog Sportsbook, BetWWTS, WagerWeb, William Hill, BetFair and Blue Square. And those are only the cases that became public, usually through postings on online industry discussion boards or in gaming industry newsletters. Just how many sites either paid or never reported their cases will never be known, but it's certain many fall into this category.

Usually, the extortionists followed the attack methodology they used against BetCris. (In Blue Square's case, they demanded 7,000 euros, or else they would send out child pornography in the company's name.) Many ended up calling Lyon for help.

"It became a personal vendetta to track these guys down," Lyon says. "I wanted them stopped. So I asked some law enforcement people, 'Is this illegal, for me to talk to them?' And they'd tell me, 'No, but we can't help you or tell you what to say. However, if you did want to say something along these lines, that would be very interesting to us.'"

January and February: Contact
By this time, Lyon and Lebumfacil had recruited Dayton Turner, an engineer from eHorse, an extorted gaming site that operated out of the same building as BetCris. Like Lyon, Turner wanted to exact a certain justice, having lived through an extortion. He agreed to go undercover. Turner and Lyon spent the next several months chatting with the extortionists while they also monitored and logged the extortionists' activities. They shared what they learned with law enforcement, mainly the NHTCU but also the FBI.

January and February's gumshoeing produced an astonishing 36-page dossier—complete with chat transcripts, log file analysis and other data. Lyon and Turner gave it the hyperbolic title "DDoS Terrorism Report." The following comes from that report.

When they were logging the DDoS attack traffic at BetCris, the team traced some of it back to a chat server. Turner and Lyon called themselves "Hardcore," made sure they masked their real location and hopped onto the chat line. (While Turner did most of the chatting, Lyon was always on the line, "managing" the conversation and chatting with Turner, but it appeared to anyone else that Turner and Lyon were one in the same.)

The leader of the chat room clique went by many names, including eXe, Key, k9, NASA, x3m1st (pronounced "extremist"), x890 and others. For simplicity's sake, we'll always call him "eXe"—even if he was going by another name at the time—and we'll always call Turner and Lyon "Hardcore." (We've also cleaned up some typos for clarity, and skipped extraneous conversation for the sake of space.)

When Turner-Hardcore-logged on, he told eXe that he had been out of the game for a while but wanted to get back into DDoS attacks. eXe took the bait and began chatting, cautiously, with Hardcore. The first few chats didn't yield much. At one point a bodyguardlike heavy named "uhdfed" came online and bullied Hardcore, proclaiming, "We have 5,000 bots, and we don't need help." He attacked Hardcore's chat client. Lyon and Turner were forced to log off, but not before their log showed uhdfed was at the same time trying to attack another site: BetCris.com.

In ensuing chats, Hardcore gathered circumstantial connections to BetCris and the gaming extortion wave. EXe asked Hardcore, "how u know about our work? about bettings & sportsbooks"; at another point, Turner saw a reference to BoDog, a sports book that had been attacked. Another time, eXe inadvertently exposed his real ISP, in Russia.

Chat sessions continued for eight weeks. Often they were jarring and discombobulated. Cyrillic characters mixed with poor English. There was foul language and other noise. Hardcore watched eXe attack Microsoft and probe SCO.com. But over time, eXe began to chat more freely with Hardcore. In a couple of long chats, they talked shop in detail, Hardcore always deferring to eXe and praising his skill. This seemed to put eXe at ease.

eXe: i shall be happy to see u again. welcome
Hardcore: :) thanks hehe
eXe: i's eat now. =)...maybe i will sleep later=)

Soon enough, eXe pointed Hardcore to a webpage with attack scripts on it, and he gave Hardcore an ICQ chat client user ID that he hacked. (Perhaps as a gesture of friendship, he gave the account the password "hardcore.") The ICQ account allowed Hardcore to chat directly with eXe, but it also led to eXe's biggest mistake when eXe conducted a file transfer over this ICQ connection. Hardcore nabbed eXe's real IP address and traced it to a dedicated broadband line in Russia, a cable modem that he determined eXe paid for himself.

March 1, 2004: Ivan
On March 1, Hardcore and eXe chatted on ICQ. EXe had been waiting for some attack code that Hardcore had promised to write for him. It was the most productive conversation Lyon and Turner would conduct.

eXe: hi how are you?
Hardcore: hey man. p
retty good...it's pretty cold here right now, what's russia like? hehe
eXe: i'm good...Russia is like the Russian vodka=)...u give me code?...
Hardcore: I still have just a little bit to do to make it functional. I'll have it for you soon dont worry :)
eXe: ok...i'm relax =)
Hardcore: i noticed you have like 4 different types of bots in there...are you testing new bots?
eXe: yes...

The two talked about zombie networks, and Hardcore pressed eXe to tell him the size of the biggest zombie network he'd ever seen. EXe bragged about a 10,000-bot network, then added, "it's no many, i seen more."

eXe: how old are you?
Hardcore: 23...how about you? :)
eXe: i am 21 =) my name is Ivan. i'm from Russia. my nationality is Russian.
Hardcore: My name is Matt :) Ive always lived in canada
eXe: i happy to meet you
Hardcore: nice to meet you too ivan :)...do you work or go to school or just do this? ive made a lot of money doing this so far :) :)
eXe: school. i'm study. inginier-mechanic. etc=) i'm learn french. my English is very bad.

Sensing a new level of openness, Hardcore pressed Ivan, but Ivan's responses were vague and confusing, and his English, as promised, proved to be very bad indeed.

Hardcore: do you make money with ddos too? I have made about $150,000 so far this year hehehe =)
eXe: well done. no all paid =(
Hardcore: nobody paid? really?
eXe: somebody

At this point the good guys' giddiness seemed to betray them. Hardcore suddenly turned loquacious and leading. He told Ivan how he attacked sites and how much money he made doing it. (His description matched the tactics employed to attack BetCris and others.) Hardcore poured out 80 straight words about his nefarious activities, and Ivan responded only with an emoticon smiley face: =).

it's pretty cold here...what's russia like? Russia is like the Russian vodka=)

Hardcore continued chatting, suggesting to eXe that he could extort money easily "with the number of bots you have." He suggested Ivan attack people who "can't use the law against you," and added, "they always pay because they want their business back and they dont want to admit they have a weakness. stupid americans."

Ivan replied with another =).

Ivan had shut down. It could be that he was just tired; it was 1 a.m. in Russia. It's also possible that Ivan sensed what Hardcore was doing. Turner and Lyon kept trying. They sent three messages to Ivan's one, but Ivan's replies maxed out at three words.

Hardcore: i read in the news about some people who got letters about dos, i figured it would be you since you have so many bots
eXe: good idea. hehe
Hardcore: did anyone pay at all?
eXe: anyone
Hardcore: i remember when you guys were going after sports books a few months ago....they must have gone down hard.. haha...
eXe: =) i go to sleep
Hardcore: ok man
eXe: see you later
Hardcore: cya
eXe: bye friend

Two weeks later, on March 13, Ivan made an even bigger mistake. He logged on to IRC chat with his real domain name. Lyon and Turner had learned the domain was registered to an Ivan. But now they also had his last name, address and phone number. They promptly sent the information to the NHTCU.

July 2004: Endgame
The NHTCU must have been pleasantly shocked to have a pro bono case worker sending a constant stream of useful documents.

The NHTCU did not condone Lyon's actions, even as they welcomed the product of his actions. "Mr. Lyon operated as a U.S. citizen, and therefore, we cannot comment on his tactics," a spokeswoman says. Investigators are not available to the press. "However, his report and his interpretation of DDoS threat proved to be an informative document."

Lyon says of the relationship: "The only answer I got from them was, 'Wow. This is great. We'll make it worth your while some day. Keep it coming.' I was hoping at the end of this we'd continue to collaborate, but I've never really heard from them."

Several people involved with the BetCris case say it was Lyon and Turner's report that cracked open the NHTCU's case, and in fact it was the BetCris case itself and how Lyon, Richardson and Lebumfacil fought it off that influenced how the NHTCU responds to online extortion attempts. (The unit would not discuss the matter.) On the NHTCU, one person close to the BetCris case says, "I think maybe [the NHTCU] weren't capable before. Not to blame them—no one was capable. Otherwise, it wouldn't have been such a problem. From what I understand, though, it was all that work [Lyon and Turner] did that helped educate the NHTCU."

"They wouldn't have made any arrests if we weren't around," Lyon says.
The spokeswoman at the NHTCU bristled at the suggestion. "Mr. Lyon's work formed a part of the investigation and assisted law enforcement in better identifying the problem with DDoS. Mr. Lyon has developed what appears to be a good defense against DDoS; however, he has not stopped it, nor can he prosecute the offenders of such attacks."

Ultimately, using Lyon and Turner's work, along with the tracing of several extortion payments, the NHTCU managed to locate three suspects, including Ivan. Significantly, they were able to work diplomatic channels with Russian authorities, and that diplomacy ultimately led to Ivan's arrest (in an Internet café, Lyon says, but the NHTCU won't confirm this) and the arrest of two others. The NHTCU describes the cooperation of Russian officials as "excellent" and says that those Russian officials anticipate a trial in late 2005.

Soon after the first three arrests, five more were made in connection with online extortion. Of the eight suspects, just two were allegedly involved in the BetCris case. Five were ultimately charged with something. Lyon, too, notes that his investigation led him to six separate online groups launching DDoS attacks. The extortion rings are proving to be deeper and more organized than even those involved suspected. Other online investigations are ongoing, and DDoS attacks continue to rise, the NHTCU says.

"Any company with an online e-trading presence needs to be aware of this type of attack," says the NHTCU spokeswoman.

In less guarded terms, Wilson at PureGig reflects on the problem: "Once we got deep into this and talked to customers about it, we started to hear more and more stories. People saying to us, 'Oh yeah, that happened to us. We were down for a week.'

We started to hear more stories. Oh yeah, us, too. We were down for a week.

"We needed to lift that veil of secrecy. Unless you talk about it, it's only going to keep happening and get worse. We need to be able to talk about online extortion and not assume it's a onetime thing or it's only going to affect gambling sites. It's only going to continue."

Today: Pay
All the while, Lyon's business grew. A second data center opened in June 2004 in Vancouver, a third came online in July, near Miami.

In May 2004, Lyon changed his company's name to Prolexic, a name that derives from his childhood. In third grade, Lyon learned he had severe dyslexia. As a child, he remembers thinking of his dyslexia simply as something that meant he learned differently from other kids. In college, the philosophers he studied were men and women with learning disabilities. "Instead of a learning disability," Lyon says, "I've decided it's a learning ability." In other words, he's decided he's not dyslexic, he's prolexic.

Another data center went live in December, in London. Two more are planned, and he has two patents pending. In January, the company moved its headquarters to Hollywood, Fla. He has close to 100 customers, many gaming websites but also mainstream financial services companies. The average client, he says, spends "less than $50,000 a year," but some spend much more for custom security services. Recently, Lyon turned 27.

Lyon understands marketing too. He never misses a chance to boast about what he now calls his "solution." He made the BetCris story the online extortion anecdote that led many general news stories about the problem. Everyone involved in this saga continues to promote one another. Wilson at PureGig fawns over Lyon. (He's still a customer of the ISP.) Lyon praises PureGig on PureGig's homepage. Richardson invested in Lyon. Lyon praises Richardson and his ISP, which also happens to be one of Lyon's customers.

It all fits together so nicely for Lyon. His eerie ability to anticipate the extortionists' moves. The fact he could build something so quickly that could fight attacks that no one had seen before. The way he turned that around into a business that benefited all the major players involved in the extortion fight.

It's enough to make a reporter paranoid. What if Lyon knew the extortionists?
"Did Barrett rig the whole thing? That's a valid question," Lyon says. "It used to come up a lot. This is why we've been an open book with law enforcement. All I can say is, I'd have to make a zombie out of myself to pull that off. I was working pretty hard when all the extortion was going on." Plus, he points out, people were arrested.

That's right. Lyon is one of the good guys. Still, Lyon's heroics weren't possible without Mickey Richardson's resolve. It's easy to forget that as Lyon worked to save him, Richardson considered paying off the extortionists. Now Richardson has a better option. Pay Lyon $50,000 a year and he's protected. He doesn't have to worry about paying extortionist's protection fees.