Frontier Communications – a systems nightmare in Sandpoint

UPDATE: After all these months of terrible connectivity, it is unclear whether the problem has actually been solved. The only group that provided what I’d consider engineer-level suggestions was the email tech support group. That ended today when I received an email from “Linda B.” in that group:

Frontier will no longer be using Support Mail. Going forward please use the chat function that can be found at https://frontier.com/Contact-Us#/residential or please call in at your earliest convenience. For Customer Service please call 1-800-921-8101

So, the only group that actually knows what they’re doing is no longer allowed to actually provide support. Their chat function only works about 20% of the time, and their phone support is absolutely worthless.

Nicely done Frontier.

 

Three months ago, we relocated from Kula, Maui to Sandpoint Idaho. Kula is semi-rural, lots of ranch land all over the place. Our Internet provider was Time Warner (dba Oceanic). Even way up the mountain, we had 230Mbps down and 25Mbps on our cable internet service. The connection was pretty damn stable and plenty fast.

Then we arrived in Sandpoint. Here there are only two non-satellite providers, Northland and Frontier Communications. Northland provides cable-based service and Frontier provides DSL. both advertise speeds of “up to 24Mbps.” In Sandpoint, that is, currently, as good as it gets — welcome to the 3rd world of connectivity. Another provider, Ting, is claiming they’ll have gigabit fiber Internet available about mid-2017 in Sandpoint. If they can make that happen, I’ll be their new best friend.

Since Northland never returned phone calls or email inquiries, we defaulted to Frontier. The install tech was friendly and knowledgable, and got us hooked up. Except that, in spite of the significant effort he put in, we haven’t been able to get a stable connection for more than a few days at a time during the 3 months we’ve had Frontier. Ping times of over 4000ms (that’s ridiculously long). DSL channels would drop out. Speeds were down in the 100’s of bits per second (no, not Mbits, not Kbits, just plain bits). The tech checked our house wiring, the neighborhood wiring, even tested our physical wires all the way back to the Sandpoint data center. We’d go hours, or days, without any connectivity. A few possibilities were hypothesized:

  • Maybe our Apple AirPort Extreme access point is shoveling data at the Frontier modem too quickly, and the modem is freaking out. This was never proved nor disproved, but if a consumer-grade wifi router can overload equipment supplied by an internet provider, that provider has some things to answer for.
  • Maybe our surge protector is supplying dirty power to the Frontier modem that is making it freak out. This seemed even more suspect than idea #1, but what the hell, we took the modem off the surge protector and things worked better for a while. Of course, later data suggests that the improved behavior was coincidental.
  • Maybe our home network is asking too much of the Frontier modem. Two laptops, iPads and iPhones, a TV and a Mac Mini to supply media to in-house devices. Sorry, I’m just not buying it.

Of course, beyond the install tech, there was Frontier’s “customer service”. Fully script-driven, unable to offer much more than “did you reboot the modem?”, reluctant to escalate to a higher-tier representative (it’s unclear if there actually are any higher-tier personnel — if there are, we never met ’em).

Frontier uses 2-channel bonded ADSL for home internet service, and they supply an ActionTec ADSL2+ modem. If you look up replacement modems for that type of connection, you’ll find there are almost none. DSL is pretty antiquated stuff. So, no replacing the possibly-crappy modem supplied by the provider.

After three months of this bullshit, I started a Twitter storm (Donald would be proud) harping on Frontier’s inability to provide basic Internet service with anything remotely resembling reliability. Frontier’s Twitter team responded in ways that make me think that (a) nobody at Frontier actually understands how to use social media, and (b) nobody at Frontier’s Twitter team knows how to look at their internal trouble-tracking database, since at least 5 different people responded via Twitter and none them seemed to know about each other’s responses, the service calls I’d logged, the problems that had been reported, etc.

The result, at least as of today, is something another Frontier tech discovered (Note: Frontier’s field techs have been consistently first-rate): Frontier’s office uses boards made by AdTran for their DSLAMs (DSL Access Multiplexors). These are essentially large slabs of electronics mounted in racks that aggregate multiple consumer connections. Think 64 modems on a single circuit board, each modem connecting to a home or business, handing off all those connections to a single pipe. As it turns out, something nobody at Frontier knew was that AdTran recommends rebooting their card every 30 days (maybe they run Windows?), else their buffer tends to fill up, and the card starts rejecting connections from those consumers.

The symptoms of this result are exactly the symptoms I’d been reporting for three months.

Now, rebooting one of those cards effectively disconnects every account wired to it for anywhere from a few seconds to several minutes. And the Frontier data center probably has hundreds of those cards, each of which apparently is supposed to be rebooted independently. And that’s the best solution AdTran appears to offer.

The thing I’d like to know is who, if anyone, at Frontier performed due diligence when procuring these crap AdTran boards? Frontier is now far too invested in AdTran to retool with Cisco or another hardware vendor; they’re effectively locked in. And AdTran appears to have no viable solution short of yanking and rebooting hundreds of their boards every month, a service disruption that would probably [further] cripple Frontier. So, a followup question for Frontier is: what the hell are you planning to do about this mess?

In about 30 days, I’m going to be asking that question of Frontier. Loudly.

Update: five hours or so after the “fix”, we’re back to 2000+ms pings, and our speed is less than 4Mbps down. Frontier: I want a fucking refund.

What if the web died but nobody noticed?

From Wired:

The Web is Dead. Long Live the Internet
— Chris Anderson and Michael Wolff, 17 August 2010

And I didn’t even know it was sick. And isn’t the Internet some thing Al Gore invented? Hmmm. In truth, it regurgitates something I’ve had to explain over and over again to people about the difference between the Internet and the Web. Sort of like the difference between the USGS 15 minute series as compared to a nice friendly tourist map.

what is the web, anyway?

Tim Berners-Lee, ‘early 90’s: “right, if you guys won’t listen, then I’ll do it myself.” I paraphrase, but that’s what it boils down to: Tim had An Idea. Nobody would listen to him, so he built it himself. At which time it seemed sort of, well, obvious. What one might refer to as “a V-8 moment”. Think chocolate and peanut butter. Or V-8, I suppose.

One original definition of the World Wide Web: “a system of interlinked hypertext documents accessed via the Internet. ” Somewhat dry, but it gets to the point: the Web is documents. Some may be static, some created on the fly, some from databases or mashups or other you-name-the-API but, as presented to the user, they are forms of a document. Of course, these days a URL may respond with data that isn’t at all human-readable – but that’s largely a presentation issue, isn’t it? Which brings us back to the browser.

Or, one can define something in terms of what it is not:

  • not networking technology
  • not hardware
  • not IP addresses

Initially, “the Web” was the part of the Internet accessed via a newfangled animal called a “web browser”, a specialized piece of software intended to access only servers configured to understand its requests. The “www.” prefix (mainly vestigial now)  to a URL identified a specific host that ran a web server, since most machines on the Internet at the time had no idea what to do with web traffic and mod rewrite didn’t exist yet.. And domain names were initially free (who knew they’d be worth something).

Ironically, one of the lasting results of the web is the W3C (http://www.w3.org/), the governing body which, 20 years later, is led by none other than Tim Berners-Lee. So we have him to thank for the web and he has us to thank for a job. You have to appreciate his staying power.

Perhaps my favorite definition of the web is from Douglas Adams: “The World Wide Web is the only thing I know of whose shortened form takes three times longer to say than what it’s short for” .

when does the web stop being the web?

You can type an IP address directly into a browser’s address field. Try using ‘72.21.207.65’ instead of ‘amazon.com’. Not a “URL” per se (I’m taking some liberties here with browser destination auto-complete, but bear with me for fun), a good old-fashioned IP address. Still the web? It’s still a document created and delivered to your browser. Is it the web because you’re using a browser? Or that the destination responds to web requests?

Does it really, in the end, matter?

when does an app stop being an app?

Traditionally, there has been an understated dichotomy between “application” developers and “web” developers. Mashups have really confused the issues. So much for the simple browser and straightforward web sites. Now you have data. And applications that can send and receive web requests and data. And different sites collaborate with their data.

Early on in the iPhone world, Steve Jobs told the developer community that they didn’t need an SDK, that they didn’t need to develop actual iPhone-native apps because they could develop “web applications”: web pages custom-engineered to feed data to and from the iPhone’s multi-touch interface and display. The development community responded by giving Mr. Jobs the finger.

making sense of all this (and I use that term loosely)

So we all get wrapped up in determining how many of the elephants in the room can fit on the head of a pin. But the really important thing is that an elephant with a pin in its ass is a dangerous thing if it’s about to fart, and a blind man will never see it coming.