Frontier Communications – a systems nightmare in Sandpoint

UPDATE: After all these months of terrible connectivity, it is unclear whether the problem has actually been solved. The only group that provided what I’d consider engineer-level suggestions was the email tech support group. That ended today when I received an email from “Linda B.” in that group:

Frontier will no longer be using Support Mail. Going forward please use the chat function that can be found at https://frontier.com/Contact-Us#/residential or please call in at your earliest convenience. For Customer Service please call 1-800-921-8101

So, the only group that actually knows what they’re doing is no longer allowed to actually provide support. Their chat function only works about 20% of the time, and their phone support is absolutely worthless.

Nicely done Frontier.

 

Three months ago, we relocated from Kula, Maui to Sandpoint Idaho. Kula is semi-rural, lots of ranch land all over the place. Our Internet provider was Time Warner (dba Oceanic). Even way up the mountain, we had 230Mbps down and 25Mbps on our cable internet service. The connection was pretty damn stable and plenty fast.

Then we arrived in Sandpoint. Here there are only two non-satellite providers, Northland and Frontier Communications. Northland provides cable-based service and Frontier provides DSL. both advertise speeds of “up to 24Mbps.” In Sandpoint, that is, currently, as good as it gets — welcome to the 3rd world of connectivity. Another provider, Ting, is claiming they’ll have gigabit fiber Internet available about mid-2017 in Sandpoint. If they can make that happen, I’ll be their new best friend.

Since Northland never returned phone calls or email inquiries, we defaulted to Frontier. The install tech was friendly and knowledgable, and got us hooked up. Except that, in spite of the significant effort he put in, we haven’t been able to get a stable connection for more than a few days at a time during the 3 months we’ve had Frontier. Ping times of over 4000ms (that’s ridiculously long). DSL channels would drop out. Speeds were down in the 100’s of bits per second (no, not Mbits, not Kbits, just plain bits). The tech checked our house wiring, the neighborhood wiring, even tested our physical wires all the way back to the Sandpoint data center. We’d go hours, or days, without any connectivity. A few possibilities were hypothesized:

  • Maybe our Apple AirPort Extreme access point is shoveling data at the Frontier modem too quickly, and the modem is freaking out. This was never proved nor disproved, but if a consumer-grade wifi router can overload equipment supplied by an internet provider, that provider has some things to answer for.
  • Maybe our surge protector is supplying dirty power to the Frontier modem that is making it freak out. This seemed even more suspect than idea #1, but what the hell, we took the modem off the surge protector and things worked better for a while. Of course, later data suggests that the improved behavior was coincidental.
  • Maybe our home network is asking too much of the Frontier modem. Two laptops, iPads and iPhones, a TV and a Mac Mini to supply media to in-house devices. Sorry, I’m just not buying it.

Of course, beyond the install tech, there was Frontier’s “customer service”. Fully script-driven, unable to offer much more than “did you reboot the modem?”, reluctant to escalate to a higher-tier representative (it’s unclear if there actually are any higher-tier personnel — if there are, we never met ’em).

Frontier uses 2-channel bonded ADSL for home internet service, and they supply an ActionTec ADSL2+ modem. If you look up replacement modems for that type of connection, you’ll find there are almost none. DSL is pretty antiquated stuff. So, no replacing the possibly-crappy modem supplied by the provider.

After three months of this bullshit, I started a Twitter storm (Donald would be proud) harping on Frontier’s inability to provide basic Internet service with anything remotely resembling reliability. Frontier’s Twitter team responded in ways that make me think that (a) nobody at Frontier actually understands how to use social media, and (b) nobody at Frontier’s Twitter team knows how to look at their internal trouble-tracking database, since at least 5 different people responded via Twitter and none them seemed to know about each other’s responses, the service calls I’d logged, the problems that had been reported, etc.

The result, at least as of today, is something another Frontier tech discovered (Note: Frontier’s field techs have been consistently first-rate): Frontier’s office uses boards made by AdTran for their DSLAMs (DSL Access Multiplexors). These are essentially large slabs of electronics mounted in racks that aggregate multiple consumer connections. Think 64 modems on a single circuit board, each modem connecting to a home or business, handing off all those connections to a single pipe. As it turns out, something nobody at Frontier knew was that AdTran recommends rebooting their card every 30 days (maybe they run Windows?), else their buffer tends to fill up, and the card starts rejecting connections from those consumers.

The symptoms of this result are exactly the symptoms I’d been reporting for three months.

Now, rebooting one of those cards effectively disconnects every account wired to it for anywhere from a few seconds to several minutes. And the Frontier data center probably has hundreds of those cards, each of which apparently is supposed to be rebooted independently. And that’s the best solution AdTran appears to offer.

The thing I’d like to know is who, if anyone, at Frontier performed due diligence when procuring these crap AdTran boards? Frontier is now far too invested in AdTran to retool with Cisco or another hardware vendor; they’re effectively locked in. And AdTran appears to have no viable solution short of yanking and rebooting hundreds of their boards every month, a service disruption that would probably [further] cripple Frontier. So, a followup question for Frontier is: what the hell are you planning to do about this mess?

In about 30 days, I’m going to be asking that question of Frontier. Loudly.

Update: five hours or so after the “fix”, we’re back to 2000+ms pings, and our speed is less than 4Mbps down. Frontier: I want a fucking refund.

Scalping at the Festival

This summer, I and my wife Sarah relocated from Kula in upcountry Maui to Sandpoint. We’ve been here quite a bit, as we have family just across the Washington border, love the climate (I’m from Colorado, Sarah’s from Montana), the people, the vibe. We love the Festival at Sandpoint.

Going to see Emmylou Harris last weekend, we probably paid a bit more attention to the goings-on around us rather than simply reveling in a “We’re at a concert!” fog; New Resident Mode vs. Tourist Mode. Some of our group got up before dawn and made the pilgrimage to War Memorial Field to stand in line and get “morning numbers”, tickets that allow the holder to enter the event at the head of the line, in the order the morning numbers were issued. Most people do this so that they can reserve space for their extended group on the grass, as close to the stage as possible. The Festival rules state that each holder of a morning number can spread out an 8’ x 8’ blanket, sufficient for four people to sit and enjoy the show. The rules also state that only one morning number per person will be issued — there was, at least up through last week, no requirement that a person present a valid event ticket in order to get a morning number.

The rules also state that there is no camping allowed in the park or on the sidewalk, but our group found a veritable tent village, mostly kids, camped out at the head of the line to get the best morning numbers. And many of those people didn’t appear to have a ticket to the event — they were acquiring morning numbers to scalp them later, for $20 and up, depending on how low a number they had.

We discovered that this scalping of morning numbers, in addition to injecting a pretty seedy ambience while waiting in line, has been a bone of contention for event-goers for years. The sentiment seemed to be that the Festival organizers had avoided the issues of the tents and the scalpers, some of whom appeared to be mothers with their teenagers in tow, demonstrating how to scalp morning numbers. On the other hand, we heard often that nobody ever cut into the line of people waiting for the show because the organizers kept a good watch on such things.

So, people are pissed off, but is it really a bad thing to be doing? It’s not technically illegal, although that may be a fine point.

The answer, as it turns out, is probably “yes, it is bad”. Let’s do some math:

Using Google Maps, I can estimate that the prime real estate at the venue, the grassy center section in front of the stage, is about 100’ x 150’, or about 15,000 square feet. Since a blanket can be 8’ x 8’ or 64 square feet, that means that the most desirable section of the venue, the place worth getting up at 4:30 am for, can support roughly 200-225 blankets. Taking into account that people rarely set out blankets in a nice, orderly way, that means that if you get a morning number higher than 150, you’re probably out of luck for the most part.

This begs the question: should the Festival be issuing 300 morning numbers or only 150?

Some more math: there were probably 25-40 people camped out in line to get the best morning numbers. Let’s assume some of them actually want to get good seats at the event for themselves. So, say 30 of those people are there mainly to scalp the numbers they get (that’s consistent with what our group saw). If only numbers 1 through 150 are worth getting, and 30 are shaved off the best numbers by scalpers:

About 20% of morning numbers are effectively stolen. Which I’d argue is bad.

After pondering this for a while last weekend, I wrote to the Festival about the issue on Sunday, and proposed that if they enforced a requirement that a morning number would only be issued to a person holding a ticket for that event, a lot of the scalping would go away. I was pleasantly surprised when, two days later, I and all ticket owners for the Festival received an official email telling us that, starting the second week of the Festival, just such a rule would be added. Three possibilities come to mind:

  1. The Festival had been planning this new rule for a long time. But it’s unlikely that, after such planning, they’d implement the new process during the second week rather than from the start.
  2. The uproar over the scalping had been so great this year from just the first few shows that someone at the Festival office decided enough was enough. Also sort of unlikely that so much energy would build that quickly.
  3. My email got through to someone to cared, and my proposing a relatively simple solution was what just what they needed to take action. This, of course, being completely unbiased, is my favorite explanation.

In any case, huge kudos go to the Festival staff for getting the new rule in place.

But I don’t think we’re done yet — some more math:

If you scan a valid ticket, electronic or hardcopy, it will scan as valid. If you scan that ticket 20 times, it will scan as valid, 20 times. If you make 20 copies of a ticket, all 20 copies will scan as valid. That means a group of scalpers can purchase a single ticket and scam a bunchnumber of morning numbers.

The straightforward solution is to associate each morning number with a ticket (it’s just a database record), so that only one morning number can be issued for any single ticket.

Better, but:

Four people, each with a valid ticket, can go early and get morning numbers. A morning number’s worth of space is good for 4 people. This group can then scalp 3 of their morning numbers for, say, $20 each, and get a $15/person profit. Still not quite there…

Better: Make the morning number valid only when presented at the gate by the person holding the associated ticket. Better still, do away with the physical morning numbers completely: For example, if you fly, and are enrolled in TSA’s Pre program, you don’t get issued a separate Pre ticket to get you into the priority line. Your ticket gets the TSA Pre symbol added to it, whether it’s a mobile ticket or a printed ticket. Similarly, a person’s Festival ticket would magically show “morning number 23” or such when rendered on their phone or printed. Or the person scanning tickets at the gate would see it on their terminal.

But some people are really determined to beat the system:

That same group of 4 could get each morning numbers, then negotiate with 3 parties in the entry line to use a blanket’s worth of the space they’ll reserve. This sounds like a lot of work, doesn’t it?

Like any padlock, the intent isn’t to totally prevent a thief. The intent is to make the theft so difficult that the thief will simply give up and look for an easier target.