Frontier Communications – a systems nightmare in Sandpoint

UPDATE: After all these months of terrible connectivity, it is unclear whether the problem has actually been solved. The only group that provided what I’d consider engineer-level suggestions was the email tech support group. That ended today when I received an email from “Linda B.” in that group:

Frontier will no longer be using Support Mail. Going forward please use the chat function that can be found at https://frontier.com/Contact-Us#/residential or please call in at your earliest convenience. For Customer Service please call 1-800-921-8101

So, the only group that actually knows what they’re doing is no longer allowed to actually provide support. Their chat function only works about 20% of the time, and their phone support is absolutely worthless.

Nicely done Frontier.

 

Three months ago, we relocated from Kula, Maui to Sandpoint Idaho. Kula is semi-rural, lots of ranch land all over the place. Our Internet provider was Time Warner (dba Oceanic). Even way up the mountain, we had 230Mbps down and 25Mbps on our cable internet service. The connection was pretty damn stable and plenty fast.

Then we arrived in Sandpoint. Here there are only two non-satellite providers, Northland and Frontier Communications. Northland provides cable-based service and Frontier provides DSL. both advertise speeds of “up to 24Mbps.” In Sandpoint, that is, currently, as good as it gets — welcome to the 3rd world of connectivity. Another provider, Ting, is claiming they’ll have gigabit fiber Internet available about mid-2017 in Sandpoint. If they can make that happen, I’ll be their new best friend.

Since Northland never returned phone calls or email inquiries, we defaulted to Frontier. The install tech was friendly and knowledgable, and got us hooked up. Except that, in spite of the significant effort he put in, we haven’t been able to get a stable connection for more than a few days at a time during the 3 months we’ve had Frontier. Ping times of over 4000ms (that’s ridiculously long). DSL channels would drop out. Speeds were down in the 100’s of bits per second (no, not Mbits, not Kbits, just plain bits). The tech checked our house wiring, the neighborhood wiring, even tested our physical wires all the way back to the Sandpoint data center. We’d go hours, or days, without any connectivity. A few possibilities were hypothesized:

  • Maybe our Apple AirPort Extreme access point is shoveling data at the Frontier modem too quickly, and the modem is freaking out. This was never proved nor disproved, but if a consumer-grade wifi router can overload equipment supplied by an internet provider, that provider has some things to answer for.
  • Maybe our surge protector is supplying dirty power to the Frontier modem that is making it freak out. This seemed even more suspect than idea #1, but what the hell, we took the modem off the surge protector and things worked better for a while. Of course, later data suggests that the improved behavior was coincidental.
  • Maybe our home network is asking too much of the Frontier modem. Two laptops, iPads and iPhones, a TV and a Mac Mini to supply media to in-house devices. Sorry, I’m just not buying it.

Of course, beyond the install tech, there was Frontier’s “customer service”. Fully script-driven, unable to offer much more than “did you reboot the modem?”, reluctant to escalate to a higher-tier representative (it’s unclear if there actually are any higher-tier personnel — if there are, we never met ’em).

Frontier uses 2-channel bonded ADSL for home internet service, and they supply an ActionTec ADSL2+ modem. If you look up replacement modems for that type of connection, you’ll find there are almost none. DSL is pretty antiquated stuff. So, no replacing the possibly-crappy modem supplied by the provider.

After three months of this bullshit, I started a Twitter storm (Donald would be proud) harping on Frontier’s inability to provide basic Internet service with anything remotely resembling reliability. Frontier’s Twitter team responded in ways that make me think that (a) nobody at Frontier actually understands how to use social media, and (b) nobody at Frontier’s Twitter team knows how to look at their internal trouble-tracking database, since at least 5 different people responded via Twitter and none them seemed to know about each other’s responses, the service calls I’d logged, the problems that had been reported, etc.

The result, at least as of today, is something another Frontier tech discovered (Note: Frontier’s field techs have been consistently first-rate): Frontier’s office uses boards made by AdTran for their DSLAMs (DSL Access Multiplexors). These are essentially large slabs of electronics mounted in racks that aggregate multiple consumer connections. Think 64 modems on a single circuit board, each modem connecting to a home or business, handing off all those connections to a single pipe. As it turns out, something nobody at Frontier knew was that AdTran recommends rebooting their card every 30 days (maybe they run Windows?), else their buffer tends to fill up, and the card starts rejecting connections from those consumers.

The symptoms of this result are exactly the symptoms I’d been reporting for three months.

Now, rebooting one of those cards effectively disconnects every account wired to it for anywhere from a few seconds to several minutes. And the Frontier data center probably has hundreds of those cards, each of which apparently is supposed to be rebooted independently. And that’s the best solution AdTran appears to offer.

The thing I’d like to know is who, if anyone, at Frontier performed due diligence when procuring these crap AdTran boards? Frontier is now far too invested in AdTran to retool with Cisco or another hardware vendor; they’re effectively locked in. And AdTran appears to have no viable solution short of yanking and rebooting hundreds of their boards every month, a service disruption that would probably [further] cripple Frontier. So, a followup question for Frontier is: what the hell are you planning to do about this mess?

In about 30 days, I’m going to be asking that question of Frontier. Loudly.

Update: five hours or so after the “fix”, we’re back to 2000+ms pings, and our speed is less than 4Mbps down. Frontier: I want a fucking refund.

Bad Form(s): A Brief Rant

A lot of interaction on the web is achieved through the use of forms, panels of text and fields used to input, classify and store data. Forms have been around pretty much since the beginning, and at first were something resembling sorcery: getting a form to look good, fields aligned, was tricky. As HTML and CSS became more sophisticated, along with improved Javascript capabilities, forms have become became both easier to build and more complex to debug. And the temptation to add unnecessary glitz is frequently the culprit behind the worst forms out on the web.

Input Fields are the workhorses of forms. They can restrict the number of characters input and the type of data entered. They can choose to display or mask the input as in the case of passwords. While there are far more options than these, this small set of variables is more than enough for poorly-implemented forms to become nearly unusable, leading to frustration and outright silliness.

Some of this silliness has declined over the decades because developers have gravitated toward some standards of behavior. Sadly, one still to this day encounters forms with some of the same lameness that we saw in the 90’s or 00’s — and we can’t blame the COGs (Crusty Old Guys) for all of it. Too often, new developers fall into the same traps of bad form design, disregard of user experience, cool-philia, laziness or the “I am IT and I decide” mindset that has consistently been focused on the mechanics of a site even when in opposition to the success of people using a site. For some reason, I encounter these issues most frequently on banking and government sites, where some noob has clearly discovered the modern equivalent of the blink tag and can’t resist inflicting their new-found toy on users.

Standard Information

Data like phone numbers, ZIP codes, Social Security numbers are pretty standardized in the US. Phone numbers, for example, are expected to have 10 digits, but they may be formatted in a number of ways:

  • (800) 555-1212 (traditional)
  • 800-555-1212 (business traditional)
  • 800.555.1212 (standard geek)
  • 8005551212 (machine version)

While a company may want to format a phone number in a particular way, all they actually need is those 10 digits — they shouldn’t really be concerned about how a person might format their number since it’s so easy to parse out the digits and store just that; storing digits only allows more efficient sorting anyway. A number can then be reformatted as desired on subsequent read and display. Still, I constantly find forms that throw errors if a user includes hyphens, periods or, gasp, spaces in a phone number — or doesn’t include one of those. Some forms simply limit the phone number field to 10 characters and leaving the user to guess what’s going on, adding no benefit but inflicting a developer’s idea of proper form entry.

ZIP Codes

The US Post Office instituted ZIP (Zone Improvement Plan) codes in 1963, and extended those codes with the “5+4” format in 1983. That’s all before forms existed, but the 5+4 format isn’t widely used even now. But it still manages to cause some forms (or the developers behind them) indigestion.

Synchronicity is Key

One particularly idiotic situation occurred when I was logging for the first time into an account that had been set up through an separate process. To verify my identity, I was asked to enter my ZIP code, at which time my verification was rejected. Checking back with the administrator, I found that they had my ZIP code had been recorded as a 5+4 value. The web form, however, only allowed 5 digits, making a match impossible unless the developers actually used some common sense. I’ve seen this occur as well when a form on a web page allows or requires something different than what is allowed on that same form on a mobile app.

The Future is Here (Occasionally)

“The future is already here — it’s just not very evenly distributed.” – Michael Gibson

I sometimes run across forms that ask for the ZIP code first, then use that information to automatically populate the fields for city and state. Smart, and rare, a form that actually helps you fill in the blanks. With technology that’s been in use since at least 1998 and information that’s been available since 1963…

Divertimento: Non-Regional Postal Codes

From Wikipedia:

In Canada the amount of mail sent to Santa Claus increased every Christmas, up to the point that Canada Post decided to start an official Santa Claus letter-response program in 1983. Approximately one million letters come in to Santa Claus each Christmas, including from outside of Canada, and all of them are answered in the same languages in which they are written.[13] Canada Post introduced a special address for mail to Santa Claus, complete with its own postal code:

SANTA CLAUS
NORTH POLE  H0H 0H0

Gotta love Canadians.

Form input and the Database

Nearly all data input through forms ends up in a database. To store data, most databases receive those data in the form of an SQL (Structured Query Language) query. For example:

insert “xyz1234pplr” into table passwords where user = “John Yaya”

[Note to geeks: these examples are simplified for a lay audience; this is not a coding class, so please back off.]

That query is evaluated by a parser and the value is stored. That parser evaluation, however, is were we can get into trouble.

Code Injection

Code injection is one of the oldest, most common and most insidious forms of hacking. Parsers look for meaning as part of their evaluation process, and some characters can cause some really interesting things to happen. Take our example from above, and let’s add something destructive:

insert “xyz1234pplr|‘/bin/sh unlink /'” into table passwords where user = “John Yaya”

The vertical bar character “|” is used in some systems (like Unix variants) as what’s called a “pipe“, a way of passing the output from one program to be input into another program. The inserted phrase:

|‘/bin/sh unlink /’

Means “pass the command to the main system to unlink the filesystem root.”

In times before developers and operating systems were careful about input (actually this is still a problem), this query might have been executed, resulting in the effective deletion of the entire filesystem on the database server. These days, rather than directly destroy a filesystem, someone might try to use injection to give themselves administrative privileges, and from there they can get into all sorts of trouble.

Input Character Restriction

Sadly, some IT groups continue the ancient (by IT standards) and lame (by any standard) practice of trying to protect against injection by restricting what characters can be entered into a form. So things like “|” or “>” might be disallowed. Problem is, good passwords these days make use of all sorts of punctuation.

Institutional Arrogance : Personal Capital IT

Recently, I opened an trial account with Personal Capital (PC), a Mint-like web site where you enter all you financial accounts and it advises you about your investments (I’m intentionally not including a link out here because I can’t with good conscience make it easy for anyone to go there). But I was unable to add some of my accounts because the passwords appeared to be getting rejected. After some back and forth, I was informed that the IT staff at PC reads your passwords when you add a financial account and strips out any characters they think are unsafe — even if those characters are perfectly fine with the financial institution. To be clear: 

  1. You create an account with a bank, establish a login and password.
  2. You create an account with PC and add the bank and login credentials to your PC account.
  3. PC reads your password when you supply it and then strips out any characters they feel are objectionable according to their internal policy. You are never informed about this action, and the now-corrupt password is stored rather than the real password that you and your bank agreed on.
  4. PC uses the corrupt password to attempt to log into your bank.
  5. The bank rejects the corrupt password.
  6. On login rejection, PC displays a message to you indicating that the bank doesn’t like your password.
  7. If this happens enough, the bank locks your account. It is then up to you to resolve the mystery with your bank, because at this point neither you nor your bank know why someone has been trying to log into your account with a bad password.
  8. At no time does PC let you know they caused the entire problem.

PC’s support personnel were very defensive about their actions, maintaining that they’re following OWASP guidelines and are justified in their policy. They informed me that the solution was for me to change all my passwords to my banks to conform to their (PCs) requirements.

One of the most myopic and arrogant IT abuses I’ve seen in a while. 

Input Escaping

A better method of defending against injection attack is by input escaping. Basically, before you parse any input from a field, you “escape”, or defang, any characters you feel are dangerous. Escaping can take a lot of forms, but in essence it means flagging a character in a way that tells the system “this is just a character, don’t use it as something else.” For example, the vertical bar character “|” in some environments is used as a “pipe”, a way of passing data from one program into another program. If that character is treated as a pipe when entered into a database, it might be interpreted as an operation, and text that seemed innocuous might get executed rather than simple entered as text. Our nefarious query from above becomes:

insert “xyz1234pplr\|‘/bin/sh unlink /'” into table passwords where user = “John Yaya”

The added backslash “\” tells the parser that the vertical bar is just a vertical bar, and the pipe is neutralized. Smart systems store the escaped string in the database so that later retrievals are safe as well.

It Really Isn’t Rocket Science

Finding a well-built form, while not rare, does continue to be uncommon in my experience. As an engineer, I’ve embraced the rule of parsimony (or Occam’s Razor if you prefer) from early in my career, which effectively means “don’t make it fancier than it needs to be.”

Modern techniques can provide all sorts of functionality to the user experience. The trick is keeping a firm eye on the goal: that technology should be used to enhance the experience, not provide a place to show off at the user’s expense.

Apple Pay and MCX : It's Not (All) About Interchange Fees

Credit cards get stolen, in one form or another, every day, as do debit cards. For me, one of the fundamental differences between the two, as I’ve unfortunately experienced personally, is this: bogus credit card charges can be disputed, and in the end, you won’t have to pay them if the issue is resolved. Fraudulent debit card charges can also be resolved – but until they are, your bank account is probably empty. It’s one reason we use credit cards far more than debit cards.

Apple Pay makes that even more secure, dramatically reducing the possibility of fraud. And it’s stupidly easy to use. And the way it’s implemented, with near-field connections and secure tokens, a merchant never sees your name, never sees your credit card, has no idea who you are or what your spending habits are: it’s like using cash at Radio Shack and declining to fill out any personal information.

Think about that.

There’s a lot of buzz generated about the feud developing between Apple Pay and the Merchant Customer Exchange (CTX), a consortium strongly supported by Walmart. Most of the press seems to think this is all about avoiding those fees that MasterCard, Visa and American Express charge for processing purchases through their networks. MCX, with their product, CurrentC, plans to avoid those charges by having customers tie their CurrentC purchases directly to their checking accounts. CurrentC also avoids the necessity of having an iPhone 6 in your pocket by flogging that old mostly-dead horse, QR codes. MCX-loyal merchants like CVS and Rite-Aid have gone so far as disabling their touch-to-pay POS terminals (also disabling Google Wallet), preferring to wait for their own MCX solution, to be launched sometime next year. That’s right, these merchants are now preventing you from using certain payment methods in their stores with the idea that this will somehow incline you to be loyal to their own “real soon now” homegrown methods.

While there are a lot of details left to be made public, as someone who’s been in the security business for a long time, the Apple Pay model looks to me like a lot of problems solved elegantly using newer, better tech, while the MCX model looks like someone trying to warm over processes invented almost 20 years go. And personally, I’m not inclined to use a system that requires me to tie my checking account directly into their system; I effectively quit using PayPal years ago for that very reason.

I agree that this is about the money – but not, in fact, about the interchange fees. MCX wants to build a payment network that centers more on a “loyalty program” model, one that allows merchants to “provide valuable messaging” to their customers, based on their intimate understanding of a “customers purchasing history and habits”. In other words, they want to track their customers’ every move.

Merchants are used to paying the interchange fees, and long ago built those fees into their pricing structures. I’m sure they’d love to find a way to strip that 2% to 4% off and save that money (although I’m highly skeptical we, the customers, would see those savings should they do that). But in the end, what terrifies the merchants is the specter of their customers becoming truly opaque to them: They are terrified of losing their ability to use us as a marketing channel.

Fidelity Full View: almost Intuit quality

I had been a Mint user for years, until Intuit purchased the group and added their characteristic bureaucracy and “we don’t really care” customer service stamp to the operation. One of the final blows was when, shortly after the acquisition, Mint suddenly reclassified HELOCs as credit cards. A large number of people, including me, posted separate threads on their support site complaining about the change. Their answer, over the next two years, was essentially “gosh, there’s just nothing we can do to fix that, it’s just too hard.” Two and a half years later, from what I understand, Intuit first fixed the problem then, days later, broke things even more badly to the point that now even mortgages are classified as credit cards. Sweet, Intuit.

In the middle of this I discovered that my bank, Fidelity Investments, offered a web app called Full View, that offered most of the same functionality I’d liked in Mint, and in fact uses the same aggregator, Yodlee. Account aggregation, budgeting, transaction classification – all good. The Fidelity app wasn’t (and still isn’t) even close to Mint in terms of usability or gloss, but it worked.

That was the case until a few weeks ago, when Fidelity, with much fanfare, released the new version of Full View. (cue Fraü Blücher’s horses in the background).

What a damn mess.

First off, they’ve clearly gone the portal route, cobbling together a lot of prefab widgets at the cost of a pleasing, well-integrated user experience. This is a classic example of the user experience delivered when an IT group researches solutions and then uses the results to determine what users want (translation: what IT wants to build, including all the little gadgets and bits of coolness that appeal to geeks but mostly get in the way of normal users). That would be bad enough but manageable if those widgets actually worked. And they don’t work all that well. IT is not the same as development: I would never want a member of a development team maintaining servers – nor would I ever want an IT engineer developing an application, much less designing a UX.

Let’s start with data handling, because I get too pissed of talking about all the 404 errors that get thrown transitioning from the dashboard to subordinate views and back to speak of it at length. Sufficed to say: OMG. Entering data, modifying the name of a transaction’s vendor for example, is pretty basic stuff. Anyone out of high school understands the concept. But with Full View, I’ve had to re-enter changes over and over, across sessions, across days, because the changes get lost. This isn’t just bad session handling (that’s pervasive across the new version) but somehow they’ve managed to corrupt the data after it should have been stored persistently in the database. You’d have to go out of your way and write code to do that. They did – although I don’t think it had anything to do with intention.

Then let’s talk about character set handling, things like punctuation in a title, for example “Gold’s Gym”. On their Transactions page, this appears to be handled correctly, but in their dashboard view making this change results in “Gold's Gym”. Their customer service folks offered the solution of “well, you should just use the Transactions page for that,” a secondary page that takes forever to load. I haven’t encountered this sort of crap for probably ten years – at least not from a professional group. And forget autocomplete or any sort of elegant categorization.

Other features like adding a note to a transaction, or splitting a transaction, have been moved from the main page exclusively to this subordinate Transactions page, an “improvement” as described by Fidelity Customer Service. UX FAIL.

And in the days of HTML5, H.264, Canvas, all the vector tools available, all of Full View’s data visualization appears to be based on Flash. Seriously?

Finally, the data displayed in Full View, even when “refreshed”, lag the main Portfolio page by around a full day. When I called this out to Fidelity, the response was sort of “oh well, that’s a really hard problem to solve.” Which puts them right in the Intuit category of competence. This is a bank that, on one web site, displays two views of your financial data, one that is up-to-date and one that is from yesterday. And claims there is no way to resolve that.

At the company I work for, producing a user experience that is delightful is one of our cardinal goals. We frequently use the term “delight” in our proposals, even in our contracts.  With the pre-Intuit Mint, there was a fair amount of delight in using the app, in the nimble UI, in the data visualizations.

There is no delight in Fidelity’s Full View: it’s a portal-oriented, forms-based UI that looks, and behaves, like something from the 90’s, and functionally appears to be built by a team stuck in that decade – the lack of stability, the demonstrable lack of code re-use, the almost complete lack of aesthetics, is downright discouraging. The UX blunders, the coding errors, the clear misses in session and data handling border on spectacular. Fidelity’s customer service is, at least in their interactions with me, in full-on “circle the wagons” mode, denying that there is actually a problem, that the app is performing just fine. In other words, they’re basically useless.

This is four weeks after they launched the new version. Which doesn’t bode well for anything getting fixed real soon.

 

Geiger Trumps GlobalEntry

photo by pennuja (http://www.flickr.com/photos/pennuja/)
photo by pennuja (http://www.flickr.com/photos/pennuja/)

Last October, I was diagnosed with prostate cancer [Prostate Cancer: diagnosis]. After considering all the options, I elected to go through a procedure called Brachytherapy. This involves volumetric measurement of one’s prostate, and the subsequent insertion of around 100 rice-grain-sized “seeds” made of titanium and filled with Iodine 125, a radioactive isotope. The seeds are permanent, the idea being a long-term, highly-localized dose of radiation to kill off the cancer. The half-life of I125 is about six months. t the end of the procedure, I was issued a wallet card identifying me, the procedure and the isotope: the seeds can be detectable when crossing international boundaries.

On a recent flight to Mumbai, there were TSA agents in the jetway at Seatac, checking passports. As I approached one of them, a little box on his belt began to buzz. He actually looked a bit scared for a moment, like “uh oh, this is the big one” and asked “is there anything you want to tell me?” Ah. I showed him my “I’m radioactive – but it’s okay” card and he waived me on. No big deal.

I travel quite a bit, much of it internationally, and for reasons unexplained I tend to get pulled into secondary screening every time I enter back into the U.S. I decided to go through the effort to get a GlobalEntry ID in the hopes it would streamline my travel experience. On my return from Mumbai, Seatac Immigration was jam-packed, so I was glad to be able to walk past the crowds to a bank of 4 GlobalEntry kiosks, nobody using any of them. I placed my hand in the scanner, posed for a photo, tapped a few things on the touch screen and was on my merry in about 30 seconds. Awesome.

I thought I was home free – until, as I handed my Customs Declaration to the CBP officer, one of those same little boxes buzzed. My radiation card didn’t much impress the officer – other officers separated me from my luggage and led me to a waiting area where they brought out a shoebox-sided device with a handle on the top that started ticking when it got near me. The agent explained “this is a very expensive instrument that looks for radiation.” I said “um, yeah, it’s a Geiger counter.” He was surprised and responded “oh, you must be a scientist.” Said Geiger counter recorded my pattern but couldn’t match it against some database of “okay” forms, and so the sample had to be emailed to “a scientist on the Internet” (FEMA?) for verification. I ended up sitting for 30 minutes before I was cleared.

I’d like to think that in the age where we have TSA, CPB, PreCheck, GlobalEntry and the concept of a trusted traveler, the systems might somehow be connected in a productive way. Once I’d been identified with a specific isotope, and determined to have a proper explanation, it would be great to have that included as part of my trusted traveler profile. I posed that as a suggestion on the CPB website and received a call from a rather surly CPB agent who basically said “can’t do it, not now, not never. Tough luck.”

Ah, the pairing of government with technology…