Frontier Communications - a systems nightmare in Sandpoint

UPDATE: After all these months of terrible connectivity, it is unclear whether the problem has actually been solved. The only group that provided what I'd consider engineer-level suggestions was the email tech support group. That ended today when I received an email from "Linda B." in that group:

Frontier will no longer be using Support Mail. Going forward please use the chat function that can be found at https://frontier.com/Contact-Us#/residential or please call in at your earliest convenience. For Customer Service please call 1-800-921-8101

So, the only group that actually knows what they're doing is no longer allowed to actually provide support. Their chat function only works about 20% of the time, and their phone support is absolutely worthless.

Nicely done Frontier.

 

Three months ago, we relocated from Kula, Maui to Sandpoint Idaho. Kula is semi-rural, lots of ranch land all over the place. Our Internet provider was Time Warner (dba Oceanic). Even way up the mountain, we had 230Mbps down and 25Mbps on our cable internet service. The connection was pretty damn stable and plenty fast.

Then we arrived in Sandpoint. Here there are only two non-satellite providers, Northland and Frontier Communications. Northland provides cable-based service and Frontier provides DSL. both advertise speeds of "up to 24Mbps." In Sandpoint, that is, currently, as good as it gets -- welcome to the 3rd world of connectivity. Another provider, Ting, is claiming they'll have gigabit fiber Internet available about mid-2017 in Sandpoint. If they can make that happen, I'll be their new best friend.

Since Northland never returned phone calls or email inquiries, we defaulted to Frontier. The install tech was friendly and knowledgable, and got us hooked up. Except that, in spite of the significant effort he put in, we haven't been able to get a stable connection for more than a few days at a time during the 3 months we've had Frontier. Ping times of over 4000ms (that's ridiculously long). DSL channels would drop out. Speeds were down in the 100's of bits per second (no, not Mbits, not Kbits, just plain bits). The tech checked our house wiring, the neighborhood wiring, even tested our physical wires all the way back to the Sandpoint data center. We'd go hours, or days, without any connectivity. A few possibilities were hypothesized:

  • Maybe our Apple AirPort Extreme access point is shoveling data at the Frontier modem too quickly, and the modem is freaking out. This was never proved nor disproved, but if a consumer-grade wifi router can overload equipment supplied by an internet provider, that provider has some things to answer for.
  • Maybe our surge protector is supplying dirty power to the Frontier modem that is making it freak out. This seemed even more suspect than idea #1, but what the hell, we took the modem off the surge protector and things worked better for a while. Of course, later data suggests that the improved behavior was coincidental.
  • Maybe our home network is asking too much of the Frontier modem. Two laptops, iPads and iPhones, a TV and a Mac Mini to supply media to in-house devices. Sorry, I'm just not buying it.

Of course, beyond the install tech, there was Frontier's "customer service". Fully script-driven, unable to offer much more than "did you reboot the modem?", reluctant to escalate to a higher-tier representative (it's unclear if there actually are any higher-tier personnel -- if there are, we never met 'em).

Frontier uses 2-channel bonded ADSL for home internet service, and they supply an ActionTec ADSL2+ modem. If you look up replacement modems for that type of connection, you'll find there are almost none. DSL is pretty antiquated stuff. So, no replacing the possibly-crappy modem supplied by the provider.

After three months of this bullshit, I started a Twitter storm (Donald would be proud) harping on Frontier's inability to provide basic Internet service with anything remotely resembling reliability. Frontier's Twitter team responded in ways that make me think that (a) nobody at Frontier actually understands how to use social media, and (b) nobody at Frontier's Twitter team knows how to look at their internal trouble-tracking database, since at least 5 different people responded via Twitter and none them seemed to know about each other's responses, the service calls I'd logged, the problems that had been reported, etc.

The result, at least as of today, is something another Frontier tech discovered (Note: Frontier's field techs have been consistently first-rate): Frontier's office uses boards made by AdTran for their DSLAMs (DSL Access Multiplexors). These are essentially large slabs of electronics mounted in racks that aggregate multiple consumer connections. Think 64 modems on a single circuit board, each modem connecting to a home or business, handing off all those connections to a single pipe. As it turns out, something nobody at Frontier knew was that AdTran recommends rebooting their card every 30 days (maybe they run Windows?), else their buffer tends to fill up, and the card starts rejecting connections from those consumers.

The symptoms of this result are exactly the symptoms I'd been reporting for three months.

Now, rebooting one of those cards effectively disconnects every account wired to it for anywhere from a few seconds to several minutes. And the Frontier data center probably has hundreds of those cards, each of which apparently is supposed to be rebooted independently. And that's the best solution AdTran appears to offer.

The thing I'd like to know is who, if anyone, at Frontier performed due diligence when procuring these crap AdTran boards? Frontier is now far too invested in AdTran to retool with Cisco or another hardware vendor; they're effectively locked in. And AdTran appears to have no viable solution short of yanking and rebooting hundreds of their boards every month, a service disruption that would probably [further] cripple Frontier. So, a followup question for Frontier is: what the hell are you planning to do about this mess?

In about 30 days, I'm going to be asking that question of Frontier. Loudly.

Update: five hours or so after the "fix", we're back to 2000+ms pings, and our speed is less than 4Mbps down. Frontier: I want a fucking refund.

Scalping at the Festival

This summer, I and my wife Sarah relocated from Kula in upcountry Maui to Sandpoint. We’ve been here quite a bit, as we have family just across the Washington border, love the climate (I’m from Colorado, Sarah’s from Montana), the people, the vibe. We love the Festival at Sandpoint.

Going to see Emmylou Harris last weekend, we probably paid a bit more attention to the goings-on around us rather than simply reveling in a “We’re at a concert!" fog; New Resident Mode vs. Tourist Mode. Some of our group got up before dawn and made the pilgrimage to War Memorial Field to stand in line and get “morning numbers”, tickets that allow the holder to enter the event at the head of the line, in the order the morning numbers were issued. Most people do this so that they can reserve space for their extended group on the grass, as close to the stage as possible. The Festival rules state that each holder of a morning number can spread out an 8’ x 8’ blanket, sufficient for four people to sit and enjoy the show. The rules also state that only one morning number per person will be issued — there was, at least up through last week, no requirement that a person present a valid event ticket in order to get a morning number.

The rules also state that there is no camping allowed in the park or on the sidewalk, but our group found a veritable tent village, mostly kids, camped out at the head of the line to get the best morning numbers. And many of those people didn’t appear to have a ticket to the event — they were acquiring morning numbers to scalp them later, for $20 and up, depending on how low a number they had.

We discovered that this scalping of morning numbers, in addition to injecting a pretty seedy ambience while waiting in line, has been a bone of contention for event-goers for years. The sentiment seemed to be that the Festival organizers had avoided the issues of the tents and the scalpers, some of whom appeared to be mothers with their teenagers in tow, demonstrating how to scalp morning numbers. On the other hand, we heard often that nobody ever cut into the line of people waiting for the show because the organizers kept a good watch on such things.

So, people are pissed off, but is it really a bad thing to be doing? It’s not technically illegal, although that may be a fine point.

The answer, as it turns out, is probably “yes, it is bad”. Let’s do some math:

Using Google Maps, I can estimate that the prime real estate at the venue, the grassy center section in front of the stage, is about 100’ x 150’, or about 15,000 square feet. Since a blanket can be 8’ x 8’ or 64 square feet, that means that the most desirable section of the venue, the place worth getting up at 4:30 am for, can support roughly 200-225 blankets. Taking into account that people rarely set out blankets in a nice, orderly way, that means that if you get a morning number higher than 150, you’re probably out of luck for the most part.

This begs the question: should the Festival be issuing 300 morning numbers or only 150?

Some more math: there were probably 25-40 people camped out in line to get the best morning numbers. Let’s assume some of them actually want to get good seats at the event for themselves. So, say 30 of those people are there mainly to scalp the numbers they get (that’s consistent with what our group saw). If only numbers 1 through 150 are worth getting, and 30 are shaved off the best numbers by scalpers:

About 20% of morning numbers are effectively stolen. Which I’d argue is bad.

After pondering this for a while last weekend, I wrote to the Festival about the issue on Sunday, and proposed that if they enforced a requirement that a morning number would only be issued to a person holding a ticket for that event, a lot of the scalping would go away. I was pleasantly surprised when, two days later, I and all ticket owners for the Festival received an official email telling us that, starting the second week of the Festival, just such a rule would be added. Three possibilities come to mind:

  1. The Festival had been planning this new rule for a long time. But it’s unlikely that, after such planning, they’d implement the new process during the second week rather than from the start.
  2. The uproar over the scalping had been so great this year from just the first few shows that someone at the Festival office decided enough was enough. Also sort of unlikely that so much energy would build that quickly.
  3. My email got through to someone to cared, and my proposing a relatively simple solution was what just what they needed to take action. This, of course, being completely unbiased, is my favorite explanation.

In any case, huge kudos go to the Festival staff for getting the new rule in place.

But I don’t think we’re done yet — some more math:

If you scan a valid ticket, electronic or hardcopy, it will scan as valid. If you scan that ticket 20 times, it will scan as valid, 20 times. If you make 20 copies of a ticket, all 20 copies will scan as valid. That means a group of scalpers can purchase a single ticket and scam a bunchnumber of morning numbers.

The straightforward solution is to associate each morning number with a ticket (it's just a database record), so that only one morning number can be issued for any single ticket.

Better, but:

Four people, each with a valid ticket, can go early and get morning numbers. A morning number’s worth of space is good for 4 people. This group can then scalp 3 of their morning numbers for, say, $20 each, and get a $15/person profit. Still not quite there…

Better: Make the morning number valid only when presented at the gate by the person holding the associated ticket. Better still, do away with the physical morning numbers completely: For example, if you fly, and are enrolled in TSA's Pre program, you don't get issued a separate Pre ticket to get you into the priority line. Your ticket gets the TSA Pre symbol added to it, whether it's a mobile ticket or a printed ticket. Similarly, a person’s Festival ticket would magically show “morning number 23” or such when rendered on their phone or printed. Or the person scanning tickets at the gate would see it on their terminal.

But some people are really determined to beat the system:

That same group of 4 could get each morning numbers, then negotiate with 3 parties in the entry line to use a blanket’s worth of the space they’ll reserve. This sounds like a lot of work, doesn't it?

Like any padlock, the intent isn’t to totally prevent a thief. The intent is to make the theft so difficult that the thief will simply give up and look for an easier target.

Island Pet Movers: poor transition from small company

We first learned about Island Pet Movers about 3 years ago when we were moving to Maui. At the time, Kari, the owner, was very involved with operations personally, and the touch and feel of using her service were nothing short of wonderful. We felt fully prepared every step of the way — and moving a dog from the mainland to Maui is far more complicated than the other way around.

So, as we prepared to relocate back to the mainland, we contacted Island Pet Movers to get our two dogs from Kahului to Seattle. One of our dogs is snub-nosed, which presents some complications for flying, and we figured Kari’s company could get us past all the hurdles.

We initially contacted Island Pet Movers via email, and were bounced to the website (effectively, “don’t bother us —  go fill out the forms.”). So we filled out the contract on the website and, … nothing. After a day or so, we tried to get anyone from Island Pet Movers to respond to email or phone calls without any luck. My wife stumbled on Kari’s direct number and called it, at which point Kari made it excruciatingly clear to us that she is no longer involved in day-to-day operations, and that we were never, ever to call her number again. She said that upon completing the online contract we’d surely immediately received 3 documents (we heard that mantra, “3 documents” too many times), and that it was probably in our junk mail, the implication being that we were too stupid to check there. Those 3 documents arrived 3 days later, as it turned out, and covered a bunch of “this is how you pay us” details and links to a useless FAQ, but very little about what to expect, as we prepared, as we arrived at the airport, all those trivial things.

Then we had a date change for travel, and emailed Island Pet Movers about it. Crickets. Our online account on their site (labeled "2015 client login”, which tells you how well their site is maintained) still showed the original travel date. Two days later we finally talked to an actual person, who blew us off with “yeah, we know about the change. Our database doesn’t work so you won’t see any changes there”. Changes like a different flight date, or uploaded required documents, or photos of the carriers, things like that. Utterly useless - our “account” page never reflected any changes, ever. We never received a “received your change - got it” communication of any sort - we had to pester them for that information.

We set all this up about 6 weeks ahead of time, only to be told we wouldn’t get flight information until about 2 weeks before departure. No pressure, just our air travel, house sale, moving pod, all of that tied to the same day that “would probably be okay”. It turns out the “no flight confirmations until 2 weeks before travel date” is actually imposed by the airline. That would have been good information, but Island Pet Movers doesn’t really volunteer any details like that unless you demand explanations. When we did finally receive the flight info, all looked good — except we then discovered that, because we lived on a neighbor island, there would be no Island Pet Movers person meeting us at the airport unless we paid extra. We were on our own. That was another detail we had to painfully extract from them.

Then pretty much weeks of no contact, no information, no preparation, just waiting and hoping these people actually knew what they were doing, which ran contrary to the impression we’d developed by that point. The concept of putting the effort into “keeping the customer warm” doesn’t appear to exist for these people.

We received a call from the company the day before travel, were told to get to Air Cargo early, and that was about it. No information about what to expect for process, forms we’d have to fill out, things like that. Until the call that day, we didn’t even know what we’d need to supply along with our dogs, things like food, or stickers, or water, or leashes.

On the morning of travel, we arrived at Air Cargo early, only to find that the staff there had the travel date for our dogs as the following day (Kari subsequently blamed Air Cargo for that). Then there were 8 pages of forms to fill out, and nobody (us or Air Cargo)  was sure who should be filled in as the sender (us or Island Pet Movers?) or the consignee (same question), or if the forms should show Honolulu as the destination where Island Pet Movers would take charge of the dogs and forward them on, or if Seattle should be indicated. All these were questions the staff at Air Cargo had for us, questions for which we had no answer. Questions that should have been dealt with prior to that day by Island Pet Movers preparing the way. Our “flight sheet” had a number we were supposed to call if there were emergency problems on the day of travel — but that number led to voicemail tree.

We sent an urgent email to Island Pet movers, received no response. Finally, we and the Air Cargo decided we had it mostly figured out, and we left to return home, shut down the house and get back to the airport for our flights. Nearly home (upper Kula), we received a call from Air Cargo saying Kari had contacted them and told them what actually had to be on the forms, and that we had to return to get the correct paperwork. We turned around and headed back down the 22 miles to the airport. Nearly there, we received an email from Island Pet Movers telling us there was no need to return to the airport, basically telling us the Air Cargo people were clueless and it was all Air Cargo’s fault. We returned anyway, since those clueless Air Cargo people were the only ones who’d actually helped us get our dogs on the way. The Island Pet Movers email also told us we should’ve used the emergency number (the one on the flight sheet that got us nothing but voicemail).

That tone pretty much describes the experience with Island Pet Movers these days:

  • Don’t bother us with any live communication. Go to the website or email us and we’ll get back to you when we get around to it.
  • You don’t need to know any actual details, so don’t bother us. We’ll tell you things when/if we think you should know them.
  • The airline/Air Cargo people are at fault if they don’t understand our process.
  • You, our customer, are at fault if you don’t understand our process.

The thing is, when you pay someone like Island Pet Movers to get your pets to a location, you expect that they’ll take care of all the administrative details like magic, and your dogs will go through a seamless tunnel to their destination. But that doesn’t happen with this outfit:

  • Their contract forms are terrible and unclear to read.
  • Their customer account page is connected to a database that never updates because it’s been broken for some time. Their customer login prompt is nearly 2 years out of date.
  • Supposedly automated processes like the “3 documents” delivery don’t appear to work reliably.
  • Information is handed out sparingly, even grudgingly. Our entire fiasco at Air Cargo might have been avoided had Island Pet Movers supplied a simple page or two of useful guidance.
  • Contact numbers on day-of-travel paperwork like the flight sheet lead to voicemail.
  • Fixes like Kari calling Air Cargo come too late to be of much use.

I fully understand that transitioning a company from being run by the founder to other staff is hard. Someone who’s been performing that function has such knowledge and experience that a lot of it has become unconscious, which means it frequently doesn’t get translated to staff, or policy, or paperwork. That transition is where this company has failed pretty miserably:

  • The staff is inexperienced, curt, sometimes petulant, and rarely volunteers information beyond the basics. It’s not even clear the staff understands the process themselves.
  • The overall vibe when challenged by a customer is very defensive.
  • There is almost no preparation for arrival at the airport with one's pets, probably one of the most stressful things many people will do in their lifetimes. We had flight info, an address for Air Cargo, and a time to get there. The rest was “you figure it out”.
  • Printed/emailed information provided is sadly lacking in any useful detail.
  • Their FAQ is a joke.

At no point in our engagement with Island Pet Movers were we offered a way to provide feedback. Now, weeks after our travel, we’ve received nothing from that company. Not a “Congrats you made it!” Not a “How did we do?” And truthfully, our impression at this point is that nobody at Island Pet Movers actually cares enough to know. And without that feedback, and paying attention to/acting on that feedback, I don’t see how Kari’s company will ever get any better, ever get to the point where she can truly extract herself from day-to-day operations and stop fire-fighting. Much less repair the reputation she's building.

The bottom line

Our experience with Island Pet Movers is that they’re generally unresponsive, frequently petulant and sometimes even rude. They dropped the ball nearly every step of the way. They were, from our perspective, a waste of money, and were basically a nightmare to work with.

"Kill" vs. "No Kill" Shelters: an Ecosystem

My wife and I have owned lots of dogs over the years, and they've all been rescues, either from the pound or some rescue group. We have pretty strong feelings about adopting a dog that needs a home as opposed to going to breeders or such.

This week, there was a Facebook post from Maui Pitbull Rescue (MPR):

https://www.facebook.com/pages/MPR-Maui-Pitbull-Rescue/145160815524156

In their feed, on March 23, 8:46pm, was this statement along with some photos of the dog:

“Alert!!!!! This pit mix is # 55 at MHS. He is to be euthanized. He was found two weeks ago and now no one has claimed him. Can anyone help. We don't have room at MPR at this time. He loves other dogs too.”

"MHS" is Maui Humane Society. What are they doing?

"Kill" Shelters

As it turns out, it's a pretty interesting story. MHS is what’s called in some circles a “kill shelter”, in that they euthanize some animals. So, why do they do that? Well, some facts: MHS is partially funded by Maui County. As such, they operate under some fundamental requirements:

  • They are required to accept all animals that arrive at their door. This includes:
    • mongoose
    • goats
    • chickens
    • wild ducks
    • turtles
    • bunnies
    • mice
    • rats
  • They are required to euthanize animals

Where do all those animals come from? Some are dumped dogs; people get tired of caring for a pet, drive them out to some remote location, kick them out of the car and drive away. Or people are going on vacation and, rather than arrange for someone to take care of their pet, they dump them. Or a dispute with a landlord means someone can no longer keep their pet. Or someone ends up in jail. Or someone gives a child a bunny for Easter, which thrills them only for a short time. There are hundreds of reasons why people dump pets. But the common factor is they're throwing away an animal that has depended on them for care, that has probably bonded with some or all of the household, that cannot understand why, all of a sudden, they are on their own.

A lot of these animals end up at Maui Humane Society. From their Fiscal 2013-2014 Annual Report

  • An average of 23 pets arrived daily
  • 3,000 pets were spayed or neutered
  • The number of feral cats received decreased 64% year over year
  • The number of strays is decreasing year over year

The breakdown:

  • Animals Received:
    • Cats & Kittens: 4,923
    • Dogs & Puppies: 2,308
    • Other Animals: 1,399
    • Total: 8,630
  • Animals Adopted:
    • Cats & Kittens: 769
    • Dogs & Puppies: 723
    • Other Animals: 241
    • Total: 1,733
  • Animals Reunited:
    • Cats & Kittens: 263
    • Dogs & Puppies: 531
    • Other Animals: 8
    • Total: 802
  • Animals Transferred:
    • Cats & Kittens: 1
    • Dogs & Puppies: 242
    • Other Animals: 16
    • Total: 259

What’s not in the Annual Report is some additional math:

  • Total Received: 8,630
  • Total Processed: 2,794
  • Not Accounted For: 5,836 (68%)

And here’s the breakdown of the unaccounted:

  • Cats & Kittens: 3,890
  • Dogs & Puppies: 812
  • Other Animals: 1,134

One has to assume that a significant portion of the unaccounted-for animals are euthanized. Why? Lots of reasons, but the main ones include:

  • Lots and lots of feral cats
  • Terribly injured/sick animals
  •  Malnourished animals; animals with mange
  • Behavior problems / not placeable

“No Kill” Shelters

"No Kill" shelters have no facilities for, or interest in, euthanasia. On Maui, there are a couple of them:

Maui Pitbull Rescue (MPR). Their website states:

"Maui Pitbull Rescue (MPR) is the only no-kill pitbull rescue shelter in the state of Hawaii.”

Hawaii Animal Rescue Foundation (HARF). On their website:

"We are a group of experienced animal welfare people in Hawaii that are buying land and building a NO KILL shelter." [emphasis theirs]

How do they operate without euthanasia? Well, as it turns out, they don’t, actually. These so-called “no kill” shelters are highly selective about what animals they take in. In some cases, they even survey the animals at MHS and take those animals they feel are the most adoptable. There are two holes in that logic, however:

  • The animals they refuse to take on have to go somewhere, namely MHS
  • If they find they can’t get an animal adopted, they take that animal to MHS — even if they “rescued” that animal from MHS in the first place

Holding the Bag

Where does that leave Maui Humane Society? Holding the bag, essentially. They are, as defined by their mission, the dumping ground for unwanted animals — but that includes animals rejected by no kill shelters, too.

There’s also a stigma that gets attached to MHS because they euthanize animals. That stigma is exacerbated when the no kill shelters make a huge deal about their not euthanizing animals, and especially when they send out alerts about an animal that is about to be put down at MHS. Now, that animal mentioned in an alert may just get adopted, but there are other possible outcomes:

  • MHS is negatively portrayed by the groups for whom they’re actually doing the dirty work
  • MHS is further discredited by people that don’t understand the shelter ecosystem
  • The animal may get adopted in a well-intended knee-jerk reaction to the alert, but then that adoption doesn't work out, and the animal finds itself back at MHS

It really is an ecosystem, as the flow of pets from sources to people to shelters to people, or to euthanasia, resembles a closed system: strays and abused animals have to go somewhere.

Some Other Perspectives

A lot of people have done a lot of thinking, and writing about this. People who work within that ecosystem and really know what they're talking about.

From SPCA, Los Angeles

“spcaLA does not euthanize for space or for time. We do not euthanize what we determine are adoptable animals.
"We will euthanize when an animal requires medical treatment that goes beyond our ability to humanely provide, or has a condition that puts other shelter animals or workers at risk.
"We will also choose euthanasia when an animal has negative behaviors, such as unmanageable aggression towards other dogs, or aggression towards people that goes beyond our ability to correct, especially if that behavior presents a safety concern to a potential adopter or to the community.
"We do not feel it is responsible to place a dangerous animal in the community. We also do not feel it is responsible to imply that we would.
"There are few organizations with the money and facilities to keep an animal that is ill or unsafe around people. In fact, keeping such animals while thousands of healthy, adoptable animals are euthanized because there is no place to keep them could be considered an unconscionable decision.
"While “no kill” is a popular phrase in today’s animal welfare environment, we do not find its use responsible. We discourage the use of the phrase “no kill.” It hides the problem. We instead want to be very clear to our community what our choices are and how our decisions are made.”

AHeinz57 Pet Rescue & Transport

“Bridging the Gap Between No Kill vs Traditional
“It’s not fair that our rescue gets to boast that we do not euthanize animals when we have to turn animals away because we don’t have room. The animals we don’t have room for end up at the traditional shelters because they do NOT turn animals away.”

Rescue with Your Eyes Open

So, next time you’re feeling sad or angry about animals getting put down, or better, thinking about finding an animal to rescue you, keep in mind the whole story. And if you do adopt a pet, take that adoption seriously, understanding that you're now, literally, responsible for that animal's life.

Hawaiian Airlines and CheapFlightsFares/Riya Travel: predatory partners

This January, I was asked by a client to fly out to NYC for a meeting. They suggested that I book first class from Kahului, a suggestion I was more than happy to consider. It was pretty last minute, the client request coming on a Monday and my departure being that Saturday, so I figured I should move fast to ensure I’d get seats. But I was also at the beach, so it looked like I’d need to book my flights using Hawaiian Airlines’ app.

Right off, I need to say that I think the Hawaiian Airlines iPhone app is pretty bad. I’ve been a road warrior for the last several years, and watched as the capabilities of apps for carriers like Alaska Airlines and Delta Airlines grew better and better. Hawaiian is, in my opinion, still functioning at a technology level the other airlines were at years ago.

My client suggested specific flights, and who was I to argue, so I wanted to search by flight number. Nope. The flight I was most concerned about was the direct from Honolulu to JFK, but that was the second outgoing leg. On the Hawaiian app, I couldn’t view downstream flights. Giving up, I decided I’d just call Hawaiian and do this the old-fashioned way. I looked up their number on my browser and gave them a call. 

That was my first mistake.

The agent was difficult to understand, something I’ve become more or less accustomed to in these days of call center outsourcing. But he was also inept, botched my request a number of times, gave me incorrect information and then nearly booked my return for April even though I was clear I was coming back in a few days. At the end of all this, I was surprised to hear him ask:  

"I'd like to know why you're booking with Hawaiian because you could save a lot of money booking with Alaska.”

Er, wha? Warning bells started clanging, and so I asked “aren’t you Hawaiian Airlines?” Nope. Turns out he’s with CheapFlightsFares (CFF), a discount aggregator. This was too weird, so I told him to cancel the entire transaction, made him repeat that cancellation back to me (you really need to do that), and hung up.

Less than 90 seconds later, my phone rang. A woman identifying herself as “Miranda” told me she was a customer service manager (she implied she was with Hawaiian Airlines, but that turns out not to be true), that she’d been monitoring the call, and wanted to know why I canceled the transaction. Now, at this point part of my brain was saying “If she really was monitoring the call, she clearly knows how badly things went." She informed me that, when Hawaiian Airlines’ phones are really busy, they automatically transfer calls to CFF so that their customers don’t have to wait a long time to speak to someone. I was still skeptical, but she was able to convince me that she, handling this personally, could get everything straightened out.

That was my second mistake.

Upon giving Miranda my Amex number for the purchase, she informed me that “because of all the fraud going on", I would need to receive and accept a DocuSign transaction. I’ve used DocuSign before. It’s cool, but usually it’s for things like buying a house. When the DocuSign email came through from CFF, I reviewed the document and saw the price: $4,964. Whoa. But I had expected the first class fare to be high, and hadn't been able to view those flights on the Hawaiian app for comparison. I processed the request, and Miranda told me I’d get my confirmation number the next day, which was also very weird. But we ended the call and I returned my attention to jumping in the ocean.

The next morning, I looked at my Amex account online. Three charges had come in: one for $4,054 from Hawaiian Airlines. Another for $1 from “TRAVEL AGENCY SERVICES” in Cleveland (Note: when the transaction went from Pending to Processed, the billing entity magically changed to "RIYA TRAVEL & TOURS IRVING TX". And a third charge for $914 from the same Cleveland outfit, which also changed to Riya Travel. Hmm. I logged onto the Hawaiian Airlines site, and searched for the same flights I’d booked - at least their web site provided sufficient capability to do this. It turns out the fare is, in fact, $4,054. So it looks like CFF booked the flight for me and then tacked on $915 in travel agency fees. Almost a thousand dollars for...what? I’m not as a rule against handling charges, but that is ridiculous.

I take this stuff seriously, and I don’t just roll over. I called Hawaiian Airlines, who basically told me “you booked this with an outside party, sorry can’t help.” Let’s be clear here:

  1. I was connected to CFF by calling the Hawaiian Airlines customer service number listed on the Hawaiian Airlines web site.
  2. The handoff from the Hawaiian Airlines number to a 3rd party was done without any indication that I was no longer actually speaking to Hawaiian Airlines.
  3. The person who answered the phone never at any time identified themselves as a 3rd party representative. Only after I confronted the guy did he admit to the fact.

At this point, as far as I’m concerned, Hawaiian Airlines should demonstrate some integrity if not good management and take responsibility for the system they put in place. When you shovel your customers over to a 3rd party in a way that leads them to believe they are still doing business directly with you, anything that occurs after that is on your plate. Especially when that 3rd party is a predatory loan shark of a business that gouges customers with exorbitant, bogus fees in the name of "service".

Next steps: I went back to my Amex account and marked the agency charges to be monitored for a potential dispute. Then I sent an email to CFF calling out their practices (I used the word “despicable” among others). Finally, I composed, printed and mailed a formal complaint to Hawaiian Airlines’ Consumer Affairs department.

On Wednesday evening I received a call, the a number from Ontario. It’s Chris from CFF, the guy who sent me the DocuSign email. He was a bit upset that I’d started dispute proceedings with Amex. Then he says:

"The $914 is a Hawaiian Airlines charge, not ours.”

I call bullshit and tell him “Well, since I’ve filed a formal complaint with Hawaiian Airlines, they should be able to tell me that themselves.”

Chris gets really flustered and tells me he’s getting his manager. Sure.

Roger, "The Manager", comes on the line and tries to reason with me:

"Those are Hawaiian Airlines charges."
"That's not what the charges on my Amex account look like. I'll take this up with Hawaiian and see what they say."
“But sir, first class is expensive - that’s just how much it costs.”
"Well, then why isn't that the price listed on the Hawaiian Airlines web site?"
"There are always taxes and fees.”
“18% fees? Sorry, I’m not buying it. We’ll see what Hawaiian Airlines says.”
"You contacted Hawaiian Airlines?! What did they say?”
“Nothing yet - but if your claim is correct, that these extra charges are theirs, this whole issue will just go away. If it turns out the extra charges aren’t theirs, I suspect they’re going to want to have a chat with you.”

Roger tells me he’s going to speak to "their accounting department", which appears to be in the office and on duty at 10pm Ontario time. He gets back after a few minutes and says:

“We’re going to refund the $914 charge. You need to drop the dispute immediately.”
“Thanks, but I’ll drop the dispute when I see the credit show up on my Amex.”
“No no, sir, you need to stop that process right now!”
“Well, Roger, that isn’t going to happen. If I see the credit on my Amex, I will then tell Amex to drop the dispute. Until then I'm going to leave the dispute ticket open. If I don't see the refund by tomorrow, I'll process the dispute with Amex and then it's your problem.”

Shortly after my call with CFF, I received an email from Hawaiian Airlines, effectively calling bullshit on CheapFlightsFares', or Riya Travel’s, claim that the extra charges were actually Hawaiian Airlines’ requirement. Unfortunately, Hawaiian Airlines also effectively washed their hands of the matter:

"We apologize for the inconvenience.
However, we need to refer you Cheap Flights where you booked your reservation. As per checking your ticket, it only shows $4,054.00 value.
Please do not hesitate to contact us again for any other concerns."

I particularly like the part "where you booked your reservation", as in "sorry but you're the idiot that didn't book directly with us". Irony appears to be alive and well at Hawaiian Airlines.

As of this update, CFF/Riya has told me it will take at least 5-6 weeks to process a refund for the charge. I've also filed a dispute with Amex, who is looking into it. Amazingly, my letter to Hawaiian Airlines resulted in a canned customer satisfaction survey. You can guess the score they got.

CFF appears to process charges under at least one alias “Travel Agency Services” in Cleveland, which also appears to be Riya Travel in Irving, TX, and their phone calls come from Ontario. I suspect they’re actually part of the global Riya conglomerate, but that’s anybody’s guess. Regardless of whether I get a refund, I think Hawaiian Airlines acted really badly here in a number of ways:

  • They handed off their customer to a 3rd party without telling them about it.
  • They engaged the services of an agency that is clearly shady at best.
  • They shunned their customer rather than taking responsibility for an issue they created.

And if these exorbitant surcharges are common, and people became aware of how they’d been ripped off, I can imagine Hawaiian Airlines and CheapFlightsFares/Riya Travel getting sued. At the very least, Hawaiian Airlines has violated my trust, and I can’t say if I will ever fly with that airline again. Sadly, my experience seems similar to other things I’ve heard and read about Hawaiian Airlines: their service while you're in the air is great, but the way the treat their customers on the ground is terrible. As of this writing, I haven't seen any refund from CheapFlightsFares or Riya Travel. Amex and I will be speaking about this tomorrow.

Hawaiian Airlines, you can do better. You really can. And in any case, thankfully you're not the only carrier I can fly with.

Update: Return Flight Troubles

Just to add insult to injury, Hawaiian Airlines' incompetence didn't end with the above comedy. On checking in for my return flight, I was surprised that my mobile boarding passes were missing the TSA Pre logo. I've been a member of PreCheck, or more precisely GlobalEntry, since the very first days the program was running, and I actually have a pretty solid understanding of how that system is supposed to work. Hawaiian agents at the airport didn't have any interest in the missing endorsement, and I nearly missed my flight because of being stuck in the standard security line. On contacting Hawaiian Airlines' Consumer Relations (that phrase now just makes me chuckle sadly), their response was that I didn't seem to understand that my Trusted Traveler number was not a Hawaiian number, and that I must have made a mistake somewhere. Let's look at the facts:

  1. I booked a single itinerary for round-trip flights from Kahului to JFK. Four flights.
  2. On booking the itinerary, the flights did not appear on my "My Flights" page on the Hawaiian Airlines site even though the flights had been booked using my name and Hawaiian Airlines frequent flyer number. I had to manually add the flights.
  3. Even after adding the flights to "My Trips" in my Hawaiian Airlines account, the corresponding "My Flights" page in their mobile app never showed those flights. Nor was I able to manually add the flights to the mobile page -- I'm not even sure adding flights to the mobile page even works at all.
  4. Using the "Check In" button on the mobile app for the outbound flights, my mobile passes had PreCheck endorsements. Using the same button on the same app for return flights on the same itinerary, the endorsements were omitted. During the check in flow, the Hawaiian mobile app even displayed my Trusted Traveler number correctly. Their system is clearly behaving inconsistently.
  5. Hawaiian Airlines' Customer Service group made it clear they had no interest whatsoever in actually trying to find and fix any problems on their end.

Resolution

As it turns out, Riya Travel doesn't appear to like being called by American Express. Within 48 hours of American Express contacting them with a "please explain this charge" request, they resolved the dispute with Amex and credited back their sleazy service charge. This sort of support is exactly what I should have seen from Hawaiian Airlines, since Riya was at the time masquerading as Hawaiian Airlines, with Hawaiian's blessing, when this whole thing happened. The reality is that Hawaiian Airlines couldn't have cared less about it.

So, in a nutshell:

  1. Hawaiian Airlines' systems and customer support are so bad (not just inept, but actually negligent and even offensively disinterested) they don't deserve anyone's business. Yes, their in-flight experience is lovely and their flight crews are great - but if you have any problems with your booking, you are on your own.
  2. Riya Travel is an opportunistic snake pit of an agency, and you should check your wallet any time you even come close to doing business with them.
  3. American Express is, put simply, awesome.