I have been subjecting my twitter followers to some rather cryptic ranting about DigitalOcean (DO) lately. I’ve been a DO customer since February 2016 when I moved from Linode to DO. I don’t do a lot with my virtual private servers (VPS). I host about 15 WordPress installations for various projects (Mostly family and friends sites). I run several command line applications that eat up a chunk of CPU So I’ve always known that shared hosting wasn’t for me and I need at least a VPS to be happy. This also serves as my own personal playground. I’m not a big deal to DO because I spend ~$50 a month. My contribution to them is very small in the grand scheme of things.

As you probably already know DO makes it very easy to spin up servers and until recently I’ve never had a day’s trouble from them. This came to a screeching halt when I actually needed the support team to reply to me about a concerning email I received.

On August 29th I received an email with the subject “Fwd: Trojan Detected -- Please Shut Down! -- [ATS #58539 ] -- IP: 104.236.252.178”.

I didn’t see the email until September 4th, when DO shut down my Droplet with no warning or support ticket. The reason I didn’t see the email until September 4th was because it was marked as spam in Gmail, with a warning that “emails like this have been used in phishing attempts”. After checking headers and verifying the email was legit, I removed the email and whitelisted the source in my Gmail settings. I immediately opened a new ticket with DO support asking why my droplet was shut down and locked without any notice or support ticket.

I opened the following ticket at Mon, Sep 4, 2017 at 9:57 PM (US Central Time):

I closed the ticket 3 minutes later because I saw previous ticket in my ticket list: “Fwd: Trojan Detected -- Please Shut Down! -- [ATS #58539 ] -- IP: 104.236.252.178”, I didn’t recognize it so I opened it to see they were warning me that my machine was serving malware. I saw another ticket “[#575508] Droplet disabled -- Heodo botnet controller” created at September 5, 2017 12:50 AM (assuming UTC). I did not receive an email about this issue (I also checked my spam folder).

I immediately responded to the ticket saying “yes mount the ISO”, then later after I recovered from a 7 day old backup, I said screw it and told them to destroy the droplet:

FIVE DAYS LATER including asking them via Twitter why there was no response, they replied:

It took support FIVE DAYS to unlock my droplet. If I did not subscribe to their backup service I would have been completely down and unable to access my data from September 4th to September 10th. Around 6 days with no communication or response from DO.

On Friday, September 08, 2017 3:17 AM (UTC) while I’m still waiting for them to unlock my droplet I get another email:

Apparently the droplet they locked is still serving malware? I replied and let my frustrations show:

They thanked me for a prompt response and appreciate my actions. I would have appreciated the hell out of them unlocking my droplet instead of waiting another TWO DAYS.

On Thursday, September 07, 2017 3:00 AM (UTC) I received yet another email complaining that the droplet was being abused and they provided the hostname pointed to the droplet. But the host wasn’t pointed at my droplet! I let them know this at Thursday, September 07, 2017 12:47 PM (UTC):

You can tell by my lack of grammar I was furious beyond words.They FINALLY replied to this ticket 17 days later. After once again being badgered via social media.

So we’re 18 or so days after this whole scenario and I have lost 100% of my faith in DO’s support. I never did spin up my droplet after they unlocked it. I should have mounted the filesystem and done some forensics to see what happened. What I can tell you is that I restored immediately from a 7 day old backup and reset ALL the passwords and did all the package updates and all of the WordPress related updates. (I was already running WP updates every 24 hours as it was). I’m willing to concede I likely had a weak root password because when i went back to check what it was, (In LastPass) it was only 6 characters and it was dictionary based. That’s completely my fault and where I assume the original issue originated. I’ve scanned and locked down the new droplet and have not had any issues or alerts since.

Takeaways:

Digital Ocean was quick to respond via social media but it never had a real impact. I was just sent canned responses.

Secure your shit. Remember “The only truly secure system is one that is powered off, cast in a block of concrete and sealed in a lead-lined room with armed guards -- and even then I have my doubts.”

DigitalOcean had many chances to make this right but no one I interacted with cared or was empowered to. I would have been happy with an apology and some reassuring words that it wouldn’t happen again. Like I mentioned earlier I’m nothing to their bottom line. How your company treats the little customer is everything I need to know about your company as a whole. I really want to like DO. I hope this was a perfect storm of customer service failure. I won’t be recommending them to anyone anytime soon.