Download: MP3 (64.6 MB)

My website crashed. It crashed hard. Nothing was loading. I had no dashboard access. It gets worse: I should’ve had daily backups, but something was improperly configured and the most recent backup was nearly 3 months ago.

I feared the worst.

Everything was down. I was getting complaints. My host tried reverting the database only to cause even more problems: orders from the database were getting repeated. Customers were getting charged multiple times. Overdrafts were happening as a result.

It wasn’t pretty.

In this episode, I tell the story of how I (nearly) lost everything, how I got it back, and what I learned from an experience that I wouldn’t wish on anyone.

Show Notes
  • Disaster Strikes (The Backstory)
  • 00:42 Ben: “You doing ok today, Sean?”
  • 00:44 Sean: I’m doing good today. I had a little bit of space—some distance—from a near catastrophe yesterday. I should probably explain what happened.
  • 01:20 Sean: Yesterday, I decided to be a good business owner and update the plug-ins on my WordPress site. I’ve got a lot of plug-ins. This site is massive. There’s a lot going on. I’m not exactly sure how many plug-ins there are but it’s between 40 and 50. I’m not just talking about stupid little plug-ins, I mean serious plug-ins that I actually need to power the core functionality of the site. It’s pretty crazy.
  • 03:54 There was something like 10 plug-ins that needed updates—I did about 3 or 4. Just a few, select, very inconsequential updates. Apparently I was not very smart to do that because once I did, the site went blank. I mean white-screen blank. I mean view-the-source-of-the-page-there-is-no-code blank. I couldn’t log into the dashboard. I couldn’t do anything. It was just dead.
  • 04:31 I freaked out. I tried deleting the plug-ins, I tried renaming the entire plugins folder in an attempt to jumpstart WordPress into running without them—all to no avail.
  • 04:51 I contact my developer (the guy who built the site) who happened to be online. I said, “Help me out! What do we do? I need to get this back up and running!”
  • 05:01 At the time, there was a half dozen people in the Community chat, I had just put out a new podcast, sent the email to 20,000 people, traffic from recent posts to Twitter and Facebook is coming in, and who knows if people were attempting to place any orders.
  • 05:26 My developer said, “Yeah, if you’re renaming the plug-ins folder and you can’t even access the dashboard, you got problem. You need to revert to a backup.” Normally I can revert to backups myself, but I can’t access my dashboard! I have to talk to my web host and my web host does not have phone support, which meant I had to submit a ticket. So I’m sitting there like an idiot waiting for emails to come in while my website’s down. It’s a very helpless feeling.
  • 06:35 I email my host to let them know what was going on and I said, “Can we revert to last night’s backup?” The nice thing was, the podcast that already posted that day was previously scheduled. That meant it would have been saved in the previous night’s backup.
  • 07:00 After about 15 or 20 minutes of refreshing, I notice my site comes back up. They’re clearly doing stuff behind the scenes. The guy from my host contacts me and says, “Hey, we got it back up. It looks like there was something with a plug-in so we disabled it.”
  • 07:24 It’s true, the site is back up, but when I look, all of the posts and data is from a week ago. Multiple podcasts were missing, blog posts were missing, and it’s all the way back to my sabbatical. At this point, I’m starting to freak out.
  • 07:44 Before the host had emailed me (again it was a period of about 15 minutes), I went and logged into my Amazon S3 bucket as I have backups configured to go to Amazon. I look in there and the last backup is from July 29. It’s supposed to be daily backups, and the last one was July 29 which also coincides with when we relaunched the new version of the website. So something did not get reconfigured. There had been no backups being made for months.
  • 08:21 I hadn’t heard back from the host yet, so I sent a follow-up email saying, “Apparently I didn’t have my backup properly configured—don’t revert to the last backup!” Fortunately, they had some sort of backup, because what they reverted to was a only a week old.
  • The Nightmare Continues
  • 09:02 I’m just beginning to start assessing what is missing. We’ve got multiple podcasts gone—which is a huge chunk of my week. The chat backlog was gone. The forum threads and all of the replies from the past week were gone. This meant not only were new member introductions gone, but orders from the past week were obviously gone, which means if anyone signed up for the Community in that time, it was like they didn’t exist. The same with Learn Lettering accounts.
  • 10:04 I emailed the host to ask if they had anything more recent than a week ago, because this is pretty devastating. Of course, this is all email, so I have no idea what’s going on behind the scenes. It’s like an adventure, Ben.
  • 10:08 Ben: “Yeah, you keep hitting the refresh button and you don’t know what you’re going to get.”
  • 10:15 Sean: Exactly. So I’m refreshing and the site comes back up! And then it goes back down. I’m just wondering what is going on. Well, here’s a little fun fact: as they’re repeatedly reverting databases and trying to figure things out, something triggered with the orders that contained memberships within a certain timeframe. The membership orders freaked out. I don’t even understand the logic that’s happening behind the scenes, but because of this blackout, the dates and data gets confused. Since memberships are a recurring deal, the system decides to go ahead and bill these people. I now had some dozen orders that were re-billed—and not only re-billed, but when they reverted the database multiple times, they were re-billed multiple times. Charged credit cards. Order emails sent.
  • 11:11 Ben: “Nice. Free money!”
  • 11:13 Sean (Laughing): No, Ben! Bad. Very bad!
  • 11:18 Meanwhile I’m getting emails from people saying, “Hey, I just got charged again.” It’s nuts, Ben. I’m still recovering, but I’m going to give you more of the story. What I just find out today is not only do we have the issue of people getting re-billed, but that caused someone to overdraft. So they’ve garnered overdraft fees as a result of this fiasco. I said, “Let me know what the overdraft fee is so I can compensate you for it.” I’m doing damage control, I’m doing customer support, and all of my links are bad.
  • 12:17 I’m steeling myself against the worst. I’m steeling myself against thinking about what I’m going to have to do if everything is lost. When my mind starts to go there, it’s insane. It would be weeks if not months of recovery—just from one week lost. It’s not just re-creating whatever was made during that time, but damage control as well.
  • 12:51 Any thoughts so far?
  • 12:56 Ben: “There are words that I can’t say that I would’ve said if that had happened to me. My stomach sank for you. When I saw this going on yesterday, the chat was going up and down and the backlog missing, and people were like, ‘What’s going on?’ I realized I could’t reach you through the chat. I knew I’d have to text you. So I text you and say, ‘Are you okay?’ and you said, ‘No.’ You just said ‘No,’ and you didn’t say anything else and I was like, ‘Man. This is bad. This is really bad.’”
  • Fortunately, All Was Not Lost
  • 13:30 Sean: I honestly didn’t know how bad it was at this point. So I contact support and say, “Do we have anything more recent?” We knew something with a plug-in update corrupted the database. I don’t know what part of it but it was to the degree that the site wouldn’t load.
  • 13:51 Fortunately, they still had a copy of the current database—even though it was corrupt. What we were able to do was selectively import things like forum posts, blog posts, podcasts, and other content types back into the week-old database. Essentially, everything was recovered.
  • 14:16 Ben: “But you’re still dealing with some of the fallout of that.”
  • 14:20 Sean: True. But it could’ve been a lot worse, that’s for sure.
  • 14:24 Ben: “Right. That’s a good thing to say. That’s a good perspective to have.”
  • Breaking Down the Story
  • 14:30 Sean: With this show, I thought I’d tell the story, dissect the situation, and break it into stages: look at how I responded, how I should have responded (if different), how this can apply to other catastrophic events, and then possible ways to prevent catastrophes.
  • The Fear of Losing Everything
  • 15:28 This is the place I was at. If the July 29 backup was the last I had, it would have essentially been losing everything. I can’t even begin to think of what I would have to do.
  • 16:07 This is the stage where something potentially catastrophic happens and you’re not sure that any permanent damage has been done yet but you’re fearing the worst. You just got news of something and you’re not sure how bad just yet.
  • 16:32 Ben: “Yeah, even for the most optimistic person it’s actually very natural to go to the worst case scenario in your mind. That sometimes can be construed as a negative or pessimistic thing, but really if you’re not thinking the worst, you can’t even begin to prepare yourself emotionally or mentally for dealing with the situation. The alternative is to ignore them but that’s very dangerous, because then you have no way of equipping yourself for whatever possibilities might be.”
  • 17:31 Sean: I think it’s a very natural go-to response because you’re trying to prepare yourself and protect yourself. It’s similar to at a primal level, if you sense imminent danger you get this shot of adrenaline preparing you to up and run. I think this a similar, natural reaction but it may not be the most helpful for you right in that moment.
  • 18:00 There’s two ways you could respond to it:
    • You can imagine every worst case scenario.
    • You can choose to remain calm and operate on what you know now.
  • 18:44 Ben: “There’s a difference between feeling the fear of losing everything and giving it feet. You can acknowledge how you feel but choose to operate based on things that you know instead of things that you don’t know. The power comes when you’re making choices based on things that you know instead of letting the situation control you.”
  • 19:30 Sean: Yeah, the “what if” game will destroy you.
  • 19:37 Ben: “Yeah.”
  • 19:38 Sean: At this stage, you can’t afford to panic. You need your focus now more than ever in order to be able to deal with this crisis. You don’t know yet, but there may potentially be something you can do. There may not, but there may in some circumstance be something you can do right now in the middle of this crisis that could resolve some portion of it. You’re not going to have the presence of mind to recognize those moments if you’re resorting to panic before you even have confirmation.
  • 20:17 Someone in the chat had asked, “What if all the orders are gone? What are you going to do?” Well, it would have been a disaster. I start to think, “People would have paid, they would have received order email confirmations, but there would be absolutely no record of them on the site. No record of the order, no account created, the reports would be off—” No, I had to stop myself! I couldn’t let myself go there. If worst came to worst, I’d be recovering from the damage for weeks if not months. I don’t need to spend any time contributing to that until I’m 100% certain that it’s unrecoverable. It’s wasted energy.
  • 21:07 Ben: “Maybe it is wasted energy, but if I just can’t get over a specific fear, I do that exercise of thinking through the possibility and making a choice about what I’m going to do if that is the case. Once I do that, then I can let it go and I can move on. Sometimes you can get past those things without having to do that, but I think it is also true that sometimes those things are just too strong for you to ignore until you deal with them.”
  • 22:00 Sean: So what would dealing with them look like?
  • 22:03 Ben: “Well I think dealing with them looks like saying, ‘Okay, I’m obviously feeling very strongly or negatively about this possibility.’ If it happens, instead of saying ‘What if?’ saying ‘IF this happens—if this is true—this is my plan. This is what I’m going to do.’ Making a decision about what you will do frees my mind from worrying about the possibility of it happening. At least I know I have a plan. I’ve got something in place to take care of that if it does happen.”
  • 22:48 Sean: I guess that my question is: Where do you draw the line in acknowledging those fears and coming up with your response or plan of action and doing that too much? So it in the face of potential catastrophe, there’s a lot of things that could potentially go wrong. You’re going to have a lot of potential situations. You could go down every single one of those streets and say, ‘What if?’ and come up with your plan of action, but where do you draw the line? At what point is it counterproductive to go down every single one of those roads when it hasn’t necessarily been confirmed yet?
  • 23:36 Ben: “Well, you have to remind yourself that the ultimate goal is to deal with what you can that’s in front of you, based on what you know. Only you can really know whether or not some possibility is something that you can ignore for now so you can focus, and whether it’s really just too overwhelming for you not to deal with what’s in front of you. All I’m saying is I want to give permission to those things. Hopefully it’s just a very few, but those things that feel so overwhelming that you just have to deal with them before you can truly focus on the problem in front of you.
  • 24:18 “I definitely wouldn’t say that you should do that for every single possibility—that would be a waste of time, absolutely. But if the worry of something and your inability to get over that is keeping you from being able to focus your energy and your efforts on what you can do now, dealing with it and moving past it is better than trying to fight a losing battle against it.”
  • 24:51 Sean: I like that. The focus being on what can you do right now? What can you focus on now? What can you actually affect now? That’s the goal. Find that thing. The litmus test is if you have other things that are pressing you so much that you can’t focus on what you need to do now, then acknowledge them—not in a “What if?” sense, but in a “Here’s what I will do,” sense.
  • Dealing With Stress
  • 26:50 Sean: People in the chat are mentioning backup systems. I already have backups. The problem was I didn’t have the key configured properly. Normally, I have daily backups I can revert to at any time. It was just a mishap in this one particular instance.
  • 27:11 Continuing with the different stages of reaction, let’s say worst has come to worst and you’ve lost everything. This is tough. This is a really tough one.
  • 27:33 Ben: “I think I remember you saying something like, ‘If I lost a week’s worth of stuff, I think I’m going to throw up.’”
  • 28:06 Sean: I definitely said that. It’s not a great situation to be in. Fortunately, I didn’t lose everything. But if I put myself in the position of assuming it really was all gone, how would I approach this? How could I break this down?
  • 28:53 I think the first question is: What do you still have?
    • What do I still have have that is of value? Acknowledge the good. Be thankful for what you do have. As difficult as it is in this time, realize that it probably could be worse.
  • 29:28 Ben: “Yeah, you said it—that can be really difficult to do. I don’t think it’s something that you can force either. If you’re not ready to truly acknowledge that and experience some of the relief that could come from thinking about things that way, don’t try to force it.”
  • 30:09 Sean: That’s true. It definitely requires some space from the situation.
  • 30:14 Ben: “You know when you’ve got enough space. You know when you are emotionally in the right place where you can receive that from yourself.”
  • 30:26 Sean: The second thing I would ask myself: What’s the most important thing to rebuild?
    • Prioritize. What has the biggest impact on your life? There’s likely a lot of things that have to be rebuilt, but just like with any task list, if you’re staring at a mountain of things you won’t be able to get any of them done.
    • Break it down. List in order of prominence what needs to be rebuilt.
  • 30:57 For me, if I had lost stuff, podcasts and shownotes are very important to me. They certainly have the greatest impact on the largest number of people, but when I really think about it, I realize that customer orders are much more important. Those are the paying customers. That may be just one person, but their experience is really important. The other people may enjoy consuming my content but they’re just consuming for free. They’re not the ones who have compensated me for a product and then had a bad experience. So I would need to focus on the paying customers and resolving their situations first. It’s a matter of ordering it in prominence. What is the first thing that needs to be rebuilt?
  • 31:53 Ben: “That’s kind of helpful. It almost makes me want to play the ‘What if?’ game—not in a negative way but just imagining if everything was lost. What would I want to rebuild first? Thinking about it when you’re not experiencing a catastrophe can be a neat way to help you prioritize those things in your own mind and maybe even change the way that you’re doing business right now. It can help you be aware of what you might need to give more importance to.”
  • 32:27 Sean: That’s a good piece of advice. It gives you clarity now on what’s important.
  • 32:58 Ben: “It may also serve to help you feel more grateful for the fact that it hasn’t happened to you or that you do have what you have.”
  • Rebuilding Is an Opportunity to Start Fresh
  • 33:32 Again, a lot of these require space from the emotional trauma of the situation. I have to really emphasize the space needed from the situation because otherwise this type of thing would almost seem offensive.
  • 33:58 To use the example of a forest fire, that’s obviously a terrible thing if someone has a house in that forrest and it burns down, but fires are a natural occurrence. It helps reset everything. There are actual, real tangible benefits from fires happening in a natural sense.
  • 34:37 Ben: “Yeah, in the movie Batman Begins, with Christian Bale…”
  • 34:42 Sean: Are we going there?
  • 34:43 Ben: “…his mansion burns down. It’s destroyed. At the end of the movie, Alfred says, ‘We can take this opportunity to build out the East Wing.’ Basically hinting at the idea that instead of having this cave with all of his Batman stuff, he can now expand it and really make it useful and functional.”
  • 35:19 Sean (Hesitantly): I… I don’t know if I’m saying exactly that…
  • 35:21 Ben: “Be like Batman.”
  • 35:22 Sean: That’s exactly what I’m saying.
  • 35:25 I don’t know if you’re like me, but I have this folder on my hard drive titled “From PC”…
  • 35:33 Ben (Laughing): “Yeah, I do have that.”
  • 35:34 Sean (Laughing): I never go back there. I don’t ever want to go back there! In the years since I’ve switched to Mac, I’ve never gone back there. If it was gone, I would have no idea. But it’s on the hard drive and in the back of my mind, I know. My hard drive’s bloated. I’m sure a lot of people can relate. It’s just a mess of stuff. If you get a new laptop, before you dump all of your files on it, something feels nice about it. It’s like the smell of new leather. Maybe you throw on a few files from your current client project, but it’s just what you need. Something about that feels good.
  • 37:44 What can we possibly glean from a terrible situation? Find the teachable moments.
    This is a learning opportunity. Not only for you, but everyone you know through your teaching. It’s what I’m doing right now with this episode. Fortunately, it did not end up being a catastrophic event for me—I was able to recover. But as an illustration, yesterday, I wasn’t planning to talk about data loss on today’s show. But as a result of this, maybe it gets someone to do a backup. Maybe is causes someone to find something positive about a situation where they lost something.
  • 38:56 Ben: “Yeah, we are definitely benefiting from that today.
  • The Importance of Community In Times of Crisis
  • 39:02 Ben: “You mentioned that it was troubling to you when you had to send an email into support since you couldn’t talk to a live person. So you got on Skype with your developer so that you can at least commiserate with somebody. One of the really valuable things that I think may be more obvious to some personality types than others is Community. Find people to share your struggle with.
  • 40:00 “You instinctively reached out to somebody who could understand what you’re going through and understood the terminology behind the things that were going wrong. They could at least be a support for you. I think that is immensely important when you’re going through something. Whether it’s a mild catastrophe or big catastrophe, find people you can talk to who at least can commiserate with you if not help solve the problem.”
  • 40:37 Sean: This is super timely but Eric just joined the Community chat room right now while we’re live streaming. I don’t know if you know, Ben, but Eric has recently gone through a very relevant experience to the topic we’re on today. He had his actual apartment burn down a fire and everything in it. He’s a lettering artist. He lost physical artwork and I believe his computer was also in there along with all of their belongings.
  • 41:18 So he just went through this situation and he came in the chat just a minute ago and says, “Super relevant topic.”
  • 42:04 The reason that I brought up Eric is it reminds us that the things we’re talking about here are not just hypothetical. We have, within the Community, people for whom this is a very real thing. We’re talking about fires happening in the forest and resetting things and providing an opportunity to rebuild—that’s someone’s real situation right now! These are not just flippant words, people are going through the literal definition of the experience we’re describing.
  • 42:40 Wouter is another one of our Community members—he shared Eric’s story. He posted a topic in the Community telling us what was going on, and we banded together to help. We said, “How can we help?” Eric made an Amazon list and also used one of those funding sites to help raise money just to rebuild, and the Community was able to come together and contribute towards that. I just think that’s so cool.
  • 43:27 It’s echoing and backing up what you said about finding people. People that at the very least you could commiserate with but I think it even more than that, you’re going to find that people are more than willing to help.
  • Preventing Catastrophic Failure
  • 43:49 Sean: First of all, can all catastrophic events be prevented? No. I’m not trying to say that. But in my case, I’m usually made aware of things that I could have done better. I’m usually made aware of weaknesses in my system and places that I can prepare.
  • 44:19 Again, this is another thing that requires clarity and distance from the situation where you’re able to assess what you can do better in the future. I also want to make a subtle distinction here that I think is really important:
    • Could have done = past-focused
    • Will do better = future-focused
  • 44:32 The phrase “could have done” is past-focused. I don’t think that’s very productive. I would rather say “will do better.” It’s future-focused, not living in regret. Not thinking about things you “should have done but can’t change now.” Instead, it’s a more purposeful reflection: What will you do?
  • 45:05 Ben: “Yeah, it’s completely useless to say those words to yourself. There’s no use for them whatsoever.”
  • 45:25 Sean: It’s a downward spiral. It just makes you feel worse and worse.
  • The Two Voices
  • 45:32 Ben: “Rachel was sharing something with me the other day that I really liked. She said there are these internal voices that guide us and lead us to action or lead us through different emotions or thought patterns.
  • One voice comes from up in the throat and it’s kind of negative. Then there’s another voice that comes from down in your gut and it’s more positive.

    If you can’t tell which voice is which, then ask yourself this: Is it stealing life away from me? Or is it giving me life? Is it moving me in the right direction?

  • Backups
  • 49:24 Sean: I had systems in place for backups, but I wasn’t regularly checking to see if they were working—which defeats the whole purpose! You have this supposed peace of mind, only to find out it had been three months since the last backup. If I had checkpoints every two weeks or even once a month just to make sure it was working, that could have saved me if something had gone worse than it did.
  • 51:48 You don’t know what you have until it’s gone.
  • 51:54 Just imagine that you lost everything. All of your data is gone. Your website is gone. It’s just gone. What would you pay to get it back?
  • 52:17 Follow-up question: Why are you not paying a fraction of that amount now to prevent it?