04/26/2018 (Thu) 07:10:34
I messed up during the migration. While waiting for the final database copy, I thought I'd upgrade/patch some servers, unfortunately I upgraded the box I was copying from. And I rebooted it without thinking about it. Now I manage around 20 servers that run hundreds of containers, so keeping track of everything isn't as easy as if I just had a couple. Also in my day job I get paid to professional move websites all the time. I make good money doing it, so I'm not a complete newb but I have no accounting for how I made such egregious error. I had a plan and just made a mistake.
Last year when we got the dedicated server, we built state of the art encryption into it. To the normal internet it looks like a website (unlike anything like Endchan) but with a special certificate our frontend server have, it runs the Endchan site. We split the drive into 3 sections, the camouflage (fake site / web server and relay), the app (lynxchan, megud, other site utilities), and the database. Then we encrypted the app and data drives. I held the keys to app, and Balrog the keys to data. We did have the foresight to give a back up of these password to other people just incase one of us got hit by a bus. Well Balrog and his backup StephenLynx both failed to find the correct password from a year ago.
So the two events combined means we can't access the latest copy of the Endchan data. So where does that leave us, well we had a full copy from 4/19 that we were syncing with the live data during the migration time. When it stop copying, it left the database in an inconsistent state. The mongo repair tool has done a great job recovering what you see;
we have most of posts but are missing a lot of database files (a great example of why you never store images in a database; we should only be missing the new files from last week on).
I fucked up, Balrog fucked up, and StephenLynx doesn't even remember us giving him the password. If just one of 4 events had gone another direction, we would not have been here.
I proposed throwing in the towel. My recent battles with pedospammers has not been fun but Balrog put it well: "we're still one of the only imageboards worth a damn and people need us goddamn it"
I considering just running the board with no content, even thought about switch imageboard software. Balrog said we should take it to a vote and let you guys decide. But to be honest we recovered more than I was expecting and the lynxchan repair process (rebuilding HTML files) is still running (going to take a long time on these older servers).
To be clear why we migrated, our donations levels were so low, we could no longer afford that server. Our previous server was awesome, a full year of uptime (Another reason why we never needed these passwords because the machine never went down).
In retrospect our set up was probably more complex than needed but it was just an attempt to help secure you're guys data the best we could. Our new drives are no longer encrypted.
Message too long. Click here to view full text.
Edited last time by odilitime on 04/26/2018 (Thu) 17:57:36.