Hi guys

I don't know if it has anything to do with the crash but the latest recorded webinar:



is the same as this one



if you click it.

Kind regards

beske

Big Mike 

Hey guys, Wow. What a difficult 25 hours it's been. Here is the instant replay from Twitter and our down message: Mike

Can you share the vendor?

beske 

Hi guys I don't know if it has anything to do with the crash but the latest recorded webinar: is the same as this one if you click it. Kind regards beske

Thanks for reporting. @bobwest already had me fix it once, lol, but yes this "fix" was lost during the crash and had to be re-applied. Good to go as of now.

Tyro 

Can you share the vendor?

WebNX.

OK,

The top of threads "thread summary" is now being updated again correctly.

The journal challenge calculator is now functioning as well, so the "Can you help ..." section in threads (below top post in a thread) is replaced with the Journal Challenge calculator for April's challenge.

Side note, the April Journal Challenge has been extended one week to May 7, due to the data center fire and delay earlier this week.

Mike

Hi guys,

There has been a problem with our newsletter system since we had to migrate (forcefully) to a new temporary server.

I have a work-around in place as of right now.

I do believe that a small number of users, around 100, probably received many duplicate emails today as I was doing testing before realizing what was happening.

Our newsletters always send in an order based on who has most recently logged in, meaning the fists people to receive our emails are always the users that most recently visited our site.

Apologies if you were one of these ~100 that received many duplicates. I didn't realize that a tiny % of emails were actually working, I though it was not working at all.

OK, so in other news, newsletter is going out now. Slowly. This temp server is nothing compared to our last server...

For those interested in these details -- I am currently doing my due diligence to find a new data center, and then have a new server built for us.

I do hope that within the next 2 weeks, we can migrate to a new server (again) that is equal or better in terms of performance to what we had previously.

Mike

Mike and crew.... what an amazing job recovering the site....Thank You very much.

Bob

Big Mike 

Hey guys, Wow. What a difficult 25 hours it's been. Here is the instant replay from Twitter and our down message: April 4th 2021 approx 5:30PM ET: Our data center experienced an electrical outage. Generators ran as expected, but then an anomaly occurred which led to damage. All servers went off-line. 11:30PM ET: Just received an update. The electrical problem was severe. Fire suppression systems kicked in. Right now, the data center still has no power, because they are waiting for an emergency inspector to clear the systems and building. The data center has advised us that many systems have extensive water damage due to the fire suppression systems. We are not sure if our servers do, or don't. If we choose to restore from off-site backups, we lose 17 hours of database sync (posts, uploads, new users, memberships, everything past midnight until 5pm). In our history of 11 years, we've never lost a single post, much less 17 hours of content. Our primary backup server was on-site in the same data center, with minute-by-minute snapshots of our database. I'm making the decision to "hold" for tonight, get some sleep, and wait for more news from the data center in the morning. Hopefully, specific news regarding our servers and their status, at least the 8TB NVMe storage array that holds our database and files. Even if the rest of the server is lost, possibly that component could be salvaged. April 5 morning: Data center isn't providing much useful info. Sounds like they are still waiting for the inspector. I have begun the process of restoring from an off-site backup. April 5 afternoon: Off-site restore progress: database servers 100%, email 100%, app servers 100%, web servers 100%, caching servers 100%. The problem is the file server has a huge amount of file attachments (screenshots, indicators, etc) and the site cannot be brought on-line without them. Currently estimating 7pm Eastern US. --------------- 7:30 PM Eastern: Site is on-line. Let me discuss what has been LOST. Our off-site backups were only up until midnight April 4. New user registrations, posts, Elite memberships, uploads, etc - made after midnight on April 4, and until approx 5:30PM April 4 when the site went down due to the data center fire, have been lost. In more than 11 years, we've never lost a single post. It pains me that we lost 17 hours of data, but there was simply no way around it. Our primary and our secondary backup servers were both in the same data center, but on different hardware. This backup was minute-by-minute (real-time without the performance loss). Unfortunately, our off-site was only nightly at midnight. So in the event of a literal FIRE, like this, we lost that data. I will be updating our disaster recovery to make it minute-by-minute off-site as well soon. RIGHT NOW: We are running at a new data center, on new hardware. The site will be a bit slower until we can figure out a long term solution, please bear with us. If you find any problems, please report them here. Thank you everyone, for your patience during this, and your understanding. Mike

There has been a problem with mail for the last ~12 hours on the site. I've corrected it now. This is a gremlin left over from the data center fire...

Mike

Hi guys,

I've been working to refine our new webinar experience...

What should happen is:

- Around 1 hour prior to webinar starts, you receive an email
- If you click the link to join, you are sent to the FIO waiting room
- You see a live countdown timer for the webinar
- You can chat using our FIO chatbox
- You are reminded how to use the Zoom Q&A feature to ask questions in the webinar

When the webinar switches to live status (Shane makes the event live), then everyone in the waiting room will instantly be transported into the live Zoom event (sent to the Zoom software).

Mike