Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
2. Yes, I have been fighting the CSS for many tests now. Thought I had it for every possibility, but some were still haywire. I believe the next one will actually be correct. (actually, I sent a second email immediately after to everyone, with corrected styling, I hope).
3. See new attachment below, this will be our new format going forward. It is ambiguous, without "register now" or "recording!". I think this will work.
It seems you are blocking Twitter and ads. So that's two elements of the home page that are designed to be in place, which is probably why it looks weird.
Still, I do plan to make the home page the first priority for being responsive HTML (eliminating tables) when I can.
Wow. What a difficult 25 hours it's been. Here is the instant replay from Twitter and our down message:
April 4th 2021 approx 5:30PM ET: Our data center experienced an electrical outage. Generators ran as expected, but then an anomaly occurred which led to damage. All servers went off-line.
11:30PM ET: Just received an update. The electrical problem was severe. Fire suppression systems kicked in. Right now, the data center still has no power, because they are waiting for an emergency inspector to clear the systems and building.
The data center has advised us that many systems have extensive water damage due to the fire suppression systems. We are not sure if our servers do, or don't.
If we choose to restore from off-site backups, we lose 17 hours of database sync (posts, uploads, new users, memberships, everything past midnight until 5pm). In our history of 11 years, we've never lost a single post, much less 17 hours of content. Our primary backup server was on-site in the same data center, with minute-by-minute snapshots of our database.
I'm making the decision to "hold" for tonight, get some sleep, and wait for more news from the data center in the morning. Hopefully, specific news regarding our servers and their status, at least the 8TB NVMe storage array that holds our database and files. Even if the rest of the server is lost, possibly that component could be salvaged.
April 5 morning: Data center isn't providing much useful info. Sounds like they are still waiting for the inspector. I have begun the process of restoring from an off-site backup.
April 5 afternoon: Off-site restore progress: database servers 100%, email 100%, app servers 100%, web servers 100%, caching servers 100%. The problem is the file server has a huge amount of file attachments (screenshots, indicators, etc) and the site cannot be brought on-line without them. Currently estimating 7pm Eastern US.
---------------
7:30 PM Eastern: Site is on-line.
Let me discuss what has been LOST.
Our off-site backups were only up until midnight April 4. New user registrations, posts, Elite memberships, uploads, etc - made after midnight on April 4, and until approx 5:30PM April 4 when the site went down due to the data center fire, have been lost.
In more than 11 years, we've never lost a single post. It pains me that we lost 17 hours of data, but there was simply no way around it.
Our primary and our secondary backup servers were both in the same data center, but on different hardware. This backup was minute-by-minute (real-time without the performance loss).
Unfortunately, our off-site was only nightly at midnight. So in the event of a literal FIRE, like this, we lost that data.
I will be updating our disaster recovery to make it minute-by-minute off-site as well soon.
RIGHT NOW: We are running at a new data center, on new hardware. The site will be a bit slower until we can figure out a long term solution, please bear with us. If you find any problems, please report them here.
Thank you everyone, for your patience during this, and your understanding.
a) I have reached out to the people that became Elite Members during that period, and have already manually upgraded them
b) I tried to reach out to people who created new user accounts in general (to the site), but due to a completely unrelated problem that just so happened to coincide with yesterday, I don't have their email addresses
c) The data center will "evaluate" our servers and see what condition they are in, supposedly in two days. When this happens, if the database and file system are mountable, then we CAN view some meta details such as:
-> posts
-> new users
-> direct messages
However, it will be impossible for us to "restore" these items and merge them into the live production system. Once we made the decision to stop waiting for the data center, there was no turning back.
We can, however, find out the names of the people involved with these items, and reach out to them, and ask them to resubmit the content.