Add news
March 2010April 2010May 2010June 2010July 2010
August 2010
September 2010October 2010
November 2010
December 2010
January 2011
February 2011March 2011April 2011May 2011June 2011July 2011August 2011September 2011October 2011November 2011December 2011January 2012February 2012March 2012April 2012May 2012June 2012July 2012August 2012September 2012October 2012November 2012December 2012January 2013February 2013March 2013April 2013May 2013June 2013July 2013August 2013September 2013October 2013November 2013December 2013January 2014February 2014March 2014April 2014May 2014June 2014July 2014August 2014September 2014October 2014November 2014December 2014January 2015February 2015March 2015April 2015May 2015June 2015July 2015August 2015September 2015October 2015November 2015December 2015January 2016February 2016March 2016April 2016May 2016June 2016July 2016August 2016September 2016October 2016November 2016December 2016January 2017February 2017March 2017April 2017May 2017June 2017July 2017August 2017September 2017October 2017November 2017December 2017January 2018February 2018March 2018April 2018May 2018June 2018July 2018August 2018September 2018October 2018November 2018December 2018January 2019February 2019March 2019April 2019May 2019June 2019July 2019August 2019September 2019October 2019November 2019December 2019January 2020February 2020March 2020April 2020May 2020June 2020July 2020August 2020September 2020October 2020November 2020December 2020January 2021February 2021March 2021April 2021May 2021June 2021July 2021August 2021September 2021October 2021November 2021December 2021January 2022February 2022March 2022April 2022May 2022June 2022July 2022August 2022September 2022October 2022November 2022December 2022January 2023
News Every Day |

Site Downtime

Today was a nightmare.  I was up super early (right at 5AM – thanks bladder), but made a tactical error (insofar as sleep goes).  I looked at my phone.  I had a notification on Mastodon telling me that my personal blog story was unreachable with a 404 error.  OH NO!  I instantly was jerked awake, and sprung to action (at 5AM).  Discovered that quite a lot of stuff was down.  Decided to write a story ab  out what happened (given that’s what I do), so if you want to read my horror story, check this out:

My web server has a total of 9 domains on it. Of the nine, sox of them were down.  I’ve had this happen before, usually rebooting the server will fix it. So I did that, and it didn’t fix it.  Uh-oh.  I dug into it deeper and discovered that two major problems had occurred.

  1. My server’s hard drive filled up
  2. The MySQL server crashed

Both of which would prevent websites that used WordPress from working, as they require the MySQL database behind them to be running, and with the drive full and databases down, nothing would work.  Several of my sites were down.  No worries, I have backups for that.

There are two kinds of backups made on my server daily.  The first is a full server backup, I get a tarball of each of the domains, the theory being I could restore the entire server from that.  In fact, in prior server issues, I’ve used these full server backups before.  It worked for the other domain names today – but not the Rangers site.  I keep a weeks’ worth of this type of backup.  Problem is the three most recent backups (including today’s) were corrupt, they didn’t have a backup of the Rangers site (again, due to drive filling up).  That left four backups going back to a week ago.  I restored all of them.  NONE of them had the database.

It sent me into a MAJOR panic, as without the database, my site was gone.  With no database, all the files on the server were pointless as this is all driven from WordPress. This Rangers site has been around since 1998, and while I don’t do the daily updates anymore, there’s a lot of archives here, and a few bits I still maintain.  I’m very proud of the uniform number coverage, so the very real possibility this morning of having lost all that work was seriously distressing to me.

So OK, that brings me to the second kind of backup.  All my sites that use WordPress (of which this is one) do a nightly backup via a software plugin called BackupBuddy. The problem is that BackupBuddy requires you to have database access to use it, and without that, I couldn’t use that backup, so I thought I was screwed since the other kind wasn’t working.  In this section of the recovery, I discovered that my root problem is that when my system filled up it was doing a sql dump, and it appears that the drive full situation while accessing SQL caused a mess which meant that most of my domain names that used databases lost their access to the data – even if it was still there.  I had to reset the passwords on everything, and then it let me clean up the right way.

Fortunately this cleanup allowed me to reinstall my Rangers site from scratch.  I had to pretend WordPress was a virgin install – it basically reset the database.  Given my old WordPress install was actually still THERE, just unreferenced because of the new database, I was then able to restore from one of the existing BackupBuddy backups. It took some time to restore everything (as the tarball for my Rangers site is about 3.18 Gb).  But EVENTUALLY I got everything going.

I didn’t declare everything restored until about 2:20PM on Monday afternoon, and I first discovered the problem right at 5AM.

So now what?  I cleaned up all the mess, the drive is back to about 50% usage, which is where it normally lives with the week’s worth of backups there, but I need to make sure this doesn’t happen again.  the full server backups I will be going to remote backups. I used to, but Dropbox decided some time ago to do away with their command line backup option.  I discovered today in fixing everything that the server maintenance software I use has an option to backup to Google Drive, something I will be signing up for.  Will cost me $75 a year to do that, but I think the cost is worth it for piece of mind.

Bottom line is that everything is working here again.  It was all due to to me losing database access due to a drive filling up situation.  What a mess.  Serious stress bomb this morning too.  Normally I take my blood pressure every morning at home, my wife and I figured this was a good day to skip that today, there was no way I was going to get a good reading today. I will tonight now that everything’s running again.

The moral of the story.  Don’t let a database heavy web server fill up.  All kinds of chaos ensues when it happens.



How to Fall Asleep in 60 Seconds (OR LESS)

EASY chili recipe for dinner #food

New Honda Civic Type R review: Is it really better?

Masturbation is KILLING Your Strength! (NEW RESEARCH)

Read also

Obi not quitting presidential race, LP insists

Holocaust Memorial Day: What is it and how many lives have been lost to genocide?

Power surge blamed for countrywide blackout

News, articles, comments, with a minute-by-minute update, now on

News Every Day

How to deal with rejection — latest news 24/7. You can add your news instantly now — here

News Every Day

Masturbation is KILLING Your Strength! (NEW RESEARCH)

Sports today

Новости тенниса
Большой шлем

Полиция на Australian Open задержала гражданина США за футболку с символом Z

Спорт в России и мире

Прежняя полноценная жизнь: каковы шансы у Романа Костомарова на выздоровление

All sports news today

Sports in Russia today


Звезды ММА и шоу-бизнеса примут участие в Гран-при «Нашего Дела» в Москве

Новости России

Game News

Bodybuilder Fighting Club 1.4.8


В Москве задержали главного "айтишника" таможни – источник

Губернаторы России


Mash: у певца Валерия Меладзе массово отменяют концерты в городах России

Ида Галич пожаловалась на медицину за границей

В МИД заявили, что Россия не ждет улучшения отношений с США

Рукописи не горят… Предыстория издания II тома В. Серошевского «Якуты»

Moonberry Jam выступит с Дмитрием Носковым в джаз-клубе Игоря Бутмана

Ольга Бузова похвасталась, что уже в 3 года начала учить английский

Эксперт назвал реальную цену дома из клипов Моргенштерна

Ани Лорак решила отказаться от Русского

Прямая трансляция матча Рыбакина – Азаренко в полуфинале Australian Open

В ФТР оценили шансы Хачанова в полуфинале Аustralian Open

Зинченко рассказал, как мотивировал партнеров после перехода в «Арсенал»

В Томском ТЮЗе началась репетиция спектакля-променада по творчеству Волкова

Эксперт Михаил Александров: МИД РФ упрекнул Армению в нелояльности, забыв о своих собственных обязательствах

Мособлдума приняла закон об объединении двух наукоградов с Серпуховом

Стали известны подробности госпитализации вдовы Кобзона в Москве

Вильфанд сообщил о погоде в Москве в начале февраля

В Архангельске прошёл XI Кубок России по хоккею с мячом среди ветеранов

Эксперт Михаил Александров: МИД РФ упрекнул Армению в нелояльности, забыв о своих собственных обязательствах

По стопам Роберта Паттинсона

Студентка из Краснодара поинтересовалась у Путина, возьмут ли ее на работу в аппарат президента после окончания университета

Подмосковные коммунальные предприятия выполнили 70 процентов заданий на противогололёдную обработку

В детской поликлинике на Лосевской закончили работы по демонтажу оконных стеклопакетов


Экс-схиигумену Сергию дали семь лет колонии за разжигание ненависти и призывы к самоубийству

Путин в России и мире

Персональные новости

Ольга Бузова

«Я хочу Jony!» На съемках шоу «Конфетка» Ольга Бузова сделала неожиданное признание

News Every Day

Masturbation is KILLING Your Strength! (NEW RESEARCH)

Friends of Today24

Музыкальные новости

Персональные новости