[admin mode] on the 31st we were up and down like a yoyo, I did send a post talking about one of the measures I took but the downtime continued to occur afterwards.
At first it looked like we were running out of connections again so I assumed it was a ddos attack and activated mod_evasive. The problems continued. Next, I added more lines to sysctl.conf to further reduce bad packets getting in, but that didn't help either.
My final clue was that the uptime was only a few minutes, suggesting that the server was actually rebooting. It only did so when the fbxl DNS was set correctly, so I decided to think about what that could mean, and I decided it could be that my rebased back-end was maybe getting overloaded to the point it was triggering a reboot. I went in and reduced my worker threads from 20 to 4. After that the reboots stopped and everything has been rock solid since. I think federation updates have been marginally slower but safe to say given a choice you'd prefer a working site that's normal speed over a non-working site that's lightning fast.
I intended to write something earlier but I wanted to wait until I felt confident everything was working to report on the outcome.
At first it looked like we were running out of connections again so I assumed it was a ddos attack and activated mod_evasive. The problems continued. Next, I added more lines to sysctl.conf to further reduce bad packets getting in, but that didn't help either.
My final clue was that the uptime was only a few minutes, suggesting that the server was actually rebooting. It only did so when the fbxl DNS was set correctly, so I decided to think about what that could mean, and I decided it could be that my rebased back-end was maybe getting overloaded to the point it was triggering a reboot. I went in and reduced my worker threads from 20 to 4. After that the reboots stopped and everything has been rock solid since. I think federation updates have been marginally slower but safe to say given a choice you'd prefer a working site that's normal speed over a non-working site that's lightning fast.
I intended to write something earlier but I wanted to wait until I felt confident everything was working to report on the outcome.
- replies
- 0
- announces
- 0
- likes
- 1