[admin mode] completed some maintenance tasks on the database(hence the inaccessibility today), it shrank the database a shocking amount so I'd call that a win. Should improve performance, I've still got more to do later but it's fine for now.
- replies
- 0
- announces
- 0
- likes
- 2
The database cleanup was just running a couple of built in pleroma scripts that clean up old data (cleaning out 25GB of remote posts!), but the real issue seems to have been a number of faulty packets with invalid addresses being aimed at fbxl social in particular (once I stopped accepting packets for fbxl domains the server came back and once I started again)
My hypothesis so far is that either intentionally due to malicious attack or unintentionally due to a misconfigured server a bunch of these malformed packets were sent my way, and filled up connection slots in the kernel, locking up not just http but telnet and icmp. Once I added some configuration changes to increase the number of connection slots and also to filter any packets with bad addresses (sysctl calls them martian packets since they come from "alien" address spaces) the connection issues ceased.
My hypothesis so far is that either intentionally due to malicious attack or unintentionally due to a misconfigured server a bunch of these malformed packets were sent my way, and filled up connection slots in the kernel, locking up not just http but telnet and icmp. Once I added some configuration changes to increase the number of connection slots and also to filter any packets with bad addresses (sysctl calls them martian packets since they come from "alien" address spaces) the connection issues ceased.
