Xyon
Puts the Fun in Dysfunctional
- Joined
- Aug 9, 2009
- Messages
- 6,925
- Reaction score
- 793
- Points
- 203
- Location
- 10.0.0.1
- Website
- www.orbiter-radio.co.uk
- Preferred Pronouns
- she/her
As pretty much everyone will have noticed, the forum was closed twice in quick succession this week for a couple of maintenance windows. The reason given was to "address some CVEs in critical system components", or something along those lines.
Here's what I did, and why, and why we've had so many bumps along the road. I've tried my best to keep the forum error free, to the extent that I've been elbow deep in the board's PHP code a couple of times this week - sometimes with disastrous results, I don't mind admitting. :lol:
To begin in the beginning, first, I've had my eye on a few of the more difficult elements of forum maintenance for some time. Our system runs on Digital Ocean, and has done since our migration six years ago. For the entirety of that time, it had run on a single Debian droplet, which I did not perform major updates on, preferring instead to do a migration to a new droplet with the newer major version on it instead. Various planning concerns stopped those migrations from happening - notably some worries I had about database consistency - so the board continued on the original droplet, with package updates, for the entire six year period.
Until, that is, this week. During the run-up to the first downtime period, I generated a new CentOS 7 droplet with Digital Ocean, and restored a forum backup to it (something I do often to test the restore, actually - a backup is only as good as its last successful restore, after all!). Then, when the maintenance window was deployed, I took the current database state and laid it over the top of the restored forum, effectively migrating the site. Digital Ocean offer floating IP addresses, so there was no DNS change necessary - from the end-user's point of view, things should have been seamless. Regardless, I did this three times ahead of the event on my own kit, testing everything I could think of in the restored boards.
The original Debian droplet ran on PHP 5.4. The new CentOS one ran PHP 7. So, the first task was to update the board to the latest version of vBulletin 3.8, 3.8.11, which supports PHP 7.1. As I quickly discovered, though, not all our plugin code supports PHP 7, and I was quickly forced to bring the droplet back to PHP 5.6 to make the thing run at all.
The update to 3.8.11 went perfectly; there were no issues with the upgrade itself. The plugins, however, suffered from a series of issues related to obsolete function calls, and in a lot of cases they relied on methods in vBulletin 3.8 which had had their signatures changed. There are precious few of the plugins we use that are still maintained upstream; I had to manually change several hundred lines of PHP code to solve all the HTTP ERROR 500 code responses - notably in the blogs and the project tools areas.
So, now that it's all done, what have we gained? Well, the new droplet is faster, has more resources, and runs a kernel patched to protect against cve-2019-11477, a particularly nasty looking vulnerability related to TCP SACK requests and their ability to cause a linux server to kernel panic, if done in theright wrong way.
Next on the list are some fixes / code updates to OHM, and then the long, slow process of updating all the plugins to make the forum work under PHP 7.
My apologies, however, for the amount of disruption the work has caused - several elements of this journey were unpleasant surprises, which they should not have been given the level of testing performed prior to the work being carried out. I've done what I can to get things working perfectly once more, and I hope your forum experience continues to be the best despite my handiwork.
Cheers!
Here's what I did, and why, and why we've had so many bumps along the road. I've tried my best to keep the forum error free, to the extent that I've been elbow deep in the board's PHP code a couple of times this week - sometimes with disastrous results, I don't mind admitting. :lol:
To begin in the beginning, first, I've had my eye on a few of the more difficult elements of forum maintenance for some time. Our system runs on Digital Ocean, and has done since our migration six years ago. For the entirety of that time, it had run on a single Debian droplet, which I did not perform major updates on, preferring instead to do a migration to a new droplet with the newer major version on it instead. Various planning concerns stopped those migrations from happening - notably some worries I had about database consistency - so the board continued on the original droplet, with package updates, for the entire six year period.
Until, that is, this week. During the run-up to the first downtime period, I generated a new CentOS 7 droplet with Digital Ocean, and restored a forum backup to it (something I do often to test the restore, actually - a backup is only as good as its last successful restore, after all!). Then, when the maintenance window was deployed, I took the current database state and laid it over the top of the restored forum, effectively migrating the site. Digital Ocean offer floating IP addresses, so there was no DNS change necessary - from the end-user's point of view, things should have been seamless. Regardless, I did this three times ahead of the event on my own kit, testing everything I could think of in the restored boards.
The original Debian droplet ran on PHP 5.4. The new CentOS one ran PHP 7. So, the first task was to update the board to the latest version of vBulletin 3.8, 3.8.11, which supports PHP 7.1. As I quickly discovered, though, not all our plugin code supports PHP 7, and I was quickly forced to bring the droplet back to PHP 5.6 to make the thing run at all.
The update to 3.8.11 went perfectly; there were no issues with the upgrade itself. The plugins, however, suffered from a series of issues related to obsolete function calls, and in a lot of cases they relied on methods in vBulletin 3.8 which had had their signatures changed. There are precious few of the plugins we use that are still maintained upstream; I had to manually change several hundred lines of PHP code to solve all the HTTP ERROR 500 code responses - notably in the blogs and the project tools areas.
So, now that it's all done, what have we gained? Well, the new droplet is faster, has more resources, and runs a kernel patched to protect against cve-2019-11477, a particularly nasty looking vulnerability related to TCP SACK requests and their ability to cause a linux server to kernel panic, if done in the
Next on the list are some fixes / code updates to OHM, and then the long, slow process of updating all the plugins to make the forum work under PHP 7.
My apologies, however, for the amount of disruption the work has caused - several elements of this journey were unpleasant surprises, which they should not have been given the level of testing performed prior to the work being carried out. I've done what I can to get things working perfectly once more, and I hope your forum experience continues to be the best despite my handiwork.
Cheers!
