Starting sometime on Wednesday 18th Jun, the forums started to show some problems.
Initially individual threads could not be read, resulting in a number of posts whose titles could be seen but whose content wasn't accessible. Several "what's happening?" posts can still be seen in News. The next day, the list of threads in a forum also became unviewable, and late Saturday the site was locked entirely.
We've had hacks before, causing our hosting provider Lycos to take action, but that's not the problem this time.
It looks like Lycos had instigated some new kind of sweep to check for "insecure files", which is basically broken and flags perfectly safe files as being a problem. The software we use for the forum is well-established and used on 100,000s of sites worldwide; it has no currently known vulnerabilities. The problem progressed as it did because each night Lycos found a new "insecure file", made it unreadable (so breaking whichever part of the forum functionality it represented) and escalating the warning to the customer. At the third alert, the site was locked. Our queries and reports - in some considerable detail - to customer support were treated in typical unhelpful manner. They assured us that the site would not be locked if we followed their instructions, which we did despite having no explanation for what was going on.
(if I've missed anything, please update me - I didn't read the forums on the first day of problems so haven't seen all the symptoms first-hand)
The more paranoid/cynical may now remove their tinfoil hats. A limited-edition range of commemorative designs will be available shortly in the foyer.
-jmb
Forum - and site - outage, June 2008
Moderator: Site Administrators