I suppose that folks may have noticed a change in forum performance.
The reason for this is that I have diagnosed the occasions when the forum locked up (I hope). What was happening was that the number of clients exceeded our ability to serve them with the way we were configured.
Each apache thread consumes between 20 and 160 MB of memory depending on what the user is doing, especially when viewing larger files. We were configured to support up to 150 concurrent server threads which, depending on the specific usage pattern at the time, would cause threads to be waiting on memory since we were badly over-committed. This is also known as thrashing. I reduced the number of concurrent threads to prevent this situation. A side effect of this change is that initial page load times may increase depending on the number of concurrent users, the limited number of threads in the pool mean that folks may have to wait until one completes what it was doing for some earlier request.
We are planning to move to a server in a week or two that has quadruple the memory of our current virtual machine. When that happens we can safely increase the thread count and database key cache and make the site appear to be more responsive.
We had a database failure that caused many private messages to be lost. Database backups were so slow and so memory intensive that they could not complete in less than about 25 minutes. The backups were 15G and much too large for our little machine. To help ease this problem, I moved image attachments from being stored in the database to being stored in the file-system, which reduced the total backup size to 1.5G and the backup time to about three minutes. There is another database table that can be treated in a similar fashion which may reduce the backup size by another Gigabyte. I plan on doing this in the next couple of days.
We have taken steps to reduce the number of invalid login attempts by folks attempting to hack your user accounts. The incidence is now one every few days, down from several per day.
Additional security holes are being closed today should things go as planned as we update some of the forum code.
On the docket for further work is to update the gallery software, then upgrade the site to vBulletin 4, and then to migrate to the new server with double the memory and ssd-based disk storage. The new server is already up and I am using it for tests prior to cut-over.
All of this explanation is to beg your forgiveness of some of the issues and down-time we have had. There will be more at each step of this process as we attempt to make this site more robust and modern.
we apologise for all of the inconvenience