Changes to forum structure

[Dark0ne]

As you might know we've been struggling with our forum server after the past few weeks and have been working towards getting them on the same Cloud setup that's worked so well for us with the main sites themselves.

 

However, we'd struggled to understand why, all of a sudden, the load times on the forums went from fine to absolutely awful in little to no time at all. There wasn't any sort of traffic spike and the servers weren't being constantly DDoS'd so there was no real reason for it. We knew it wasn't our database because the forums are running on the same database cluster that the sites are using; so the sites would have been slow as well.

 

We came to the conclusion that it must be related to how heavily we've integrated our forum software, Invision Board, with the sites themselves. All membership information, profiles, file comments and discussions and bug reports, image comments, video comments and Premium Membership information is handled and called through Invision Board and each time we add a new game to the site we were creating 17 new forum categories to support it, to the point where we had close to 3,000 forum categories on the forums. It seems Invision Board doesn't play nice with growth like that. At all. For the technically minded, Invision Board seems to store things like your read topics for every single forum inside variables before every single page load, so if there's 3,000 forums, that's a lot of server resources being used on each page load.

 

We've been experimenting with a number of different tweaks and changes to the forum servers over the weekend. Because it was the final weekend of the Steam Summer sale we once again saw record numbers of users on the sites (36,000 new members this weekend alone and 7,500 page views a minute). We were very pleased to see the sites held excellently during this time with page load times well within normal ranges and the traffic helped us to not only benchmark potential issues with the sites in light of a Fallout 4 release that's not too far away but also to analyse what would help relieve the load on the forum servers.

 

Today we've made some relatively large changes to the forum category structure. Most notably, almost all games on the network now only make use of a single forum for all discussion (with the exception of the big games like Skyrim, Fallout 3 and so on). We've done this not because we particularly want to but because the forum software itself cannot handle having 3,000 categories (and growing each time we add a new game to the network). All file, image, video and article comment topics now take up a single category each (as opposed to a category for each game on the network) and all the old topics and posts for files already uploaded have been moved to these single categories. While you can still access the "forum view" mode for file and image comments you will no longer be able to access the file and image comment forum category directly. Because all the files and image comment topics for all games on the network are now mingled together in a single forum category it seems pointless to open up the category for viewing; you'll never be able to find what you're looking for by browsing through that category anyway.

 

Making this small change has not only reduced our forum category count from 3,000 to 1,200 categories (and we haven't finished doing spring cleaning yet which will likely drop that figure to below 1,000) but it's drastically (DRASTICALLY) reduced the load times on the forum server. Essentially, the forums are now back to being quick and, in fact, better than ever.

 

We'll now manually add more forums (like forums for "mod requests") as and when they are needed. As it is, a lot of the games we support have no need for forums here and they're added automatically as a token gesture. They might be used, they might not. Point is, they're there, and we can always add more categories for popular games as and when they're necessary.

 

We've still got some spring cleaning to do and we're still going to go ahead with moving over to the cloud setup for the forum server. The cloud setup has showed its worth to us many times already, but case-in-point, this weekend the NMM virtual machine that handles logins and retrieving metadata for the Nexus Mod Manager became overloaded due to the sheer number of requests being made of it. As soon as we found out about the issue we were able to double the RAM and CPU cores available to the virtual machine, on the fly, within 5 minutes. Problem gone.

 

Aaaaaand, as I type this, I've just been informed of a DDoS taking place against our servers. It's being looked in to as we speak and I can't speak highly enough of our friends (well, actually my friends, two of whom I lived with in Reading after university and one of whom is a childhood friend from my early secondary school days) at Krystal.co.uk who handle a lot of our hosting requirements for the sites and are always there for the sites whether it's 3pm or 3am if we're having issues. If you're a Brit who needs hosting or has a crap hosting provider and is looking for reliable hosting with great service you really can't do better than these guys. And I wasn't paid for that little advertisement, they just deserve the exposure for the times they're there for the servers, fighting off the kiddies out there who want to ruin it for everyone else (they were also up at 3am on Friday morning helping us to bring back the forum server that decided it didn't want to wake up from a restart).

 

[setiweb]

As always thanks for the update! :)

[AurianaValoria1]

Thanks for the heads-up! :)

[Deleted54170User]

Thanks for the Update.

 

In kind I'd like to say thanks to you and your friends for being persistent in taking on the challenge of our growing Cyberspace population.

[MotoSxorpio]

Thanks for the summary.

 

Doesn't actually seem like there was or is a DDOS attack, as I logged in and began surfing two minutes before this thread was created. I had 6 threads open from notifications and made two replies within minutes (timer allowing). So, I'm thinking your forum category solution is working very well.

 

Thanks again for keeping us up-to-date.

[JimboUK]

It must have been a pretty sad and sorry DDoS, I didn't notice anything slowing. It's a shame about the forum, the forum side for files was good for searching mod comments when faced with a problem, preferable to wading though pages on the site itself.

[Dark0ne]

In response to post #26349034.

jim_uk wrote: It must have been a pretty sad and sorry DDoS, I didn't notice anything slowing. It's a shame about the forum, the forum side for files was good for searching mod comments when faced with a problem, preferable to wading though pages on the site itself.

If you visit the forum thread for the file in question then you can still search within the thread, if that's what you mean?

[M48A5]

I have read the latest notification of the changes being made to the nexus and even after re-reading it, I am still confused.

 

Is the thread "(Game Name) Uploaded files" being discontinued?

 

Having looked in Skyrim, Fallout 3, Fallout New Vegas and Obilivion, the Uploaded files section is missing. That section was one that I checked religiously everyday to see if I could help someone with my limited knowledge.

 

If I understand your reply to Jim correctly, the only way to see if someone needs help would be to go to the individual mod page in Nexus Mods. Am I correct in this?

[Dark0ne]

In response to post #26349444.

M48A5 wrote:

I have read the latest notification of the changes being made to the nexus and even after re-reading it, I am still confused.

 

Is the thread "(Game Name) Uploaded files" being discontinued?

 

Having looked in Skyrim, Fallout 3, Fallout New Vegas and Obilivion, the Uploaded files section is missing. That section was one that I checked religiously everyday to see if I could help someone with my limited knowledge.

 

If I understand your reply to Jim correctly, the only way to see if someone needs help would be to go to the individual mod page in Nexus Mods. Am I correct in this?

That's correct.

[SydneyB]

Thanks for your continuous improvement of the sites, Dark One and Nexus team :)