Downloading and file serving changes

[Dark0ne]

Happily and thankfully a week after I wrote my 2,600 word End of Year review/essay we managed to stabilize our new server setup and run through some quick but efficient code updates that have enabled us to run the sites almost down-time free over the past week. No database desyncs, no maintenance windows, just a few times where we've taken the pages down for a few minutes to perform code updates. Paint me f'ing happy on that. And vindicated in my own eyes. No doubt we've still got work to do, but the important thing is that things are way better now.

 

In the same article I also mentioned that during January we would be decommissioning all 21 of our file servers (2 static content servers, 15 file servers, 4 Premium file servers) and replacing them with new servers. The old file servers were coming to the end of their life-line and running out of hard-drive space fast so we needed to upgrade and update the whole line up. We're now replacing those 21 file servers with 26 new file servers (4 static content servers, 20 file servers, 2 Premium file servers with probably more Premium servers to come). As you can imagine that's quite a job; we've got to set up each server with our custom server modules and configurations as well as transfer the 4TB or so of current files there are on to every single one of those servers (so we've got to transfer about 100TB of data over the next month). It's time consuming to be sure. We're currently setting up the first new servers which requires these new servers sync from the old ones, this is causing some issues with the network which is causing some downloads to fail (more than usual) or not start at all. Once we've got these first new servers setup we can sync the files between all the new servers, from the new servers themselves, thus reducing the load on the old setup in the mean-time.

 

I know there's been some issues with downloading reliably through NMM recently, especially when downloads don't start at all. In those instances it's important (for your own sake, not mine) that you know how to do the simplest of things: how to add mods to NMM without using the "download with manager" button. It's so simple I might as well just write it here:

 

• Instead of clicking "Download with manager" click "download manually" on the file you want to download.
  
• A window will pop up with a list of file servers to download from. Pick a file server. Typically the fastest server will be the one with the least users, not the one that's closest to you.
  
• Your download should start within 1-30 seconds. Let the download finish.
  
• When your download has finished, open NMM.
  
• Open the folder where your file downloaded to.
  
• Drag the file in to the "mods" window in NMM.
  
• OR: in the "mods" window, click the button that looks like a jigsaw piece with a plus (+) symbol next to it and navigate to your file on your HDD.
  
• NMM will add your mod just as if you'd used the "download with manager" button.
  

 

That's it. If you download from the Nexus and don't change the filename then NMM will recognise the file and the file will work exactly as if you had downloaded it through NMM itself. There seems to be some myth that if you download manually then NMM won't show you any new mod versions. That's wrong/a lie. NMM cannot show you mod versions for mods not downloaded via the Nexus (so if you download a mod from the Workshop and add it to NMM then NMM won't show you any new mod updates/versions for that file), hopefully for very obvious reasons, but it will show you updates for any Nexus mods whether you downloaded the file directly through NMM or manually via the site.

 

At times like these you will find it easier and quicker to download your files manually and then add them to NMM rather than using the "download with manager" button and have it failing on you a lot.

 

While we're working on the file servers we're directing a bit of attention to how NMM handles downloads, and especially downloads that fail. Right now NMM will try and find a working server that fits your download location preferences as well as possible. However if that server fails then it doesn't automatically try another server, it just fails. We're trying to add more fail safes in to the program so you're left with no failed connections.

 

Similarly we're having to implement a download queuing system based on feedback I've personally been getting from a few...interesting individuals. With the ever increasing interest in projects like Skyrim GEMS and S.T.E.P. there's a lot of users who are downloading large quantities of mods all at once. I'm getting support tickets from users wondering why they're struggling to download 200 mods all at the same time in NMM. Hopefully I don't need to tell you why downloading 200 files at the same time isn't good for the servers (or respectful to the other users of the site). Now this is partially our fault as we haven't hard-coded any limits in to NMM in this regard. NMM will let you download as many files as you want. They probably won't all work, but it'll let you do it. We've got hard-coded limits on our servers (which I believe are 16 connections for normal users and 64 for Premium, with an increase for Premium as they can use 4 threads, 4 multiplied by 16 is 64). With currently 15 file servers that's a theoretical maximum of 240 concurrent downloads, but that works on the assumption that NMM will pick servers that you aren't already maxing out, which it won't. But I digress, trying to download 200 mods all at once is really stupid and us not having hard-coded limits in the program is equally stupid. As such we are going to be introducing a download queuing system. Normal members will be able to download 5 files concurrently, Premium members will be able to download 10 files concurrently. You can add as many files as you want to your download list but only 5 (or 10) will download at any one time. Once a download finishes a new download in the queue will start automatically. Such functionality is commonplace on many sites and will enable us to provide a more stable and fair service to all users.

 

We hope to get this new NMM version out to coincide with the completion of our transition over to the new file servers. It's likely this version will be another "forced" update, in that you'll need to update to this version in order to be able to use NMM online, because it's updating and fixing things that are the cause of issues on the network. I'll have a word with the programmers to see if we can update the error you receive when trying to login to an unsupported version of NMM rather than the generic error that is currently used.

 

In the meantime I hope you're enjoying a much smoother site experience, even if the downloads are currently a little iffy. The important thing is: we're getting there. Like I said we would.

 

[TrueOtaku2]

In response to post #11073787.

Brilliantly written! Love the effort you guys are putting into the site for our sake. I figured out the manual to NMM method after noticing that NMMs mods are all nice in their vanilla zip form in its folder. I also fixed any problems I had downloading manually by trying different servers (Im in the states, so I used the London and Amsterdam servers). Alot of problems users face they can fix themselves with some ingenuity instead of mass ticket submitting. It would definitely make it easier on you guys.

[frostryder]

In response to post #11073787. #11075590 is also a reply to the same post.

I'm glad to hear about the queuing feature, as oftimes there are several mods I'd like to download, and I'd prefer it not try to download them all at the same time.

Another feature I'd love to see is to NOT have it go to London as it's first attempt at a server. EVERY.SINGLE.TIME. I'm planning to write a module to implement this, during my 'free time'. How do we send in coded solutions we've come up with?

[Arthmoor]

People seriously try and download 200 files at once? o.O

[spydar59]

Start charging them 100 dollars a mega-download. That will cure them of that real quick. And if that doesn't work, send them the Red Square White Exclamation Point note. You know, the one that reads "Really, are you that selfish?" Besides that, I am one of the people that read your notes. You are kind enough to tell us what is going on, and how things are working out. Not many people or places care to do that anymore. So thank you. If I had this many problems with my computer, I would have already mailed it to a fish. (For all you youngsters out there that means throwing it in a lake). I hope you get it worked out the way you want. Keep up the great work and again, a big thank you.

[starfis]

Wonder what will happen when huge and highly anticipated Immersive Armors v7 will be released? I'm sure that a lot of eager people like myself will be downloading it at the same time, hopefully your new setup withstands. Thank you for the info about what is going on, it is always a very interesting reading.

[erelde]

About the 2600 word rant you did for the new year review :)

 

What your thoughts (Dark0ne&everyone) about an upcoming Fallout 4 and after that probably TES 6 and their effect on the Nexus ?

[smon67]

Just like to say thanks to all involved here, great site, good support & info, hope all go`s well with the upgrades.

 

[choochoo1]

Im glad ya'll are working on the issues with NMM. Installing by hand aint terrible, nothing we didn't have to do before NMM. I look forward to when everything is all done and running great again. and thanks for keeping us informed.

[yaq]

I want to thank you and the whole team behind Nexusmods for their never-ending work on these sites (and everything having to do with the pages!). That's a lot to do but you handle it very fine!

Don't let yourself get discouraged from occasional trolls or other demotivational posts/news.

 

edit: special thanks for keeping us updated on whats going on, thats definitely something special AND very appreciated!

Edited January 8, 2014 by yaq