Site updates: stability, premium donations, endorsements, downloading, apostrophes and full-stops

[Dark0ne]

In response to post #16033580. #16033965, #16034610 are all replies on the same post.

There's no bug. We have 20 file servers and all of them are at full capacity. There's no point buying more file servers in the short term as they take 10 days to requisition, by which time the traffic will have died down to the normal levels and it will have all been for nought. All we can do is ride it out.

Instead, I signed contracts yesterday with a high level CDN that is used by major video game companies (including Valve/Steam and CCP) that should ensure that, in the future, this can't happen again. We'll simply replace our current 20 file servers with the CDN service, which will not only mean we can serve practically unlimited bandwidth in the future, but will also mean download rates should improve drastically.

Once we're on the CDN and have tested the feasibility to ensure it's 100% doable then I'll make a news post about it. We're 95% certain it'll work for us. Edited June 28, 2014 by Dark0ne

[wickedGRL]

In response to post #15940985. #15943120, #15944040, #15946380, #15994885 are all replies on the same post.

Then it's time Nexus dukes it out with Steam don't you think?

J '.' J

Still the old way was never a problem for many. Edited June 28, 2014 by wickedGRL

[Thandal]

<snip>

Then it's time Nexus dukes it out with Steam don't you think?

 

 

Oh, that fight was over long ago. Valve lost!. :tongue:

 

The Steam Workshop is for dead-simple mods, and players who require dead-simple mods.

The Nexus is for everything else, especially all those games not supported by the Steam Workshop. :thumbsup:

[Octopuss]

 

In response to post #16033580.

Or...you could just be patient. Not discounting the fact that a lot of people coming back to download are people who already have accounts on the Nexus but have bought another game in the Steam sale; e.g. Skyrim players buying New Vegas, Dragon Age players buying Skyrim, and so on.

 

You have valid arguments, but when you can't download 8kB file for 3 days, and basically can't download anything for a week, any arguments go aside and only frustration and anger remains. It wasn't exactly spectacular experience to download things before (but of course I do admit it was huge improvement over the previous state), but this is completely ridiculous.

 

The CDN thing, is that something conceptually similar to those akamai servers which you're being redirected to when you open some popular sites for example? That's great news. Out of curiosity, will it cost more than maintaining the current set of file servers?

[Dark0ne]

I don't think I've ever seen the Akamai stuff you mentioned. A CDN system is simply a vast network of servers located all across the world. When you go to download a file the CDN will pick the closest server to you (and this will especially benefit those folks in Australia and the far east, for example) and serve the download. We have 20 file servers, the number of servers in the CDN is practically infinite in comparison. Based on current estimates it will cost about 50% more, though until we're on there and do a test we can't know for sure. It may well rise to being 100%-150% more.

 

Our current file servers are capped in speed based on their hardware. 2xSCSI drives in RAID-0 can serve about 500-600mbit before they start to max out their I/O. All 20 file servers are doing that (so yeah, we're pushing about 10Gbit a second at the moment). The CDN simply won't have that issue and will scale practically indefinitely. If it can handle Steam, it'll be able to handle the Nexus.

[DrakeTheDragon]

In response to post #16049670.

Akamai is a CDN. We use it primarily for image hosting in our business projects. But file type doesn't really matter anyways. It works somewhat like that:

You still have all the images on your server(s) and tell Akamai the address. When someone requests an image of our's from the Akamai url (our ads contain image-links to Akamai instead of our own servers), Akamai acts just like a proxy, it accesses our servers and downloads the image to its own, at the same time passing it on to the requestee. Next time someone requests the same image from Akamai it's on their servers already and they won't query our's anymore.

There's also sorts of 'update' cycles, when Akamai checks our servers again instead of directly serving the image from one of its own, but it either happens only rarely or not even reliably at all to begin with. We learned to fight that (mis)behaviour of the CDN by taking care no changed file will ever keep the same name. You'll never know if the CDN will 'ever' update the file again otherwise.

Out-sourcing the Nexus file hosting to large-scale distributed CDNs sounds like a really promising idea. I only hope this CDN isn't as unreliable in regards to re-checking for changed files as Akamai was in our observations. Else you'll need to come up with some clever automatic versioning in filenames every time some uploaded mod file changes.

A new file will always be requested from the Nexus the first time, as it doesn't yet exist on the CDN after all. A changed file however, which already existed on the CDN, will only be updated in the CDN's next update cycle, and until then the CDN will keep serving the old version. In case of Akamai we never knew when this would happen, and sometimes even if at all.

[Octopuss]

Looks like Akamai is CDN as well. So yea, good news. Maybe you will have more time to work on other things if this will mostly be handled externally?

Btw I am surprised you can still afford to keep the site running, the costs must be enormous, especially since all the changes over the last months.

[AxelDominatoR]

In response to post #15940985. #15943120, #15944040, #15946380, #15994885, #16037245 are all replies on the same post.

It was a problem for us when specific fileservers crashed regularly because people tended to use some more than others. We also have a badwidth cap on each fileserver and we needed to balance the spread over all of them otherwise that would have caused a chain reaction of instability and failures.

[aqualung99]

Is there another place getting setup (I hope?) where we can go to actually get these files (these files donated by the authors) instead of here? Where we can actually d/l stuff?

 

[TurboTurkey]

Wow you came to New Mexico? What'd you think of it?