CDN caching old versions of files

[Deleted133263User]

This question was asked in a topic which is now locked ( Dark0ne wanted to move the focus into the newer topic reference CDN and the Network Diagnoser software ) - So shifting this question to its own topic :

 

http://forums.nexusmods.com/index.php?/topic/2058534-august-holidays-fileserver-to-cdn-switchover-and-wysiwyg-editor-for-descriptions/page-17&do=findComment&comment=19153709

 

Dark0ne

 

I don't understand how the new CDN works exactly, but I think last night we witnessed a flaw with it :

 

All the UnOfficial Skyrim Patches ( USKP plus UDGP, UHFP, UDBP, UHRP ) were updated

 

 

Quite a few of them had been corrupted by a beta test version of Wrye Bash .. So needed re-uploading

 

Arthmoor did a quick turn around on fixing the problem in an attempt to stop too many people getting messed up with Wrye Bash / Nexus Mod Manager throwing those errors and confusing users as to what to do - And re-uploaded the files

 

 

The files were fine and fixed - confirmed by downloading the same re-uploaded files from AFK Mods site instead of Nexus

 

 

The reason a load of people had to go AFK Mods instead ? .. Even after the upload of fixed files to Nexus ( New time stamp etc at upload time confirming the upload had been delivered to nexus ), people were still getting the same issues as the previous upload.

 

 

We can only conclude that the CDN was still serving people cached old versions of the same problematic files

 

If that is the case - How are people to know when it is safe to re-download a new update to a file, if the timestamp of the upload is no guarantee that you will receive the newly uploaded file ?

Or is this something to do with the virus check system ?

Drake the Dragon replied ..

I can't speak for every CDN out there, but if it's anywhere like the Akamai CDN we're using in my job, "replacing" files will not exactly work 'reliably' or at all.

 

There's a couple hundred or thousand CDN hosts all over the globe and they all have to 'update' the file they're hosting without even knowing the source file has changed. So they do it in regular 'intervals' rather than every single time, and the bigger the network the less regular these will be. In our job it came up to the point at which updates 'in time' were not to be expected at all anymore and we needed to circumvent this by introducing versioning into the actual filenames themselves.

 

If a requested file doesn't exist on the host, it will connect to the source and obtain a copy, which from then on will rely on the host for whenever it's requested again. But if an old version of a file with the same name exists on the host, the host will only rarely query the source again to check if it changed, if it will at all. So our way to 'bust' this unwelcome cache was to change the filenames as well whenever a file's contents changed. Then it's a new file not yet present on the host and as such it's guaranteed the correct file will be obtained from the source by all means.

 

I'm not sure how the internal naming of the Nexus' archive files works exactly, but if you re-upload a file and the internal filename won't change, then you can almost 100% guaranteed forget about any CDN to ever realize the change in a timely fashion, if at all.

 

CDN and "re-upload" don't go well together.

 

But I'd wait for one of the devs or Dark0ne's reply to this before jumping to conclusions based only on my own knowledge of CDN. It wouldn't surprise me though, if it's indeed just like I said.

And I followed that with ..

So assuming the above is all correct : Further questions for Dark0ne ..

 

The only sure fire method would be a slight name change to files, and delete the old one.

 

Would this ensure NMM will also not have problems with updating files, and notify / download correctly an actual new file rather than a cached old version ?.

 

Dark0ne did the necessary contacting ..

I'll get an official response from one of the programmers, or they'll respond here in due course. Cheers for the heads up.

 

And Axeldominator seemed to agree there was a problem ..

Yes, we may definitely have issues with reuploading files using the same filename at the moment as we don't know the internals of the caching mechanism either. I will have to speak with the guys from the CDN tomorrow to solve this and other performance problems that may be caused by the same caching mechanism too.

In the meantime please upload a file using a different filename. I'm really sorry for the issue :sad:

 

That topic is now locked, and no further response was made.

 

Around the nexus since then I have seen similar problems people are having with other files, and going by their description of downloading problems I am guessing the problem still exists

 

Is there any update as to what has happened, fixable or no fix ? ( understandable if it is not fixable, and the site is not willing to go back to the older system .. Very understandable :smile: )

[AxelDominatoR]

I'm working on a fix for this issue, I'll let you know as soon as I have some (good, obviously) news!

[Deleted133263User]

:) Ah good to know thank you.

 

Glad I made a new topic, there will be mod authors scratching their heads over this one I reckon.

[Arthmoor]

For what it's worth I can vouch for changing the name of a file fixing things as far as caching.

 

I would assume as well that if you are uploading an update with a new version number that the adjusted filename the site makes out of that would count as well. So this should only be a real problem with re-uploading a file without changing the version number.

[Deleted133263User]

A bit of a necro post .. But @ AxelDominatoR did you ever get around to finishing the fix ?

 

The problem is still not solved to date anyway.