Nexus Mods API released

[matortheeternal]

It'd be great if the 200 responses would be documented through swagger so we can know the data format to expect for a successful request without having to actually test the request ourself. I'm happy to help out with adding that if the Nexus Staff don't feel like they don't have time to add it themselves.

 

I must say that this change really makes it feel like Nexus Mods has moved forward in supporting developers of third party tools. It's a great step to take and I wholeheartedly approve. :thumbsup:

Edited February 12, 2019 by matortheeternal

[AL12]

In response to post #67632606. #67634316, #67634966, #67636786, #67637301, #67650701 are all replies on the same post.

MrJohn wrote: What is the reasoning for the user-rate limiting?

DoctorKaizeld wrote: what user rate limiting?

al12rs wrote: The new API features an account based limit of 2500 daily requests, after which you get an extra 100 each hour.
If you are with no requests left, all nexus requests are blocked (this includes downloading manually).
Downloading a file is 2-3 requests, checking a mod for file updates is 1-2 requests. Logging In uses requests as well.

Limiting API calls is standard procedure to avoid overloading servers with requests, slowing them down. Otherwise the API functionality might get compromised for everyone because someone is abusing it.
This particular implementation is pretty simplistic, but should serve it's purpose well.
Nexus staff mentioned that they are monitoring usage and will adjust the limits in case they deem it necessary.

DoctorKaizeld wrote: Ooooh that

damanding wrote: Among other things it slows down file scraping from people downloading all mods in mass to illegally host on their own sites.

BrinaSair wrote:

In response to post #67632606. #67634316, #67634966, #67636786, #67637301 are all replies on the same post.

MrJohn wrote: What is the reasoning for the user-rate limiting?

DoctorKaizeld wrote: what user rate limiting?

al12rs wrote: The new API features an account based limit of 2500 daily requests, after which you get an extra 100 each hour.
If you are with no requests left, all nexus requests are blocked (this includes downloading manually).
Downloading a file is 2-3 requests, checking a mod for file updates is 1-2 requests. Logging In uses requests as well.

Limiting API calls is standard procedure to avoid overloading servers with requests, slowing them down. Otherwise the API functionality might get compromised for everyone because someone is abusing it.
This particular implementation is pretty simplistic, but should serve it's purpose well.
Nexus staff mentioned that they are monitoring usage and will adjust the limits in case they deem it necessary.

DoctorKaizeld wrote: Ooooh that

damanding wrote: Among other things it slows down file scraping from people downloading all mods in mass to illegally host on their own sites.

I admit this is concerning to me. I check for mod updates through MO2 about once a week. My understanding was that this is just a ping to see if the current version is the same as the version on my computer. But from what you are saying the API will treat this the same as attempting to download the entire mod for every mod managed by MO2, whether it needs an update or not and whether the mod is currently being used or not.

Per MO2 I currently have ~3400 mods for Skyrim SE and almost 1900 for Fallout 4. The vast majority are not being used, but MO2 lists them regardless. MO2 has only one means of checking for updates - a clickable link labelled "Check all for update". It is not possible to select a sub-set or selection of mods to check - all of the mods listed in MO2 are checked. So I cannot, for example, check only half one day and half another day. Now, I already have separate installations of MO2 for Skyrim SE and Fallout 4, so I should not have any issues with Fallout 4 if the user limit is 2500. I also usually check the two of them for updates on different days, so again there is no real issue with Fallout 4.

But for Skyrim this means that I will be locked out of nexus before the check for updates is even finished, let alone before I can start going to the nexus pages to see about downloading said updates. I don't mind being limited in how many files or how many MB I can download per day or even per hour - that is entirely reasonable. But counting checks of the current version as downloads - even when the versions match - just seems incredibly unreasonable.

We do a lot of trickery to get around the api limit. Like using a single call to get list of mods that have been modified in the last moth and cross referencing those with your modlst. Another thing is that all mods with the same nexus mod id (so patches, duplicated mods etc) will only be checked once.
But your setup seems like it would be a really great stress test for the dev build, to determine if indeed the limits are too harsh, and if we can improve things on our end.
We did for example add an option to updateCheck a single mod (right click menu).
Here is the link to the discord server with the dev builds: https://discord.gg/6GKR9jZ

[ff7legend]

In response to post #67632451. #67632701, #67633716, #67637026, #67643171 are all replies on the same post.

al12rs wrote: Congrats for the release!

As people on the Mod Organizer 2 discord know, we are already testing dev builds with the new API, so you can rest assured MO2 will get an update in time to support the change. (It has been in the works already for a while)

About MO1, there hasn't been any discussion yet about updating it. Since the update does take quite a considerable amount of time an effort, it could be that there is no one with enough free time to dedicate to it.

Moksha8088 wrote: Wondering if the tried, true, and trustworthy Nexus Mod Manager found on GitHub will also be updated with this feature?

al12rs wrote: There are sill people that are fond of NMM, despite the fact that that it's no longer supported. I do believe someone of the long term fans might update it, even though I personally don't agree on that trustworthiness statement.
Like with MO1 it all depends on whether there is someone there willing to put in the time and effort.
Nexus staff will, understandably, not update NMM.

ff7legend wrote: Yours truly is sticking with NMM for the simple fact that any attempt to migrate a 1,000+ Active Mods installation from NMM to Vortex is sure to FAIL on an EPIC SCALE. I'm not about to play literal Russian roulette with my modded Skyrim SE installation. Hopefully NMM will be updated to support the new API at some point since I cannot migrate my entire modded install from NMM to Vortex.

Silarn wrote: BardicKnowledge is, I believe, the main maintainer of NMM and he's well aware of the changes and has been informed along with the rest of us. I assume he will be updating it at some point.

I sure hope so. I cannot & will not migrate to Vortex due to the sheer size/scope of my modded Skyrim SE installation.

[MrMason]

In response to post #67632606. #67634316, #67634966, #67636786, #67637301, #67650701, #67671731 are all replies on the same post.

MrJohn wrote: What is the reasoning for the user-rate limiting?

DoctorKaizeld wrote: what user rate limiting?

al12rs wrote: The new API features an account based limit of 2500 daily requests, after which you get an extra 100 each hour.
If you are with no requests left, all nexus requests are blocked (this includes downloading manually).
Downloading a file is 2-3 requests, checking a mod for file updates is 1-2 requests. Logging In uses requests as well.

Limiting API calls is standard procedure to avoid overloading servers with requests, slowing them down. Otherwise the API functionality might get compromised for everyone because someone is abusing it.
This particular implementation is pretty simplistic, but should serve it's purpose well.
Nexus staff mentioned that they are monitoring usage and will adjust the limits in case they deem it necessary.

DoctorKaizeld wrote: Ooooh that

damanding wrote: Among other things it slows down file scraping from people downloading all mods in mass to illegally host on their own sites.

BrinaSair wrote:

In response to post #67632606. #67634316, #67634966, #67636786, #67637301 are all replies on the same post.

MrJohn wrote: What is the reasoning for the user-rate limiting?

DoctorKaizeld wrote: what user rate limiting?

al12rs wrote: The new API features an account based limit of 2500 daily requests, after which you get an extra 100 each hour.
If you are with no requests left, all nexus requests are blocked (this includes downloading manually).
Downloading a file is 2-3 requests, checking a mod for file updates is 1-2 requests. Logging In uses requests as well.

Limiting API calls is standard procedure to avoid overloading servers with requests, slowing them down. Otherwise the API functionality might get compromised for everyone because someone is abusing it.
This particular implementation is pretty simplistic, but should serve it's purpose well.
Nexus staff mentioned that they are monitoring usage and will adjust the limits in case they deem it necessary.

DoctorKaizeld wrote: Ooooh that

damanding wrote: Among other things it slows down file scraping from people downloading all mods in mass to illegally host on their own sites.

I admit this is concerning to me. I check for mod updates through MO2 about once a week. My understanding was that this is just a ping to see if the current version is the same as the version on my computer. But from what you are saying the API will treat this the same as attempting to download the entire mod for every mod managed by MO2, whether it needs an update or not and whether the mod is currently being used or not.

Per MO2 I currently have ~3400 mods for Skyrim SE and almost 1900 for Fallout 4. The vast majority are not being used, but MO2 lists them regardless. MO2 has only one means of checking for updates - a clickable link labelled "Check all for update". It is not possible to select a sub-set or selection of mods to check - all of the mods listed in MO2 are checked. So I cannot, for example, check only half one day and half another day. Now, I already have separate installations of MO2 for Skyrim SE and Fallout 4, so I should not have any issues with Fallout 4 if the user limit is 2500. I also usually check the two of them for updates on different days, so again there is no real issue with Fallout 4.

But for Skyrim this means that I will be locked out of nexus before the check for updates is even finished, let alone before I can start going to the nexus pages to see about downloading said updates. I don't mind being limited in how many files or how many MB I can download per day or even per hour - that is entirely reasonable. But counting checks of the current version as downloads - even when the versions match - just seems incredibly unreasonable.

al12rs wrote: We do a lot of trickery to get around the api limit. Like using a single call to get list of mods that have been modified in the last moth and cross referencing those with your modlst. Another thing is that all mods with the same nexus mod id (so patches, duplicated mods etc) will only be checked once.
But your setup seems like it would be a really great stress test for the dev build, to determine if indeed the limits are too harsh, and if we can improve things on our end.
We did for example add an option to updateCheck a single mod (right click menu).
Here is the link to the discord server with the dev builds: https://discord.gg/6GKR9jZ

The limits we've put in place are our first 'best guess' at what is reasonable. If we find that legitimate users are having problems then we are fully on board with changing the limits.

[BrinaSair]

 

In response to post #67632606. #67634316, #67634966, #67636786, #67637301, #67650701 are all replies on the same post.

 

 

 

MrJohn wrote: What is the reasoning for the user-rate limiting?

DoctorKaizeld wrote: what user rate limiting?

al12rs wrote: The new API features an account based limit of 2500 daily requests, after which you get an extra 100 each hour.

If you are with no requests left, all nexus requests are blocked (this includes downloading manually).

Downloading a file is 2-3 requests, checking a mod for file updates is 1-2 requests. Logging In uses requests as well.

 

Limiting API calls is standard procedure to avoid overloading servers with requests, slowing them down. Otherwise the API functionality might get compromised for everyone because someone is abusing it.

This particular implementation is pretty simplistic, but should serve it's purpose well.

Nexus staff mentioned that they are monitoring usage and will adjust the limits in case they deem it necessary.

DoctorKaizeld wrote: Ooooh that

damanding wrote: Among other things it slows down file scraping from people downloading all mods in mass to illegally host on their own sites.

BrinaSair wrote:

In response to post #67632606. #67634316, #67634966, #67636786, #67637301 are all replies on the same post.

 

 

 

DoctorKaizeld wrote: Ooooh thatal12rs wrote: The new API features an account based limit of 2500 daily requests, after which you get an extra 100 each hour.

If you are with no requests left, all nexus requests are blocked (this includes downloading manually).

Downloading a file is 2-3 requests, checking a mod for file updates is 1-2 requests. Logging In uses requests as well.

 

Limiting API calls is standard procedure to avoid overloading servers with requests, slowing them down. Otherwise the API functionality might get compromised for everyone because someone is abusing it.

This particular implementation is pretty simplistic, but should serve it's purpose well.

Nexus staff mentioned that they are monitoring usage and will adjust the limits in case they deem it necessary.DoctorKaizeld wrote: what user rate limiting?MrJohn wrote: What is the reasoning for the user-rate limiting?

damanding wrote: Among other things it slows down file scraping from people downloading all mods in mass to illegally host on their own sites.

I admit this is concerning to me. I check for mod updates through MO2 about once a week. My understanding was that this is just a ping to see if the current version is the same as the version on my computer. But from what you are saying the API will treat this the same as attempting to download the entire mod for every mod managed by MO2, whether it needs an update or not and whether the mod is currently being used or not.

 

Per MO2 I currently have ~3400 mods for Skyrim SE and almost 1900 for Fallout 4. The vast majority are not being used, but MO2 lists them regardless. MO2 has only one means of checking for updates - a clickable link labelled "Check all for update". It is not possible to select a sub-set or selection of mods to check - all of the mods listed in MO2 are checked. So I cannot, for example, check only half one day and half another day. Now, I already have separate installations of MO2 for Skyrim SE and Fallout 4, so I should not have any issues with Fallout 4 if the user limit is 2500. I also usually check the two of them for updates on different days, so again there is no real issue with Fallout 4.

 

But for Skyrim this means that I will be locked out of nexus before the check for updates is even finished, let alone before I can start going to the nexus pages to see about downloading said updates. I don't mind being limited in how many files or how many MB I can download per day or even per hour - that is entirely reasonable. But counting checks of the current version as downloads - even when the versions match - just seems incredibly unreasonable.

We do a lot of trickery to get around the api limit. Like using a single call to get list of mods that have been modified in the last moth and cross referencing those with your modlst. Another thing is that all mods with the same nexus mod id (so patches, duplicated mods etc) will only be checked once.

But your setup seems like it would be a really great stress test for the dev build, to determine if indeed the limits are too harsh, and if we can improve things on our end.

We did for example add an option to updateCheck a single mod (right click menu).

Here is the link to the discord server with the dev builds: https://discord.gg/6GKR9jZ

 

 

Okay, knowing that MO2 can query which mods have updated within the last month relieves most of my worries. If I only have to worry about mods that have updated within the last month then the worst case scenario is when Skyrim SE updates in a manner that requires most mods with a dll to update, and that would be less than 100 (currently about 60 or 70, I think). It is also nice to know that a means is being developed to check some mods for update without checking every mod. Between these I no longer see a 2500 / day limit ever causing an issue.

[CLYung]

In response to post #67632451. #67632701, #67633716, #67637026, #67643171, #67672576 are all replies on the same post.

al12rs wrote: Congrats for the release!

As people on the Mod Organizer 2 discord know, we are already testing dev builds with the new API, so you can rest assured MO2 will get an update in time to support the change. (It has been in the works already for a while)

About MO1, there hasn't been any discussion yet about updating it. Since the update does take quite a considerable amount of time an effort, it could be that there is no one with enough free time to dedicate to it.

Moksha8088 wrote: Wondering if the tried, true, and trustworthy Nexus Mod Manager found on GitHub will also be updated with this feature?

al12rs wrote: There are sill people that are fond of NMM, despite the fact that that it's no longer supported. I do believe someone of the long term fans might update it, even though I personally don't agree on that trustworthiness statement.
Like with MO1 it all depends on whether there is someone there willing to put in the time and effort.
Nexus staff will, understandably, not update NMM.

ff7legend wrote: Yours truly is sticking with NMM for the simple fact that any attempt to migrate a 1,000+ Active Mods installation from NMM to Vortex is sure to FAIL on an EPIC SCALE. I'm not about to play literal Russian roulette with my modded Skyrim SE installation. Hopefully NMM will be updated to support the new API at some point since I cannot migrate my entire modded install from NMM to Vortex.

Silarn wrote: BardicKnowledge is, I believe, the main maintainer of NMM and he's well aware of the changes and has been informed along with the rest of us. I assume he will be updating it at some point.

ff7legend wrote: I sure hope so. I cannot & will not migrate to Vortex due to the sheer size/scope of my modded Skyrim SE installation.

I'm fighting the hassle now with it. NMM will no longer download mods on my computer so I tried to migrate to Vortex.

ALL of the mods I was using are now F.U.B.A.R. and I'm going to have to start all over. Vortex has pissed all over my system and now I have to go through all directories to find files that I must now remove so that I will be able to again play FO4

Thank You NEXUS for shutting down NMM from downloading updated files! Edited February 22, 2019 by CLYung

[WishkahKid]

I don't quite understand this. Is the only reason this new (API) is being introduced is because of Vortex?

[Thandal]

I don't quite understand this. Is the only reason this new (API) is being introduced is because of Vortex?

 

It's being introduced so that others can write small programs of their own that interact with (some of) the Nexus data the way Vortex can. Some of those programs might extend or enhance the capabilities of Vortex, but that's up to those who write them. :cool:

[DFX2K9]

In response to post #68188981.

Thandal wrote:

I don't quite understand this. Is the only reason this new (API) is being introduced is because of Vortex?

 

It's being introduced so that others can write small programs of their own that interact with (some of) the Nexus data the way Vortex can. Some of those programs might extend or enhance the capabilities of Vortex, but that's up to those who write them. :cool:

I'm glad you mentioned games being able to use this as well. Because I've fiddled around with a number of IDEs as a hobby. And one of the things I thought would be cool to do would be to make the game Nexus/Vortex-aware to some extent in the way Steam Workshop games are for the workshop.

Like, if someone boots up the game without booting Vortex, the game can go check for mod updates and let the user know. That's super handy.

That password verification is also EXTREMELY nice, since it takes a lot of the security-load off of the application developer who wants to do that kind of integration. Props for this in particular.

[DFX2K9]

In response to post #67632451. #67632701, #67633716, #67637026, #67643171, #67672576, #67920921 are all replies on the same post.

al12rs wrote: Congrats for the release!

As people on the Mod Organizer 2 discord know, we are already testing dev builds with the new API, so you can rest assured MO2 will get an update in time to support the change. (It has been in the works already for a while)

About MO1, there hasn't been any discussion yet about updating it. Since the update does take quite a considerable amount of time an effort, it could be that there is no one with enough free time to dedicate to it.

Moksha8088 wrote: Wondering if the tried, true, and trustworthy Nexus Mod Manager found on GitHub will also be updated with this feature?

al12rs wrote: There are sill people that are fond of NMM, despite the fact that that it's no longer supported. I do believe someone of the long term fans might update it, even though I personally don't agree on that trustworthiness statement.
Like with MO1 it all depends on whether there is someone there willing to put in the time and effort.
Nexus staff will, understandably, not update NMM.

ff7legend wrote: Yours truly is sticking with NMM for the simple fact that any attempt to migrate a 1,000+ Active Mods installation from NMM to Vortex is sure to FAIL on an EPIC SCALE. I'm not about to play literal Russian roulette with my modded Skyrim SE installation. Hopefully NMM will be updated to support the new API at some point since I cannot migrate my entire modded install from NMM to Vortex.

Silarn wrote: BardicKnowledge is, I believe, the main maintainer of NMM and he's well aware of the changes and has been informed along with the rest of us. I assume he will be updating it at some point.

ff7legend wrote: I sure hope so. I cannot & will not migrate to Vortex due to the sheer size/scope of my modded Skyrim SE installation.

CLYung wrote: I'm fighting the hassle now with it. NMM will no longer download mods on my computer so I tried to migrate to Vortex.

ALL of the mods I was using are now F.U.B.A.R. and I'm going to have to start all over. Vortex has pissed all over my system and now I have to go through all directories to find files that I must now remove so that I will be able to again play FO4

Thank You NEXUS for shutting down NMM from downloading updated files!

I can feel pain for those uses NMM because of huge mod installs.

I happened to switch over to vortex at a time where I already wanted to reinstall everything fresh, so the inconvenience was a given. And I've never installed more then a few dozen mods at a time, a habit I got from NMM because it seemed not to like going big whenever I tried.