Meta Servers and mod meta lookup

[agc93]

So this was originally raised on Discord, but I think this might be an easier format. I'm basically looking for a bit of context around meta servers and modmeta-db, and how they're used in Vortex.

 

Looking at the code for modmeta-db, both the lib and the server, shows a heap of code for looking up by logical file names, expressions, and some other file details. However, the actual lookup method that's exported seems to ignore all of that and just use by_key with the MD5 hash no matter what's provided. Likewise, since Vortex just calls into lookup(), it also only lookups using MD5/by_key.

 

Is this expected? What are the other routes (by_name/by_expression) actually used for?

 

If it is expected behaviour, does that mean that any mod source that doesn't store MD5 hashes of its archive downloads is essentially a complete non-starter for meta server integration? I'm hoping not since that seems to also rule out using any of Vortex's in-built dependency or conflict management functionality.

[Tannin42]

I'm not sure what you're looking for here, maybe it would help if you described what you're trying to achieve?

 

The meta server has two use cases, only one is currently actively being used:

 

a) User adds a file into Vortex manually, Vortex has no information about it so it creates a md5 hash, looks it up, gets some details (e.g. which source the file is available on), can use those details to look up further information through the source-specific api.

b) Mod rules can reference other files that may not be installed on the current system, Vortex uses the meta server to find those files and figure out how to download them.

 

For use case a) you _need_ the md5 hash because Vortex doesn't have any other information about the file. Starting with a file "foobarmod_123456_v1_42_13.zip", how would you do a lookup based on a file expression? What file expression?

For use case b) you can use wildcard patterns, e.g. your rule could say "i depend on the mod with filename pattern foobarmod_*.zip with version >= 1.40.3" and then the meta server would be used to resolve that expression (that is what by_name and by_expression are used for).

 

> If it is expected behaviour, does that mean that any mod source that doesn't store MD5 hashes of its archive downloads is essentially a complete non-starter for meta server integration?

 

What _does_ your mod source store that you would want to uniquely identify the mod with?

 

The modmeta-db protocol requires md5 hashes because that's how we look up files we don't know anything about, that's just - not optional. We can't use different identifiers for different sources because if we have just a file on disk we don't know what source it is. The whole point of the meta server is to abstract away the differences of mod sources.

If the mod source doesn't store md5 hashes, just start storing them, if you have the file you can calculate the md5 hash.

But Vortex can't use different identifiers for different sources when it doesn't know (and shouldn't have to know) which source it's request is going to end up with.

If you don't want this you don't want to use the modmeta-db protocol.

 

> I'm hoping not since that seems to also rule out using any of Vortex's in-built dependency or conflict management functionality.

 

Why is that? You can use the api to add rules to installed mods independent of how you installed the mod (store.dispatch(actions.addModRule(...))) but then the mod rule needs to reference the dependent mod somehow.

Currently this is limited to the mod id for already installed mods, the md5 hash, a file expression or a logical file name.

Once we release collections that will probably support ways to reference mods on specific repositories using ids custom to that repo but that is still a while off.

[agc93]

Sorry, should have definitely included a bit more context.

I'm not sure what you're looking for here, maybe it would help if you described what you're trying to achieve?

 

The meta server has two use cases, only one is currently actively being used:

 

a) User adds a file into Vortex manually, Vortex has no information about it so it creates a md5 hash, looks it up, gets some details (e.g. which source the file is available on), can use those details to look up further information through the source-specific api.

b) Mod rules can reference other files that may not be installed on the current system, Vortex uses the meta server to find those files and figure out how to download them.

 

For use case a) you _need_ the md5 hash because Vortex doesn't have any other information about the file. Starting with a file "foobarmod_123456_v1_42_13.zip", how would you do a lookup based on a file expression? What file expression?

For use case b) you can use wildcard patterns, e.g. your rule could say "i depend on the mod with filename pattern foobarmod_*.zip with version >= 1.40.3" and then the meta server would be used to resolve that expression (that is what by_name and by_expression are used for).

 

> If it is expected behaviour, does that mean that any mod source that doesn't store MD5 hashes of its archive downloads is essentially a complete non-starter for meta server integration?

 

What _does_ your mod source store that you would want to uniquely identify the mod with?

 

> ...that seems to also rule out using any of Vortex's in-built dependency or conflict management functionality.

 

Why is that? You can use the api to add rules to installed mods independent of how you installed the mod (store.dispatch(actions.addModRule(...))) but then the mod rule needs to reference the dependent mod somehow.

Currently this is limited to the mod id for already installed mods, the md5 hash, a file expression or a logical file name.

 

Sorry I wasn't all that clear. I'll have to experiment a bit with the API (the mod rule stuff isn't documented so I'm flying a bit blind here) and hopefully be able to narrow down the problem space.

 

For the specific use case, it's the Beat Saber extension: there is already an existing community mod source that stores a lot of information about mods (names, downloaded file names, hashes of individual files inside the archive, other metadata), but doesn't include the MD5 of the archive file. My extension can't control that and I don't want to have to build my own layer indexing almost a thousand packages by hash, ideally.

 

So, if we say that use case a) is out (source doesn't give hash), is the best way of telling Vortex about mod rules going to be in the installer? Or should I listen to `start-install-download` and do the required API calls in there?

 

The way I see it, I should be able to then add mod rules (in the installer/event listener) using logical file name/file expression that a meta server can respond to, pointing Vortex back to the correct mod dependency on the upstream mod source. Does that sound like the correctly flow, or am I misunderstanding this?

 

Sorry if this is confusing, but it can be a bit tricky to understand the high-level flow from reverse-engineering the code (and nexus_integration has a lot of code )

[Tannin42]

Where do you get the dependency information from?

 

You can set rules from the installer. in your installer you probably currently only return copy instructions like this:

{ type: 'copy', source: ..., destination: ... }

 

but there are other instructions an installer can generate (see https://nexus-mods.github.io/vortex-api/globals.html#instructiontype and https://nexus-mods.github.io/vortex-api/interfaces/iinstruction.html),

e.g. you can set mod attributes during installation like this:

{ type: 'attribute', key: 'author', value: 'agc93' }

 

and rules

{ type: 'rule', rule: { type: 'requires', reference: { fileExpression: 'foobarmod_*.zip', version: '>=1.0.5' } } }

 

Technically that reference should then be used with the metaserver to look up and download the dependency, however internally we don't use that functionality yet (nexus mods has no lookup using file expressions or logical names) so this is mostly untested.

I'm happy to work on improving this with you but please don't be surprised if you run into problems.

 

One important thing: When you work with fileExpression or logical name, you have to make sure that after installation of the mod, the mod has attributes (fileName and/or logicalFileName) that still match what the rule references, otherwise the rule won't be considered fulfilled, you might end up having a mod that downloads a dependency, installs it but then still claims it's not fulfilled.

[agc93]

Where do you get the dependency information from?

 

 

Thankfully the upstream API does return dependency information. It's very awkward, but workable (based on name and version, with file names derived from those).

 

 

 

You can set rules from the installer. in your installer you probably currently only return copy instructions like this:

{ type: 'copy', source: ..., destination: ... }

 

but there are other instructions an installer can generate (see https://nexus-mods.github.io/vortex-api/globals.html#instructiontype and https://nexus-mods.github.io/vortex-api/interfaces/iinstruction.html),

e.g. you can set mod attributes during installation like this:

{ type: 'attribute', key: 'author', value: 'agc93' }

 

and rules

{ type: 'rule', rule: { type: 'requires', reference: { fileExpression: 'foobarmod_*.zip', version: '>=1.0.5' } } }

 

Technically that reference should then be used with the metaserver to look up and download the dependency, however internally we don't use that functionality yet (nexus mods has no lookup using file expressions or logical names) so this is mostly untested.

I'm happy to work on improving this with you but please don't be surprised if you run into problems.

 

Awesome, I'll have to look into this. At current, the installer does set mod attributes, but I cheated and just use a call to setModAttributes to set metadata during installation. Doing this the "proper" way with instructions might cut down on the jank. I'll test out using the rules and a meta server to return responses for file expression (or logical file name) and let you know if I see any more issues.

 

One important thing: When you work with fileExpression or logical name, you have to make sure that after installation of the mod, the mod has attributes (fileName and/or logicalFileName) that still match what the rule references, otherwise the rule won't be considered fulfilled, you might end up having a mod that downloads a dependency, installs it but then still claims it's not fulfilled.

 

 

That makes sense. Hopefully moving the metadata work into the installer instructions will cut down on odds of missing this during install etc.

[Tannin42]

Yeah, using setModAttributes during installation is a bit problematic because technically the mod doesn't exist at that time.

It seems to work at the moment but if the installation process changes in the future for whatever reason that call may break so please don't rely on this behavior.

[agc93]

Ignore the below, was using the wrong property name (version vs versionMatch). Will continue testing.

 

-----------

 

Okay so ran into my first issue before meta servers even get involved. My extension is now returning requires rules as follows:

return {
    type: 'rule',
    rule: {
        type: 'requires',
        reference: {
            logicalFileName: dependencyName,
            version: `=${dependencyVersion}`
        }
    }
}

However, Vortex seems to be just ignoring the version part of that rule. If I install a dependency (dependencyName in that example) with the wrong version, Vortex seems to think it's satisfied the requirement just fine.

 

Is this expected? If not, I can raise an issue with all the details but want to make sure I'm not still missing something in how Vortex handles dependency rules.

Edited May 3, 2020 by agc93

[agc93]

...

and rules

{ type: 'rule', rule: { type: 'requires', reference: { fileExpression: 'foobarmod_*.zip', version: '>=1.0.5' } } }

 

Technically that reference should then be used with the metaserver to look up and download the dependency, however internally we don't use that functionality yet (nexus mods has no lookup using file expressions or logical names) so this is mostly untested.

I'm happy to work on improving this with you but please don't be surprised if you run into problems.

...

 

Okay so I have now updated the extension to do this correctly: rules are returned for mods based on logical file names and version matches that are correctly picked up in Vortex. It gives me warnings about missing dependencies and they show up as rules on individual mods in the Mods list.

 

How do I go about getting Vortex to query my metaserver for where to find those missing dependencies? I have one configured but the only requests it receives are the one query for MD5 hash on first mod install, seemingly nothing for the dependencies. Does this need `modmeta-db` to be updated to use keys other than md5hash?

[Tannin42]

Hmmm, you are testing with 1.2.x, right? However, this code is almost certainly buggy because we've only ever used it in development towards mod collections, if you provide me a working sample I'll look into it on monday.

 

The way it should work is that during install of the mod it should check the dependencies (logging "process dependencies"). Then, if there are unsolved requirements, it should give a dialog along the lines of "The mod has unsolved dependencies, do you want to install them now?"

In that process it should have looked up the reference via metadb and then use the sourceURI from the result (if any) to do the download.

That sourceURI has to be a supported protocol (direct download via http/https, nxm or anything you implemented via context.registerProtocol).

[agc93]

Hmmm, you are testing with 1.2.x, right? However, this code is almost certainly buggy because we've only ever used it in development towards mod collections, if you provide me a working sample I'll look into it on monday.

 

The way it should work is that during install of the mod it should check the dependencies (logging "process dependencies"). Then, if there are unsolved requirements, it should give a dialog along the lines of "The mod has unsolved dependencies, do you want to install them now?"

In that process it should have looked up the reference via metadb and then use the sourceURI from the result (if any) to do the download.

That sourceURI has to be a supported protocol (direct download via http/https, nxm or anything you implemented via context.registerProtocol).

 

Yep, testing on release version of 1.2.10.

 

What do you need in terms of working sample? The Beat Saber extension is now using these rules (latest CI build here), but I can probably try and extricate a minimum working sample if you need that?

 

Definitely didn't see any dialog, but I do get the warning notification about missing dependencies.