Mod Distribution/Automation Program

[Solver]

Backups are a must.

 

Also, the program should handle UPKs *and* the modified ini, as modpatcher does. The program should have an "install mod" or whatever option that lets the user select upk and/or ini files, and then handles everything automatically. Makes a backup, does hashes, patches resources, etc. The user shouldn't have to use the command line or make backups manually or do anything else.

[Daemonjax]

Well, modpatcher could easily be integrated into any GUI, since it's a command-line tool that has nothing to do with anything else that needs to be done. :D

 

Same goes for Gildor's decompressor.

 

I just wouldn't distribute them with the program (unless dreadylein/Gildor wanted to, of course).

 

In fact, each and every requirement could be a seperate program:

 

1) Patching the UPK bytecode (moderately time-consuming)

2) Decompressing UPKs(already done -- Gildor's decompressor)

3) Filehandling of the little <filename>.uncompressed_size files (easy)

4) SHA hash patching of the main executable file (already done -- XHSAPE)

 

Then we just need another program (a GUI, or yet another command line tool) to tie them all together.

 

The other possible features are really all just different aspects of the thing, mostly polish (especially the GUI). A nice GUI isn't easy.

Edited October 20, 2012 by Daemonjax

[dose206]

Right now XHSAPE does SHA patching of the main executable, nothing else.

 

This program would have to do what XSHAPE already does, but it also has to do way more.

 

We can do the project using more than one programming language if need be. The separate parts of the program could communicate with each other using network packets over loopback on some port :D Doing that isn't as hard as it sounds, and might even be preferable... at least for the GUI and Backend.

 

Okay, gotcha.

 

So if I understand the pipeline correctly, it works as follows:

1. decompress .upks

2. patch in changes to .upk bytecode

3. update ..._filesize file to reflect that the new .upk is a different size

4. mod the .exe to reflect that the new upk has a different SHA hash (which essentially points the .exe to our modified file instead of using the original)

 

I'd assume we want to allow Player A (from your earlier example) to be able to load the mod that Player B made AND the mod(s) that Players C, D, E, etc. have made as well, assuming that none of those mods directly overwrite each other. Assuming my basic understanding of the pipeline is correct, that means that when adding any new mod, we have to ensure that patching the bytecode ONLY patches the affected lines of that .upk. file, in order to avoid overwriting any earlier mods that made changes to other lines of that .upk

 

It seems to me that in order to distribute any particular mod, all we'd need are 1. the new lines of code and 2. the .upk file and location within that file where the new lines should be patched.

 

This points to, perhaps, a custom file format? If I make a mod, all I need to do is put my changes and instructions together in the right way and our program will be able to read it.

 

Is this off base?

[Crashdown1]

I know this isn't the place to ask, and I really don't want to seem like I don't appreciate what you guys are doing. I don't think anyone here that you guys and what you are doing is amazing, so thank you :).

 

I was just wondering though Daemonjax, you said you were working on a program to make hex editing the files a little easier. Is that included in this program or is that something else.

[Anbar]

I'd assume we want to allow Player A (from your earlier example) to be able to load the mod that Player B made AND the mod(s) that Players C, D, E, etc. have made as well, assuming that none of those mods directly overwrite each other.

 

i wouldnt over-complicate it for now... its highly likely that files will overwrite if they try to "load" multiple mods.. almost certain in fact.

 

keep it simple and get it working for 'just' one mod is my advice fwiw.

[dose206]

 

i wouldnt over-complicate it for now... its highly likely that files will overwrite if they try to "load" multiple mods.. almost certain in fact.

 

keep it simple and get it working for 'just' one mod is my advice fwiw.

 

Theoretically it doesn't seem that complicated. We're already talking about programmatically isolating lines of code, erasing them and replacing them with something else. In an effort to avoid copyright issues we want to avoid distributing complete .upks, which means we'll HAVE to selectively edit individual lines of code. If we read through an entire file, copy the entire contents, then parse out and edit the section we're aiming for, we shouldn't have issues with completely overwriting an earlier mod. The only tricky part would be determining if a given mod will overwrite lines already altered by an earlier mod, but that doesn't have to be our concern, rather an issue for mod developers to concern themselves with.

 

However, your advice to keep KISS (keep it simple, stupid) in mind is, as this type of advice will always be, timely and certainly welcome in my eyes.

[Daemonjax]

I think dose206 got it right. We should have no trouble with multiple mods writing to the same upk, but we can also prevent any potential problems from occurring with some simple checks. The user can bypass them and go full steam ahead if they choose, but I'd like something in there that checks if you try to load two mods that write to the same byte(s) in a upk.

 

The general idea of things:

 

1. Load a .xcom.mod file
2. IF upk is compressed:
       Backup original
       Create uncompressed copy of it
       Rename <filename>.upk.uncompressded_size
3. IF bytes in .xcom.mod file don't match up with expected values (with user option to bypass check with a big warning sign)
       Patch in changes to .upk bytecode         
       Update SHA hash in exe
   ELSE IF upk was compressed, then undo all changes

 

So, the .xcom.mod file should not only contain the changes to be made, but ALSO it should contain the original values of the bytes it expected to be at those locations.

 

For each byte to be changed we can have at least 4 fields:

 

FILENAME,

OFFSET,

EXPECTED_VALUE,

MOD_VALUE

 

Sound good so far? Any other ideas? For the backup system... I think it would be best to only keep one copy... just the original compressed file, and after that, just uncompress and reapply whatever mods to it... rather than keeping X copies of slightly modified upk files. Kinda like a differential backup system. This idea somewhat changes the above, but whatever. What do you guys think?

Edited October 20, 2012 by Daemonjax

[dose206]

 

The general idea of things:

 

1. Load a .xcom.mod file
2. IF upk is compressed:
       Backup original
       Create uncompressed copy of it
       Rename <filename>.upk.uncompressded_size
3. IF bytes in .xcom.mod file don't match up with expected values (with user option to bypass check with a big warning sign)
       Patch in changes to .upk bytecode         
       Update SHA hash in exe
   ELSE IF upk was compressed, then undo all changes

 

This seems like a good solution. However, I'm a little confused by the Else If in your pseudo code. Wouldn't it never be true that the upk would be compressed at that point, since it is uncompressed in step 2?

 

I'm going to try and get my rig set up for modding tonight (downloading all the tools that are currently being used/ been made by others here). Hopefully I'll have time tomorrow to take one of the minor mods with well documented bytecode changes and write up a little java goodness that will automate it. However, with an exam early Monday morning, studying might get precedence... :)

Edited October 20, 2012 by dose206

[Daemonjax]

However, I'm a little confused by the Else If in your pseudo code. Wouldn't it never be true that the upk would be compressed at that point, since it is uncompressed in step 2?

 

Yes, but it may have failed due to the bytes not being what the mod expected... like if the upk was changed by Firaxis and those bytes are different now. Of course, the user can bypass that (I wouldn't), but that's up to the user.

 

 

The other thing we could do is in additional to storing the byte's offset within the file, we could also store the hash value of the bytes around the area to be modified, and directly search through the upk to find a match entirely by hash value. The reason I say we should use hash values is so that we avoid entirely the problem of distributing any part of the game's assets.

 

Both ways have advantages disadvantages, and neither is particularly difficult to do. Perhaps using both methods simultaneously would be best, but then it might be slightly more complicated for the user to decide if they want to go through with the changes in certain circumstances. But, whatever. Choices are good, imo.

 

Hmm... perhaps it could work like this:

 

Player A (the modmaker) is creating his mod. He modifies a function from the unpacked files of a upk. Now, to create the mod, he needs the:

 

1) original unpacked file (filename.upk.original)

2) modified unpacked file (filename.upk.modified)

3) filename of the upk it was in

 

Not too complicated.

 

We can make it easier for him by having a configuration file that simply points to a static folder where all his unpacked files are, so he won't even need to worry about #1.

 

Then he just uses this program, creates a mod package, and shares the file to other players. We can even have it automatically zip it up. And NONE of the game's assets would be contained in the file he shares. Nothing at all.

 

I dunno, just thinking aloud, really :D

Edited October 20, 2012 by Daemonjax

[dose206]

 

The other thing we could do is in additional to storing the byte's offset within the file, we could also store the hash value of the bytes around the area to be modified, and directly search through the upk to find a match entirely by hash value. The reason I say we should use hash values is so that we avoid entirely the problem of distributing any part of the game's assets.

 

Both ways have advantages disadvantages, and neither is particularly difficult to do. Perhaps using both methods simultaneously would be best, but then it might be slightly more complicated for the user to decide if they want to go through with the changes in certain circumstances. But, whatever. Choices are good, imo.

 

 

We'd have to use both methods, if we want to support mods on top of mods. If we were only looking at the hash values, and they had been modified previously, our program would have no way of knowing where to patch in the new code. I do love the idea of using hash values, either around the area to be modded, or as the "expected value" that you mentioned previously, but we would need to have the byte's offset as well.

 

 

Player A (the modmaker) is creating his mod. He modifies a function from the unpacked files of a upk. Now, to create the mod, he needs the:

 

1) original unpacked file (filename.upk.original)

2) modified unpacked file (filename.upk.modified)

3) filename of the upk it was in

 

Not too complicated.

 

We can make it easier for him by having a configuration file that simply points to a static folder where all his unpacked files are, so he won't even need to worry about #1.

 

Then he just uses this program, creates a mod package, and shares the file to other players. We can even have it automatically zip it up. And NONE of the game's assets would be contained in the file he shares. Nothing at all.

 

 

I was thinking it could be even more simple. Our modder creates a single file that contains, like you mentioned previously, the name of the file to be modified (filename.upk.original, and it could optionally check for filename.upk.modified, if the user knows this file has been modified previously). It would also contain the byte offset and perhaps the hash values, and then of course the new code to patch in. That's it, at least starting out. All our modder has to do to distribute his mod is send this one file out, and all our user has to do is run this file through our program. There could be some config settings for the user, such as where all the unmodified upks are, as well as the modified ones, but it seems to me that it doesn't have to be more complicated than that. Unless, of course, we want to support more heavily the mod makers as well. I guess it depends on exactly what the scope of this part of the program needs to be.

 

EDIT: After reading your suggestion again, it sounds like we're suggesting essentially the same thing

 

EDIT2: I reread through ALL the relevant threads in an effort to make sure I have all the proper software and understand the process properly. I haven't actually tried anything yet so pardon my ignorance, but aren't we talking about doing (essentially) what dreadylein's Modpatcher is already doing? If not, then I'm having trouble understanding what that program does.

Edited October 21, 2012 by dose206