R&D - New Modding Tools

[Drakous79]

When you click on UPKmodder v0.65.zip, there is an arrow icon in bottom right corner.

Edited December 23, 2013 by Drakous79

[Amineri]

For someone unfamiliar with google drive...

 

Is there a way to download UPKmodder v0.65.zip as a single file instead of browsing it?

 

It's a little unfortunate that Google Drive parses into the zip file. Down at the lower right there should be a download button (there is in my Chrome browser, anyhow).

Edited December 23, 2013 by Amineri

[Bertilsson]

Found it! Thank you

[Bertilsson]

Is there any chance that someone could make a small example on how to use the tool?

 

I'm guessing first step is to adjust UPKmodder v0.65\Config\upk_config.ini to reflect location and GUID of some local UPK's?

 

Next step would reasonably be to run UPKmodder.jar and to start a new Project...

 

But what is next? Create a mod and populate the mod data manually into the tool?

[Amineri]

I've included a few sample projects in the distribution (they should be in the UPKmodderProjects subfolder within the decompressed folder.

 

You should be able to open the project files (which are the .xml files in the projects' folder) -- for example UPKmodderProjects\Expanded Perk Tree EW\Expanded Perk Tree EW.xml. From there you can open individual modfiles by double-clicking on the file in the project pane. Alternatively you can directly open a sample modfile with the "Open File..." menu item.

 

This will let you look at a variety of sample upk_mod files (most of which are full function replacements).

 

There are some definite usability improvments we're still trying to make. The projects pane isn't as useful as it could be, for example. And associating target UPKs is still a file-by-file basis, which is a bit cumbersome.

 

When I'm creating a new modfile, I typically start off by copying the header from another function and then modifying it as necessary. I then open the function I'm going to change in UE Explorer's Token view, copy the data into a temporary Notepad++ document and use a macro to reformat it into something more readable by UPKmodder (stripping out the leading/trailing stuff around the hex, and the internal memory positions). I then copy the code into the [ CODE ] block in the BEFORE section. This triggers the limited parser in UPKmodder, which will color code the various hex parts accordingly, so I can double check that things are working as expected. I then fill out the memory/file positions (either directly from UE Explore's buffer view or from the UPKmodder tree view)

 

The next steps involve either doing another copy/paste of the hex into the AFTER / CODE section (if I'm primarly making modifications), using updated code from a EU-version mod that I'm trying to fix up, or writing entirely new code, depending on the particulars.

[dubiousintent]

I've included a few sample projects in the distribution (they should be in the UPKmodderProjects subfolder within the decompressed folder.

 

You should be able to open the project files (which are the .xml files in the projects' folder) -- for example UPKmodderProjects\Expanded Perk Tree EW\Expanded Perk Tree EW.xml. From there you can open individual modfiles by double-clicking on the file in the project pane. Alternatively you can directly open a sample modfile with the "Open File..." menu item.

 

This will let you look at a variety of sample upk_mod files (most of which are full function replacements).

 

There are some definite usability improvments we're still trying to make. The projects pane isn't as useful as it could be, for example. And associating target UPKs is still a file-by-file basis, which is a bit cumbersome.

 

When I'm creating a new modfile, I typically start off by copying the header from another function and then modifying it as necessary. I then open the function I'm going to change in UE Explorer's Token view, copy the data into a temporary Notepad++ document and use a macro to reformat it into something more readable by UPKmodder (stripping out the leading/trailing stuff around the hex, and the internal memory positions). I then copy the code into the [ CODE ] block in the BEFORE section. This triggers the limited parser in UPKmodder, which will color code the various hex parts accordingly, so I can double check that things are working as expected. I then fill out the memory/file positions (either directly from UE Explore's buffer view or from the UPKmodder tree view)

 

The next steps involve either doing another copy/paste of the hex into the AFTER / CODE section (if I'm primarly making modifications), using updated code from a EU-version mod that I'm trying to fix up, or writing entirely new code, depending on the particulars.

Added this to the 'Features' page on the Google Code wiki page. Couldn't see how to create a new page, so you will need to move it if you want it elsewhere.

 

-Dubious-

Edited December 28, 2013 by dubiousintent

[Amineri]

 

Couldn't see how to create a new page, so you will need to move it if you want it elsewhere.

 

I think you create a link as if linking it to a page name (although it uses full URI instead of shorthand names). Instead of a full red "dead" link it has a small blue '?' next to the text.

[Amineri]

Uploaded a v0.70 to the Downloads page. Most of the changes are to do with the functionality of the project pane, as well as various bug fixes and usability improvements :

 

 

 

 

Removing the RESIZE= command line should now allow for non-resize apply/revert operations without app restart

Fixed Search error that occurred if searching to the end of an object

0B and 0b tokens should now display in non-bold blue

Fixed hex parser to handle hex without a trailing space

Displays warning dialogue if attempt to close tab without saving (does currently not display on application exit)

Display '*' character to left of file name if file has been edited since last save

Target upk associations are stored in the project xml file, and are common across all modfiles with the same UPKFILE= header entry

Added "Test Install Status" command to Menu -- checks install status of active modfile

Added "Delete Project" command to Menu

Added pop-up menu commands to Project Pane - current commands are (New Project, Remove Project, Create New File)

Altered tab and font styles / sizes (r106)

Fixed bug that would cause a name reference in the last position of a line to cause the parser to fail

Fixed format so that Operand 0x0C not parsing correctly

Fixed crash bug when KEYWORD= token was used

 

 

 

It can be found at the UPKmodder Google Code site : https://code.google.com/p/upk-modder/ in the Downloads area (on the left side of the page).

[Amineri]

I was thinking it would be good to move the discussion on how "best" to use the upk file format information to enable patches into a separate forum so that wghost's upk format changes don't get derailed. I hadn't meant to highjack this thread into a UPKmodder-only thread, but more as a general how-to-create-tools thread, so I'm cross-posting/replying to some conversations here.

I think, its just a matter of taste. :smile: I prefer to locate object data by its export table offset, extract it and change separately. And I don't like search-replace style, as search string uniqueness is not guaranteed. I prefer to always know what exactly I'm working with and where exactly it is. This principle is what driven me to look deeper into UPK file format in the first place. :smile:

I'm developing small mods mostly and moving functions preserves compatibility with other mods, as no other objects are affected. Mods which change the same data are incompatible in most cases anyway, so I'm not concerned with it. People use a bunch of different tools to distribute/install mods and they are still compatible.

But different people may use different approaches. This is especially true for big projects. And the best way is to provide a support for any type of function expansion. And that's exactly what I decided to do. :smile:

 

I think we're all in agreement that using object names and relative offsets is preferable to using global file offsets, for a variety of reasons. The only advantage to global offsets is simplicity, and that simplicity comes at quite a high cost in terms of handling patches and various resizing operations.

 

You both have convinced me of the merits of being able to use relative offsets instead of requiring S&R, and I'm planning to incorporate their usage into the next release of UPKmodder. However they aren't purely beneficial in all cases, even when performing user-installs. Having the control to overwrite previous changes without going through the extra steps of reverting is a powerful tool, but it can mess things up just as easily as S&R can (but in different ways).

 

Even with ORIGINAL and MODDED_HEX sections to enable un-installs, using relative offsets can result in troubles that are difficult to recover from. Consider the following case :

1. User downloads MOD A and installs it, changing Function A
2. User downloads MOD B and installs it, which also changes Function A
3. If the installer uses relative offsets and doesn't verify that the contents of Function A were as expected when installing MOD B, then the contents of MOD A are partially written over. This can easily result in crashing/aberrent behavior (likely not at startup, but duing gameplay)
4. If the user then attempts to un-install MOD B, the only option is to revert the code back to the vanilla original state, which can still result in crashes / aberrant behavior, as now MOD A is effectively only partially installed.
5. The user would have to fully re-install MOD A over the top, or revert to a clean copy of the upks (via backups or Steam verify) and re-install
6. The situation gets more complex if more mods are involved... consider if MOD A, B, and C are installed, and MOD D partially overwrites different parts of them

For this reason when I add the relative offset feature to UPKmodder, it will still use BEFORE and AFTER (MODDED) hex in addition to the relative offsets, and will perform the following checks when performing apply/revert :

1. Look up absolute file position using specified Object name from UPK header
2. Navigate via relative offset to the correct file position
3. Read from file and verify that contents are as expected (If doing Apply operation, verify that BEFORE hex is currently in place, if doing Revert operation, verify that AFTER hex is currently in place)
4. If contents are as expected, perform operation as normal
5. If contents are not as expected, raise a warning dialogue indicating that contents did not match and that the operation cannot be undone -- user can select whether to overwrite or not

For development of a large mod (especially if using components from several authors) this is critially important, akin to merging files in a version-control repository.

 

For a mod installer for an end-user (who may be installing multiple mods from different authors), this is also quite important. My recommendation would be that the installer checks all of the hex to be installed prior to performing the actual install. For the GUI a confirmation dialogue will work, but for a command-line installer I'd think that aborting with an error message should be the default behavior, unless overridden via a command-line flag to force overwriting (alternatively a command-line flag to skip the error-checking process and simply overwrite).

 

 

 

That extra level of complexity is only because you haven't discovered how much easier life is once you isolate the function in a separate file and work on it directly (which doesn't necessarily mean that you will replace the entire function when finished).

In combination with a method for quickly and easily injecting such files into the upk (which wghost has provided) it is fewer steps involved and generally more efficient.

 

I understand that it is very much a matter of preference and that you most likely have a very fast and stable routine doing it via search and replace... And that the upk modder tool may be currently oriented towards before and after hexes.

 

But for someone who isn't that experienced at doing that way (like me) it seems a lot more complicated to keep track of at minimum 3 different sets of hexes (Original, current and next version).

And I wouldn't be surprised at all if the origin of needing to search for the header in the entire upk was really the result of losing track of what was the current hex before last change.

 

Please note that I have absolutely no interest in putting you in a spot or debating for the sake of debating. I have very much respect for you (especially since you have literally learned me most of the modding stuff I know).

I only want things to be as easy and efficient as possible and towards that goal I see very limited benefit of using before and after hexes. Especially when things are finally moving towards higher level modding (which you are a very large contributor to).

 

However having the option to do it either way is a very good thing, but even better would be if there was more or less consensus regarding a more or less standardized way to install mods which could be consistently used by all.

But I do understand that is very unlikely to ever happen, and if it did happen then something would likely change and overturn it right away. But nevertheless I think it is a good thing to strive for. :smile:

 

 

If I understand you right, you are referring to using Gildor's UPK extract utility to unpackage the upk and work on the individual components.

 

I tried this method, but I found it too cumbersome for me eventually, but that is primarily a matter of taste / experience (as well as the lack of nice tools like wghost's back then). I think that as things scale up working with the full upk starts to become more a matter of efficiency/necessity. With more changes integration testing becomes ever-more-important, and that can only be done by running the modified full upk. I think JL and I (while rebuilding LW for EW) are now somewhere around the 130-140 mark in number of modified functions, and we've still only got a bit over 1/2 of the EU LW functionality re-built. It's not uncommon for us to build a function that decompiles perfectly well in UE Explorer, but to have to modify it several more times after doing integration testing.

 

UPKmodder is currently designed around my design preferences. As much as I tried to make it general, I too have my blind spots, and I'm the one doing all the designing of it so far, with most of my feedback from JL and XMTS :tongue:. So ... definitely more of this type of feedback is helpful in trying to make it more broadly useful.

 

As I mentioned above, I am planning to incorporate usage of relative offsets with controls to allow the user to bypass search errors (thus allowing to edit the AFTER hex while the mod is still applied and overwrite the contents without doing search and replace).

 

 

 

debating for the sake of debating

 

:sweat: My grandmother used to say that this was my Norwegian blood coming into play...

[Drakous79]

Imagine a mod, that has the following setup:

• Original hex: A A A A
• Modded hex in the first release: B B B B
• Modded hex in the first update: C C C C

Patching issues:

• Patching is easy without checking for original hex. Original hex may be changed by other mods or official patch.
• Patching and not exactly knowing the final code (the result) can go wrong as Amineri suggested. Some safeguards should be fine.

If I'd take A A A A mod example:

// [original_hex] A A A A
// [compatible_hex] B B B B
// [modded_hex] C C C C

If (([original_hex] == vanilla hex) OR ([original_hex] == compatible hex))]
{
    patch;
}
else {
    prompt;
}

But something like this is viable when checking own mods. Would be better to check some pseudo code or generated hex from such code.

 

Complex overwrites can check for function's size, like if expected size differs than prompt.

Edited December 29, 2013 by Drakous79