Jump to content

UPK file format


wghost81

Recommended Posts

Yes, all child objects are chained together though links, not positions. I linked a number of articles, explaining cooking process and seekfree concepts. Those articles say, that script packages are seekfree, but native script packages are not, as they contain crazy dependences. But regardless, cooker definitely has some rules, he tries to follow anyway.
Link to comment
Share on other sites

  • Replies 111
  • Created
  • Last Reply

Top Posters In This Topic

dubiousintent, document is not complete yet, so adding a link may be better.

 

Amineri, adding Import Table entries is possible. You'll need to add all missing names to nametable and construct all missing objects.

 

I still don't like the idea of expanding objects in-place, though. My policy is to make any change as small as possible. But I can add support for both methods.

Link to comment
Share on other sites

 

I still don't like the idea of expanding objects in-place, though. My policy is to make any change as small as possible. But I can add support for both methods.

 

I think appending is probably the safer approach for actually installing mods, but using the "in-place" method has already been useful to me in development. There was a function I was sure I was going to have to increase in size (after spending about 30 minutes trying to figure out a way to make it fit) so I went ahead and wrote it with the increased size.

 

Later I ran into another problem which prompted a design change. This design change meant that the function no longer had to be increased in size. Since I had done the increase in-place I was able to completely undo any changes made to the upk without having to resort to a backup or a Steam verify.

 

You know how it is when developing code ... you change your mind, and then change it again ;)

 

 

 

Amineri, adding Import Table entries is possible. You'll need to add all missing names to nametable and construct all missing objects.

 

I knew I'd have to create the additional name table entries. The two things I want to import are both struct members of a struct that is being imported, as some other struct members are used. Is it as simple as creating the import table entries that reference the package and and outer import object? As far as I've seen Import Table entries don't otherwise refer to any other sort of data structure within the upk.

Link to comment
Share on other sites

2Amineri

 

iMobilityBonus Name Table Entry:

0F 00 00 00 // length = 15

69 4D 6F 62 69 6C 69 74 79 42 6F 6E 75 73 00 // "iMobilityBonus"

00 00 00 00 10 00 07 00 // NameFlags (don't know the meaning, seems to be the same for all XCOM names)

 

iLargeItems Name Table Entry:

0C 00 00 00 // length = 12

69 4C 61 72 67 65 49 74 65 6D 73 00 // "iLargeItems"

00 00 00 00 10 00 07 00 // NameFlags

 

Import Table Entries will be the same as for XComGame.XGTacticalGameCoreNativeBase.TArmor.iHPBonus. You'll just need to change iHPBonus index to newly added iMobilityBonus/iLargeItems index.

 

If you already have the working code for export objects data offset recalculation, you could try to add new objects yourself. My own code is very ugly at the moment and requires a lot of manual support to work.

 

Keep in mind, that Unknown1 field in FPackageFileSummary seems to be equal to HeaderSize, so you'll need to adjust it too. And Import Table is located before Export Table.

 

If you'll be able to do this and use in actual mod, it will be a good confirmation of all our current assumptions.

Edited by wghost81
Link to comment
Share on other sites

I think increasing an objects size might actually be trickier since only some of the object entries' position fields have to change (only those that are later in the file than the object being increased in size).

 

X built a nice class that parses the header for a upk, including namelist, import list and object list, so it's pretty easy to loop over all objects in a upk and check / update their file position. The part I'm a bit hazy on is the "Depends" part of the package file. I haven't parsed anything to do with those (it wasn't needed even in order to increase the size of a function in-place). If the Depends data comes before the actual object data that would make sense, but I'd have to update the file position if I start inserting space back in the tables themselves.

 

I think I'm going to defer some of this experimentation on new features, as we're reaching the stage of a minor "file management crisis" in our Long War development ;). JL and I have been using the prototype UPKmodder app to update Long War, and so far we're up to ~120 functions changed. This is not even half of the functionality from the final EU version, plus we have all of the new stuff we'd like to add. So far UPKmodder's project info / management is kind of subpar.

 

However if we could eventually go a bit crazy and add lots of new objects as we like, it could create a lot of niceties. I've been able to confirm increasing the size of a fixed localization string array and being able to configure additional localization entries into the larger array. However we havn't been able to add additional enumeration entries, so the config/localization arrays had to be indexed using numbers instead of enum names.

 

If we can add new enum entries to existing enum containers, though, we could extend the enums, allowing them to be used by the config/localization loader, making the extended config/localization files that much easier to read.

 

I'm a little worried about potential performance impacts though. I noticed that the namelist is sorted alphabetically, and I can't imagine that was by accident. Plus adding a lot of objects far from their containers could slow down the de-serialization and hence the loading process. I don't think it would be a big deal if we added 5 or 10 new objects, but if we start adding 100s it could start to become noticable? And we could add that many if we start adding enum values ... off hand I'd want to add 16 new soldier classnames (8 for the Long War subclasses, and 8 more for the MEC variants derived from those subclasses). Not to mention adding new research projects, foundry projects, items, and perks. It could get a little crazy @_@.

 

If you go here: http://hem.bredband.net/bertrich/XCOM/ReverseLookup.htm to Bertilsson's enum lookup, you can see there are 5084 enumerated values defined in Enemy Within. So the 100's number isn't that crazy :)

Link to comment
Share on other sites

XCOM script packages are loaded when launching a game, so any slowdowns would affect game loading only. While loading a package, engine reconstructs all actual objects, and when it is done, file objects order becomes irrelevant. At least it is how I understood it from various articles concerning the subject. :smile:
Link to comment
Share on other sites

I think appending is probably the safer approach for actually installing mods, but using the "in-place" method has already been useful to me in development. There was a function I was sure I was going to have to increase in size (after spending about 30 minutes trying to figure out a way to make it fit) so I went ahead and wrote it with the increased size.

 

Later I ran into another problem which prompted a design change. This design change meant that the function no longer had to be increased in size. Since I had done the increase in-place I was able to completely undo any changes made to the upk without having to resort to a backup or a Steam verify.

 

You know how it is when developing code ... you change your mind, and then change it again :wink:

{ You may undo function expand }
EXPAND_UNDO=XGStrategyAI.GetAltWeapon
That command not only restores function size, it also restores the code, which is not possible with in-place modified functions. (or at least I think that is what it does, and in theory could do).
In my book that is a lot better from any perspective and would actually be the preferable mod method for any code change regardless if resizing is needed or not.
The dead space loss is so incredibly tiny that the only scenario where I can imagine it worth to even consider is if you create a gigantic mod and plan to redistribute the development file itself.
And even then I don't really see the argument, as that method by it's very definition is to re-distribute several MB of redundant data, while having an issue with kilobytes added data...
If we could have the option to FUNCTION_RESTORE_ORIGINAL=XGStrategyAI.GetAltWeapon and if all changes were always done to appended objects, then this would allow for partial mod uninstallations, making it possible in many (not all) cases to very easily undo unwanted features of large mods like Long War.
That way the dead space would not even be dead space, it would actually be uninstall and version data.
And if it is still an issue I personally think a cleanup/optimization tool would be the proper way to go, which would reverse lookup and remove all appended objects that are not the last version or original (missing the hash tag).
Edited by Bertilsson
Link to comment
Share on other sites

 

 

That command not only restores function size, it also restores the code, which is not possible with in-place modified functions. (or at least I think that is what it does, and in theory could do).

 

When using BEFORE/AFTER info, the in-place undo returns the upk to its original state, even replacing the original code (from the BEFORE block). I've used HxD to verify that the upk is identical size and identical at the byte level after an in-place resize apply+revert.

 

The append method doesn't technically restore to the original state, although functionally it does. Whether that really matters is really (I think) the core of the issue we're discussing here. As you say, having duplicate code could be seen as a feature, a detriment, or ultimately irrelevant.

 

 

 

The dead space loss is so incredibly tiny that the only scenario where I can imagine it worth to even consider is if you create a gigantic mod and plan to redistribute the development file itself.

 

I'm not worried about the space but about having two (or more) possible locations where there could be code. When you're dealing with 100's of hex replacements done over an extended period (and with multiple modders) these are the things I worry about :)

 

 

 

And if it is still an issue I personally think a cleanup/optimization tool would be the proper way to go, which would reverse lookup and remove all appended objects that are not the last version or original (missing the hash tag).

 

I think this would be possible, but not trivial. If you append new block A, then append new block B, and then undo the application of new Block A, in order to clear up the space you have to find the redundant block (probably possible with the extra tags), then rewrite the file to remove the excess space, and then correct the file offsets of the blockB (and any other blocks after that point). That algorithm is identical to what the in-place resizing does.

 

 

 

making it possible in many (not all) cases to very easily undo unwanted features of large mods like Long War

 

It's technically always been possible to undo any of the changes in Long War. JL posted up the complete hex changelist to the Nexus site. Whether reverting a particular piece of code results in a stable game (as in doesn't crash) is another issue. I'm not quite certain how having the original code embedded within the upk (which would only be for expanded functions) and requiring a file position change within the Export Table entry is any easier for a user than having before/after hex and using HxD.

 

-----------

 

I'll say it once again ... the extra memory usage is not my concern at all. My concern is actually having duplicate (and extraneous) hex for the same function object at the same time within the upk. Technically I know that the Export Table entry only points to one of the objects, so execution isn't affected. It adds an extra level of complication when trying to manage what's going on. I've had occasion (several times) to do a raw hex search for a functio header over the entire upk. With the duplicate entries it pops up twice. With a little more effort I can resolve which one is active, but it's just another step that is prone to error while developing.

 

If you're even been brain dead after staring at code (and hex bytecode to boot) for several hours, I'm sure you can appreciate how easy it is to make that kind of error :D

 

-------------

 

In the end, in terms of installation, I don't think there is any practical significant difference. Both methods will execute correctly, file size difference is negligible, and both methods can be functionally undone.

 

In terms of development it probably depends on the coder and the coding methodology. I personally prefer the in-place method while developing mods. YMMV :cool:

Link to comment
Share on other sites

I think, its just a matter of taste. :smile: I prefer to locate object data by its export table offset, extract it and change separately. And I don't like search-replace style, as search string uniqueness is not guaranteed. I prefer to always know what exactly I'm working with and where exactly it is. This principle is what driven me to look deeper into UPK file format in the first place. :smile:

 

I'm developing small mods mostly and moving functions preserves compatibility with other mods, as no other objects are affected. Mods which change the same data are incompatible in most cases anyway, so I'm not concerned with it. People use a bunch of different tools to distribute/install mods and they are still compatible.

 

But different people may use different approaches. This is especially true for big projects. And the best way is to provide a support for any type of function expansion. And that's exactly what I decided to do. :smile:

Edited by wghost81
Link to comment
Share on other sites

It's technically always been possible to undo any of the changes in Long War. JL posted up the complete hex changelist to the Nexus site. Whether reverting a particular piece of code results in a stable game (as in doesn't crash) is another issue. I'm not quite certain how having the original code embedded within the upk (which would only be for expanded functions) and requiring a file position change within the Export Table entry is any easier for a user than having before/after hex and using HxD.

It seems a lot easier to tell someone to install this mod:

UPK_FILE=XComStrategyGame.upk
FUNCTION_RESTORE_ORIGINAL=XGFacility_Barracks.PickAClass
{End of mod}

Than telling the same person all steps involved in search and replacing specific bytes.

And if multiple functions were changed the difference would be bigger. Both from the mod authors perspective and the end-user perspective.

 

Also it was on the condition that all changed functions were appended as new objects and not only expanded functions, and that there was actually a FUNCTION_RESTORE_ORIGINAL option available in the mod tool.

And yes while it would be situational whether or not a feature could be disabled by restoring one or more functions to vanilla, I don't think it would be very rare.

 

 

I'll say it once again ... the extra memory usage is not my concern at all. My concern is actually having duplicate (and extraneous) hex for the same function object at the same time within the upk. Technically I know that the Export Table entry only points to one of the objects, so execution isn't affected. It adds an extra level of complication when trying to manage what's going on. I've had occasion (several times) to do a raw hex search for a functio header over the entire upk. With the duplicate entries it pops up twice. With a little more effort I can resolve which one is active, but it's just another step that is prone to error while developing.

 

 

That extra level of complexity is only because you haven't discovered how much easier life is once you isolate the function in a separate file and work on it directly (which doesn't necessarily mean that you will replace the entire function when finished).

In combination with a method for quickly and easily injecting such files into the upk (which wghost has provided) it is fewer steps involved and generally more efficient.

 

I understand that it is very much a matter of preference and that you most likely have a very fast and stable routine doing it via search and replace... And that the upk modder tool may be currently oriented towards before and after hexes.

 

But for someone who isn't that experienced at doing that way (like me) it seems a lot more complicated to keep track of at minimum 3 different sets of hexes (Original, current and next version).

And I wouldn't be surprised at all if the origin of needing to search for the header in the entire upk was really the result of losing track of what was the current hex before last change.

 

Please note that I have absolutely no interest in putting you in a spot or debating for the sake of debating. I have very much respect for you (especially since you have literally learned me most of the modding stuff I know).

I only want things to be as easy and efficient as possible and towards that goal I see very limited benefit of using before and after hexes. Especially when things are finally moving towards higher level modding (which you are a very large contributor to).

 

However having the option to do it either way is a very good thing, but even better would be if there was more or less consensus regarding a more or less standardized way to install mods which could be consistently used by all.

But I do understand that is very unlikely to ever happen, and if it did happen then something would likely change and overturn it right away. But nevertheless I think it is a good thing to strive for. :smile:

Edited by Bertilsson
Link to comment
Share on other sites

  • Recently Browsing   0 members

    • No registered users viewing this page.

×
×
  • Create New...