UPK file format

[XMarksTheSpot]

That's great news, good job! :thumbsup:

[Bertilsson]

Sounds more like you are a genius who still make a few human mistakes on occasion.

 

So is this discovery limited to only variables or does it work on functions and other stuff as well?

[wghost81]

I may still turn to be an idiot, so lets take it slowly. :smile:

 

What I just checked.

 

I added new namelist entry. Game started normally.

 

I added a new object (int variable) and linked it to existing non-native structure. UE Explorer showed my new variable and game started normally (started a new game, started a battle; loaded an existing game, started a battle).

 

I added a new variable to XGGameData.TAlienPod native structure. Again, all went good.

 

But. I haven't tried to actually USE those new variables, i.e. write/read something. This is the next step in confirming that everything works, especially in cases where native objects are involved. But, since script packages are loaded and deserialized at the very start, I assume, engine deserialized new variables and reconstructed UPK objects successfully.

 

Theoretically, this works with any type of object, beginning with class.

Edited December 27, 2013 by wghost81

[wghost81]

An experiment with writing/reading to/from new variable seems to went good. :smile:

 

What I did so far.

 

Added a new variable XGGameData.TAlienPod.wghost81NewVar:

struct native TAlienPod
{
    var XGGameData.EAlienPodType eType;
    var XGGameData.ECharacter eMain;
    var XGGameData.ECharacter eSupport1;
    var XGGameData.ECharacter eSupport2;
    var XGGameData.EItemType eMainAltWeapon;
    var XGGameData.EItemType eSupport1AltWeapon;
    var XGGameData.EItemType eSupport2AltWeapon;
    var XGGameData.EItemType wghost81NewVar;

    structdefaultproperties
    {
        eType=EAlienPodType.ePodType_Soldier
        eMain=ECharacter.eChar_None
        eSupport1=ECharacter.eChar_None
        eSupport2=ECharacter.eChar_None
        eMainAltWeapon=EItemType.eItem_NONE
        eSupport1AltWeapon=EItemType.eItem_NONE
        eSupport2AltWeapon=EItemType.eItem_NONE
        wghost81NewVar=EItemType.eItem_NONE
    }
};

Made a quick change to XComAlienPodManager.GetPodCharArray:

simulated function array<XGGameData.ECharacter> GetPodCharArray(out TAlienPod kPod, out array<XGGameData.EItemType> arrAltWeapon)
{
    local array<XGGameData.ECharacter> arrEnemies;

    // End:0x50
    if(kPod.eMain != 0)
    {
        arrEnemies.AddItem(kPod.eMain);
    }
    // End:0xA0
    if(kPod.eSupport1 != 0)
    {
        arrEnemies.AddItem(kPod.eSupport1);
    }
    // End:0xF0
    if(kPod.eSupport2 != 0)
    {
        arrEnemies.AddItem(kPod.eSupport2);
    }
    arrAltWeapon.AddItem(kPod.eMainAltWeapon);
    arrAltWeapon.AddItem(kPod.eSupport1AltWeapon);
    kPod.wghost81NewVar = kPod.eSupport2AltWeapon; // write to new var
    kPod.eSupport2AltWeapon = kPod.wghost81NewVar; // read from new var
    arrAltWeapon.AddItem(kPod.eSupport2AltWeapon);
    return arrEnemies;
    //return ReturnValue;    
}

Game worked.

 

Anyway, this is a huge change and it must be extensively tested before we can say for sure we aren't bound by any limitations from now on. :smile:

[Drakous79]

Wonderful! :thumbsup:

[Amineri]

I'm quite impressed.

 

I take you are completely de-serializing the package file (following the format in your earlier document), then adding the additional object (class, function, class variables, local variable, enum, struct element, etc) and then re-serializing the package file?

[Amineri]

I've been doing some thinking on this, and have a concern.

 

If I understand correctly what wghost has done above (inserting a new structure element), since the structure elements must have consecutive object ID's (from her document "Child objects are stored in Export Table in reverse order: ChildN, …, Child2, Child1, Owner. So next child index is always lesser than previous child index." ) that means that to accomplish the above requires inserting a new object identifier in the middle of the Export Table. The total number of objects in the table increases by 1, and all objects after the new structure element have the reference values increased by 1.

 

Functionally, this is equivalent to what happens when Firaxis recompiles their source code after having introduced a new element. Indeed, if they added a new structure element as wghost did, the Export Table and some reference values would be changed. Excepting the GUID and some versioning info, I'd expect the two files to be byte-level identical.

 

This is effectively the same as Firaxis releasing a new patch, and would render unusable any mods that directly utilize hex code with Export Table references (which is almost any mod beyond config file or variable changes). If many modders were to use this, then each such mod would effectively work like a different "patch version" of the game, making mod interoperability almost unworkable.

 

Offhand the only way I can see around this would be to require all mods to transform their references into a more resistant form -- the fully contexted name. The patcher would then resolve these to the Export Table (or Import Table or Name Table) reference values when patching. The same reference name would be converted into a different Object Table index for different versions (either different patch versions released by Firaxis or different re-serialized packages created via the method wghost outlines above).

[wghost81]

No-no-no! The general idea was to add new objects without the need to completely recook package! All existing references stay the same. I just haven't updated UPK format document to say: it is possible to break objects order and re-link last (or any other) object to newly created object, added to the end of the list.

 

So it is not equivalent to package recooking. And of course, new patch will break any existing mods, which use object references. But, again, adding new references in the manner I did, won't break anything.

 

The process is a bit complicated and I need to rewrite UPK Tools to handle this in a safe way. Right now I added new objects half-manually. :smile:

 

Step by step actions, needed to accomplish this:

 

Adding a new name:

1. Change header size entry to new value.

2. Increase NameCount by 1.

3. Change Import, Export and Depends offset to new value.

5. Copy all existing Name Table data.

6. Write new Name Table entry.

8. Copy Import Table data.

7. Recalculate all export objects data offsets and re-write Export Table.

8. Copy objects data.

 

Adding a new object:

1. Change header size entry to new value.

2. Increase ExportCount by 1.

3. Change Depends offset to new value.

4. Copy all existing Name, Import and Export Table data.

5. Write new Export Table entry.

6. Recalculate all export objects data offsets and re-write Export Table.

7. Copy objects data.

8. If you want new object to be a child of existing object, you need to modify Next field in old object's last child serialized data to point to the newly added object.

 

You can add new Import Table entries, process is similar to adding new Name Table entries.

Edited December 28, 2013 by wghost81

[Amineri]

 

 

8. If you want new object to be a child of existing object, you need to modify Next field in old object's last child serialized data to point to the newly added object.

 

So the objects contained "within" another object (e.g. local vars/parameters in a function, struct members in a struct, enumerated values in an enum, or functions/class variables in a class) are stored as a linked list and don't have to explicitly be adjacent values? Presumably them being adjacent is a form of optimization (so that when reading the data in it avoids having to skip around too much), but if it isn't required that is good news indeed.

 

If I try and paraphrase ....

 

1) Add a new name to the end of the namelist (readjusting all filepositions)

2) Add a new object to the end of the objectlist (readjusting all filepositions)

3) Link new object into linked list of objects contained in desired container

 

By adding the new name/object to the end of the list you preserve the indices. It was not realizing that step 3 could be made that led me to think that you'd have to insert the new object adjacent to the others in the container.

 

I was hoping that we'd be able to add new Import Table entries. XComStrategyGame.upk doesn't import the iMobility stat from TArmor, which makes it current nigh-impossible to display in the character sheet. I also want to import the iNumLargeItems field from TArmor in order to change how MEC customization works.

 

Given that you're writing code to completely rewrite the file positions for all of the Export object data offsets anyhow, are you considering re-thinking your position on the in-place vs append methodologies for expanding functions? Once you have the code written to rewrite all of the file positions resizing an object in-place is quite easy.

[dubiousintent]

OK, I'm ready to release an alfa version of UPK format description document. It is incomplete and may contain errors, so be careful.

 

I'm not taking any credits for discovering all these things, I just gathered the info from different sources.

 

For the last two days I tried to use this info to add new objects into the package. But I had no success. :sad: Even adding a new name to Name Table causes the came to crash. But an interesting thing is, UE Explorer decompiles my modified UPK correctly: I was able to add new names and objects and even add new variables to existing structures... but this all doesn't work with actual game. :sad: I don't know either I made some mistakes or XCOM just doesn't allow it. :sad:

Hmm. Nice. Do you want the current wiki article to link to your PDF file, or for me to transform your document into wiki format? (Linking is easy and you can update it when and as you desire as long as the URL doesn't change. The wiki can do tables, but that will take a bit more work. I'd sort of prefer to wait until you feel it is 'ready'.)

 

-Dubious-