R&D - New Modding Tools

[wghost81]

Trying to implement some simple C++ code to read upk header. I found some pieces of information on Unreal Wiki and on Eliot's homepage, but this is slightly different from what I see in XCOM upk files.

 

That's what I found so far.

 

First bytes of the package header seem to be:

uint32_t Signature;
uint16_t Version;
uint16_t LicenseeVersion;
uint32_t HeaderSize;
uint32_t FolderNameLength;
string   FolderName;
uint32_t PackageFlags;
uint32_t NameCount;
uint32_t NameOffset;
uint32_t ExportCount;
uint32_t ExportOffset;
uint32_t ImportCount;
uint32_t ImportOffset;

Signature=0x9E2A83C1 is an upk identifier. Version numbers denote package and license version info. HeaderSize is a size of entire header, including all name, import and export tables and some other information, which I couldn't figure out yet. FolderNameLength and FolderName are 5 and "None" (null-terminated c string) for XCOM. PackageFlags are unknown to me and Count-Offset pairs contain all the information needed to locate name list and object tables.

 

I wrote a simple code to extract name lists an object tables into memory arrays. Now I want to try and reconstruct full object names from this two tables. It wouldn't be so hard. But reverse procedure of obtaining hex code from variable name will be a pain. Searching linear arrays will take a lot of time. Need to try some other approach, like binary trees or something.

Edited November 21, 2013 by wghost81

[Amineri]

Good stuff.

 

I've been experimenting using Java -- sorry for the ugly code but I'm not very proficient with the language.

 

I wrote a much more primitive parser to extract just the header information I was looking for, and it may not be very robust:

 

 

    public void parseUPKHeader(Path thisfile) throws IOException
    {
        try(FileChannel fc = FileChannel.open(thisfile))
        {
            ByteBuffer buf = ByteBuffer.allocate(100);
            buf.order(ByteOrder.LITTLE_ENDIAN);

            fc.position(0x19);
            fc.read(buf);
            
            numNamelistEntries = buf.getInt(0);
            System.out.println("Namelist entries : " + numNamelistEntries);
            
            posNamelist = buf.getInt(4);
            System.out.println("Namelist start pos: " + posNamelist);
            
            numObjectlistEntries = buf.getInt(8);
            System.out.println("Objectlist entries: " + numObjectlistEntries);
            
            posObjectlist = buf.getInt(12);
            System.out.println("Objectlist start pos: " + posObjectlist);
            
            buf.clear();
        }
        catch (IOException x) 
        {
            System.out.println("caught exception: " + x);
        }
    }

 

 

 

This works for XComGame.upk, but I'm not sure about about upks in general.

 

I've got classes to do the basic reading of the namelist and objectlist tables as well:

 

Namelist:

 

 

    public Boolean parseNamelist(Path thisfile) throws IOException
    {
        int iCurrPosition = m_iPositionNamelist;
        int iStringLength;  
        
        ByteBuffer buf = ByteBuffer.allocate(100);
        buf.order(ByteOrder.LITTLE_ENDIAN);

        // Read the bytes with the proper encoding for this platform.  If
        // you skip this step, you might see something that looks like
        // Chinese characters when you expect Latin-style characters.
        String encoding = System.getProperty("file.encoding");

        if (m_iNumNamelistEntries == -1 || m_iPositionNamelist == -1)
        {
            return false;
        }
        try(FileChannel fc = FileChannel.open(thisfile))
        {
            
            for(int count = 0; count < m_iNumNamelistEntries ; count++)
            {
                StringBuilder testString = new StringBuilder();
                fc.position(iCurrPosition); // seek to current namelist entry
                if(DEBUG)
                {
                    System.out.println("Position: " + fc.position());
                }
                fc.read(buf);
                iStringLength = buf.getInt(0);
                buf.rewind();
                testString.append(Charset.forName(encoding).decode(buf), 4, 3+iStringLength);
                m_arrNameListStrings[count] = testString.toString();
                if(DEBUG)
                {
                  System.out.println("Namelist[" + count + "]: " + m_arrNameListStrings[count]);
                }
                buf.clear();
                iCurrPosition += 12 + iStringLength;
            }

        }
                catch (IOException x) 
        {
            System.out.println("caught exception: " + x);
            return false;
        }
        return true;
    } 

 

 

 

 

Objectlist:

 

 

    public Boolean parseObjectlist(Path thisfile) throws IOException
    {
        int iCurrPosition = m_iPositionObjectlist;

        ByteBuffer buf = ByteBuffer.allocate(BYTES_PER_OBJECT_ENTRY);
        buf.order(ByteOrder.LITTLE_ENDIAN);

        if (m_iNumObjectlistEntries == -1 || m_iPositionObjectlist == -1)
        {
            return false;
        }
        try(FileChannel fc = FileChannel.open(thisfile))
        {
            for(int count = 1; count <= m_iNumObjectlistEntries ; count++)
            {
                fc.position(iCurrPosition); // seek to current namelist entry
                if(DEBUG)
                {
                    System.out.println("Position: " + fc.position());
                }
                fc.read(buf);
                
//                IntBuffer temp = buf.asIntBuffer();
                for (int I = 0; I < INTEGERS_PER_OBJECT_ENTRY ; I++)
                {
                    m_arrObjectlist[count][I] = buf.getInt(4*I);
                }
                if(DEBUG)
                {
                    System.out.print("Objectlist[" + count + "]: ");
                    for (int I = 0; I < INTEGERS_PER_OBJECT_ENTRY ; I++)
                    {
                        System.out.print(m_arrObjectlist[count][I] + ", ");
                    }
                    System.out.println();
                }
                buf.clear();
                iCurrPosition += BYTES_PER_OBJECT_ENTRY;
            }

        }
                catch (IOException x) 
        {
            System.out.println("caught exception: " + x);
            return false;
        }
        return true;
    }     

 

 

 

These appear to be working, at least for XComGame.upk.

 

I've got some initial code that is mostly working for matching up the object list with the contextualized names:

 

 

    public Boolean constructObjectNames(upkNameList kNamelist)
    {
        int iTempNameIndex;
        int iOwnerIndex, iPrevOwnerIndex;
        int iType;
        String sTempString;
        
        for(int I = 1; I <= m_iNumObjectlistEntries; I++)
        {
            iType = getObjectlistEntry(I, 0);
            
            if(iType !=0)
            {
                iPrevOwnerIndex = -1;
                iTempNameIndex = getObjectlistEntry(I, 3);
                sTempString = kNamelist.getNamelistEntry(iTempNameIndex);
                m_arrObjectNames[I] = sTempString;
                if(iType > -400 || iType < -1000)
                {
                    iOwnerIndex = getObjectlistEntry(I, 2);
                    while(iOwnerIndex <= m_iNumObjectlistEntries && iOwnerIndex > 0 && iOwnerIndex != I && iPrevOwnerIndex != iOwnerIndex)
                    {
                        m_arrObjectNames[I] = m_arrObjectNames[I].concat(".");
                        iTempNameIndex = getObjectlistEntry(iOwnerIndex, 3);
                        if(iTempNameIndex > 0)
                        {
                            if(iTempNameIndex > kNamelist.getNumNamelistEntries())
                            {
                                iOwnerIndex = -1;
                            }
                            else
                            {
                                sTempString = kNamelist.getNamelistEntry(iTempNameIndex);
                                m_arrObjectNames[I] = m_arrObjectNames[I].concat(sTempString);
                                iPrevOwnerIndex = iOwnerIndex;
                                iOwnerIndex = getObjectlistEntry(iOwnerIndex, 2);
                            }
                        }
                        else
                        {
                            iOwnerIndex = -1;
                        }
                    }
                }
//                if(DEBUG)
                {
                    System.out.println("Objectlist[" + I + "]: " + m_arrObjectNames[I]);
                }
            }
        }
        return true;
    } 

 

 

 

Currently this appears to work for the first ~55,000 objects out of the 56,228 objects in XComGame.upk.

 

For some reason I'm starting to get bad object/name list indices at that point.

 

Here is some sample output from the middle of the object reference list:

 

 

Objectlist[37364]: GetProjectileDamage.XGAction_Fire
Objectlist[37365]: kView.Destroyed.XGAction_Fire
Objectlist[37366]: Destroyed.XGAction_Fire
Objectlist[37367]: kView.MarkOrDestroyView.XGAction_Fire
Objectlist[37368]: MarkOrDestroyView.XGAction_Fire
Objectlist[37369]: iReflect.Projectile_OnInit.XGAction_Fire
Objectlist[37370]: kProjectile.Projectile_OnInit.XGAction_Fire
Objectlist[37371]: Projectile_OnInit.XGAction_Fire
Objectlist[37372]: bValidShot.Projectile_OnShutdown.XGAction_Fire
Objectlist[37373]: KillCount.Projectile_OnShutdown.XGAction_Fire
Objectlist[37374]: Victim.Projectile_OnShutdown.XGAction_Fire
Objectlist[37375]: TargetUnit.Projectile_OnShutdown.XGAction_Fire
Objectlist[37376]: kProjectile.Projectile_OnShutdown.XGAction_Fire
Objectlist[37377]: Projectile_OnShutdown.XGAction_Fire
Objectlist[37378]: vTarget.Projectile_OnDealDamage.XGAction_Fire
Objectlist[37379]: kTarget.Projectile_OnDealDamage.XGAction_Fire
Objectlist[37380]: kProjectile.Projectile_OnDealDamage.XGAction_Fire
Objectlist[37381]: Projectile_OnDealDamage.XGAction_Fire
Objectlist[37382]: ReturnValue.CanDisplayDamageMessage.XGAction_Fire
Objectlist[37383]: CanDisplayDamageMessage.XGAction_Fire
Objectlist[37384]: ReturnValue.WeaponWasFired.XGAction_Fire
Objectlist[37385]: WeaponWasFired.XGAction_Fire
Objectlist[37386]: ReturnValue.IsDoneWithAbility.XGAction_Fire
Objectlist[37387]: kAbility.IsDoneWithAbility.XGAction_Fire
Objectlist[37388]: IsDoneWithAbility.XGAction_Fire
Objectlist[37389]: strRep.InitialReplicationData_XGAction_Fire_ToString.XGAction_Fire
Objectlist[37390]: ReturnValue.InitialReplicationData_XGAction_Fire_ToString.XGAction_Fire 

 

 

[Amineri]

It looks like my hypothesis that the objectlist entries were all exactly 68 bytes (17 words) is being tested.

 

Most entries, including those for most objects of interest appear to conform to this, but near the end of the objectlist, if it is parsed according to this rule I start to see:

 

• Objectlist[55041]: -395, 0, 0, 4580, 0, 0, 0, 458756, 12, 11374309, 1, 1, 91, 251938573, 1106949789, -2013702505, -1915016781,
• Objectlist[55042]: 536870913, -395, 0, 0, 4782, 0, 0, 0, 458756, 12, 11374321, 1, 1, 646, -529985599, 1094784748, -2049321541,
• Objectlist[55043]: -2065696142, 536870913, -395, 0, 55042, 1088, 0, 0, 0, 458756, 12, 11374333, 1, 0, 0, 0, 0,
• Objectlist[55044]: 0, 0, -395, 0, 0, 4795, 0, 0, 0, 458756, 12, 11374345, 1, 1, 4864, -1657728135, 1302639489,
• Objectlist[55045]: -1703016810, 877446254, 536870913, -395, 0, 55044, 1088, 0, 0, 0, 458756, 12, 11374357, 1, 0, 0, 0,
• Objectlist[55046]: 0, 0, 0, -395, 0, 0, 10805, 0, 0, 0, 458756, 12, 11374369, 1, 1, 2254, 1297293924,
• Objectlist[55047]: 1290797989, 2080432809, 522670513, 536870913, -395, 0, 0, 12479, 0, 0, 0, 458756, 12, 11374381, 1, 1, 40,
• Objectlist[55048]: -1230367986, 1200576940, 1666547358, 1037699589, 536870913, -395, 0, 55047, 23594, 0, 0, 0, 458756, 12, 11374393, 1, 0,
• Objectlist[55049]: 0, 0, 0, 0, 0, -395, 0, 55047, 23821, 0, 0, 0, 458756, 12, 11374405, 1, 0,
• Objectlist[55050]: 0, 0, 0, 0, 0, -395, 0, 0, 13774, 0, 0, 0, 458756, 12, 11374417, 1, 1,

I've manually highlighted the -395 values to try and make the pattern appear more clearly. I don't know what the -395 represents, but the first valid string pointed to was "CH_BaseChar_ANIMTREE".

And of course is the first identifier that UE Explorer lists under the Content tab instead of the Classes tab. Definitely am re-inventing stuff that EliotVU has already figured out in UE Explorer ^_^.

[Amineri]

It turns out the word 11 (counting from 0) is indicating additional words that are being added onto the structure.

 

When I modified for that, everything fell into place nicely and entry in the table pointed to a valid string.

 

Here is my modified (yet still hacked up) Java function that reads in the objectlist:

 

 

    public Boolean parseObjectlist(Path thisfile) throws IOException
    {
        int iCurrPosition = m_iPositionObjectlist;
        int iExtraBytes = 0;
        
        ByteBuffer buf = ByteBuffer.allocate(BYTES_PER_OBJECT_ENTRY);
        buf.order(ByteOrder.LITTLE_ENDIAN);

        if (m_iNumObjectlistEntries == -1 || m_iPositionObjectlist == -1)
        {
            return false;
        }
        try(FileChannel fc = FileChannel.open(thisfile))
        {
            for(int count = 1; count <= m_iNumObjectlistEntries ; count++)
            {
                fc.position(iCurrPosition); // seek to current namelist entry
                if(DEBUG)
                {
                    System.out.println("Position: " + fc.position());
                }
                fc.read(buf);
                
                iExtraBytes = buf.getInt(4*11);
                buf.rewind();
                for (int I = 0; I < 17 + iExtraBytes ; I++)
                {
                    m_arrObjectlist[count][I] = buf.getInt(4*I);
                }
                if(DEBUG)
                {
                    System.out.print("Objectlist[" + count + "]: ");
                    for (int I = 0; I < 17 + iExtraBytes ; I++)
                    {
                        System.out.print(m_arrObjectlist[count][I] + ", ");
                    }
                    System.out.println();
                }
                buf.clear();
                iCurrPosition += 4*(17 + iExtraBytes);
            }

        }
        catch (IOException x) 
        {
            System.out.println("caught exception: " + x);
            return false;
        }
        return true;
    }     

 

 

[wghost81]

Thanks for the update, Amineri, got my code working too. :smile:

 

A took a function I worked with (ProcessPodTypes) and a variable from this function (iNumPods) and tried to locate it's entry:

Object table index: 0X51CF (20943)
Object type: 0XFFFFFEE6 (unknown)
Parent class reference: 0
Owner reference: 0X51D4
Index to name list table: 0X16D8
Object file size: 0X28
Object data offset: 0X47EA6E
Name from name list table: iNumPods
Owner chain: iNumPods -> ProcessPodTypes -> XGStrategyAI

Object type (not only for this entry) doesn't seem to correspond with what you wrote in one of the previous posts. For local variables I always get 0XFFFFFEE6.

[wghost81]

Quick update on object entry format. First, there is no null-object in object table inside upk file, so it's first element has index 1, not 0. Name list first index is 0. Object list entry consist of 17 static uint32 fields + additional fields. NumAdditionalFields is defined in Field11 (counting from 0), so total entry size is 17 * 32 + NumAdditionalFields * 32. Seems to work! Hooray! :smile: Edited November 21, 2013 by wghost81

[Amineri]

I independently discovered the same thing... objectlist start index at 1 instead of 0, while namelist start index is 0.

[wghost81]

OK, I've made a pretty straightforward console utility which takes upk file and full object name as an arguments and finds the corresponding entry in object list.

>FindObjectEntry.exe XComStrategyGame.upk XGStrategyAI.ProcessPodTypes.iNumPods
Object table index: 0X51CF (20943)
Object type: 0XFFFFFEE6 (unknown)
Parent class reference: 0
Owner reference: 0X51D4
Index to name list table: 0X16D8
Object file size: 0X28
Object data offset: 0X47EA6E
Name from name list table: iNumPods
Object name: XGStrategyAI.ProcessPodTypes.iNumPods

Now, if we want to "repair" some function, we need to search it's hex-code for object references, extract those objects names from old upk, find a new reference in new upk and replace old reference with a new one. Not so hard for variables and functions, but I'm starting to think about object arrays...

 

Yes, seems we're re-inventing UE Explorer. :smile: Sad, there is no upk format description and no developer files for that dll.

 

PS Sometimes being sick and spending more time at home increases productivity... :smile:

Edited November 21, 2013 by wghost81

[Amineri]

As a double check I processed XComStrategyGame.upk as well and can confirm:

Objectlist[20943]: iNumPods.ProcessPodTypes.XGStrategyAI

So we're in sync which is good.

 

Search functions are in my code as well. Tomorrow I'm going to work on reading in the Patch 4 EU and compare to the release EW upk and see if I can't get a handle on how that mapping might work.

 

----------------------------------------------

 

The objectlist index is the references used for almost everything.

 

Virtual functions instead use the namelist index. These functions are always prefaced with the 1B token, and have the format : 1B ## ## ## ## 00 00 00 00 <parameters> 16

[wghost81]

Seems, if I'll try to search byte-code for object references I end up writing a decompiler. :smile: This isn't such a horrible thing, but it will take a huge amount of time. And I'm not sure I'm up to this task. :smile:

 

Another approach would be to take a list of objects, used by function, find objectlist indexes by names and replace all occurrences with new indexes.

 

When I make changes to upk, I export entire function into separate binary file with UE Explorer, then copy and rewrite it's decompiled code, "compile" a new byte-code manually, using tokens from UE Explorer, modify separate binary file of that function and import modified function into upk. As a result, I have all my changes as separate binary functions with their full names, for example, XGStrategyAI.ProcessPodTypes.Function. Since I also have decompiled code, making a list of objects wouldn't be too hard. And with this list I can "repair" all my functions to match object names with new indexes. At least, I think I can. :smile:

 

Final utility should be able to find function by its name in new upk, check if the size is correct, "repair" object references and patch upk file with modded function. If it works, I could easily adapt Larger Pods to new EU patch.

 

Furthermore, local variables and function params are stored somewhere in function or upk header, so finding their names wouldn't be too hard. Hard part would be to locate function or external objects calls — it will require decompilation.

Edited November 21, 2013 by wghost81