R&D - New Modding Tools

[wghost81]

I have more bad news. Some functions received additional (optional) parameters in EW. So, even if decompiled code still looks the same, function call is not: it has 4A for omitted optional values, making it longer and ruining all jump offsets as a result. So much for easy automation. :sad:

 

While manually re-writing Larger Pods code for EW, I realized what I really need: I way to quickly get an object's reference by name. New UE Explorer with Disassembled Tokens view helps a lot. By if I could right-click object name and copy its reference into memory that would be fantastic!

 

I also noticed, Disassembled Tokens view omits some tokens, like 0x38 (primitive cast), for example:

 

    (23-29;065/049) [9A 38 3A 1B 6B 11 00 00 00 00 00 00 16 38 3A 24 12 16]
    	NativeFunctionToken(18/18)
    	GetCommanderType() == 18

        (24-26;066/04B) [3A 1B 6B 11 00 00 00 00 00 00 16]
        	ByteToIntToken(12/11)
        	GetCommanderType()

            (25-26;068/04C) [1B 6B 11 00 00 00 00 00 00 16]
            	VirtualFunctionToken(10/10)
            	GetCommanderType()

                (26-26;071/055) [16]
                	EndFunctionParmsToken(1/1)
                	)

        (27-28;072/057) [3A 24 12]
        	ByteToIntToken(4/3)
        	18

            (28-28;074/058) [24 12]
            	ByteConstToken(2/2)
            	18

        (29-29;076/05A) [16]
        	EndFunctionParmsToken(1/1)
        	)

 

[Bertilsson]

I also noticed, Disassembled Tokens view omits some tokens, like 0x38 (primitive cast), for example:

I think it shows same details as hoovering with mouse in view buffer.

So return values and other stuff like the 38 token which doesn't show up when hoovering are not included.

[Amineri]

I have more bad news. Some functions received additional (optional) parameters in EW. So, even if decompiled code still looks the same, function call is not: it has 4A for omitted optional values, making it longer and ruining all jump offsets as a result. So much for easy automation. :sad:

 

While manually re-writing Larger Pods code for EW, I realized what I really need: I way to quickly get an object's reference by name. New UE Explorer with Disassembled Tokens view helps a lot. By if I could right-click object name and copy its reference into memory that would be fantastic!

 

 

Definitely -- if anything has been changed in the original source code of a function then do an automatic update of the references won't work. This includes even adding an extra optional parameter to a function that is called within the target function (from a hex perspective optional parameters are just parameters with a few extra rules).

 

My goal for the first release is to enable auto-updating of references at minimum but I've got a host of other plans for things to include to make creating mods in general easier. This would include re-creating/adjusting a mod if the original source has changed.

[wghost81]

Amineri, I'm not criticizing or something. What you're doing is great and I can't wait to test your tool! :wink:

[johnnylump]

For anyone reading this thread, Amineri & XMarkstheSpot's efforts are far and away the most important development efforts for modding X-Com right now. We desperately need a mechanism to quickly update mod hex whenever an official patch comes out. Without it, large-scale mods are going to be unfeasible until Firaxis/2k stop patching the game entirely -- something we won't be sure of for some time, I imagine. In the meantime, everyone who wants a particular mod or modlet back in action quickly should understand that this development effort is the means to getting those mods quickly up to speed every time a patch comes out.

[Dethraker]

@johnnylump

 

Yea, been following this thread closely trying to see if I can help out in some way. Unfortunately most, if not all, of this is way beyond my personal expertise. Good stuff in here though and I can't wait to see the result!

[troch22]

For anyone reading this thread, Amineri & XMarkstheSpot's efforts are far and away the most important development efforts for modding X-Com right now. We desperately need a mechanism to quickly update mod hex whenever an official patch comes out. Without it, large-scale mods are going to be unfeasible until Firaxis/2k stop patching the game entirely -- something we won't be sure of for some time, I imagine. In the meantime, everyone who wants a particular mod or modlet back in action quickly should understand that this development effort is the means to getting those mods quickly up to speed every time a patch comes out.

 

Ive been following this thread from page one. By page 3 im lost on what your talking about, lol Its way over my head. But I do find it fascinating how everyone involved here comes together to spearhead the future of modding this game.

 

I find myself checking in on this every day for the same reason as Johnny pointed out. And im not a modder just the end user who enjoys the mods you guys release. With that said, hats off to Amineri, Xmarksthespot and all the others involved.

[wghost81]

One of the most important results of modding efforts is file format documentation. If we had one for upk files, it would be a lot easier from the beginning. So I ask everyone involved: keep you notes and, please, share them in any form, so we could have that documentation eventually.

[Amineri]

For those of you interested in the technical side of the things I'm working on, here is a snippet of work from trying to document/etc the work on interpreting the unrealhex bytecode.

 

 

 

*** Regarding the operands and various tokens...

Here is a more concrete example demonstrating why I organized the way I did. Remember that the rule for the Element Interface is that leaves don't contain data.

I'll break down the line:

// if(m_arrTeamLoadoutInfos[PlayerIndex].m_eTeam == 16)
07 51 00 9A 38 3A 35 96 9F 00 00 98 9F 00 00 00 00 1A 00 10 A0 00 00 01 BF 9F 00 00 38 3A 24 10 16 

The first comment line, goes into a LineNode with a single StringToken leaf, which contains the entire line of text.

And here is the breakdown of the hex line:

• OperandNode -- type is JumpIfNot
  • OperandToken containing "07 "
  • OffsetToken containing "51 00 "
  • OperandNode -- type is IntegerEqualityTest
    • OperandToken containing "9A "
    • OperandNode -- type Typecast
      • OperandToken containing "38 "
      • GenericToken containing "3A " (this represents the type of typecast -- byte-to-int in this case
      • OperandNode -- type Struct
        • OperandToken containing "35 "
        • ReferenceToken containing "96 9F 00 00 " (this reference is to m_eTeam struct member)
        • ReferenceToken containing "98 9F 00 00 " (this reference is to the structure type containing m_eTeam)
        • GenericToken containing "00 00 " (I'm not even sure what these two byte do)
        • OperandNode -- type ArrayIndex
          • OperandToken containing "1A "
          • OperandNode -- type LocalVariable
            • OperandToken containing "00 "
            • OperandReference containing "10 A0 00 00 " (this reference is to PlayerIndex)
          • OperandNode -- type ClassVariable
            • OperandToken containing "01 "
            • OperandReference containing "BF 9F 00 00 " (this reference is to m_arrTeamLoadoutInfos)
    • OperandNode -- type TypeCast
      • OperandToken containing "38 "
      • GenericToken containing "3A " (again this is the type of typecast, byte-to-int)
      • OperandNode -- type ByteConst
        • OperandToken containing "24 "
        • GenericToken containing "10 " (this is the value 16)
    • GenericToken containing "16 " (this is the terminator for the equality test parameters)

So you can see that very definitely in practice Operands are very hierarchical (there are examples that go much deeper). Also branching and the types of allowed branches are controlled by the particular Operand -- all of which is defined within the operand_data.ini.

The above example used:

• 07 1 J G ; JumpIfNot
• 9A 1 G G 1 ; == binary operator, integer
• 38 1 1 G ; Primitive Cast
• 35 1 R R 2 G ; Struct member -- member reference, struct reference, variable
• 24 1 1 ; byte const
• 00 1 R ; local variable
• 01 1 R ; class variable

The particular operand is what fairly tightly controls what branches are possible.

A number indicates a number of generic tokens

The "J" indicates a jump offset

The "R" indicates a reference

A "G" indicates that the next byte will be an operand and that a new OperandNode must be formed.

If it seems incomprehensible, well... the goal of working on this tool is to make it less necessary to understand such details.

I'm pretty sure it is replicating some of that UE Explorer already does. However I ended up re-implementing it for a few reasons:

1. Writing code in Java so that it will be compatible on OS X, and UE Explorer is Windows only
2. Have to identify references they can be updated using automated tools
3. Will be built-in to editor to allow quicker feedback as to hex correctness

Ideally some of the coloring and mouse-over information will be available similar to the UE Explorer token/buffer views. This isn't intended to replace UE Explorer but simply to provide tighter integration while editing/creating mods. There's no way that this will ever do a full parsing and resolve all objects, etc in a upk.

Right now we're primarily working on things such as the modfile structure as well as integrating into an editor that can display styles. (I have a crappy prototype that can display in a plain text editor, but it's really yuck :p)

The goal of the above mapping is create a document model that can be edited yet still display text highlighting based on content. Hopefully it will come together eventually ^_-

[wghost81]

If it seems incomprehensible, well... the goal of working on this tool is to make it less necessary to understand such details.

One of the modding goals is to understand these kind of details, to break down the file formats and understand how it works. At least for me. :smile: So I would appreciate any info in any form about upk structure and scripting. :smile:

 

Keep up the good work!

Edited December 5, 2013 by wghost81