R&D - New Modding Tools

[Amineri]

Update:

 

I have a command-line version of the tool that can do the following:

• Parse a file in my "kind-of-like-ToolBoks" xcommod file format
• Read and interpret the namelist and objectlist of multiple pairs of before/after upk files (used for updating references from one XCOM version to another)
• Output an xcommod file with all references tagged (and also strip tags from references)
• Replace tagged references with stringnames (and also revert stringnamed references to hex)
• Write a consolidated modfile with all excess formatting removed
• Apply / Revert a modfile to referenced upk files

 

My current command-line arguments are:

Usage: XComModTool <options> -i <InputFile> -o <OutputFile>
Options:
-V, --VERSION : version info
-h, --HELP : this help file
-v, --VERBOSE : verbose output
-w, --WRITETOUPKS : write hex to configured upks (before->after)
-r, --REVERTUPKS : revert hex in configured upks (after->before)
-c, --COMPRESSEDOUTPUT : outputs compressed modfile
-m, --MIRROREDOUTPUT : outputs mirrored modfile
-x, --REFS_STRINGTOHEX : converts string references to tagged hex references
-s, --REFS_HEXTOSTRING : converts tagged hex references to string references
-t, --REFS_TAG : tags references
-u, --REFS_UNTAG : removes reference tags

Upk accessibility is configured via a config file with a format like so:

// config file for matching upk versions
NUM_UPKS=2

UPKFILE=XComGame.upk
BEFORE=C:/Games/XComGame_EU_Patch4.upk
AFTER=C:/Games/XComGame_EU_Patch5.upk
OUTPUT=C:/Games/XComGame_test.upk
//AFTER=C:/Games/XComGame_EW_release.upk

UPKFILE=XComStrategyGame.upk
BEFORE=C:/Games/XComStrategyGame_EU_Patch4.upk
AFTER=C:/Games/XComStrategyGame_EU_Patch5.upk
OUTPUT=C:/Games/XComStrategyGame_test.upk
//AFTER=C:/Games/XComStrategyGame_EW_release.upk

In theory any number of upks can be configured although function code pretty much just sits in XComGame and XComStrategyGame.

 

BEFORE/AFTER are used for reference updating. BEFORE would be the upk corresponding to hex code that needs updating, while AFTER is the upk that the hex code is being updated to. OUTPUT is the upk file to which hex is applied/reverted.

 

Any of the BEFORE/AFTER/OUTPUT files can be identical.

 

---------------------

 

The format of the xcommod file allows for both "//"-style and "/* */"-style commenting.

 

General format is:

UPKFILE=<root upk filename corresponding to UPKFILE in upk_config.ini>
FUNCTION=<unique function name>

[BEFORE_HEX]
[HEADER]
<hex that will not be parsed for reference marking>
[/HEADER]

<hex that will be parsed for reference marking>
[/BEFORE_HEX]

[AFTER_HEX]
[HEADER]
<hex that will not be parsed for reference marking>
[/HEADER]

<hex that will be parsed for reference marking>
[/AFTER_HEX]

Each FUNCTION= scope can contain multiple BEFORE/AFTER hex pairs.

Each UPKFILE= scope can contain multiple function changes.

 

The current limits on a single modfile are:

• 50,000 lines of text (including comments)
• 1,000 BEFORE/AFTER replacement blocks
• Each BEFORE/After block is limited to 10,000 bytes (not characters, but hex bytes)

In practice I think that larger mods will be broken into multiple modfiles.

 

To apply hex changes the function limits scope to that defined by the FUNCTION=declaration. My current version which reads and parses both patch 4 & 5 versions of XComGame.upk and XComStrategyGame.upk (so 4 upks total) and applies a single 246 byte replacement takes 1.25 seconds to execute. Most of the time is spent parsing the upks -- applying the hex requires only 5 ms.

 

------------------------------------

 

I think ultimately usability would be improved with a good GUI but that's an art/science pretty far afield from my core, so any assistance would be most appreciated.

[Amineri]

For those interested in helping my trace down various crash cases I've uploaded a very early alpha build of the tool to the Long War Misc Files section.

 

I've included both a jar file (with a batch file to run it) as well as a copy of the Net Beans project folder with all of the .class source code.

[XMarksTheSpot]

Update:

 

[snip]

 

------------------------------------

 

I think ultimately usability would be improved with a good GUI but that's an art/science pretty far afield from my core, so any assistance would be most appreciated.

 

Nice work on the tool :thumbsup:

I myself am working with Java on a regular basis and have pondered lending a hand, but this thread seemed to move rather fast and you appear to handle yourself quite well anyway :)

My Java work is mainly focused on user interface programming, so that last bit had my interest piqued again, maybe I can help out with that?

[wghost81]

Wow, great!

 

I'm C/C++ programmer, never worked with java, never liked to make UI. :smile: Can help only with bugtesting.

[Amineri]

So, a few notes

 

command-line parameters

 

-v, --VERBOSE : verbose output

Writes a lot of extra stuff to the console. Helpful for identifying where code is crashing.

 

-----------

-w, --WRITETOUPKS : write hex to configured upks (before->after)
-r, --REVERTUPKS : revert hex in configured upks (after->before)

 

Both of the options cannot be active at once. If both -w and -r are in the command-line, the last one (rightmost) takes precedence.

 

-------------

-c, --COMPRESSEDOUTPUT : outputs compressed modfile
-m, --MIRROREDOUTPUT : outputs mirrored modfile.

 

Both options cannot be true (only one output file can be specified). If both -c and -m are in the command-line, the last one (rightmost) take precedence.

 

Compressed output removes all comments and excess whitespace (including EOLS markers). It includes only the UPKFILE=, FUNCTION=, and [..]-style context markers. It is basically a dump of the internal binary data structures.

 

Mirrored output attempts to preserve all comments and whitespace. Hex values are drawn from the binary data structures, but are omitted for ||<string>|| references. Purpose is to apply/remove tagging to a modfile.

 

-------------

 

Both a (-c or -m) and a (-w or -r) option should be invokable at the same time. Result will be both an output text file and update to the upks as defined below.

 

------------

-x, --REFS_STRINGTOHEX : converts string references to tagged hex references
-s, --REFS_HEXTOSTRING : converts tagged hex references to string references
-t, --REFS_TAG : tags references
-u, --REFS_UNTAG : removes reference tags

 

There are sort of 3 different states that a modfile can be in :

 

1) "Raw" hex, basically what can be pasted in from UE Explorer token view (with header info from the UE Explorer Buffer view)

2) Hex with {{ and }} markers indicating where references are located

3) Mixture of hex and ||<string>|| marked variable names

 

The -t command mutates a type 1 file into a type 2 file

The -u command mutates a type 2 file into a type 1 file

The -s command mutates a type 2 file into a type 3 file

The -x command mutates a type 3 file into a type 2 file

 

Because of this only 1 of the 4 options can be active. If multiple tags are present the last one (rightmost) takes precedence.

 

Graphically, something like:

Raw hex modfile                  Hex w/ marked references                     Hex w/ string references
       -t |---------->  <-----------| -u         -s |---------------->   <----------------| -x

Technically the -s and -x options work on a token-by-token basis (so a line with mixed marked/named references will be updated).

 

However the -t and -u options work on a line-by-line basis. Any line with non-hexadecimal values will be skipped when marking (excepting "//"-style comments). The -u option simply mirrors the file stripping any {{ and }} tokens.

 

--------------------------

 

upk configuration:

UPKFILE=XComGame.upk
BEFORE=C:/Games/XComGame_EU_Patch4.upk
AFTER=C:/Games/XComGame_EU_Patch5.upk
OUTPUT=C:/Games/XComGame_test.upk

When a modfile is read it stores both a internal binary format as well as the original string data. In the internal binary format any marked references (either with {{ <hex> }} or named ||<string>||) are looked up in the BEFORE upkfile and the references are then updated to the AFTER upk file.

• {{ <hex> }} references are resolved to stringname in BEFORE, and the reference with the identical stringname in AFTER replaces the value (in the binary data).
• ||<string>|| references are directly looked up in AFTER and the hex is stored in the binary data structure.

Applying a modfile to a upk does the following:

1. BEFORE/AFTER blocks are turned into SEARCH/REPLACE blocks depending on -w or -r. For writing (-w) SEARCH = BEFORE and REPLACE = AFTER. For reverting (-r) SEARCH = AFTER and REPLACE = BEFORE.
2. For each SEARCH/REPLACE block, use FUNCTION= declaration to find objectfile entry. This search is done in the AFTER upk. This gives a (file position / file range) valid for replacement.
3. Code checks that there are identical numbers of SEARCH/REPLACE blocks and that each SEARCH/REPLACE block has an identical number of bytes.
4. Code then searches for every SEARCH block's file position. If any SEARCH block is not found then the entire write/revert operation is aborted and an error printed to the console (indicating the first failed block).
5. If all SEARCH block seeks find valid positions then the REPLACE blocks are written to the upk designated with OUTPUT

 

Key is that function hex file locations are found using the AFTER upk, but the change is applied to the OUTPUT upk. This is primarily for testing purposes to avoid corruption of a upk while the code is in alpha state. Function hex file locations are not read from the OUTPUT upk because the OUTPUT upk is not parsed (saves run-time and memory).

 

It is valid to make the AFTER and OUTPUT upks be the same file.

 

It is also valid to make the BEFORE and AFTER upks be the same file, in which case no references will ever be changed.

[Amineri]

 

 

Nice work on the tool :thumbsup:

I myself am working with Java on a regular basis and have pondered lending a hand, but this thread seemed to move rather fast and you appear to handle yourself quite well anyway :smile:

My Java work is mainly focused on user interface programming, so that last bit had my interest piqued again, maybe I can help out with that?

 

 

Would be very much appreciated!

 

I've glanced at the SWING and JavaFX stuff but know no details. Will PM you.

[Amineri]

Some further notes:

 

In order to update references from raw hex a "2-pass" system is required.

 

Pass 1) Reads the raw hex. Since no reference tags are present no reference value are updated. Output modfile with tagged references. Writing to upk will write the original reference hex.

Pass 2) Read the reference-tagged hex. Since references are now present the values are updated from the BEFORE upk to the AFTER upk. Writing to upk will write the updated reference hex.

 

----------------

 

Note that it is possible to make the -i <Input File> and -o <Output File> be the same file.

 

The input file is opened, read and cached internally, then closed. Then the output file is opened and written to. This makes it safe to write to the input file (excepting of course bugs that mess up the output).

 

Once the code is debugged and stable, I expect that making the output file = input file will become fairly commonplace.

 

A batch file (or java wrapper) could run two successive calls:

XComModTool -m -t -i MyModFile.txt -o MyModFile.txt  // opens raw hex file and tags references
XComModTool -m -s -i MyModFile.txt -o MyModFile.txt  // opens file and converts references to strings

Alternatively, to mark references in a raw hex file and then write updated references into upks:

XComModTool -m -t -i MyModFile.txt -o MyModFile.txt  // opens raw hex file and tags references
XComModTool -w -i MyModFile.txt  // opens file and writes to upks

I've provided a lot of options in terms of separating output:

a) To allow for better debugging

b) To allow for viewing intermediate steps while modding.

[Amineri]

So on reviewing my own code I realized I need to add another command line switch. It's going to be:

-U, --UPDATE_REFS : updates any tagged references or named references

I had put a flag internal to the code to control whether any tagged ({{ <hex> }}) or named (||<string>||) references would be updated but I failed to add a command-line parameter for it. By updating I mean the "look up in BEFORE upk and replace with value in AFTER upk" step.

 

I'll have a new version posted some time today.

[Amineri]

Version 0.11 updated (in Long Misc Files area)

• Added -U, --UPDATE_REFS command-line option
• Fixed stupid bug when declaring both input and output files
• Added sample modfiles

[Amineri]

I found a bug in my reference parser data that may cause the code to fail to mark references correctly when there is a long int (0x1D) or float (0x1E) constant in the line.

 

In the operand_data.ini the following lines:

1D 4 			// integer constant
1E 4 			// float constant

need to be changed to:

1D 5 			// integer constant
1E 5 			// float constant

I'm not releasing a full version update for this but it will be folded into the next update.