R&D - New Modding Tools

Amineri · February 18, 2014

Technically UPKmodder allows for resizing at any point within a given object.

In practice functions always have to have their memory/file size in the header redefined, which means that at least those two words of the header have to be changed. Since the position of the 0x53 token changes as well, in practice all of our function resized start with these last 2 header words and continue to the 53 token. If any of the footer flags had to be changed they could also be included.

But in general the resize is pretty generic, which is why I was able to apply the same resize operation to an object of type enum:

MODFILEVERSION=4
UPKFILE=XComGame.upk
GUID=5B 06 B8 18 67 22 12 44 85 9B A8 5B 9D 57 1D 4B // EW Patch 1
FUNCTION=EShipType@XGGameData
RESIZE=28

// increase number of EShip enums

[BEFORE_HEX]
[HEADER]
7C 2E 00 00 76 5F 00 00 00 00 00 00 7C 2E 00 00 
0B 00 00 00 // length
[/HEADER]
54 2D 00 00 00 00 00 00 // eShip_None
52 2D 00 00 00 00 00 00 // eShip_Interceptor
55 2D 00 00 00 00 00 00 // eShip_Skyranger
51 2D 00 00 00 00 00 00 // eShip_Firestorm
5A 2D 00 00 00 00 00 00 // eShip_UFOSmallScout
59 2D 00 00 00 00 00 00 // eShip_UFOLargeScout
56 2D 00 00 00 00 00 00 // eShip_UFOAbductor
5B 2D 00 00 00 00 00 00 // eShip_UFOSupply
57 2D 00 00 00 00 00 00 // eShip_UFOBattle
58 2D 00 00 00 00 00 00 // eShip_UFOEthereal
53 2D 00 00 00 00 00 00 // eShip_MAX

[/BEFORE_HEX]


[AFTER_HEX]
[HEADER]
7C 2E 00 00 76 5F 00 00 00 00 00 00 7C 2E 00 00 
10 00 00 00 // length
[/HEADER]
54 2D 00 00 00 00 00 00 // eShip_None
52 2D 00 00 00 00 00 00 // eShip_Interceptor 	= 1
55 2D 00 00 00 00 00 00 // eShip_Skyranger   	= 2
51 2D 00 00 00 00 00 00 // eShip_Firestorm   	= 3
5A 2D 00 00 00 00 00 00 // eShip_UFOSmallScout 	SCOUT			= 4
59 2D 00 00 00 00 00 00 // eShip_UFOLargeScout	DESTROYER		= 5
56 2D 00 00 00 00 00 00 // eShip_UFOAbductor	ABDUCTOR		= 6
5B 2D 00 00 00 00 00 00 // eShip_UFOSupply		CRUISER		= 7
57 2D 00 00 00 00 00 00 // eShip_UFOBattle		BATTLESHIP	= 8
58 2D 00 00 00 00 00 00 // eShip_UFOEthereal	ETHEREAL		= 9
6F 29 00 00 00 00 00 00 // eMPT_SniperSquaddie  // +8  // stand-in for FIGHTER			= 10
5F 29 00 00 00 00 00 00 // eMPT_HeavySquaddie   // +8  // stand-in for RAIDER				= 11
73 29 00 00 00 00 00 00 // eMPT_SupportSquaddie // +8  // stand-in for HARVESTER			= 12
5B 29 00 00 00 00 00 00 // eMPT_AssaultSquaddie // +8  // stand-in for ASSAULT CARRIER	= 13
5A 29 00 00 00 00 00 00 // eMPT_AssaultColonel  // +8  // stand-in for TERROR SHIP		= 14
53 2D 00 00 00 00 00 00 // eShip_MAX
[/AFTER_HEX]

In this case the entire object's hex is being replaced, although I could have omitted the first 16 bytes of the header since they didn't change.

I suspect you'd have to write another function to resize and move the enum, then change the internal hex. The above isn't patch-safe for two reasons: (1) There are references in the header which aren't recognized/parsed (2) the namelist references in the main body aren't recognized/parsed.

Once we pretty much finish up with Long War EW I'll likely get back to UPKmodder and update it to better handle the things some things we did using raw hex -- mostly variable definition changes, with a couple of enum redefinitions.

wghost81 · February 19, 2014

PatchUPK resize functionality is generic too, so you can resize enums or any other objects as easily, as functions. The difference is it operates within "scopes" with minimal scope being "Object" and UPKModded search-replace style creates local scope, defined by BEFORE_HEX data. All I need to do is introduce another (smaller) scope, called "Selection", for example, and use BEFORE_HEX to define selection range.

PS I must admit, that in this particular case (local resizing) search-replace style works better than offset-data style. :smile: But since I designed PatchUPK with repeat-ability in mind, I haven't thought about this before. :smile:

Edited February 19, 2014 by wghost81

Amineri · February 19, 2014

Hmmm, yes, I see. UPKmodder's maximal scope for a single upk_mod file is a single object. To affect multiple objects you must have one upk_mod file per object being altered (this made the design simpler). However within a single upk_mod file you can limit the scope of the change implicitly via the BEFORE hex. You can also define several such changes by including multiple BEFORE/AFTER blocks.

I still have plans to allow for offset-data style in a later version by allowing for optional tags prior to each BEFORE or AFTER block that define the location/size of the block (location relative to the start of the object, though). I'd still do the error-checking step of testing the FIND hex against the file, but instead of searching it would immediately go to the specified location. If the FIND hex isn't found it would present a dialogue to the user allowing "blind" replace of the FIND hex with the REPLACE hex.

Amineri · February 23, 2014

A new use for the generic resizing in UPKmodder was discovered by johnnylump and XMTS.

It's possible to use the resize capability within an actionscript object to add additional actionscript / sprites without having to remove code from other portions. I think this is definitely a case where the "expand-in-place" is superior as some of the flash objects (as in upk-level objects) are quite large.

For example, the SquadSelect SWF file (when extracted) is 135k, while the StrategComponents SWF file is 153k. These SWF files comprise the bulk of the upk-level object containing them, although not quite all of it.

Using resize-in-place it's possible to rewrite a small portion of the SWF file (like a single flash-level object) and resize it accordiningly, which allows for much easier editing of the embedded SWF files. There are actually decompile/recompile tools for SWF files, which actually makes creating new code here easier than in the upk itself (since we don't have a re-compile tool that will use existing object info from an existing upk).

Here is an example that was created by JL and XMTS that alters the SituationRoom :

MODFILEVERSION=4
UPKFILE=UICollection_Strategy_SF.upk
GUID=7B 52 25 96 F3 8E 7C 48 8F 11 3B 40 43 3F BD 3F
FUNCTION=SituationRoom@gfxSituationRoom
RESIZE=18

[BEFORE_HEX]
FF 09 A4 00 00 00 39 00 14 00 FF 0A 03 00 00 00 5F 31 00 89 06 06 01 00 38 00 16 92 39 80 40 00 40 00 40 00 40 00 3F 03 02 00 00 00 07 00 40 00 FF 0A 03 00 00 00 5F 32 00 8B 06 16 03 00 38 00 16 EE 39 80 05 00 40 00 40 00 40 00 40 00 3F 03 02 00 00 00 07 00 40 00 FF 0A 03 00 00 00 5F 33 00 8B 06 16 05 00 38 00 16 C0 30 C0 0A 00 40 00 40 00 40 00 40 00 3F 03 02 00 00 00 07 00 40 00 FF 0A 03 00 00 00 5F 34 00 8B 06 16 07 00 38 00 18 8F 5C 2E 0F 00 40 00 40 00 40 00 40 00 3F 03 02 00 00 00 07 00 40 00 00 00
[/BEFORE_HEX]

[AFTER_HEX]
FF 09 BC 00 00 00 39 00 06 00 FF 0A 03 00 00 00 5F 31 00 89 06 06 01 00 38 00 16 92 39 80 40 00 3F 03 02 00 00 00 07 00 FF 0A 03 00 00 00 5F 32 00 8B 06 16 03 00 38 00 16 EE 39 80 01 00 40 00 3F 03 02 00 00 00 07 00 FF 0A 03 00 00 00 5F 33 00 8B 06 16 05 00 38 00 16 C0 30 C0 02 00 40 00 3F 03 02 00 00 00 07 00 FF 0A 03 00 00 00 5F 34 00 8B 06 16 07 00 38 00 18 8F 5C 30 03 00 40 00 3F 03 02 00 00 00 07 00 FF 0A 03 00 00 00 5F 35 00 8B 06 16 09 00 38 00 16 C0 02 40 04 00 40 00 3F 03 02 00 00 00 07 00 FF 0A 03 00 00 00 5F 36 00 8B 06 16 0B 00 38 00 18 8F 40 90 05 00 40 00 00 00
[/AFTER_HEX]

If you are familiar with actionscript hex, you'll recognize that the FF 09 at the beginning marks the beginning of a flash-level object block, and the next 4 bytes are the size of the object, which is changing from A4 to BC. This matches with the RESIZE operation amount of 18, as 0xA4 + 0x18 == 0xBC.

Not only does this allow for easier modifying of the actionscript components, but also for adding additional sprite data. This method could even be used to add additional image / icon data, or to modify compressed imagery (where the replacement won't be the same size as the original).

Amineri · February 23, 2014

An update on actionscript.... the object containing an SWF object has a fairly long and variable-sized "header" preceding the actual embedded SWF file (which has the "FWS" leading string replaced with "GFX"

For my example I looked at the SituationRoom object in the UICollection_Strategy_SF.upk.

The objectlist entry for "SituationRoom" looks like so:

FC FF FF FF // type -- Core:SwfMovie@GFxUI
00 00 00 00 // parent - none
2A 00 00 00 // owner (gfxSituationRoom)
3D 02 00 00 00 00 00 00 // namelist reference ("SituationRoom")
00 00 00 00 00 00 00 00 // unknown
04 00 0F 00 // flags
2B 69 02 00 // object size
53 11 69 00 // object position
01 00 00 00 // unknown
00 00 00 00 
00 00 00 00 
00 00 00 00 
00 00 00 00 
00 00 00 00 
00 00 00 00

Of particular note are the object's file size and file position, which are 0x2692B and 0x691153.

The embedded SWF file begins with :

47 46 58 08 30 67 02 00

47 46 58 in ASCII is "GFX", the 08 is the version, and the next 4 bytes are (in little endian) the SWF filesize, 0x26730. This is 0x1FB = 507 bytes smaller than the full object. This extra 0x1FB bytes is entirely contained within a "header" preceding the SWF embedded file within the object.

The entire hex header is variable size, as it contains string elements :

FF FF FF FF 46 00 00 00 00 00 00 00 2D 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 C6 01 00 00 00 00 00 00 69 01 00 00 00 00 00 00 04 00 00 00 00 00 00 00 00 04 00 00 94 02 00 00 00 00 00 00 47 00 00 00 00 00 00 00 08 00 00 00 00 00 00 00 D7 00 00 00 00 00 00 00 D8 00 00 00 00 00 00 00 90 02 00 00 00 00 00 00 7E 02 00 00 00 00 00 00 08 00 00 00 00 00 00 00 04 00 00 00 54 47 41 00 0B 02 00 00 00 00 00 00 10 00 00 00 00 00 00 00 8A 00 00 00 00 00 00 00 04 00 00 00 19 00 00 00 67 66 78 63 6F 6D 70 6F 6E 65 6E 74 73 2E 63 6F 6D 70 6F 6E 65 6E 74 73 00 1D 00 00 00 67 66 78 47 61 6D 65 70 61 64 49 63 6F 6E 73 2E 47 61 6D 65 70 61 64 49 63 6F 6E 73 00 29 00 00 00 67 66 78 53 74 72 61 74 65 67 79 43 6F 6D 70 6F 6E 65 6E 74 73 2E 53 74 72 61 74 65 67 79 43 6F 6D 70 6F 6E 65 6E 74 73 00 17 00 00 00 67 66 78 58 43 6F 6D 49 63 6F 6E 73 2E 58 43 6F 6D 49 63 6F 6E 73 00 0C 02 00 00 00 00 00 00 10 00 00 00 00 00 00 00 B4 00 00 00 00 00 00 00 2C 00 00 00 9A 01 00 00 9B 01 00 00 9C 01 00 00 9D 01 00 00 9E 01 00 00 9F 01 00 00 A1 01 00 00 A0 01 00 00 A2 01 00 00 A3 01 00 00 A4 01 00 00 A5 01 00 00 A6 01 00 00 A7 01 00 00 A8 01 00 00 A9 01 00 00 AA 01 00 00 AB 01 00 00 AC 01 00 00 AD 01 00 00 AE 01 00 00 AF 01 00 00 B0 01 00 00 B1 01 00 00 B2 01 00 00 B3 01 00 00 B4 01 00 00 B5 01 00 00 B6 01 00 00 B7 01 00 00 B8 01 00 00 B9 01 00 00 BA 01 00 00 BB 01 00 00 BC 01 00 00 BD 01 00 00 BE 01 00 00 BF 01 00 00 C0 01 00 00 C1 01 00 00 45 00 00 00 4B 00 00 00 70 00 00 00 75 00 00 00 B0 01 00 00 00 00 00 00 30 67 02 00
ÿÿÿÿF.......-................Æ.......i...................”.......G...............×.......Ø...............~...................TGA.................Š...............gfxcomponents.components.....gfxGamepadIcons.GamepadIcons.)...gfxStrategyComponents.StrategyComponents.....gfxXComIcons.XComIcons.................´.......,...š...›...œ.......ž...Ÿ...¡... ...¢...£...¤...¥...¦...§...¨...©...ª...«...¬.......®...¯...°...±...²...³...´...µ...¶...·...¸...¹...º...»...¼...½...¾...¿...À...Á...E...K...p...u...°.......0g..

I'm sure there are several interesting portions here, but what I am interested in is the last 4-byte word of this "header" : 30 67 02 00, which is the size of the SWF object within the upk-level object.

Hence there SWF object size is actually declared twice (and immediately adjacent):

30 67 02 00 // end of upk-object header -- file size of SWF movie
47 46 58 // beginning of SWF file, ASCII "GFX"
08 //version number
30 67 02 00 // size of SWF movie object

The positioning of the file size is consistent with the position of an unrealscript filesize -- again it's the final 4-byte word of the header. The second instance of the size is for the flash interpretter in order to conform with Flash file specification.

Slightly later in the Flash file is some embedded XML:

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> 
<rdf:Description rdf:about="" xmlns:xmp="http://ns.adobe.com/xap/1.0/"> 
<xmp:CreatorTool>Adobe Flash Professional CS5</xmp:CreatorTool> 
<xmp:CreateDate>2011-11-15T11:20:44-05:00</xmp:CreateDate>

Here we can see that Firaxis is using Adobe Flash Professional CS5 to create the Flash files, which are later cooked into the UPK.

--------------

At any rate, in order to resize an actionscript object both of these values must be adjusted. Since the current RESIZE operation in UPKmodder only allows a single hex change, if this header is separate from the change (quite likely), it would require a second-pass modification.

Eventually I'll look into figuring out a way to do both within the same file.

dubiousintent · February 24, 2014

Added this to the 'Flash SWF Editing' wiki article.

Eventually I'll look into figuring out a way to do both within the same file.

Why not a "[Pass#]" switch? Put each pass' collection of code into a table, and then process each table in subsequent passes.

-Dubious-

Edited February 24, 2014 by dubiousintent

Amineri · February 26, 2014

Just in time for Enemy Within patch 2's release, I have an updated version of UPKmodder v0.80 :

v0.80 (r129)
Added bulk apply/revert functionality accessible through context menu in project pane
-- any file beginning with __ or in a directory starting with __ cannot be applied/reverted in this manner, but can be manually applied/reverted by opening the file
Fixed parse error when null string was at end of line
Fixed error that prevented RESIZE operations from being applied if UE Explorer was open
Added stability improvements to RESIZE operation -- memory copy of UPK header info is reloaded after RESIZE
Fixed operand_data.ini issues with 0x59 (array Sort), 0x43 (Delegate), 0x52 (Interface Cast), 0x6F (NativeFunction), 0x09 (assert), and 0x7E (InStr) operands

The big feature is of course the ability to apply or revert in batches. Entire folders or even projects can be applied with a single command. Handy for us on Long War with our 500+ changes :)

Otherwise some minor bugfixes and a couple of stability improvements.

wghost81 · February 27, 2014

Using UPKModded to update my mods. Fantastic tool! I don't have words to describe how helpful it is. Thank you very-very much for this!

Some feedback on usability.

1. I can't understand the purpose of upk_config.ini. You need to set UPKFILE anyway and when undating references you need to manually select upk files too.

2. To replace old code by copy/paste you need to delete it first, then paste a new code. Replacing selected text with text from clipboard doesn't work.

3. For Update References to work properly I needed to lookup/convert source references first, then close the dialog, re-open it and lookup/convert destination references. If I try to lookup/convert destination references right after lookup/convert source references nothing happens.

4. As I see, converting destination updates GUID, but it doesn't updates UPKFILE path.

5. At some point GUID wasn't updated after updating destination references. That was my first attempt to work with this feature, so I messed up some things. After I learned how it works, I see GUID updated each time I convert destination references (but, again, UPKFILE is not updated).

6. Code structure sometimes get messed up after references conversion: end of line gets "eaten", so some lines get merged into one. I suspect it can be related to different end of line standards: 0x0A without 0x0D or 0x0D followed by 0x0A.

Amineri · February 27, 2014

Thanks for the kind words. :)

Before I delve into responses to specific question, I should mention that I uploaded a v0.81 "hotfix" that fixes an issue that was preventing bulk apply/revert operations of an entire project (v0.80 could only apply/revert files/subfolders). No other changes, I've "Stress-tested" the feature by bulk-applying all ~500-600 upk_mod files in Long War EW, verifying that the 3 affected upks were still parsable by UE Explorer and launched without CTD, and then bulk-reverting the entire set of 500-600 files again. I then verified using HxD that the upks were byte-level identical to decompressed vanilla versions.

----------

Now response to specific questions:

Let me preface by saying that you can also post issues (either bugs or usability improvements) to the issues list here : http://code.google.com/p/upk-modder/issues/list

No account is required to post issues.

1. I can't understand the purpose of upk_config.ini. You need to set UPKFILE anyway and when undating references you need to manually select upk files too.

Right, I keep meaning to take that config file out, but I keep forgetting. It was how I was thinking about implementing the UPKFILE= to actual upk file mapping before XMTS talked me into a better method. It's currently used in some unit test code which I'd break if I removed it, but it doesn't have any functionality in the release version.

Apologies ...

2. To replace old code by copy/paste you need to delete it first, then paste a new code. Replacing selected text with text from clipboard doesn't work.

This isn't part of the way I do things, so I'd never noticed that it didn't work. I guess it must be something in the way the DefaultStyledEditor works in Java (we didn't create a new editor, just used the Java in-built functionality)

3. For Update References to work properly I needed to lookup/convert source references first, then close the dialog, re-open it and lookup/convert destination references. If I try to lookup/convert destination references right after lookup/convert source references nothing happens.

Are you converting to names as an intermediate step? The name conversion is more intended as a step for handling cases were some of the symbols don't match from one version to another. My usual steps are :

1) Look up Source References

2) Look up Destination References

3) If everything mapped then the "Convert Source Refs to Dest Refs" button will activate -- this directly converts the numerical source refs to dest refs

However, if one of the intermediate names doesn't map to a destination refererence (like in one function were Firaxis renamed a field className to ClassName), then that button doesn't activate, since doing such a conversion would result in a file with references to two different versions of UPK and no way to tell them apart.

To handle that case is what I added the "Convert Hex References to Names" button for. It's there to convert everything to a name. It is a bit glitchy that you have to close and re-open the dialogue, but if you do so you can then convert the names that do match to dest refs back to references values, leaving just the names that didn't map to deal with manually.

4. As I see, converting destination updates GUID, but it doesn't updates UPKFILE path.

That's a good point. I guess I assumed that the UPKFILE=<tag> wouldn't be changing. Behind the scenes it's a fully pathed filename, but I could extract the upk name and replace it if necessary.

5. At some point GUID wasn't updated after updating destination references. That was my first attempt to work with this feature, so I messed up some things. After I learned how it works, I see GUID updated each time I convert destination references (but, again, UPKFILE is not updated).

I think if you convert to names the GUID is changed to "None", since a file with all names isn't necessarily tied (in the way numerical references are) to a particular version. Any time you convert to destination references it should be updating the GUID, as you say.

I should also point out that the smallest unit for conversion is a single line, so it's quite possible to copy lines to another file and therefore convert a file "piecemeal". I sometimes do this when some lines don't convert properly.

6. Code structure sometimes get messed up after references conversion: end of line gets "eaten", so some lines get merged into one. I suspect it can be related to different end of line standards: 0x0A without 0x0D or 0x0D followed by 0x0A.

I think the issue is that an earlier version of UPKmodder required a trailing space even after the final hex character (if the final hex was directly followed by a '\\n' it would be treated as non-parsable). This was fixed in v0.70, by which point the Reference Update function had already been written and tested.

So I think that what is happening is that the Reference Update code is assuming that each hex line will have at least one trailing space, which it overwrites. However, if the final hex is instead immediately followed by a '\\n' character then instead that is overwritten by a space, resulting in the loss of the newline.

Definitely a glitch I've noticed as well, thought I forgot to add it to the issues list.

------------------

Thanks a bunch for the feedback, and I'm glad you found it useful.

dubiousintent · February 27, 2014

FYI: Added wghost81's questions and your answers as a "FAQ" section to the "Features" page on "code.google.com".

-Dubious-

R&D - New Modding Tools

Recommended Posts

Link to comment

Share on other sites

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Recently Browsing 0 members