XCOM Modding: Need of a centralized source of information?

[Amineri]

Well, when I create my modlets, I put them into the Custom Mod text format for ToolBoks. That format includes a string for the current game version, a la :

 

 

Works with XCOM Enemy Unknown version:

Patch 3 ( Tuesday, December 11, 2012 4:36 PM - Changelist: 347752 )

Including this with any notes about when the research was done would at least warn the user how much to trust the information.

[johnnylump]

All right, here's a draft of rescripting instructions. Please take this and edit it to your heart's content, or throw some suggestions at me how to improve it.

 

 

This will attempt to serve as a tutorial for modders who wish to rescript functions within X-Com: Enemy Unknown 2012.

It will assume that you already know how to use UExplorer and a hex editor together to make simple changes to upk files, like editing single values or changing signs in an equation. It will also assume you'll do smart things like back up your originals before making changes.

This also assumes you've found the function you want to change, and you know what it does.

First you'll need lots of practice reading hex code in UExplorer. It helps to be able to look at the hex and recognize certain patterns, so you can find ways to copy those patterns to build new code.

It also helps if the following equation makes sense to you: 19 + 01 = 1a. Windows calculator has a programmer mode that can help a lot with hex math and going between decimal and hex.

Make sure you have the unreal bytecode list with the upk editing tools. Some key hex to know:

05: Switch

06: Jump

07: JumpIfNot -- An If statement

16: End Parameters

1b: Function call

04: Return

0b: Null byte

(You should of course know by now that what these bytes mean depends on the context in which the game encounters them. The 0b in 2c 0b is not a null byte; it's an decimal-eleven, because 2c means an integer constant will follow.)

Rescripting is a very powerful way to change the game. It's also slow and has no tolerance for mistakes; if you have one byte off, the game won't load, or will crash suddenly, and you'll have no feedback why -- particularly if the code looks OK in UExplorer. So expect a great deal of trial and error and frustrations. Posting nonfuctional code on the Nexus is useful for getting other eyes on it to find your mistake.

Here are some limitations: You cannot add parameters to a script header. There are also some critically important functions that are denoted as "native," which I think means they're written in C++ and located in the exe file -- and you can't do anything to them at the time of this writing.

In addition, a key limitation is that no script may grow beyond or shrink from its original (actual) size. Think of the upk as a library that would collapse if any one book (function) is out of place or has pages added or subtracted. But you can pull a book off the shelf, change what's on the pages, and put it back.

These length restrictions mean for every thing you want to add to a function, you have to find something to take out. Sometimes you might get lucky and be able to replace one variable with another, but usually that's not the case; you'll need to remove some code.

So the first step is to examine the vanilla function you want to edit and see if you can take anything out without breaking the game. If it's a small function, or one that efficiently does exactly what it needs to, you're probably out of luck. And even if you can take out a little, you may not be able to take out enough for the new stuff you want to add.

Good candidates for removal are:

* Debugging code

* Multiplayer code

* Difficulty-specific code (if you are willing to combine difficulty-oriented settings. For example, UFO stats are different for the four difficulties, and they are all set in the same function. If you combine all the difficulties into you'll have a ton of extra space to play with).

* Code that doesn't do anything (These are likely artifacts of things considered but not included in vanilla).

* Inefficiently written vanilla code that can be tightened to achieve the same effect

I like to know how many bytes I'll have to play with, so at this point I overwrite all the code I'm getting rid of with null bytes: 0b (zero, b). These great little bytes do nothing but take up space.

Warning: Doing so will usually prevent the script from working without further changes. You're past the point of no return. (Well, except for control-z-ing your way back to your original code.)

Make sure you get rid of ONLY the code you want to get rid of. This is why recognizing 16 (end params) and other bits of hex is useful, because you can figure out where exactly one line ends and another begins.

It may be useful at this point to save your changes and open the function in UExplorer. It will look like a mess, most likely, because your jumps are all messed up now. But you can see if your function calls and other operations are still intact, at least (that is, you didn't cut out part of something you needed to keep). What will be messed up is the structure of the function -- things you want inside brackets after if-statements won't be; random jump calls will appear, etc. We'll fix that later.

Back to the hex editor: Once you have your null bytes in place, cut and paste all your null bytes to the location(s) you want to add new code.

Again, if you aren't confident what you are doing, save your changes and open the function again again in Explorer. It will still be a mess, but you should see all the lines you want to keep. If Explorer can't read the function, you've done something wrong.

Now you research and add hex for the new scripting you want to add, overwriting all the 0b's with hex that actually does something. This is the hardest part, of course, and you have far too many options for me to explain here. Look what other functions do what you want to do and copy it, and see if it works. If you are calling functions from another class, you might have to reference the class in the function call.

Once that's done, load again in UExplorer and see if it looks okay. Again the structure will be a mess, but the statements should be there.

(One note: One thing that often catches me is not putting enough 16s after a function call; it will show up in Explorer all right, but the game won't load. One of things you should check is whether you are missing a 16 at the end of a line).

Now there's two things left to fix: jumps and virtual size. These are interrelated concepts.

What happens when you add or remove certain elements to a function, like variable references, it gets bigger or smaller, but only in a certain -- virtual -- way. I think of it as the code ballooning when it's called up.

These changes are picked up and must be addressed in the structure of the function (that is, jumps), and in the header.

We'll address the header change first.

The header is always the first 48 bytes of the function. Don't touch anything in it except the virtual size. The virtual size is the second-to-last set of 4 bytes. The last four bytes are the actual size; the virtual size should always be >= the acutal size.

Here's a sample header:

CB 02 00 00 47 55 00 00 00 00 00 00 CB 02 00 00

7A 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00

5A 00 00 00 E3 0B 00 00 0D 00 00 00 09 00 00 00

For this function, the virtual size is 0D 00 00 00.

We need to change it, but we don't know what it should be.

In UExplorer, open your *edited* function. Select "View Tokens" (we'll also use this for fixing jumps, below).

Look for the last line. It should say EndOfScript. In front of that is the position of that line / line number / offset, like 0x109. Add one (in hex) to the number (disregarding the 0x), so you'd get 10a in this case. That's your virtual size.

You'll need to enter it like this: 0a 01 00 00 ... reversing the order of bytes, but not the individual bytes themselves.

If EndofScript isn't there, no worries! That means your virtual size is too small. Go add something to it -- usually a multiple of four, so 0d becomes 11, and then 15. (Usually each new variable adds 4 bytes, but that's not always the case). Then reload in UExplorer and check if EndofScript is there. Again, add 1 in hex to the line number of the EndofScript line, and that's the correct Virtual Size. You can go wrong by having the VS too large or too small, so always make sure it correlates with your EndofScript token.

Jumps (06, 07)

(Note: I believe this also applies to switch (05) statements; however, I have not worked with them extensively and will avoid giving direction regarding them here. YMMV).

If you've added or removed elements that have changed the virtual size of the function, every jump DESTINATION that is AFTER those new / removed elements will have changed. So you'll need to find every 06 and 07 that points to a destination AFTER your changes, and update them. Again, it's not the location of the jump that matters, it's the location of where it points to. So some jumps early in the fucntion may need to change, and some at the end (if they point back to the beginning) may not.

Jumps look like this in the hex:

06 4e 01: This means jump to position 014e in the code. Notice the destination bytes are reversed, but internally in order.

07 4a 02: This means that if the following condition is not true, jump to position 024a in the code.

To find where they should now be pointing to, go back to tokens mode.

If you want to go to some new code you just created (like you've added a full conditional statement), then enter the line number of the destination token after the 06/07.

If you're updating jumps from vanilla code I find it is often useful to open up the vanilla / unmodified version of the function separately in Explorer, in case you have a lot of jumps to change and can't tell where they need to point to.

For each jump, look at the unmodified code, and the content of the line (not the number) where the jump formerly pointed to.

Now find this line in your modified code. If the line number has changed, enter the new line number in your modified code's jump.

Example

In the unmodified file, Object mode, you see this:

// End: 0x2f8

if (conditional)

{

statements

}

some other statement

In hex, the jump will look like this:

07 f8 02 (conditional hex) (statement hex)

In token mode, you'll see the JumpIfNot where the 07 is, but you won't see the hex of the destination after it, so you'll have to switch back and forth between token and object mode. But in token mode you WILL see another line with the number 0x2f8. That's the line where the jump does to. Look at its content.

Now, go to your modified file, and find the line with the same content as the destination line, note its line number, and enter that in the bytes after the 06 or 07.

Save and reload your file in Explorer to make sure the structure is right. Once completed for every jump, this method should update loops and else statements correctly.

Good luck.

Outstanding Questions: Can you add local variables?

Edited March 13, 2013 by johnnylump

[Amineri]

Another key hex value (although not useful for recognizing things when looking at a hex dump) is good old 00

 

00 when placed before a four byte address represents a local variable (depending on location, of course).

Here is an example of how I put assemble a replacement bit of hex code:

 

 

07 9F 00 -- jump if not (with offset)

9A -- "==" operator

00 F3 76 00 00 -- local variable iWeapon

2C 08 -- int byte constant 8

16 -- execute "==" operation

I put in each hex code, along with a little descriptor. Something to keep in mind is that all operations are what is called in math as "prefix". Normally you'd write something like 4 + 5 (which is called "infix" since the operator is in the middle), but for hex code it looks more like + 4 5. This is why the operator goes first, then the two values in all the hex code.

Above, the hex code starts with a jump-if-not operator. This is how an "if" statement is implemented. If the next thing isn't true, it jumps to an offset specified by the two bytes immediately following the 07 hex code. This is why the 00 if 07 9F 00 doesn't represent a local variable -- because of the context it is in, it is part of a jump offset. The Unreal code is "little endian" as JL pointed out, so the 9F 00 offset would be read by us normal humans as 0x009F (note - I'll use "0x" to denote hex code for larger hex code numbers). In UE Explorer 1.2 Token View, that 0x009F is what you'll see. This is what JL said above, I'm just repeating it ^_^.

Next is 9A hex for the "==" operator, which compares the next two integers and returns true if they are equal.

The 00 F3 76 00 00 represents a local variable (you figure the address by mousing over the hexcode in UE Explorer).

2C 08 is the 8, and finally the 16 hex tells the CPU/ interpretter to execute whatever operation is "on the stack" ... in this case the 9A "==" operator.

The above hex looks like :

if(iWeapon == 8)

when decompiled.

In this example, I wasn't changing the body of the "if" statement, just the condition that caused it to trigger.

[anUser]

The main page in the wiki is up, 'though pretty empty

 

http://wiki.tesnexus.com/index.php/Category:XCOM_Modding

 

great tutorial johnnylump, that is going straight ahed to the wiki. I also found that BlackAlpha did an excellent job documenting how to modify single values in his modding tools pack readme, so that sould cover #2, as for the ini... we'll see, there's already a lot of stuff out there, it'd be a matter of gathering and perhaps re-writting it. Links to the tools you need step by step, at least right at the beggining, would help not scarying out newbies

 

I'll try now to create a few sample categories and functions so it doesn't look so much as a fake, and if you like it we can actually start discussing a shape for the templates to post cracked code.

 

edit: oh I thought that XCOM_Modding would be more fitting to our pouroses rather than XCOM, so to differenciate if there's ever a wiki dedicated to the game.

Edited March 13, 2013 by anUser

[johnnylump]

@ anUser -- I updated the how-to slightly for clarity's sake, in case you lifted it before now.

 

Amineri's post would be a great start on how to actually construct new code.

[Amineri]

Also, not trying to be nitpicky, but I think there are two ways to generate links ... one for internal wiki pages, and one for external webpages. The external pages show the little blue arrow. The bonus to internal is that you can create a link to a page that doesn't exist yet, and it shows up in the red color. If someone were to then follow that link, they could then create that page. Also gives a hint to the user that the page does not exist.

 

There is a blurb about it at this Nexus wiki page:

 

http://wiki.tesnexus.com/index.php/How_to_post_a_new_wiki_page

 

I'll maybe put some stuff up there once I finish figuring out this stun thing.

 

Lastly, I believe that the package of tools did not include the most recent UE Explorer, ver 1.2.1, but the older ver 1.1. I highly recommend the updated version, and many of the tips/tricks that I will be putting up will rely on it.

Edited March 13, 2013 by Amineri

[anUser]

Thanks, I def needed some tip for formatting and so. Also I've now realized the names suggested don't make use of the alphabetical sorting for categories, I'll think better of it.

[anUser]

well, naming could become a major issue if the same were to be made for other games, since it's the same wiki for all of them... what do you think about it? Should we just go ahead and name each entry just as the function or whatever is named? Should the entry have the Class.function_name format to avoid possible duplicate values, I guess?

[anUser]

It should look more consistant now. Instead of linking let's use the category system and let the forum lists related content. Now the main page for xcom modding at the wiki should be http://wiki.tesnexus.com/index.php/Category:XCOM_Modding (the category itself) while leaving the page for an exhaustive tutorial.

 

I've included a few placeholders sub-categories and a sample function

 

now my question is, can you imagine it as full of entries as UE explorer is? would it be useful at all?

[Amineri]

Here's my first stab at a wiki page : http://wiki.tesnexus.com/index.php/XCOM_developer_shell

 

It captures some of the what I've been messing around with recently. Any feedback is appreciated!