Cooking U to UPK

[abeclancy]

First attempt at this sort of modding, is I'm working on a mod to alter how the Protect the Device missions work. Namely, 1) spawn XCOM near the objective, 2) change the mission parameters a bit, 3) call in more reinforcements, so it's a more pseudo-Avenger-Defense mission instead of "run to objective, kill all enemies", 4) do this without having to modify EVERY parcel in the game that has a ProtectDevice OSP.

 

The core thing that I need to change, is the game's algorithm of how it picks spawn locations. Namely, note how in Mission Schedules there are MinXComSpawnDistance and IdealXComSpawnDistance properties, but no MaxXComSpawnDistance so that we can "force" XCOM to spawn near the device. (There might be a way to use Kismet to move the individual XCOM soldiers to the objective location, but I couldn't get it to work properly, and I'd much rather just a) use the spawn location algorithms where possible so that enemy spawn positions and such are calculated properly, and b) make this easy for other mods to use in the future by adding this functionality to the game, rather than just some Kismet scripting that would have to be duplicated everywhere).

 

And I actually have the updated spawning algorithms working! Just some simple modifications to the XComParcelManager with its native(Core) declaration, and when I load my XComGame.u via -NOSEEKFREELOADING, my spawn algorithm works, so I can specify additional options in Mission Schedules to make XCOM spawn near objectives. The only thing that isn't working, is running my XComGame.u without -NOSEEKFREELOADING. Maybe it's cooking, maybe it's something else, maybe you just straight-up can't get custom XComGame.u's working without changing the arguments. But, a) it would be nice if it worked out of the box, and b) if cooking XComGame.u results in greater performance, then I'd like to cook it since it's a big portion of the game code, and it should be optimized.

 

You do have to be careful about modding native classes, but I think you only have issues if you start modifying native function signatures, native struct properties, maybe even adding functions to native classes won't work. But just modifying any function code within the class itself is fine as long as the function's not native.

Edited February 29, 2016 by abeclancy

[davidlallen]

Thanks for explaining. This is definitely past my level of understanding. Maybe at some point you can write up a "code walkthrough" like this one:

 

http://forums.nexusmods.com/index.php?/topic/3805750-mod-help-conditional-blademaster/page-9&do=findComment&comment=34824230

[abeclancy]

Alright, a bit of progress here. Figured out the specifics of cooking the packages, but still not able to get them to load without -NOSEEKFREELOADING.

If you CD to your compiled \Script directory, then you can run:

"\steamapps\common\XCOM 2 SDK\Binaries\Win64\XComGame.exe" CookPackages "Script\XComGame.u" -platform=pc -usermode

This'll spawn what looks like (n-1) processes to cook the file, each process seems to use 1GB of memory (including the main process). You can use -singlethread to not spawn these processes.

The resulting output files (there are a lot!) will be placed in \XCOM 2 SDK\XComGame\CookedPC, which is fine because the actual game uses CookedPCConsole (which you can specify as -platform=pcconsole). Thing is, the CookedPCConsole directory is symlinked between \steamapps\common\XCOM 2 and \steamapps\common\XCOM 2 SDK. This symlinking occurs at some point during use/installation of the XCOM 2 SDK. So, by cooking packages with "pcconsole", you might be modifying what the standard game is running.

So, BACKUP the \steamapps\common\XCOM 2\XComGame\CookedPCConsole directory, so that when you are done with the cooking process you can make sure to replace the contents of that directory if you want. Alternatively, you can revalidate your files in Steam, or just un-do the directory symlinks temporarily before you do your cook, or rename the \steamapps\common\XCOM 2 SDK\XComGame\CookedPCConsole directory so that it doesn't overwrite anything in there during your cook. Just make sure that you do something before and after you cook, just to make sure that you are testing your mod in a clean environment.

Additionally, there is a ton of extra data copied into the output folder, \CookedPC (or \CookedPCConsole). I'm curious if the argument -skipmaps (or something else) on the cooking process would significantly reduce the output filesize. Additionally additionally, The cooking process tries to use WWise for some audio thing... which I assume doesn't matter because it's just code that's being cooked here, not content. It looks like it doesn't prevent the cooking from working if that version of WWise isn't installed (which I don't think you can get from the official site anymore, it's an old version that isn't listed).

Now the bad part... is that even a cooked file doesn't seem to load without -NOSEEKFREELOADING. Maybe it's because when I tested this, I cooked as pc instead of pcconsole. Or maybe I need to -skipmaps. More testing is in order. As soon as I revalidate the XCOM 2 and XCOM 2 SDK directories, put everything back where it belongs, and make some testing copies of everything, I'm going to try cooking with various options and such and see what happens. But I did learn something interesting:

Anything in \Mods\YourMod\Script is loaded if it's a valid pack file. Doesn't matter if it's cooked or uncooked, doesn't matter if it's named XComGame.u or YourMod.u, or if the extension is U or UPK. However, XComGame doesn't load, even if you change its name, unless -NOSEEKFREELOADING is specified, so it's not just a matter of naming, it's a matter of file content, the game engine refuses to load it.

 

The biggest question that remains to me, is how to get a properly cooked XComGame package loaded without that launcher argument. Is it even possible? Nothing in the technical documents seems to describe this process in much detail, so it might actually be a case of "you can't". Which would suck, but wouldn't necessarily be the end of the world for mods that change the core game functions. There would just have to be extra careful, and maybe have fallback behavior defined for when the argument isn't specified on the launcher. Maybe bring up a pop-up in game that indicates that the argument MUST be specified, and only give the player the Exit to Desktop button. That way no one would be able to complain "the mod doesn't work". It tells you the argument isn't active and needs to be set.

Edited March 1, 2016 by abeclancy

[FxsRMcFall]

Abeclancy, you are on the right track.

 

We left cooking out in the initial drop because it comes with some caveats and is only worth it if your mod absolutely needs it. However, tools and documentation for cooking are in our future plans because as mods become more sophisticated the need for it will grow.

 

Reasons to cook:

* You are making a new XComGame script package - ie. you are rewriting / changing our base game classes to be whatever you want. Total conversion, inaccessible function, etc.

* You have a LOT of new maps. To tell whether this is necessary you will just have to benchmark and see how your map pack affects load times. If it makes them really long it might be worth it to cook.

 

The main caveat to cooking is that you are tying your mod to a specific revision of the game content. If the base game cooked data changes, it may invalidate the cooked data that your mod contains.

 

Now, to help you with your XComGame cook... The symlink to the cookedpcconsole directory is to facilitate performing cooks for mods. This is so that your references to base game cooked packages and texture file cache locations are correct. The cooking process below is just if you are cooking a new script package.

 

Before cooking, you'll want to run:

"xcomgame.com make -final_release -full"

- This will output final release versions of the base game script packages that the cooker can use. They go into a ScriptFinalRelease directory in your SDK\XComGame directory.

Then:

"xcomgame.com cookpackages -platform=PCConsole -final_release -quickanddirty -modcook -sha -multilanguagecook=INT+FRA+ITA+DEU+RUS+POL+KOR+ESN"

- The output from this process will be a bunch of packages inside of a "XComGame\Published\CookedPCConsole" folder. You'll notice that the cooked XComGame script package is much larger than the uncooked because the cooker has embedded resources that the script packages references directly into it ( and the game needs this when running in seekfree mode ). Take the XComGame.upk and size files from that folder and copy it to a "CookedPCConsole" folder in your mod just like you would with a content package.

XComGame and all the core script packages are not something that UE3 normally permits to be overridden by downloaded content, but we made changes to make it possible - so if you have the above setup the game will use your cooked base game script packages. Our future plans include an option under the "tools" menu in Modbuddy that performs these steps for you, but for now a batch script or cmd can do the same.

[abeclancy]

This is some amazing help there, FxsRMcFall! I feel like I'm definitely making more progress towards getting these custom XComGame packs loading in a vanilla XCOM 2 installation now, and knowing exactly those build commands helps immensely! Most of those command arguments arn't listed in "xcomgame.exe help cookpackages", so it would have taken a long time to figure out those extra options.

 

Your instructions are obviously having an effect, but unfortunately the game is crashing upon loading. After placing XComGame.upk (and XComGame.upk.uncompressed_size) to a new directory \XCOM 2\XComGame\Mods\YourMod\CookedPCConsole, the following error occurs just after the intro movies are done playing.

[0015.50] Warning: Warning, Detected data corruption [header] trying to read 1048576 bytes at offset 10129142 from '..\..\XComGame\CookedPCConsole\World-a.tfc'. Please delete file and recook.
[0015.50] Critical: appError called: Assertion failed: appErrorf [File:G:\BuildAgent\work\a78b974cadaf0436\branches\Shipping\XCOM2\Development\Src\Core\Src\UnVcWin32.cpp] [Line: 150]
I/O failure operating on '..\..\XComGame\CookedPCConsole\World-a.tfc'
Stack: Address = 0x4029dad5 (filename not found) [in E:\SteamGames\steamapps\common\XCOM 2\Binaries\Win64\XCom2.exe]

The exact error message may change, sometimes it's CharTextures-a or Textures-a, sometimes it's the non "-a" versions of those files. And if those published files are also copied to the \Mods\YourMod\CookedPCConsole directory, the error message changes slightly (but I don't think that's the correct thing to do anyways).

[0018.76] Warning: Warning, Detected data corruption [header] trying to read 1048576 bytes at offset 3684731 from '..\..\XComGame\CookedPCConsole\World.tfc'. Please delete file and recook.
[0018.76] Critical: appError called: Assertion failed: appErrorf [File:G:\BuildAgent\work\a78b974cadaf0436\branches\Shipping\XCOM2\Development\Src\Core\Src\UnVcWin32.cpp] [Line: 150]
I/O failure operating on '..\..\XComGame\CookedPCConsole\World.tfc'
Stack: Address = 0x4051dad5 (filename not found) [in E:\SteamGames\steamapps\common\XCOM 2\Binaries\Win64\XCom2.exe]

After running "xcomgame.exe make -final_release -full", there are a series of prompts indicating "These files need to be updated: ..\..\Development\Src\Core\Inc\CoreClasses.h. Do you want to overwrite the existing versions?" Looks like there's one popup for each package that the Make has detected needing updating, sometimes a single file sometimes up to a couple dozen. Both accepting all prompts and rejecting all prompts results in no overall change, the crash on load still occurs.

 

I'm going to redownload the XCOM 2 and XCOM 2 SDK directories in their entirety from Steam, just to make sure there isn't something wrong with my directory after having been fiddling with building processes. I'll also try and throw in some logging statements, just to make double-sure that the custom pack XComGame (maybe also try with Core) is being loaded, but I think it is.

 

Couple of things that may make a difference: I'm testing with a "Default" project configuration build, not "Debug". The game is being run without "-NOSEEKFREELOADING" as an argument.

[FxsRMcFall]

OK, based on your error... I'm guessing your cook produced some small-ish texture file caches in the published cooked data folder. These are bad, because it means that your script packages will point to offsets in these small files instead of the ones that the base game has. Anyways, to fix this...

 

Copy the following files into your "XComGame\Published\CookedPCConsole" out of the symlinked cooked directory:

 

GuidCache.upk

GlobalPersistentCookerData.upk

PersistentCookerShaderData.bin

All TFC files

 

And then run the cook script again. If everything is operating correctly, you should see the TFC files remain unmodified by the cook ( indicating your script package is pointing to all the right offsets into the TFCs ).

[abeclancy]

That worked wonderfully! Added the files once, and now subsequent cooks contain the correct offset information and appear to run flawlessly in a vanilla XCOM 2 installation.

 

Thank you very much for the assistance FxsRMcFall!

 

To anyone following this thread, I'm going to collate all my notes, instructions, and everything from this thread, rewrite it, and probably make a new thread describing the process as thoroughly as possible.

[FxsRMcFall]

Nice! A few things to note about cooking:

 

Mods that replace XComGame are the highlander of mods, there can only be one. But it lets you do pretty much whatever you want with the script source - don't need to use class replacements, screen listeners, et. al. Also, regular mods that just add new script packages can still work with an XComGame replacement mod - but they would be effectively modding a mod so it could get weird unless they use your mod's source code as a base. "Weird" could include crashes if you start renaming or deleting variables or functions in the base classes that another mod's complied script code is using.

 

Lastly, be very careful about making changes to base game classes that are part of the cooked data. One of the reasons that cooked data loading is so much faster is that it doesn't do any data type checking / version checking / tagged serialization / etc. So if you modify the data layout of a cooked object with script changes and don't update the cooked data then the object will serialize in using the old data layout. This causes random memory overwrites, crashes. So best to leave actors in the cooked data ( ie. stuff like XComLevelActor ) alone, but if you do make changes stick to just function changes and transient variables if you add variables to the class.

[abeclancy]

Sounds about right, even down to me calling them Highlander mods!

 

Indeed, I'm working on some modifications to the Protect the Device mission, and the only thing I really need in order to do everything I want, is change a few functions in XComParcelManager. No new variables or anything else, so at the very least I should be able to get back to work on the mission scripting now and know that I can release it without any further core package modifications.

 

The only variable-adding I've tested is to add in a single variable to the XComEngine.uc class, as a test to see if it is possible to create pseudo-global storage, like a new template manager type. I've noticed that crashes do occur, but where you add the variable to the class seems to matter. Add an INT before other variables and it crashes, add it after and it's fine.

 

I'm imagining what's happening, is that the deserialization from C++ object to Unreal object works when the variable is last because it doesn't interfere with the loading of the other variables in order, and the last variable that was added is either a) in its normally default state (so probably zero) or b) in an invalid state, so could be any value. It seems to be stable enough, but I imagine that I can't add variables to the class if the class is ever part of a C++ to Unreal array of objects, as the size differences between the objects would cause the deserialization of more than a single object to fail. I also don't imagine that the contents of the variable is ever going to be persisted, since GetEngine() returns the XComEngine instance every time it is called.