Forcing game to load from DefaultGameCore.ini

[Amineri]

Did some more digging around to try and discover what is causing the ever-increasing load times. In my current set-up I did not install the DGC.ini "load from file" change, nor install the developer console.

 

I was using Long War 1.9 beta1 with some of my own mods added on (fixes and such mostly). I used ToolBoks 1.21 to update the SHA values, and then realized that ToolBoks installs the exe hex change that skips the SHA checks altogether.

 

I discovered that in the My Game folder is a set of log files. For me they are in XComGame/Logs.

 

Perusing through the log files I discovered one of the instances of exceptionally long loading times :

 

[0066.22] Log: ########### Finished loading level: 57.086431 seconds

 

Nearly a minute to load the level at this point.

 

For some reason the game appears to be loading the DLC content over and over again. The following 24 loads are repeated 154 times, from line 9735 to line 17119

 

 

[0013.10] Log: ------------------ Fully loading ../../XComGame/DLC/PCConsole/DLC_Day060\CookedPCConsole\DLC_Day060_SF.upk

[0013.11] Log: ------------------ Fully loading ../../XComGame/DLC/PCConsole/DLC_Day060\CookedPCConsole\Deco_Archangel0_SF.upk

[0013.12] Log: ------------------ Fully loading ../../XComGame/DLC/PCConsole/DLC_Day060\CookedPCConsole\Deco_Carapace1_SF.upk

[0013.13] Log: ------------------ Fully loading ../../XComGame/DLC/PCConsole/DLC_Day060\CookedPCConsole\Deco_Ghost0_SF.upk

[0013.14] Log: ------------------ Fully loading ../../XComGame/DLC/PCConsole/DLC_Day060\CookedPCConsole\Deco_Kevlar1_SF.upk

[0013.15] Log: ------------------ Fully loading ../../XComGame/DLC/PCConsole/DLC_Day060\CookedPCConsole\Deco_Psi0_SF.upk

[0013.16] Log: ------------------ Fully loading ../../XComGame/DLC/PCConsole/DLC_Day060\CookedPCConsole\Deco_Skeleton1_SF.upk

[0013.17] Log: ------------------ Fully loading ../../XComGame/DLC/PCConsole/DLC_Day060\CookedPCConsole\Deco_Titan0_SF.upk

[0013.18] Log: Failed to find Package Helmet_Archangel0 to FullyLoad [FullyLoadType = 0, Tag = Command1]

[0013.18] Log: Failed to find Package Helmet_Carapace1 to FullyLoad [FullyLoadType = 0, Tag = Command1]

[0013.18] Log: ------------------ Fully loading ../../XComGame/DLC/PCConsole/DLC_Day060\CookedPCConsole\Helmet_Ghost0_SF.upk

[0013.19] Log: ------------------ Fully loading ../../XComGame/DLC/PCConsole/DLC_Day060\CookedPCConsole\Helmet_Kevlar1_SF.upk

[0013.20] Log: ------------------ Fully loading ../../XComGame/DLC/PCConsole/DLC_Day060\CookedPCConsole\Helmet_Psi0_SF.upk

[0013.21] Log: Failed to find Package Helmet_Skeleton1 to FullyLoad [FullyLoadType = 0, Tag = Command1]

[0013.21] Log: Failed to find Package Helmet_Titan0 to FullyLoad [FullyLoadType = 0, Tag = Command1]

[0013.21] Log: ------------------ Fully loading ../../XComGame/DLC/PCConsole/DLC_Day060\CookedPCConsole\Head_AsnMale_F_SF.upk

[0013.22] Log: ------------------ Fully loading ../../XComGame/DLC/PCConsole/DLC_Day060\CookedPCConsole\Hair_MaleHair_N_SF.upk

[0013.23] Log: ------------------ Fully loading ../../XComGame/DLC/PCConsole/DLC_PackIn\CookedPCConsole\Deco_Kevlar0_SF.upk

[0013.24] Log: ------------------ Fully loading ../../XComGame/DLC/PCConsole/DLC_PackIn\CookedPCConsole\Helmet_Kevlar0_SF.upk

[0013.25] Log: ------------------ Fully loading ../../XComGame/DLC/PCConsole/DLC_PackIn\CookedPCConsole\Deco_Skeleton0_SF.upk

[0013.26] Log: ------------------ Fully loading ../../XComGame/DLC/PCConsole/DLC_PackIn\CookedPCConsole\Helmet_Skeleton0_SF.upk

[0013.27] Log: ------------------ Fully loading ../../XComGame/DLC/PCConsole/DLC_PackIn\CookedPCConsole\Deco_Carapace0_SF.upk

[0013.28] Log: ------------------ Fully loading ../../XComGame/DLC/PCConsole/DLC_PackIn\CookedPCConsole\Helmet_Carapace0_SF.upk

[0013.29] Log: ------------------ Fully loading ../../XComGame/DLC/PCConsole/DLC_PackIn\CookedPCConsole\Hair_OriginalSol

 

 

Other loads are also repeated over and over. Zhang's animations are loaded 154 times, from line 9177 to line 9330

 

[0005.57] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 2 for LoadForAllGameTypes: Zhang_ANIM

Other DLC lines repeatedly being loaded:

 

[0005.57] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for CIN_LoadScreen: UILibrary_MapImages_DLC1

[0005.57] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for DLC1_1_LowFriends: DLC_Day060

[0005.57] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for DLC1_2_CnfndLight: DLC_Day060

[0005.57] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for DLC1_3_Gangplank: DLC_Day060

 

Additionally, the following set of lines is repeated the similar 154 times:

 

[0005.54] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: DLC_Day060

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Deco_Archangel0

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Deco_Carapace1

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Deco_Ghost0

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Deco_Kevlar1

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Deco_Psi0

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Deco_Skeleton1

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Deco_Titan0

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Helmet_Archangel0

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Helmet_Carapace1

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Helmet_Ghost0

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Helmet_Kevlar1

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Helmet_Psi0

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Helmet_Skeleton1

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Helmet_Titan0

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Head_AsnMale_F

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Hair_MaleHair_N

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Deco_Kevlar0

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Helmet_Kevlar0

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Deco_Skeleton0

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Helmet_Skeleton0

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Deco_Carapace0

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Helmet_Carapace0

[0005.55] Log: ------------- AddPackagesToFullyLoad: fullyloadtype 0 for Command1: Hair_OriginalSoldier

 

-------------------------

 

Here's my current hypothesis :

The game does some ini file merging when it loads, in order to handle the DLC content :

 

[0005.48] DevDlc: Merged DLC config file (../../XComGame/DLC/PCConsole/DLC_Day060\Config\XComGame.ini) into existing config file (..\..\XComGame\Config\XComGame.ini)

[0005.48] DevDlc: Merged DLC config file (../../XComGame/DLC/PCConsole/DLC_Day060\Config\XComGame.int) into existing config file (..\..\XComGame\Localization\int\XComGame.int)

[0005.48] DevDlc: Merged DLC config file (../../XComGame/DLC/PCConsole/DLC_Day060\Config\XComMaps.ini) into existing config file (..\..\XComGame\Config\XComMaps.ini)

[0005.48] DevDlc: Merged DLC config file (../../XComGame/DLC/PCConsole/DLC_Day060\Config\XComNarrative.ini) into existing config file (..\..\XComGame\Config\XComNarrative.ini)

[0005.48] DevDlc: Merged DLC config file (../../XComGame/DLC/PCConsole/DLC_Day060\Config\XComStrategyGame.int) into existing config file (..\..\XComGame\Localization\int\XComStrategyGame.int)

[0005.48] DevDlc: Merged DLC config file (../../XComGame/DLC/PCConsole/DLC_Day060\Config\XComUIShell.int) into existing config file (..\..\XComGame\Localization\int\XComUIShell.int)

[0005.54] DevDlc: Merged DLC config file (../../XComGame/DLC/PCConsole/DLC_PackIn\Config\XComContent.ini) into existing config file (..\..\XComGame\Config\XComContent.ini)

[0005.54] DevDlc: Merged DLC config file (../../XComGame/DLC/PCConsole/DLC_PackIn\Config\XComEngine.ini) into existing config file (..\..\XComGame\Config\XComEngine.ini)

 

My hypothesis is that this merging process is somehow in error, and that each time the game loads and the merge process executes, the assets above are caused to be loaded another time. So first game time it's only once, but after loading the game 100 times it's now loading those assets 100 times.

This is why the loading process grows over time. It also really only affects us modders who are constantly quitting and restarting the game over and over again when adding new features / bug-testing. Regular users don't launch and quit the game over and over like we do ^_^.

 

The issue appears to be in the XComEngine.ini / [Engine.PackagesToFullyLoadForDLC] section. In that section the following 37 lines are repeated 182 times, from line 2607 to line 9338 :

MapName=Command1
Package=DLC_Day060
Package=Deco_Archangel0
Package=Deco_Carapace1
Package=Deco_Ghost0
Package=Deco_Kevlar1
Package=Deco_Psi0
Package=Deco_Skeleton1
Package=Deco_Titan0
Package=Helmet_Archangel0
Package=Helmet_Carapace1
Package=Helmet_Ghost0
Package=Helmet_Kevlar1
Package=Helmet_Psi0
Package=Helmet_Skeleton1
Package=Helmet_Titan0
Package=Head_AsnMale_F
Package=Hair_MaleHair_N
LoadForAllGameTypes=LoadForAllGameTypes
Package=Zhang_ANIM
MapName=CIN_LoadScreen
Package=UILibrary_MapImages_DLC1
MapName=DLC1_1_LowFriends
Package=DLC_Day060
MapName=DLC1_2_CnfndLight
Package=DLC_Day060
MapName=DLC1_3_Gangplank
Package=DLC_Day060
MapName=Command1
Package=Deco_Kevlar0
Package=Helmet_Kevlar0
Package=Deco_Skeleton0
Package=Helmet_Skeleton0
Package=Deco_Carapace0
Package=Helmet_Carapace0
Package=Hair_OriginalSoldier

So, my supposition is that this has nothing to do with the SHA hash check removal, the DGC.ini load-from-file change, or the dev console, but is instead a faulty ini-merge process with the DLC.

 

Removing the excess lines from the XComEngine.ini / [Engine.PackagesToFullyLoadForDLC] section periodically should reduce the loading time. This also explains why re-verifying from Steam fixes the issue -- the XComEngine.ini file is reloaded from scratch.

 

I'll be testing this out on my next game re-install.

[Amineri]

Just confirmed that the XComEngine.ini file is getting the additional loads merged in, even with a completely vanilla version of the game. (I re-verified from Steam and am running in online mode).

 

I launched the game a second time, then quit and opened the XComEngine.ini file (the one in the My Games folder). The DLC Content is specified to be loaded twice :

 

 

 

 

[Engine.PackagesToFullyLoadForDLC]

MapName=Command1

Package=DLC_Day060

Package=Deco_Archangel0

Package=Deco_Carapace1

Package=Deco_Ghost0

Package=Deco_Kevlar1

Package=Deco_Psi0

Package=Deco_Skeleton1

Package=Deco_Titan0

Package=Helmet_Archangel0

Package=Helmet_Carapace1

Package=Helmet_Ghost0

Package=Helmet_Kevlar1

Package=Helmet_Psi0

Package=Helmet_Skeleton1

Package=Helmet_Titan0

Package=Head_AsnMale_F

Package=Hair_MaleHair_N

LoadForAllGameTypes=LoadForAllGameTypes

Package=Zhang_ANIM

MapName=CIN_LoadScreen

Package=UILibrary_MapImages_DLC1

MapName=DLC1_1_LowFriends

Package=DLC_Day060

MapName=DLC1_2_CnfndLight

Package=DLC_Day060

MapName=DLC1_3_Gangplank

Package=DLC_Day060

MapName=Command1

Package=Deco_Kevlar0

Package=Helmet_Kevlar0

Package=Deco_Skeleton0

Package=Helmet_Skeleton0

Package=Deco_Carapace0

Package=Helmet_Carapace0

Package=Hair_OriginalSoldier

MapName=Command1

Package=DLC_Day060

Package=Deco_Archangel0

Package=Deco_Carapace1

Package=Deco_Ghost0

Package=Deco_Kevlar1

Package=Deco_Psi0

Package=Deco_Skeleton1

Package=Deco_Titan0

Package=Helmet_Archangel0

Package=Helmet_Carapace1

Package=Helmet_Ghost0

Package=Helmet_Kevlar1

Package=Helmet_Psi0

Package=Helmet_Skeleton1

Package=Helmet_Titan0

Package=Head_AsnMale_F

Package=Hair_MaleHair_N

LoadForAllGameTypes=LoadForAllGameTypes

Package=Zhang_ANIM

MapName=CIN_LoadScreen

Package=UILibrary_MapImages_DLC1

MapName=DLC1_1_LowFriends

Package=DLC_Day060

MapName=DLC1_2_CnfndLight

Package=DLC_Day060

MapName=DLC1_3_Gangplank

Package=DLC_Day060

MapName=Command1

Package=Deco_Kevlar0

Package=Helmet_Kevlar0

Package=Deco_Skeleton0

Package=Helmet_Skeleton0

Package=Deco_Carapace0

Package=Helmet_Carapace0

Package=Hair_OriginalSoldier

 

 

 

This pretty much convinces me that it isn't any mod that's causing the increasing load times -- it's a bug in the vanilla code that merges the game and DLC ini files.

[Bertilsson]

I just tested what happens if I simply renamed XComEngine.ini to something else.

XComEngine.ini is then happily recreated when I launch the game, but I lose my config settings, so I guess it really has to be cleaned and not just deleted.

[Bertilsson]

I have created a little vbscript that cleans up the XComEngine.ini file from redundancy.

 

It is not pretty at all but it gets the job done.

 

It can be downloaded here: http://hem.bredband.net/bertrich/XCOM/CleanUpXComEngineIni.zip

 

But since your browser & anti-virus will probably scream that you are doing something dangerous you could also do it the hard way:

1. Open notepad.

2. Copy & Paste below text into notepad

Const MY_DOCUMENTS = &H5&
Const ForReading = 1
Const ForWriting = 2
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objShell = CreateObject("Shell.Application")
Set objFolder = objShell.Namespace(MY_DOCUMENTS)
Set objFolderItem = objFolder.Self 

ConfigFolder = objFolderItem.Path & "\My Games\XCOM - Enemy Unknown\XComGame\Config\"
Set OrgFile = objFSO.OpenTextFile(ConfigFolder & "XComEngine.ini", ForReading)

strOrgText = OrgFile.ReadAll
OrgFile.Close

arrMapName = Split(strOrgText, "MapName=Command1")
if Ubound(arrMapName) >2 then
  strOutput = arrMapName(0) & "MapName=Command1" & arrMapName(1) & "MapName=Command1" & arrMapName(Ubound(arrMapName))
  timestamp = year(Now()) & "-" & Right(Cstr(month(now()) + 100),2) & "-" & Right(Cstr(day(now()) + 100),2) & "_" & Right(Cstr(hour(now()) + 100),2) & Right(Cstr(Minute(now()) + 100),2) & Right(Cstr(Second(now()) + 100),2)
  objFSO.CopyFile ConfigFolder & "XComEngine.ini", ConfigFolder & "XComEngine.bak" & timestamp
  Set destFile = objFSO.OpenTextFile(ConfigFolder & "XComEngine.ini", ForWriting)
  destFile.WriteLine(strOutput)
  destFile.Close
  msgbox (Ubound(arrMapName)/2)-1 & " redundant sets of engine DLC packages has been removed from XComEngine.ini." & vbnewline & vbnewline & "A backup of your original file has been saved here:" & vbnewline & ConfigFolder & "XComEngine.bak" & timestamp
else
  msgbox "Everything looks fine. No action taken."
end if 

3. File --> Save As... --> CleanUpXComEngineIni.vbs (make sure to not save it as CleanUpXComEngineIni.vbs.txt by mistake).

 

When you run the script it will first check that <My Documents>\My Games\XCOM - Enemy Unknown\XComGame\Config\XComEngine.ini contains more than two "MapName=Command1".

 

If that is true, a backup of the original file will be made and the third occurance and everything after will be removed.

 

If your "My Games" folder is called something strange like "Meinen Spielen" you will have to change that on line 9.

 

Edit: Have corrected the code so that it only removes lines between the second and the last "MapName=Command1" so that the scenario described below does not cause a problem.

Edited July 7, 2013 by Bertilsson

[Amineri]

Just a small word of caution when using the above tool.

 

For some reason my earlier XComEngine.ini file (the one in the My Games folder) ended with the lines :

[WinDrv.Accessibility]
StickyKeysHotkey=False
ToggleKeysHotkey=False
FilterKeysHotkey=False
StickyKeysConfirmation=False
ToggleKeysConfirmation=False
FilterKeysConfirmation=False

The appeared just after the DLC lines, but I'm not quite sure why.

 

I renamed my original one and let the game recreate it. The new one has those lines preceding the DLC load lines.

[Bertilsson]

Code has been corrected.

[Kartarasha]

How does one find the hex offset ya'll are talking about? The hex editor I'm using, HxD, doesn't show the line. Sorry if the question is newbish.

[Zybertryx]

With the use of an additional program called "UE Explorer" and its "Token View" feature.

 

UE Explorer basically decompiles the HEX into the language-code that the developers used to write the functions so that you can view it somewhat more intelligably. Unfortunately, we can't just write in that language and have a recompiler magically hexificate it all back again for us. That's perhaps the main reason why modding XCOM is such an ordeal. We have to do it in reverse; HEX to decompiled code. :confused:

 

It's kinda like trying to make flintstone and kindle out of a naked flame. . .

Edited July 10, 2013 by Zybertryx

[Amineri]

How does one find the hex offset ya'll are talking about? The hex editor I'm using, HxD, doesn't show the line. Sorry if the question is newbish.

 

 

With the use of an additional program called "UE Explorer" and its "Token View" feature.

 

UE Explorer basically decompiles the HEX into the language-code that the developers used to write the functions so that you can view it somewhat more intelligably. Unfortunately, we can't just write in that language and have a recompiler magically hexificate it all back again for us. That's perhaps the main reason why modding XCOM is such an ordeal. We have to do it in reverse; HEX to decompiled code. :confused:

 

It's kinda like trying to make flintstone and kindle out of a naked flame. . .

 

Kartarasha, more specifically what we do is use UE Explorer to find the appropriate hex that we want to change -- when those changes are in a .upk (Unreal Package file I think?).

 

In this case, however, the items being changed are text strings within the xcomgame.exe.

 

I use HxD also (that's how I found the strings). One thing is that you need to search for unicode strings instead of 1-byte ASCII -- there is a check-box for it in the HxD search window when you search for text. HxD should find two instances of the strings. The second one is the one that needs changing.

[dubiousintent]

How does one find the hex offset ya'll are talking about? The hex editor I'm using, HxD, doesn't show the line. Sorry if the question is newbish.

 

Edit: A more polished version of these instructions may now be found in the Wiki article at 'Basic Guide to installing mods#Offset Location Values'.

 

There are two ways your question might be interpreted. 1) Given a hex-code 'offset value', how do I find that location in the file to be modified; OR 2) Given some decompiled code sequence, how do I determine the equivalent hex-code and then find the 'offset value' of the beginning location of that equivalent hex code in the packed UPK or executable?

 

Zybertryx and Amineri gave the generalized answer to interpretation #2. However, I imagine many people would appreciate an answer to interpretation #1, so here goes. (This will get added to the Wiki once any rough spots have been identified and smoothed over.)

 

When you open any hex editor (but we will use HxD here because it's in our suggested toolkit) you have to first tell it what file you want to edit. For this example we will assume we want to edit the game executable XComGame.EXE to enable INI file loading by modifying location '0x157D93A'. Note the '0x' prefix simply indicates that the following value is a hexadecimal number, and should be ignored. The actual offset value is '157D93A'.

 

1. Open HxD and then select "File | Open' and navigate your <steam install path> to '\XCom-Enemy-Unknown\Binaries\Win32\' and select 'XComGame.EXE'.

 

2. Select 'Search' in the menu bar and then drop down to select 'Goto', or use the hotkey combination of <Ctrl>+<G>. (This results in the smaller box at the top of this image.)

 

 

3. Enter the 'offset location' in the 'offset' field of the 'Goto' box, as shown in the upper part of the image. Double check that you do not have any extra characters when you either paste or type the offset value. The field has a right-justified zero by default. (Don't worry about case; HxD will automatically convert valid letters into upper case for you.)

 

4. Check that the two radio buttons in that dialog box are set correctly: type of value is on 'hex'; and 'Offset relative to' is on 'begin'.

 

5. When you click on the 'OK' button, the edit cursor ('|') in the HxD window will be positioned at that location in the hex portion of the display, ready to start replacing the byte with your changes. (Unfortunately, the editor cursor is not displayed in the image.)

 

Notice that the very first column of the display for each line is also a hex value very similar to what you entered in the 'Goto' box (actually all but the very last character). Each line displays 16 bytes, starting in position zero (notice the column labels across the top line), with each byte consisting of a pair of hexadecimal characters. In the image the very bottom line offset is '0157D930'. Our offset location is in the '0A' (or 10th) column of that row. So it is entirely possible to scroll through the file to the desired offset location. <Ctrl>+<G> is simply faster.

 

To the right of the hex bytes are displayed the 16 printable ASCII characters for each byte. Where there is not a printable ASCII character for a byte, a period is used as a placeholder.

 

Note that a dotted-box is displayed around the equivalent ASCII character for the same byte that the hex edit cursor is positioned at. You can edit either the hex byte or the ASCII character, but do not attempt to place a non-printable character in the ASCII display. For this reason, it is best to get used to entering only hex values and treat the ASCII display as a means of verification.

 

Since this is about how to locate an offset location, I'll stop here. The details about the necessary hex changes to enable INI file loading are available in the 'Recent Discoveries' section of the Wiki article 'Basic Guide to installing mods'.

 

Please point out anything you find confusing or needing more explanation.

 

-Dubious-

Edited July 11, 2013 by dubiousintent