Save game file format?

[wghost81]

Good work! I'm pretty much sure saves format is similar to packages format, i.e. it contains serialized data. In case you haven't checked it yet:

http://forums.nexusmods.com/index.php?/topic/1254328-upk-file-format/

http://forums.nexusmods.com/index.php?/topic/1253996-upk-utils/

[Drakous79]

Here's what I know and found on the web about save files.

 

The save file consists of:

• Header - 1024 bytes
• Compressed data - variable size

Compressed data are organized into numerous compressed chunks (N FCompressedChunk). There is no compressed chunks list (FCompressedChunksList) containing uncompressed/compressed size and offset of every FCompressedChunk. There is only one compressed block (FCompressedBlock) per chunk.

 

FCompressedChunk structure:

FCompressedChunk
    - FCompressedChunkHeader     // 12 bytes
    - FCompressedBlock
        - FCompressedBlockHeader // 8 bytes
        - Raw compressed data    // variable size

---

Summarized, each FCompressedChunk looks like this:

• FCompressedChunkHeader - 12 bytes
• FCompressedBlockHeader - 8 bytes
• Raw compressed data - variable size

An example:

// FCompressedChunkHeader - 12 bytes
// Number of blocks = rounded up (uncompressed size / block size)
// Block size seems to be always 0x20000 (131072)
C1 83 2A 9E    4 bytes UE3 package signature (magic)
00 00 02 00    4 bytes Block size
3B 63 00 00    4 bytes Compressed size
00 00 02 00    4 bytes Uncompressed size

// FCompressedBlockHeader - 8 bytes
3B 63 00 00    4 bytes Compressed size
00 00 02 00    4 bytes Uncompressed size

// Raw compressed data - variable size

---

PC is using LZO1X_1 compression. It is said, that iOS uses zlib.

 

Saves are protected by CRC checksum, that must be updated when repacking a file. I was not able to calculate it right.

 

For those who still have a problem with gamesave for PC or iOS version, let me explain few details. Data save is just serialized sets of classes (nothing more to add as people are doing great with finding & replaces patterns). In PC and iOS version (don’t have PS3 save) save game is compress with different algorithm. PC is using LZO1X_1 when iOS compresses data by zlib. Both saves are protected by CRC checksum. However, PC version has additional one CRC. One calculated from compressed data and 2nd generated from header.

 

Save file crc check is at 0xBE offset and it is the same customized CRC32b as from Windows version. I don't see difference between PC and iOS version, the savefile structure is exactly the same.

 

To fix it, you need to guys calculate CRC from whole compress data which is from offset 1024 to EOF() and put it just after game language code. In example, it’s at 0xEA. To calculate correct position it’s required to go though save header variable. Sum of first 3 in red (it’s just sizeof string), the 4th one is for DLC if you don’t have it will be 0 anything else it’s string size. And finally add 8 to results for sum and you are in correct place for 1st CRC.

2nd CRC has static offset so it’s not needed to do similar operation as for 1st one. The offset is 0x3FC. 4 bytes earlier, at 0x3F8 is size of block from beginning of save which will be used for generate checksum.

 

Sources:

user carpi at 360haven.com - checksum (scroll down 2 posts for python code)

user carpi at 360haven.com - saves structure, compression and checksum

user fairchild at 360haven.com - XCOM Enemy Unknown Compression Toolkit

user mudithegamer at 360haven.com - XCOM: Enemy Unknown Save Editor

[dubiousintent]

Wiki article 'Savegame file format - XCOM:EU 2012' has been created from this thread. Note there are tools written for the XBox on the 360Haven site that can be used on the PC, which are linked in the article. Free registration with 360Haven IS required to download them.

 

-Dubious-

[Amineri]

On the upk side, for those that weren't already aware, it appears that the various structures that begin with "CheckpointRecord" are what controls goes into the save file.

 

For the ammo mod I had to alter which variable in XGWeapon were saved/loaded to/from the savefile. For various reasons I had to use a new variable to store the ammo cost -- m_iTurnFired. After some experimentation I realized that this was not being preserved when quitting and reloading within a tactical mission. The vanilla CheckpointRecord for XGWeapon was :

struct CheckpointRecord_XGWeapon extends CheckpointRecord_XGInventoryItem
{
    var int iAmmo;
    var int iOverheatChance;
};

I altered the struct hex so that it is now :

struct CheckpointRecord_XGWeapon extends CheckpointRecord_XGInventoryItem
{
    var int iAmmo;
    var int m_iTurnFired;
};

for my mod, and m_iTurnFired is now saved (allowing quitting and restarting during a tactical mission without weapons all automatically reloading.

 

Presumably these structures are being serialized into the savegame file -- see wghost's articles/documents to get a handle on serialization. The easiest way to find all of the Checkpoint Records is via a search for "struct CheckpointRecord" in UE Explorer.

 

As an example, XGStrategySoldier has :

struct CheckpointRecord
{
    var TCharacter m_kChar;
    var TSoldier m_kSoldier;
    var int m_aStatModifiers[ECharacterStat];
    var XGStrategySoldier.ESoldierStatus m_eStatus;
    var int m_iHQLocation;
    var int m_iEnergy;
    var int m_iTurnsOut;
    var int m_iNumMissions;
    var string m_strKIAReport;
    var string m_strKIADate;
    var string m_strCauseOfDeath;
    var bool m_bPsiTested;
    var bool bForcePsiGift;
    var bool m_bMIA;
    var bool m_bAllIn;
    var TInventory m_kBackedUpLoadout;
    var XGCustomizeUI.EEasterEggCharacter m_eEasterEggChar;
    var array<XComGame.XGTacticalGameCoreNativeBase.EPerkType> m_arrRandomPerks;
    var int m_arrMedals[EMedalType];
    var bool m_bBlueShirt;
};

I've omitted the defaultProperties for the struct.

 

Game loading appears (at least in part) to use the state LoadGameAsync in UILoadGame:

simulated state LoadGameAsync
{
    J0x00:    // End:0x191 [Loop If]
    if(((XComOnlineEventMgr(GameEngine(class'Engine'.static.GetEngine()).OnlineEventManager).CheckpointIsSerializing || XComOnlineEventMgr(GameEngine(class'Engine'.static.GetEngine()).OnlineEventManager).CheckpointIsWritingToDisk) || XComOnlineEventMgr(GameEngine(class'Engine'.static.GetEngine()).OnlineEventManager).ProfileIsSerializing) || XComOnlineEventMgr(GameEngine(class'Engine'.static.GetEngine()).OnlineEventManager).StorageWriteCooldownTimer > float(0))
    {
        Sleep(0.10);
        // [Loop Continue]
        goto J0x00;
    }
    XComOnlineEventMgr(GameEngine(class'Engine'.static.GetEngine()).OnlineEventManager).LoadGame(m_iLoadGameAsync, ReadSaveGameComplete);
    GotoState('None');
    stop;            
}

Note the timer loop to ensure that the Checkpoint isn't still serializing or writing to disk before LoadGame can be called.

 

Unfortunately both LoadGame and SaveGame in XComOnlineEventMgr are native functions, so I haven't been able to figure out in what order the CheckpointRecords are written to / read from disk.

[wghost81]

There are some other interesting functions, worth checking too: CreateCheckpointRecord and ApplyCheckpointRecord. I think they used to control class-specific save/load process.

[FogGene]

Very good info guys, much appreciated. Sorry I haven't responded earlier, I had a couple of busy days and could only do quick forum reads. I'm going to try to reply to a few interesting things, but before that a plea if I may: Is there a chance I could borrow some XCOM: Enemy Within save games to test the explorer?

 

Last patched version, to keep things simple for now. Preferable Tactical View map savegames, but I'll take any Strategy View savegame too, since sooner or later I'll have to get around those anyway.

 

I just finished the GUI for the full save read (it's not final at all, but check the last post here for some pics if curious) and I'd like to see if it works (or more likely where it fails) when reading other saves.

 

 

---------------------------------------------

 

 

Good work! I'm pretty much sure saves format is similar to packages format, i.e. it contains serialized data. In case you haven't checked it yet:
http://forums.nexusmods.com/index.php?/topic/1254328-upk-file-format/
http://forums.nexusmods.com/index.php?/topic/1253996-upk-utils/

 

 

 

@ wghost81,

 

I had checked the upk structure you posted in that thread, but I haven't had the chance to play with UPK Utils yet. Definitely in my to-do list. I hope it'll help me to figure out what everything is once the explorer reads data across saves consistently.

 

 

------------------------------------------

 

 

 

 

Saves are protected by CRC checksum, that must be updated when repacking a file. I was not able to calculate it right.

 

For those who still have a problem with gamesave for PC or iOS version, let me explain few details. Data save is just serialized sets of classes (nothing more to add as people are doing great with finding & replaces patterns). In PC and iOS version (don’t have PS3 save) save game is compress with different algorithm. PC is using LZO1X_1 when iOS compresses data by zlib. Both saves are protected by CRC checksum. However, PC version has additional one CRC. One calculated from compressed data and 2nd generated from header.

 

Save file crc check is at 0xBE offset and it is the same customized CRC32b as from Windows version. I don't see difference between PC and iOS version, the savefile structure is exactly the same.

 

To fix it, you need to guys calculate CRC from whole compress data which is from offset 1024 to EOF() and put it just after game language code. In example, it’s at 0xEA. To calculate correct position it’s required to go though save header variable. Sum of first 3 in red (it’s just sizeof string), the 4th one is for DLC if you don’t have it will be 0 anything else it’s string size. And finally add 8 to results for sum and you are in correct place for 1st CRC.

2nd CRC has static offset so it’s not needed to do similar operation as for 1st one. The offset is 0x3FC. 4 bytes earlier, at 0x3F8 is size of block from beginning of save which will be used for generate checksum.

 

 

 

 

@Drakous79,

 

Oh man... those CRCs are the epic find of the year. Remind me to thank you again whenever I get to the repacking part of the project.

 

The rest of the structure seems to confirm my own findings.

 

 

-----------------------------------------------------

 

 

 

Wiki article 'Savegame file format - XCOM:EU 2012' has been created from this thread. Note there are tools written for the XBox on the 360Haven site that can be used on the PC, which are linked in the article. Free registration with 360Haven IS required to download them.

 

-Dubious-

 

 

 

@dubiousintent

 

Thanks for the link. To be honest I'm pretty green in python, not really being a programmer myself. But it could be handy to have that code around.

 

 

-------------------------------------------------------

 

 

 

On the upk side, for those that weren't already aware, it appears that the various structures that begin with "CheckpointRecord" are what controls goes into the save file.

 

For the ammo mod I had to alter which variable in XGWeapon were saved/loaded to/from the savefile. For various reasons I had to use a new variable to store the ammo cost -- m_iTurnFired. After some experimentation I realized that this was not being preserved when quitting and reloading within a tactical mission. The vanilla CheckpointRecord for XGWeapon was :

struct CheckpointRecord_XGWeapon extends CheckpointRecord_XGInventoryItem
{
    var int iAmmo;
    var int iOverheatChance;
};

I altered the struct hex so that it is now :

struct CheckpointRecord_XGWeapon extends CheckpointRecord_XGInventoryItem
{
    var int iAmmo;
    var int m_iTurnFired;
};

for my mod, and m_iTurnFired is now saved (allowing quitting and restarting during a tactical mission without weapons all automatically reloading.

 

Presumably these structures are being serialized into the savegame file -- see wghost's articles/documents to get a handle on serialization. The easiest way to find all of the Checkpoint Records is via a search for "struct CheckpointRecord" in UE Explorer.

 

As an example, XGStrategySoldier has :

struct CheckpointRecord
{
    var TCharacter m_kChar;
    var TSoldier m_kSoldier;
    var int m_aStatModifiers[ECharacterStat];
    var XGStrategySoldier.ESoldierStatus m_eStatus;
    var int m_iHQLocation;
    var int m_iEnergy;
    var int m_iTurnsOut;
    var int m_iNumMissions;
    var string m_strKIAReport;
    var string m_strKIADate;
    var string m_strCauseOfDeath;
    var bool m_bPsiTested;
    var bool bForcePsiGift;
    var bool m_bMIA;
    var bool m_bAllIn;
    var TInventory m_kBackedUpLoadout;
    var XGCustomizeUI.EEasterEggCharacter m_eEasterEggChar;
    var array<XComGame.XGTacticalGameCoreNativeBase.EPerkType> m_arrRandomPerks;
    var int m_arrMedals[EMedalType];
    var bool m_bBlueShirt;
};

I've omitted the defaultProperties for the struct.

 

 

 

This will help to figure out what is what in the save file. The structs look like on of the many property list entries, but I couldn't find one that exactly matched what you posted. I'm gonna try to focus on the property reader now, see if things clear up a little when we can see those properties.

 

Too bad about the native functions, now that would've sped things up :tongue:

Edited January 30, 2014 by FogGene

[Drakous79]

EW 11/22/2013 - 1:51 - Game 1 - Operation Crimson Future (EXALT Base Raid) - Brisbane, Australia - 4.03 am (Dropbox link)

EW 12/11/2013 - 23:02 - Game 23 - Operation Avenger (Temple Ship Assault) - 7:26 pm (Dropbox link)

 

No screenshots in spoilers on me3explorer forum.

Edited January 30, 2014 by Drakous79

[wghost81]

FogGene, from what I can see on your screenshots, saves structure is similar to that of the map packages, which is expectable. So you may want to check this thread as well:

http://forums.nexusmods.com/index.php?/topic/1104408-r-d-xcom-map-alterations/

You probably could read almost all the default properties with DeserializeAll tool from UPK Tools.

[FogGene]

Well, fumbling around with those saves was very instructive. Progress is slow yet, but I'll post what I've got so far.

 

This is the save structure I'm using:

 

 

 

 

NAME_TABLE

PROPERTY_LIST

PROGRESS_DATA

NAME_TABLE

UNIT_LIST

INDEX_TABLES

PROPERTY_LIST

NAME_TABLE

COMPRESSED_GAME

 

 

 

This save structure I have seems for the most part correct. Except for a couple of wrong assumptions I had made regarding signatures, the saves load fine. However, because of this I have hit a major roadblock while reading the saves:

 

- The problem is at the index tables. These are a group of int32 tables with their values ordered following an arithmetic progression. After each table there come unreadable structs of data of undetermined format (some are easy to guess, others nigh impossible). Each of these undefined structs seems to have an int32 index itself, the first 4 bytes. At least it was so for those that I could guess their structure. There are as many structs as indexes in the table, and as you probably have guessed the indexes match. Although remember they follow a specific arithmetic progression. Below there is an example of one of this tables (one entry).

 

 

http://i.imgur.com/8r3boA3.png

 

 

The problem is I can't for the life of me figure out these structs. I can skip this stuff by searching for signatures, but it would be a very unreliable way of parsing a save. Also, I think we wanna know what's in there. My guess is that something is reading the tables and the structs, and that something likely has the key to their format.

 

 

-----------------------------------------

 

As a sort of better news, I can now read each object's properties and store them away in structs. I think I have figured out pretty much the properties format of all the objects I could find. I'm having a fight with the GUI in trying to figure out how to display them on the PropertyGrid, once I do they'll be editable as well. Screenshot below (sorry for not displaying the data, like I said me and GUI....).

 

Edit: This is your save36 btw, Drakous

 

 

 

http://i.imgur.com/weBAhnv.png

 

 

 

 

------------------------------------------

 

wghost81, thanks for those links. I'll check them out asap.

Edited February 3, 2014 by FogGene

[wghost81]

The structure of unknown index table is similar to that of regular packages. As far as I can tell, no one has figured it out yet. I don't know about saves, but regular packages have header size and serial data offset in their summary info, so that unknown index table can be skipped easily.