Finding the first step?

[Lethargus]

There are two methods of editing XCom going on in this thread.

 

1) Resource Editor to extract the ini files, edit them, then repack them back into the .exe. This is where you're seeing things where people are changing the crit/aim bonus of difficulties, adding item slots to armor, etc. A surprising amount can be edited this way.

 

2) Gildor's tools to decompress the upks themselves and then UE Editor to decompile the Unrealscript. This is what you see in the psionics test stuff listed and is the primary game logic. It *might* be possible to decompress/extract a upk, UE edit the Unrealscript, then recook it back into a upk the game is willing to use, but this is not something I've really looked into. Right now I'm just curious to see how the game decides to do certain things, not actually change them. If it turns out you can get the game to use a re-cooked upk then the sky is the limit for what can be done.

 

Ideally I'd love to see a mix of the both, expand on the mod that balances the aim/crit cheats and lets you intercept the abduction mission UFOs while also not giving you a panic penalty for missed UFOs you couldn't detect. Something along those lines.

[tbkiah]

I've decompressed and extracted several files from the UPK files. Does anybody know the particular files that aren't funtions of elements in the game, and can actually be manipulated? I'm not talking about getting them to work in the game, i'm talking about finding the files that go beyond telling something in the game to do something when a particular action is taken.

 

Maybe i'm just a moron.

[Anbar]

Ideally I'd love to see a mix of the both, expand on the mod that balances the aim/crit cheats and lets you intercept the abduction mission UFOs while also not giving you a panic penalty for missed UFOs you couldn't detect. Something along those lines.

 

 

you dont get a panic penalty for missed UFOs that you dont detect.

[Lethargus]

Ideally I'd love to see a mix of the both, expand on the mod that balances the aim/crit cheats and lets you intercept the abduction mission UFOs while also not giving you a panic penalty for missed UFOs you couldn't detect. Something along those lines.

 

 

you dont get a panic penalty for missed UFOs that you dont detect.

 

I can watch panic increase on a country that literally nothing has happened in game that I could see, so if it's not a missed UFO, I'm not sure what it is. Either way I was happy to find those values actually buried in the ini file as well. There's not much you can't do by only editing the ini files it seems, short of creating an entirely new ability/item/etc.

[dreadylein]

I've decompressed and extracted several files from the UPK files. Does anybody know the particular files that aren't funtions of elements in the game, and can actually be manipulated? I'm not talking about getting them to work in the game, i'm talking about finding the files that go beyond telling something in the game to do something when a particular action is taken.

 

Maybe i'm just a moron.

 

Dont understand what exaclty you are searching :o Could you be a bit more specific ?

[tbkiah]

 

Dont understand what exaclty you are searching :o Could you be a bit more specific ?

 

The new thread explained it, i was looking for the game scripts and "hardcoded" game constants.. and i found what i was looking for... unfortunately no one has figured out how to recook the files in order to put them back in the game..

[mgreed]

Ok, BAD NEWS on modifying the UPKs:

 

[0005.22] Critical: appError called: SHA Verification failed for '..\..\XComGame\CookedPCConsole\XComStrategyGame.upk'. Reason: Bad hash

[0005.22] Critical: Windows GetLastError: The operation completed successfully. (0)

[0005.22] Init: Finished loading startup packages in 4.66 seconds

[0005.24] Log: 107636 objects as part of root set at end of initial load.

[0005.24] Log: 0 out of 0 bytes used by permanent object pool.

[0005.25] Log: Enum Cache created: 176728 bytes

[0005.25] Log: Initializing Engine...

[0010.49] Log: === Critical error: ===

SHA Verification failed for '..\..\XComGame\CookedPCConsole\XComStrategyGame.upk'. Reason: Bad hash

 

It checks for a SHA-1 hash match for a huge list of files. You can see the list and the hashes expected using resource editor. Its the 1030 in 1010.

 

The hashes there don't match what I get when I generate hashes for the original, unedited files. I don't know if they are seeded or encrypted or... whatever. I really don't know much about that sort of thing. My best guess would be that someone would have to patch the hash checks out of the main exe.

 

Oh, and if you want to verify, you don't have to modify the UPK. Just use a hex editor to change a byte of the hash listed for one of the files in the list. I tried with XcomStrategyGame.upk. then check the logs under your /USERNAME/My Games/blahblahblah directory

 

Edit:

Sad my first post is bad news :(

Edited October 18, 2012 by mgreed

[Daemonjax]

 

It checks for a SHA-1 hash match for a huge list of files. You can see the list and the hashes expected using resource editor. Its the 1030 in 1010.

 

The hashes there don't match what I get when I generate hashes for the original, unedited files. I don't know if they are seeded or encrypted or... whatever. I really don't know much about that sort of thing. My best guess would be that someone would have to patch the hash checks out of the main exe.

 

It's possible that it's SHA-0, not SHA-1. It's doesn't say SHA-1 in the output... it says SHA.

 

I think it's unlikely that they used a salted or nonced hash value to protect the data... That'd mean they were REALLY serious about people not f***ing around with their files. ;) I think the hash check is there to guard against loading corrupted files, not for security reasons.

 

Or... the UPK's could be compressed? I read somewhere that UPKs can exist in either compressed or uncompressed states. Maybe it's checking the hash value after uncompression? (Yeps, this was it!)

 

Or... Our method of extraction changes the data in the UPK slightly. Even a 1 bit difference would totally change the hash value. (nevermind this one)

 

Or... the output is a lie, and it's not SHA. It takes a while to generate the SHA hash value of a file, unless that file is very small. There's 2000+ upk files, and I highly doubt the game is generating SHA hash values for all of them in the time it takes the game to start (74 of them are over 10 megs). I could be wrong. I could write a program to test it. It's not an insignificant amount of time, though.

 

Or... Perhaps... What program are you using to generate the SHA-1 hash? The value they're looking for may or may not be the retrieved hash after it's been converted to hex format.

 

EDIT: Writing the program to test the time to generate SHA-1 hashes of all the upk files... I'll be back shortly :D

 

import java.io.File;
import java.io.FileInputStream;
import java.security.MessageDigest;


public class UPKHashCheck 
{
   public static void main(String[] args)throws Exception
   {        
       File[] listOfFiles = new File("D:\\Program Files (x86)\\XCom-Enemy-Unknown\\XComGame\\CookedPCConsole").listFiles(); 
       MessageDigest md = MessageDigest.getInstance("SHA");
       int counter = 0;
       int j = 0;

       for (int i = listOfFiles.length - 1; i >= 0; --i) 
       {
           if (listOfFiles[i].isFile()) 
           {
               
               String file = listOfFiles[i].getName();
               if (file.endsWith(".upk") || file.endsWith(".UPK"))
               {                    
                   FileInputStream fis = new FileInputStream(listOfFiles[i]);
                   byte[] dataBytes = new byte[1024]; 
                   int nread = 0; 
                   
                   while ((nread = fis.read(dataBytes)) != -1) 
                   {
                       md.update(dataBytes, 0, nread);
                   }
                   
                   byte[] mdbytes = md.digest();
                   j += mdbytes[0];
                   ++counter;                
               }
           }
       }
       
       counter += (j / Integer.MAX_VALUE);
       System.out.println(counter);
   }
}

 

Output:

 

run:

2463

BUILD SUCCESSFUL (total time: 17 seconds)

 

 

 

17 seconds proves they are not checking SHA-1 hashes for all those files, so the only 3 options are:

 

a) They're only checking some UPK files, but not others.

b) It's not SHA-1

c) Java is ridiculously slow when it comes to generating file hashes

 

If A is true, then they did it purely for security reasons and not for checking file corruption. Which is sad, because then it may be salted or nonced (it's also paranoid of them, imo).

 

MD5 drops it to 12 seconds. Still too long for all the UPK files.

 

UPDATE: Mistakenly used SHA-256. Fixed. ;)

 

 

Can I see the expected hash for one of the files? Maybe I can figure it out. If they took the extra step to salt or nonce it, I think we're boned though. None of us are going to crack SHA anytime soon. ;)

 

EDIT: Lol... just noticed the resource hacker comment. I guess there is a file list. :D I should have read your post more carefully, heh. Ok, I'll check the expected hash and see what I see.

 

EDIT2: Ok, I'm making sense of it... still working on it, though.

 

Here's the expected hash of core.upk to save someone else the time:

 

47 65 3C 8D B1 23 15 9F 22 AC C9 04 9F E0 55 B6 7A 4E 9E A2

 

Now, I'll write another program...

 

import java.io.File;
import java.io.FileInputStream;
import java.security.MessageDigest;

public class UPKHashVerify
{
   public static void main(String[] args)throws Exception
   {                
       MessageDigest md = MessageDigest.getInstance("SHA");
       FileInputStream fis = new FileInputStream(new File("D:\\Program Files (x86)\\XCom-Enemy-Unknown\\XComGame\\CookedPCConsole\\core.upk"));
       byte[] dataBytes = new byte[1024]; 
       int nread = 0;            
       int[] expectedHash = {0x47, 0x65, 0x3C, 0x8D, 0xB1, 0x23, 0x15, 0x9F, 0x22, 0xAC, 0xC9, 0x04, 0x9F, 0xE0, 0x55, 0xB6, 0x7A, 0x4E, 0x9E, 0xA2};

       while ((nread = fis.read(dataBytes)) != -1) 
       {
           md.update(dataBytes, 0, nread);
       }
       fis.close();

       byte[] mdbytes = md.digest();               
       System.out.println(md.getAlgorithm());
       System.out.print("Expected: ");
       for (int i=0; i < expectedHash.length; ++i) 
       {
           String hexString = Integer.toHexString(expectedHash[i]).toUpperCase();
  	    if (hexString.length() == 1) hexString += "0";
  	    System.out.print(hexString + " ");
   	}
                     
       System.out.print("\nActual..: ");
       for (int i = 0; i < mdbytes.length; ++i) 
       {                             
           String hexString = Integer.toHexString((int)mdbytes[i] & 0xff).toUpperCase();           
  	    if (hexString.length() == 1) hexString += "0";
           System.out.print(hexString + " ");
       }
       
       System.out.println();      
   }
}

 

Output:

SHA
Expected: 47 65 3C 8D B1 23 15 9F 22 AC C9 40 9F E0 55 B6 7A 4E 9E A2 
Actual..: 10 F0 C0 CD 56 10 20 63 9B 51 E0 F5 48 6B 46 F5 A7 60 1A E7 

 

Yeah, not good... let's look at the bits...

 

 

UPDATE: Woops. I thought I figured it out, but I didn't. ;)

 

Well, I have another idea... there may be a header of size X which the engine skips when doing the hash check. If that's true, I'll write a program that will find it.

 

UPDATE: No, that's not it.

 

I think I know what's going on... there's a file in the folder that gives a clue: Core.upk.uncompressed_size

 

The contents of the file is simply: 223449

 

Since core.upk is 59k, then it must be compressed.

 

So, I believe the game is most likely taking the hash of the uncompressed file.

 

Using gildor's decompressor with the -lzo switch, I can decompress it to that filesize. (thanks, Gildor!)

 

Running my program again to verify... and yeps... that did it:

 

SHA
Expected: 47 65 3C 8D B1 23 15 9F 22 AC C9 40 9F E0 55 B6 7A 4E 9E A2 
Actual..: 47 65 3C 8D B1 23 15 9F 22 AC C9 40 9F E0 55 B6 7A 4E 9E A2

 

Solved, I guess. Only took me 5 hours of trying every wacky idea to find a solution that took 5 seconds to actually do. I'm such a noob! :D

 

Can we unpack/repack/recompress yet?

 

I think I found a way to recompress... The LZO compression the file uses is a good clue... give me a few minutes...

 

Yeah, this should work... LZO is an obscure algorithm, but someone made a .net library for it. Here it is:

 

http://lzo-net.sourceforge.net/

 

Someone else can do that part, because I don't feel like setting up a .NET programming environment on my PC.

 

UPDATE: I might do it anyways, since I found some source code in C here: http://www.oberhumer.com/opensource/lzo/#download

 

UPDATE: LZOP works right out of the box, so that saves us the trouble. http://www.lzop.org/

 

Testing lzop now...

Edited October 18, 2012 by Daemonjax

[dreadylein]

@daemonjax nice work ;)

As far as iam aware the upk has chunk infos which point to compressed code, so we would need to decompress, make the modding, recompress and maybe adjust every chunk info ? :o

[Daemonjax]

I don't know for sure.

 

I thought it would just decompress the file on the fly, doing whatever it needed to do. Having information tucked away in a compressed file that is usable before decompression sounds more complicated than it needs to be.

 

I've never tried modding anything for the Unreal engine before... it's very very unfriendly. It's like I'm back in 1994... let's start breaking out the disassemblers and hexeditors.

 

 

Anyways, LZOP does not compress it to the exact filesize. I wouldn't care, except that the game immediately crashes when trying to load with a recompressed file. \

 

Maybe that has something to do with what you're talking about. :D

 

Working on it...

 

UPDATE: Lzop does not recognize the original compressed file as being an lzop file. I'll have to whip out my hexeditor to see what's up. :/ Told ya.

 

UPDATE2: I think I can fix the files compressed with LZOP to match what the engine is expecting. Working on it... While I'm at it, how about someone explain what these chunk infos are?

 

UPDATE3: Close but no cigar. I can get it to crash a little later, that's all. By modifying the header... The game then complains in the log file about only finding like 100 classes out of 2000 in Core. Maybe it's not impossible, but I'm taking a break from looking at this for now. I'd rather just play the game, and this is starting to feel like work :P

Edited October 18, 2012 by Daemonjax