kanet32

Ahh, I see; sorry, I misinterpreted you there.

Uhh, I don't know if you missed what I said, but, for this, modding tools aren't required. I already did it and posted the modded files earlier in the thread. (The links are around the middle of the post.) Or, did you see that and it didn't work for you?

If it helps, I figured out why my altered executable wasn't working for some people. Turns out, the pre-order version of the game has a different executable than the post-order version. Something to do with the pre-order bonuses, presumably. So, unfortunately, if you didn't have a pre-order version of the game, the executable I uploaded wouldn't've worked for you anyway. XSHAPE really should still work for everyone, though, so as long as you're willing to trust XSHAPE (which is open-source) you should be safe to use my XComGame.upk. Just in case people are actually using my modded files, I'll post updated ones when the next patch comes. I have a friend who has a non-pre-order version of the game, so I can probably get both versions of the executable so it'll work for (hopefully) everyone. Everyone playing it on Steam, anyway.

Ahh, yeah, the fact that there was a 1 in the ASCII was a coincidence. Any constant values would be stored as integer values (as in, two's complement binary-encoded integer values) or, in the case of 0s and 1s, as the special 0 and 1 bytecode. In your case, you'd be looking for a 26 in the hex somewhere. Like I said, though, changing a 0 or 1 to any other value is currently impossible, because it requires expanding the size of the file by one extra byte (the "IntConst" byte). We need to figure out how to update the file lengths in the UPK before we can do anything like that.

The reason I ask is because I think now that the UPKs aren't simple file archives. They must include metadata about the functions and classes stored in them. Hence why I'm assuming figuring out the UPK format will help us figure out how to change the stored object lengths. (Although, honestly, even if all they DID store were files, I would assume changing the lengths of those files would affect how the archive was parsed. I'm assuming the lengths of files in most archive formats isn't implicit.)

Good info... When you change the length of a function, when does it fail? When the game launches? Is it a catastrophic failure, or is it something more subtle? That information comes from Solver, in the last page of the analysing the game scripts thread. I think it was a crash when the game launched, but Solver didn't say exactly.

Ahh, yeah, definitely the float thing. Just double-checked. Yeah, for converting integer values to hex, use Google. A constant will only be stored as a float value if it's got a fractional component to it, otherwise it'll be stored as an integer. And, what do you mean, changing it from 1 to 4? You mean, in the decompiled source that UE Explorer shows?

Well, first of all, I'm not sure how you got 1 = 0x3f800000... 1 is 0x1. I'm honestly a bit bewildered about this. Second, unfortunately, there's special bytecodes for constant 0 and 1 in UnrealScript. You're looking for 0x25 for 0, 0x26 for 1. So you'd need to search for 0x26. The reason this is a problem is because if you want to change it to 2, you're going to need to change it from 26 to 2C 02, because any constant other than 0 or 1 has to be preceeded by 2C (or a different code, for different types). And as we just learned a few minutes ago, you can't change the lengths of functions. Changing 26 to 2C 02 would add one byte to the function's length, causing the altered script to be rejected. This specific change is going to have to wait until we know how to alter the object lengths in the UPK files.

So, have you just been editing the UPKs in a hex editor, or using a repacking tool? I guess we need (in the short term) to figure out what the UPK format is, so we can change the function sizes. Maybe Gildor's tools are open source, or the format is documented somewhere.

Yeah, I missed the bit where you said they were just accessor methods. (And even still, it was just a wild guess on my part.) Sorry, I've no idea here. If I had some time, I'd try reading through them following the logic from that UnrealScript decompiler I posted a while back (BytecodeReader.cs, which I assume is what you're looking at to find the instruction codes) and see if I could figure out what was happening there. To be honest, at this point I'd probably recommend not trying to make such large changes to the compiled scripts. My guess is pretty soon someone with knowledge of the UDK will come in and give us instructions on how to set it up to decompile and recompile UnrealScript. There's a lot of room to manoeuvre just in changing constants or operators around for the time being. (Though, if you do continue forging ahead learning the bytecode, by all means continue letting us know what you find.)

Well, I can tell you that most of the times I've seen any bytes over 0x60 that weren't constants, they were function calls. The fact that a few bytes later you have 0x16 (EndFunctionParams) seems to support that. So, I'd wager that the C3 04 and BE 04 are both function calls. Since they're little-endian, it's actually 0x04C3 and 0x04BE, which would be 1219 and 1214, respectively. Depending on which UPK you're looking at, those could be different functions. In XComGame.upk, they're... From ImportTable.txt: 1219 = Function'Engine.ResetDesiredRotation' 1214 = Function'Engine.PostBeginPlay' Or possibly, from ExportTable.txt: 1214 = ObjectProperty'UIScreen.m_kTacMgr' 1219 = IntProperty'UIScreen.GetCurrentView.ReturnValue' But those are both properties, not function calls, and the 0x16 really seems to imply function calls. Alternatively, if you're looking in XComStrategyGame.upk... From ExportTable.txt: 1214 = Function'UIBriefing.SetupIntroBink' 1219 = ObjectProperty'UIBriefing.OnMovieLoaded.LoadedArchetype' The first one is a function; if the second block of code in your screenshot is your modified code, maybe the problem is that your changed code is calling an ObjectProperty as though it were a function? Maybe? I haven't really dug that deep into the bytecode, so this is all guesswork. EDIT: Just re-read what you wrote, and you said both functions are from the game and just return a value, so now I have no idea why there'd be a function call in there. Maybe it's something like C# properties, where accessing a property actually calls a function? I'm used to C++, myself, where everything is much more ordered and reliable. Variables are variables and functions are functions.

This may be a silly question, but have you tried seeing what UE Explorer decompiles it to? Or does UE Explorer choke on it as well? I'm assuming you've already tried this, but since you haven't mentioned it, I figure it's possible you didn't even know about UE Explorer and have been reading through all the scripts in bytecode all this time. (If so, respect. That's some Story of Mel s*** there.)

Yeah, that's the expected result, but I believe Solver is saying that it actually DOESN'T work like that when you're calling a function. Even if execution will never reach a function call, the interpreter will check that call in advance to make sure its parameters are correct. So while execution never reaches that line, it will still cause a crash because it's being checked for correctness before execution even begins.

In this post, DaemonJax basically goes over the whole process in broad strokes. If you missed it, it might help. And yeah, UnrealScripts don't seem to be compiled so much as they're compressed. The "compiler" just seems to tokenise everything and replace the tokens with indices into various tables. There's one for native operators and library functions, one for variable and class names, one for exported functions, and one for imported functions. The most complex thing it seems to do is generate jumps for conditionals (but not for the ternary operator :?, which is actually a single three-"operand" instruction)

Actually, I don't have the job; Dreadylein's already taken care of it! Dreadylein's also worked out some more detail on the NTL format, so you might want to wait a bit before trying to write that parser.

Ah, good point. Tonight, if nobody else has done it, I'll get in there with dotPeek and take a look. The source code for that UnrealScript decompiler is probably also worth checking.

Like I said, they're in the UE Explorer directory. If you have UE Explorer installed, it's: Program Files (x86)\Eliot\UE Explorer\Native Tables\NativeTablesList_UT3.NTL or, probably a better choice: Program Files (x86)\Eliot\UE Explorer\Native Tables\NativeTablesList_UDK-2011-08.NTL Open them up in a hex editor, and you'll see a bunch of stuff like: 3D 3D 18 02 9A ==..š The 3D 3D is the operator (in ASCII). The 18 02 I'm not sure about, but there's always a couple characters after each operator or native function. I think they have to do with what type and number of parameters are allowed to follow it? And then the 9A is the bytecode. == (and a few other operators) appear twice, once at the very top, and once a bit further down. I'm not sure why, but I bet it has to do with those middle two bytes that I don't understand yet. So when we're compiling a list of these we should include both possible bytecodes. (9A and F2 in the case of ==.) EDIT: Aaaand Lein beats me to the punch again ;). Good to know about the SDK, I changed the filename in this post to reflect that.

I'd do it myself, but I'm a bit busy today. It would be fairly easy to just add the stuff in the Native Tables to that list of instructions you posted above (it's the UE3 or UDK tables we should be using, they're the most recent). The list of imports, exports, and names from the package, however, is also fairly important and those files are massive. Writing a simpler disassembler than UE Explorer is also probably doable, one that simply puts one instruction per line and replaces them with the instruction name, like an assembly language. Then you could reverse the progress to turn a modified pseudo-assembly back into bytecode. As near as I can tell, the UnrealScript compiler doesn't seem to actually do anything to the code. It's not an optimising compiler. So what UE Explorer shows is literally what's in the bytecode, just with the semantic sugar peeled off and the all the variable and function names replaced with indices into a table. I'm not sure this is worth the effort, though. It might be better to just keep doing what we're doing now until someone with better knowledge of the UDK figures out how to get the UnrealScript compiler to work with the XCOM UPKs.

EDIT: Haha, the server carked out on me, and by the time my post went through you'd already found all this. I'll leave this post up in case you didn't know about the Native Tables, since I didn't see you talking about that. Just took a look at the file; there's two 0x07s in there, which are JumpIfNotEqual. One of them needs to be changed to 0x06, which is Jump (unconditionally). I'm assuming it's the second one, since that if statement we're looking at comes second. Also, I missed your earlier post. 0x9A is ==, 0x9B is != (but, I found other bytecodes for those operators, F1 and F2, if I remember correctly). So just changing a 9A to a 9B will change an == to a !=. I mentioned this in another post, but it bears repeating: this source file from an UnrealScript disassembler has a ton of useful information on the bytecode, including many of the native bytecode instructions. For other operators and functions, you'll want to look in the Native Tables folder in the UE Explorer directory, and the ExportTable.txt, ImportTable.txt, and NameTable.txt extracted from the UPKs using Gildor's tools. (The Native Tables files are a bit confusing, but if you open them in a hex editor it only takes a little time to figure them out. It goes <operator/function> <some garbage> <bytecode>. Knowing that == is 9A and != is 9B should help you figure it out.)

Whoops, doublepost.

That's odd, I just opened an old .class file from ages ago and it doesn't look anything like that. It has what seems to be a string table at the beginning, and the rest looks very much like compiled UnrealScript. Maybe we're talking about different things here? I thought the only two states for Java were source and compiled bytecode (ie., "javac Blar.java") . Is there some middle state between those two I didn't know about? Maybe compiled applets are different than compiled console applications? Back on topic a bit, has anyone figured out where the game saving happens? I glanced through most of the main UPKs but couldn't pin down the exact spot it all happens. I have a sinking suspicion it's just dumping all those Checkpoint instances, which look like they're going to be a nightmare to parse. I looked at the save files themselves and it's bizarre--strings seem to be fractured and incomplete and sometimes you see bits and pieces of a variable name. Like, where you'd expect to see the value of bPsiGift, you see "bPsiG." Identifying (by inspection) an actual value in all that mess is impossible. EDIT: OH! I see what you're talking about. You're talking about the disassembled code. Yeah, that makes more sense. What we see in UE Explorer is the disassembled code for UnrealScript, equivalent to what you get from javap. I was talking about the compiled UnrealScript we see in the actual UPK, and the compiled Java code you get from javac. Both are pretty similar. I wouldn't mind having a tool for UnrealScript like the JBE or Visual Studio's disassembly view, where it views it as an assembly language. THAT would be more convenient.

I don't have any experience with Java bytecode. Can you actually look at it in ASCII and see what's going on? I just assumed all compiled languages had to be read in hex (or run through a disassembler of some kind). In hex, I find it mostly makes sense, but there's some stuff that's presumably meant to improve performance that's just frustrating. Like the fact that there's a separate bytecode for constant 0 and 1. Or the way conditionals are an actual, literal thing, rather than semantic sugar that the compiler just turns into CMPxx and JMP instructions. And yeah, little-endian sucks. I'm sure there's some reason why it's better in this case, but I can't imagine what it was. I guess they had to write their own interpreter by hand to read the compiled script language, and if those design choices made it easier for them to write the interpreter, I suppose I shouldn't begrudge them for it. Honestly, I think it makes sense to me because it kind of works the way I was planning something similar for a game engine I'm working on. You basically parse all the scripts that are going into the package, get every unique token, put them in a table, and then replace the text tokens with their table index. So, "CompleteAction()" in your example gets turned into 0x0122 (or, maybe 0x2201, if it's little-endian? I don't understand endianness as well as I really should). "simulated" and "state" also probably have a specific bytecode associated with them, but since they're native to the language it's not stored in any of the tables in the UPK. This way, you can retain all your semantic information, so you can decompile scripts perfectly, but it's still fairly efficient to read using (I'm assuming) some kind of FSM. So yeah, basically, I don't think it's obfuscation, I just think that it was designed primarily for the interpreter, not for hand-editing. Though, personally, I would have also allowed it to load uncompiled scripts, and compile them manually during startup, like HLSL. Maybe you can, and it's just a feature that's disabled in release builds.

Ahh, right. I'm a singleplayer person myself, so I hadn't thought of that. Presumably, they use some kind of anti-cheat techniques... although, now I think of it, I didn't see any Punkbuster install or anything. Hmph, now I'm wondering how they expect that not to turn into a massive, unregulated anarchy... Maybe it's that phone home thing the game does on launch?

Eh, I'm of the mind that if we can already do it with a minimum amount of hex editing, it'd be ridiculous of them to handwring over the possibilities of releasing an editor. Bethesda sure doesn't worry about the admittedly awful things some people do with their modding tools. And Valve practically release the full source of their games. I'm curious what you think they'd be worried about, though. Piracy, I'm assuming? I'm afraid that ship has sailed. And it's captained by a surly, bearded fellow with an eyepatch and a parrot. And, no, pointing out the copyright issue isn't annoying. It honestly hadn't occurred to me. Someone has to read the terms of service so the rest of us don't have to ;). (Not that I didn't read the terms of service, of course. Because I did. Eventually.)

Oh, really? Hmmm. I'm not particularly worried about copyright--2K'd have to be pretty churlish to go after that--but I'd rather not get banned from the forum. I still need to hang around so I can grab the save game editor when some brave soul finally makes it. Guess I'll have to check the forum terms, see if there's a way around that. Guess that makes the need for an autopatcher for the UPKs even more important. EDIT: Yeah, DaemonJax had the same thought. I wonder if there's any practical way to get in contact with Firaxis and have them clarify the rules for redistributing content. I DO recall them saying something about supporting modding. If that wasn't a brazen lie, surely they'd say it's fine.