Unicode in int files

[FMod]

A short while ago I promised to give the modding community inline language patches. So that you can make a mod - remember, you don't have to replace the whole gamecore.ini - that goes like this:

 

;Builder patch
Weapons=(iType=eItem_FragGrenade,ABILITIES[0]=eAbility_FragGrenade, ... iDamage=4,iRadius=480
XComGame.int: [eItem_FragGrenade]=The Frag Grenade's blast radius is 4 tiles\n Does 4 damage

That would patch both the exe and the description in int files.

XComStrategyGame.int in user folder and other text files can also be patched this way, you just name the file.

 

But turns out there is an issue: XComGame.int is in Unicode, while the ini is in ASCII. Unicode and ASCII can't be together in one file.

 

Now, there are some possible solutions. One would be externalizing all int tweaks and calling a file, like this:

Weapons=(iType=eItem_FragGrenade...
XComGame.int: "diffs\grenade.damage.diff"

It will take some work, and the external tweak will almost certainly have to be in proper diff format, not "new lines only" like regular Builder patches. But it will preserve the unicode.

 

 

Another solution is simply using a non-unicode XComGame.int instead. I've tested it and it works.

The only place unicode characters are actually used in English int files is in credits for localization teams, and these credits will still read, just as Josef Pawlowski instead of Józef Pawłowski. As far as sacrifices go, I think we can all live with that.

Mod-wise, it will prevent making a mod putting non-English characters in nicknames, I think that's about it. You'll still be able to make such mods, of course, Builder just won't be able to work with them.

 

Actual soldier names are in a separate file - curiously, it's not even in Unicode. I figured that if the game can eat ASCII instead of Unicode, the reverse might work too - and made a replacement unicode file, filled it with non-English names, and indeed it works just fine. So you can have soldiers' names in their native language. No katakana, but still, I thus consider my debt to multilingualism paid in full ;-)

 

And, since I don't use brute force replacement installation, the non-Unicode file will only be used if you turn the "Game text patching" option on in Builder, and maybe even only once a mod actually requests an int patch. I won't be shipping a premade file, either: your existing int files will be converted on the fly, so, if the game updates them or if you have your own tweaks already in, the changes won't be lost.

 

 

Now that I've made my case, I'm asking the modding community as to whether you consider the latter solution acceptable.

Or maybe you have a third solution that gives the best of both worlds.

[dreadylein]

Voted for the last option :P

 

Dont see major drawbacks in any of the ways so its more or less up to you.

Staying Unicode would be the proper way of course, but if it involes too much work and there is no downside, why not :)

[FMod]

Well, on my end, it's certainly easier for me to de-unicode any file I need to modify.

It's also prettier - everything that doesn't require UPK mods is going to be just one file. I mean everything, you'll be able to tweak any file it makes sense to tweak from within your patch, just put a "File.To.Tweak: content" line in.

 

On the other hand, switching away from unicode - well, no one would probably even notice if I just slipped it in. But I want to check by the community, maybe there are some issues I'm missing, and at the very least I'd rather be open about it.

 

UPK mods will still have to be supplied separately. Technically I could make them inline as well, but, since they are made with another program rather than manually, there is no advantage to it. Although I might simplify their installation somewhat.

[fys]

Better do with unicode as the game demands. More work but less trouble and complains later.

[FMod]

So far I haven't seen any signs that the game prefers either way. That it lets you replace ASCII files with Unicode ones and vice versa indicates that it processes them via WinAPI rather than directly.

 

If anyone wants to test it out more thoroughly (I don't play the game myself these days), here's how to convert your game's int file to non-Unicode:

cd "C:\Program Files\Steam\steamapps\common\XCom-Enemy-Unknown\xcomgame\localization\int"
ren xcomgame.int xcomgame.unicode
type xcomgame.unicode>xcomgame.int

Open command prompt (or FAR or Total Commander - any power user should have one of these), copy-paste and execute these three lines in order.

(I assume anyone who has their game in another folder can change the paths on their own).

 

It's that simple, conversion is built into windows. More precisely, doing any text operation on unicode without a special /u flag turns it into normal ASCII.

 

If against all odds something breaks and you need to revert, use:

cd "C:\Program Files\Steam\steamapps\common\XCom-Enemy-Unknown\xcomgame\localization\int"
copy xcomgame.unicode xcomgame.int

[BlackAlpha]

Any ideas what this will do to non-English localizations?

[FMod]

With about 97% certainty nothing. I'm going to try and verify it positively. If engine localizations are any indication, any local versions don't use the INT folders, but their own GER/Japanese/RUS/POL etc folders, with appropriate filenames.

However I know that some localizations on UE1 used to just replace the int files. But it was long ago and they were hacks not made by the original developer. That's where the 3% comes from.

 

I'm going to include it by default in the next release, so that I can see some reports if there are bugs for someone lese. Maybe you could also run a test on your user base - include selectable int file options (e.g. none, unicode, new experimental) with the next version, see if anyone has an issue. The probability of problems appearing is very small, so if there are any, they are unlikely to show without a wide rollout.

[BlackAlpha]

With about 97% certainty nothing. I'm going to try and verify it positively. If engine localizations are any indication, any local versions don't use the INT folders, but their own GER/Japanese/RUS/POL etc folders, with appropriate filenames.

However I know that some localizations on UE1 used to just replace the int files. But it was long ago and they were hacks not made by the original developer. That's where the 3% comes from.

 

I'm going to include it by default in the next release, so that I can see some reports if there are bugs for someone lese. Maybe you could also run a test on your user base - include selectable int file options (e.g. none, unicode, new experimental) with the next version, see if anyone has an issue. The probability of problems appearing is very small, so if there are any, they are unlikely to show without a wide rollout.

 

Well, so far I only support the English version. And you've already tested that. I think that if there's an area where it could go wrong, it's the non-English versions. So I can't help you for now.

 

I'm thinking about how I could make the mod compatible with other languages.

[FMod]

If there are non-English versions that use INT files (I doubt there are), the consequences of using a unicode and a non-unicode replacement int will be equal. UE3 is multilingual from the beginning, foreign versions, at least for European markets, have no need for a special obscure routine to handle their text.

 

Maybe if there are or will be Asian languages versions, they might do something differently, because the INT version can't handle their characters, but that's something we'll either know or not know. Again the consequences are bad either way, you don't want to put English right in the middle of another language.

 

I'm somewhat concerned if there may be any glitches in the English version, maybe if the developers used a very dirty hack somewhere. It's unlikely, I'd much sooner expect that they had everything in ASCII for the first 80% of the development and only touched unicode when time came to make translations; that would explain why some files that should logically be in unicode aren't.

My play-testing pretty much considered of checking an old save and starting a new game to confirming that all the text looks right.

[FMod]

Well, anyway.

The conversion operation is now automated in Builder. It's under Options, "Enable game text patches". There are automatic backups, reverting, mod uninstaller, etc to prevent any irreversible changes.

 

The latest build includes a couple sample mods with inline INT edits. You can try them out, see how it goes.

One is a combo patch to show syntax, another makes all your tanks called HWP or Heavy Weapon Platforms, like in the original game (and it's an order of magnitude less childish too).

 

That patch also has modifies XComStrategyGame.int. Since that file is properly in the user folder, which isn't as straightforward to access, it might not work for you, so please tell me if does or doesn't.