Sound replacement possible?

[wghost81]

Yes, you're using wrong notation. First, OBJECT key is unnecessary and second, modded file name should have Class.Object.Type format (see readme for more details). As I understand, you're trying to patch SoundNodeWave'SoundMissionControlMusic.MemorialScreen3' object, so your file should be named as "SoundMissionControlMusic.MemorialScreen3.SoundNodeWave". Modfile will look like:

UPK_FILE=HQSound_MemorialMusic_SF.upk
MODDED_FILE=SoundMissionControlMusic.MemorialScreen3.SoundNodeWave:AUTO

[wghost81]

BTW, you can see SoundNodeWave class inside Engine.upk. Serialized data consist of default properties and then most probably are followed by UntypedBulkData_Mirror objects, but that's just a guess.

 

HQSound_MemorialMusic_SF.upk package has 7 objects inside:

0x00000001 (1): ObjectReferencer'ObjectReferencer_320'

0x00000002 (2): Package'HQSound_MemorialMusic'

0x00000003 (3): Package'HQSound_MemorialMusic_SF'

0x00000004 (4): Package'SoundMissionControlMusic'

0x00000005 (5): SoundCue'HQSound_MemorialMusic.Memorial_Cue'

0x00000006 (6): SoundNodeLooping'HQSound_MemorialMusic.Memorial_Cue.SoundNodeLooping_0'

0x00000007 (7): SoundNodeWave'SoundMissionControlMusic.MemorialScreen3'

 

Package objects are just a package descriptions, ObjectReferencer contains two references to HQSound_MemorialMusic.Memorial_Cue and HQSound_MemorialMusic.Memorial_Cue.SoundNodeLooping_0. SoundMissionControlMusic.MemorialScreen3 contains a music, as you discovered.

[tracktwo]

Thanks! I renamed the file and it does update correctly. It still isn't a full solution because of the need to update the offsets once it's been inserted, and also even generating the .SoundNodeWave from a ogg is a little painful because of the need to set the name references for the default properties, which are specific to the particular UPK the sound lives in. So far I've been doing this by extracting the .SoundNodeWave target from the UPK using the export function of UE explorer and then overwriting the ogg data with my new sound.

 

Looking at the definition of SoundNodeWave in Engine.upk, I think you're right. But looking at the definition of SoundNodeWave, there are far more fields there than I see in the actual .UPK. The same is true for the definition of UntypedBulkData_Mirror, which is in Core.upk. I looked at the SoundNodeWave files from the sample project in the UDK, and it's slightly different: there are 7 of the UntypedBulkData_Mirror entries instead of 5. Looking at the differences between the two definitions in Engine.upk, my guess is that the 5 are:

 

RawData (not used)

CompressedPCData (this is where the ogg is for both Xcom and the sample UDK game)

CompressedXbox360Data (not used)

CompressedPS3Data (not used)

CompressedWiiUData (not used)

 

In the UDK, after the WiiU data entry it also has:

 

CompressedIPhoneData

CompressedFlashData

 

Which explains the two extra entries I'm seeing.

 

Each individual entry is only 16 bytes, though, far smaller than I would've thought the UntypedBulkData_Mirror object should be. I'm guessing the fields that are present are:

 

native const int SavedBulkDataFlags (?? maybe, it always seems to be 0 from what I've seen)

native const int SavedElementCount

native const int SavedBulkDataSizeOnDisk (This and the above are always the size of the ogg data)

native const Pointer BulkData (This is always the absolute offset of the start of the ogg data)

<variable number of bytes: all the ogg data. Since only the CompressedPCData has a non-zero size, its the only one with data here>

 

I am still unsure why the type definitions have far more variables listed than I am seeing in the file. Poking around through UE explorer some of them seem to make sense, like some of the SoundNodeWave entries being marked "editoronly" or "transient". Others just don't appear to be there when I guessed they should, like most of the other fields in UntypedBulkData_Mirror. For example, in Core.upk there looks to be a field "SavedBulkDataOffsetInFile" between the SavedElementCount and SavedBulkDataSizeOnDisk, but I always see the two "size" fields right next to each other in the upk.

 

 

 

[wghost81]

I am still unsure why the type definitions have far more variables listed than I am seeing in the file.

All the objects have both persistent and memory variables. Persistent variables get sezialized and memory variables are not.

native const Pointer BulkData (This is always the absolute offset of the start of the ogg data)

This really is an absolute file offset, not relative serialized data offset? On the other hand, UntypedBulkData_Mirror variables are the only data I know, which operate on absolute offsets, so yes, this must be it.

 

BTW, I've almost finished work on in-place object resize functionality for PatchUPK.

Edited September 22, 2014 by wghost81

[tracktwo]

All the objects have both persistent and memory variables. Persistent variables get sezialized and memory variables are not.

 

Is there any way to tell which are which? I couldn't find anything in UE explorer to determine which is which for the UntypedBulkData_Mirror type. They all look the same, but if this the real type used in the UPK only 4 of those fields can be present cause the whole object is only 16 bytes long.

 

BTW, I've almost finished work on in-place object resize functionality for PatchUPK.

 

Nice! I've hacked in support for a [REPLACEMENT_SOUND] tag to PatchUPK, the diffs are up in github https://github.com/tracktwo/UPKUtils. If you'd like to pull this into the main PatchUPK that'd be great, either as-is or as a starting point for whatever you'd like to do with it. It also has some misc other stuff to get it to build under Visual Studio that you can take or ignore. The important changes are all in ModScript and UPKUtils.

 

For example:

UPK_FILE=Weapon_LaserRifle_SF.upk
OBJECT=WP_AssaultRifleLaser.LaserRifleFire
REPLACEMENT_SOUND=tone.ogg:AUTO

Replaces the laser/pulse rifle (and carbine, and heavy rifle, but not sniper rifle) fire sound with whatever is in tone.ogg. I need the OBJECT key again because it can't derive the object from the filename.

[wghost81]

Is there any way to tell which are which?

No. You need cpp sources to tell this for sure.

I've hacked in support for a [REPLACEMENT_SOUND] tag to PatchUPK

There are no permanent offsets in object "header", it's better to use Deserialize for obtaining DefaultProperties. We better code in USoundNodeWave object as child class of UObject and handle export data serialization from there.

 

ModScript class received a major overhaul as its code was very ugly. I can update github sources right now for you to work with fresh files, but I'm not 100% sure I haven't broken anything. :smile:

Edited September 22, 2014 by wghost81

[tracktwo]

There are no permanent offsets in object "header", it's better to use Deserialize for obtaining DefaultProperties. We better code in USoundNodeWave object as child class of UObject and handle export data serialization from there.

Yeah I started down that road of creating a USoundNodeWave class (and probably UUntypedBulkData_Mirror) but I couldn't see any way to re-serialize the data to pass it off to WriteBinaryData. If you update the github sources I'll sync to it and try to add that.

 

I did replace my initial hardcoded constants with deserialization of the Default Properties - this was required because I did discover different sounds having different properties in the list. It still hardcodes offsets within the bulk data entries, though, so it's fragile. Proper deserialization of the UUntypedBulkData_Mirror would also handle the case that any of the "other" entries (e.g. what I believe are the PS3 or XBox entries) had actual data in them. I haven't seen this yet, though.

 

I think I have enough info now to take a stab at updating the wiki entry with what I believe is the object data structure, but I probably won't get to that for another day or so.

[wghost81]

tracktwo, making a fully functional serialization/deserialization code is a big task. For now I have only made serialization code for package header, as I needed it to add new variables.

 

I prefer to construct objects myself, using [MODDED_CODE] to write DefaultProperties and other things.

 

Here's an example of how you can code SoundNodeWave data using MODDED_CODE:

 

OBJECT=SoundMissionControlMusic.MemorialScreen3:AUTO
REL_OFFSET=4 // never rewrite PrevObjRef, it's an internal linker info!
[MODDED_CODE]
<Duration>           // var name
<FloatProperty>      // property type name
<%u 4>               // property size
<%u 0>               // array idx
<%f 58.742>          // property value
<NumChannels>
<IntProperty>
<%u 4>
<%u 0>
<%u 2>
<SampleRate>
<IntProperty>
<%u 4>
<%u 0>
<%u 44100>
<RawPCMDataSize>
<IntProperty>
<%u 4>
<%u 0>
<%u 10362096>
<None>           // end of list
// write all the obeject-specific serial data here
// you may use pseudo-code if needed to insert int/float values or object/name references
// copy/paste raw data hex block here

This won't shrink big objects, though. You can use BEFORE/AFTER blocks combination to do it. But, new patcher version has RESIZE key to set new object size explicitly. Example:

OBJECT=SoundMissionControlMusic.MemorialScreen3:INPL
RESIZE=12345 // new object size here (a size of entire object serial data)
REL_OFFSET=4 // never rewrite PrevObjRef, it's an internal linker info!
[MODDED_CODE]
<Duration>           // var name
<FloatProperty>      // property type name
<%u 4>               // property size
<%u 0>               // array idx
<%f 58.742>          // property value
<NumChannels>
<IntProperty>
<%u 4>
<%u 0>
<%u 2>
<SampleRate>
<IntProperty>
<%u 4>
<%u 0>
<%u 44100>
<RawPCMDataSize>
<IntProperty>
<%u 4>
<%u 0>
<%u 10362096>
<None>           // end of list
// write all the obeject-specific serial data here
// you may use pseudo-code if needed to insert int/float values or object/name references
// copy/paste raw data hex block here

This will allow to resize object in place (shrink or expand).

 

I'm updating github sources now. I've checked most of the functionality and it seems to work as intended. When adding your own keys/sections to ModScript class, always use WriteModdedData function for actual data writing. It automatically checks for resize in respect with object behavior modifier and handles backup data writing.

Edited September 22, 2014 by wghost81

[tracktwo]

Cool, thanks. And thanks for all the help with this, WGhost. The MODDED_CODE looks like a good approach, especially for handling the default property list, but there is still one problem with that for the sound data itself. Sound nodes are not relocatable because of the absolute offsets, so the modded code section needs to be able to generate them. Only PatchUPK knows the final offsets where the new object will wind up in the UPK, not the mod script. If we can expose some way to generate those through MODDED_CODE, that would work. E.g. the object-specific data in your example above would look something like:

// All the default property list code as in your previous comment goes here
 
// The raw data mirror object
<%u 0>
<%u 0>
<%u 0>
<Pointer>
// PC compressed data
<%u 0>
<%u 10000> //Size of the ogg data (not the full serialized object size)
<%u 10000> // Size of the ogg data (not the full serialized object size)
<Pointer>
// All the hex data for the ogg go here
// Xbox compressed data (?)
<%u 0>
<%u 0>
<%u 0>
<Pointer>
// PS3 compressed data (?)
<%u 0>
<%u 0>
<%u 0>
<Pointer>
// WiiU compressed data (?)
<%u 0>
<%u 0>
<%u 0>
<Pointer>

Where <pointer> emits the absolute file offset into the file that would immediately follow the position of the pointer. E.g. if the <Pointer> was to be written at offset 0x1000 in the file, it would write the value 0x1004.

 

This looks like it might be the most general way to get replacement sounds and/or new sounds into the upks, rather than a dedicated REPLACEMENT_SOUND node. But it's still a bit of work to generate the required script. Maybe a stand-alone utility that can take an ogg file and generate a mod script snippet from it would be useful? That would be nice because it could use the vorbis library to compute the new duration, channels, etc. values to populate the new default property list with values that are actually correct. So far I've been ignoring these in my experiments because the engine doesn't seem to use them, but these values are presumably exposed to scripts and would see the wrong values with modded sounds.

[wghost81]

Are you implying Pointer type variable inside an UntypedBulkData_Mirror is a file offset? Because AFAIK, unrealscript Pointer type variable is a placeholder for c++ memory pointer. Just trying to sort out terminology for better understanding.

 

Problem is, Patcher is not re-cooking package, it's just patching it. This means, MODDED_CODE doesn't know its file offset at the time of parsing.

 

The way cooking works, *_SF packages should be considered one big object and dealt with accordingly. So, in theory, stand-alone utility should make the whole package out of ogg file, but that's beyond our abilities for now. Replacing a sound inside existing package should be a two-pass operation: replacing a sound object (patcher or stand-alone utility) and recalculating and rewriting absolute offsets (stand-alone utility).

 

Or, if we can figure out SoundNodeWave data format, we can code serialization/deserialization function and handle all the offsets from there.

Edited September 22, 2014 by wghost81