[WIP/Tool] HKXPack - extract .hkx files

[ShinraStrife]

 

Thank you ^^

This tool requires Java 8 (I personally use the Oracle jre 1.8.0_71 for Windows).

 

 

Thanks. I was using 7 because I installed it when I was into minecraft and I disabled the annoying update notifications so I never got around to updating it.

Edited March 3, 2016 by ShinraStrife

[DexesTTP]

New version is out ! here

It mostly fixes issue about data read wrong and padding, and it added support for dirarrays.

I have 100% reproduction on Clothdata/physics data and Behaviors, as well as padding removal from animation files (with the file working in-game). That means that if I convert an original file to XML then back to HKX, all the data is exactly the same. Needless to say, that is really really promising.

I'll next work on solving the problem with skeletons. I still can't read the data. I'll get back to you when I have results ofc, until then... Have fun :)

[ShinraStrife]

 

New version is out ! here

It mostly fixes issue about data read wrong and padding, and it added support for dirarrays.

I have 100% reproduction on Clothdata/physics data and Behaviors, as well as padding removal from animation files (with the file working in-game). That means that if I convert an original file to XML then back to HKX, all the data is exactly the same. Needless to say, that is really really promising.

I'll next work on solving the problem with skeletons. I still can't read the data. I'll get back to you when I have results ofc, until then... Have fun :smile:

 

 

As always, nice job!

[Nightasy]

Nevermind, I got it working.

Edited March 16, 2016 by Nightasy

[ousnius]

Next version of Outfit Studio will allow exporting BSClothExtraData (physics) blocks to HKX files and import the HKX files back into the current project.

That way you can export the physics data as HKX files, convert them to XML files, edit them, convert them back to HKX files and load them into NIFs as a BSClothExtraData block. :)

[bizzclaw]

I know this is a bit of an odd question, but it's important for someone who wants to make edits to the movement animations (like me). Anyways, does this method preserve the root bone's "movement" through the scene that the animation system uses to determine the models walkframes(? - Not sure what the term is for Creation/Gamebryo, but that's what it's called in source at least).

 

In case you're not sure what I mean because I'm not very good at explaining stuff like this, here's a video of what I'm talking about from an oblivion animation.

 

The reason I'm so concerned about this is because the tools released for Skyrim did not support this, and you had to manually re-create the movement. (which is a pain)

Thanks.

Edited March 19, 2016 by bizzclaw

[DexesTTP]

No, it currently doesn't. The reason is simple : we have no idea how to extract this data from the HKX file. Concerning my method, there isn't even a way to extract the animation data from a file using this tool yet.

 

To explain it more in depth. The animation data is stored as NURBS, which are basically curves defined in several points : one at the beginning, one at the end and the rest to define the transition. We are currently only able to extract the begin/end data from the file, using the Havok library. The algorithm to get the intermediate frames isn't known yet. Several people (including myself) are looking through research papers/code about this and trying to find matches for the algorithm. Worst case scenario, the algorithm would have to be reverse-engineered. In any case, it will take time - we may be talking years - to find it.

 

If we have any updates about ways to solve this, we'll let you know.

Edited March 19, 2016 by DexesTTP

[Nightasy]

Alrighty,

So I have been trying to use this tool to make a physics variant of a body mod. I kept getting crashes in game after adjusting the physics for the body. After a bit of testing I believe that the crash is somehow related to running the bsClothExtraData.hkx through this tool. The testing condition is if I simply export the bsClothExtraData.hkx from the Sturgess outfit in Outfit Studio and then immediately import it back into the body project nothing crashes. Game runs fine and I can't get it to crash no matter how long I play. However, if I export the bsClothExtraData.hkx from the Sturgess outfit and then decompile it and recompile it using your tool it will then crash the game when it's applied to the body project. In this test I made no changes to the xml file after decompiling it. This test was ran using the latest version of your tool.

I'm going to include links to the relevant files for your review.

The Original Sturgess bsClothExtraData.hkx file:
http://www.mediafire.com/download/9h9tykma95vxlma/sturgessOriginal.hkx

The Sturgess bsClothExtraData.hkx file after having ran through your tool:
http://www.mediafire.com/download/ogsayn53f4i1x53/sturgessPostTool.hkx

Again, if I take the original and use it in game I can play without crashing. If I take the one I ran through your tool and use it in game, the game crashes after a random amount of time. No changes were made in the xml file after decompile. Hope this information is useful.

I should also iterate that the game doesn't crash right away. It crashes after a few minutes of running from one area to another in game.

Edited March 24, 2016 by Nightasy

[DexesTTP]

Ok, I checked and I believe I found the reason.

 

I handled a TYPE_TRANSFORM like a TYPE_VECTOR4. It appears that :

1- a transform is 64 bytes long at least, not 16 (so about 3/4th of the data was lost)

2- it must be handled as a TYPE_MATRIX4 for the thing to work.

My bad :P

 

There's a couple other weird things in the file : there's unknown data used as padding (that's not 0 i mean), and there's two extra values at the end of an array. I can't say for sure why, but it is consistent with RAM leftovers. I believe most of the hkx file contents are memory dumps from the Havok tools.

Let me know if it keeps happening after I update the tool. I believe it's not the cause of the problem though, as missing data should result in an instant crash while non-written data is more prone to dragged out crashes.

 

The reason I believe the crash was taking its time to happen is because it may have accumulated errors in the calculated movements due to the transform being 0 where it shouldn't have been. This simply stacks the wrong data until it's so blatantly wrong it makes the game crash. Another reason may be because you didn't meet Sturges until the actual crash, therefore it didn't happen until then. But I believe you actually tested that around him, or you put the .hkx file in another nif.

 

I'm currently finishing handling skeleton data, which I finally decoded, and I'd rather not do a build in between when I'm potentially hours away of a new feature release. So, if you want a debugged version now, hit me up on IRC and I'll dump one from the dev branch :)

[DexesTTP]

New version is out ! get it here

It adds handling of Skeleton files, as well as a fix for some data inside BSClothExtraData files.

Note that there is some extradata that is still not handled well, associated with Googles and/or belts. I'll release a new version - maybe a definitive one - when I find how to solve this problem.