As far as I know, Skyrim .hkx files need/might be converted before editing them, so you can open them in anything that opens .kf (Nifskope, etc.). This tool is good for conversion: https://www.nexusmods.com/skyrim/mods/17109/ - the author provided no readme, but make sure to read through the comment section. I found Hextol's instructions (comment from 06 March 2014) 100% working. As for working with the .kf, this guide might (or might not) be useful: http://wiki.tesnexus.com/index.php/Creating_character_animations - it requires Blender, which is free to download. It's sometimes quite annoying to work with, though. Also, animation replacers don't require FNIS, it's only needed for adding new animations.