You can't import nif in SP, use nifskope or blender + nif plugin or maybe outfit studio to open the nif and export in obj or fbx.
Now, what channels, I'm not sure there's a good answer. There's certainly not a single "one size fits all" approach. Skyrim isn't pbr as you say, if you make pbr textures and then load those in skyrim is not going to look good at all. You can:
a) Make pbr textures normally and then do some conversion on these to get something that looks good in skyrim. Like, for example, I need the metals darker then maybe I can multiply the metalness onto the albedo at a certain opacity, stuff like that. Lighten/darken stuff, more contrast or whatever. Pbr glossiness could be a starting point for normal map alpha (specular) if you do metalness workflow, otherwise the specular. The metalness map will have some similarities with what you'll want the enviroment mask (_m) to look like (you definitely want to use enviroment mapping in skyrim in metals), since you basically want enviroment mapping in metals. You most definitely want to multiply AO into the texture, potentially even more lighting. There's some baked lighting effects in SP that could be useful. You could do this conversion later in photoshop or less destructively with adjustments directly in SP on top of the layer stack.
b) Make textures specifically for skyrim in the first place with the downside that what you see in the viewport won't really match what you'll see ingame so you are a bit "blind". It's basically the same that you'll do in photoshop, just exporting and test ingame as you go.
In either case, make a custom export preset that exports exactly what you want, it's super easy to setup and that way you put your specular in normal map channel, suffixes and so on. You need some other software to compress to dds though.
Which aproach is better? B is simpler and kind of makes more sense, I'd argue you lose a lot of what SP provides, I do A. In the end you have to be familiar with PBR and Skyrim and find a workflow that works for you. Look at vanilla textures or textures from mods for a guideline of how should stuff look.
Some links that could be useful:
Edited by FrankFamily, 11 March 2019 - 11:01 PM.