Help/tutorial on Normal Maps and Alpha Channels (Photoshop)

[bientje]

Hi all,

 

So I downloaded the Photoshop .DDS plugin to try and fix some overly bright textures in my game. For example, one is from a mod that adds some ducks to Skyrim. When the sun hits the ducks they become blinding. Another example is a banner retexture that makes banners extremely bright in the sun.

 

I know we can set Glosiness strength in the .nif file through Nifskope. However, for the banner retexture there are no meshes associated with the mod, so I assume that the problem can be partially fixed by editing the texture? And even if the problems is in the meshes, I'd really like to learn how to edit textures.

 

I've been searching for hours for a decent tutorial that properly explains how .DDS normal maps work and how I can make changes to specularity and transparency through the Alpha Channel in Photoshop, but I couldn't find anything useful.

 

The furthest I've come without breaking the entire Alpha Channel is opening the original normal map in Photoshop and then (without any adjustments) re-saving it as BC7 sRGB. But even then when I re-open the file the entire Alpha Channel is completely white. So I cannot even save the normal map properly without messing up the entire structure of the file... it's kind of sad.

 

Another thing I did learn is that if I delete the entire Alpha Channel and then select the RGB channel and then create a new channel I can create a 'new' Alpha Channel. This new Alpha Channel is much more grey-ish than the original one. If I try to save the .dds with my new Alpha Channel is still shows up as a completely white Alpha Channel the next time I load up the file...

 

I read somewhere that editing the Alpha Channel in the normal map could lead to less glossiness/shininess, so my goal is to change the Alpha Channel from almost completely black (original modded texture) to slightly more transparent.

 

For the love of my sanity can someone please help me! And why are there no decent tutorials on this subject?

[thumbincubation]

I'm afraid I can't help you with Photoshop, as I haven't used it since like 6.0.

 

IF you're willing to download GIMP (free) and its .dds plugin (the newest GIMP may not even need a separate plugin) then this may help you. It is copied and pasted from a thread I did some time ago, I believe I was using GIMP 8 at that time. What follows is the "old way" I was doing them, though I believe it is more than sufficient for simple things like banners, less so for things like armor that has both leather and chainmail.

 

.............................................................................................

 

Here's the way I do them in GIMP. There is probably a better way, but most of what I do has been trial & error. This will give you a rough normal map, with the difference in scale and filters being only where you've highlighted. I don't know how to make the vanilla-style ones, where the changes are more gradual and appear more like shadowing.

 

 

 

With the GIMP .dds plugin installed...

 

1. Open the diffuse map in GIMP. Uncheck "load mip maps".

 

2. colors > desaturate the diffuse.

 

3. colors > brightness/contrast to make the details stand out, but don't go so far as to distort.

 

4. layer > transparency > remove alpha channel

 

5. filters > maps > normal map. Start with something mild. Leave it at 4 sample for now, check 'wrap,' and set scale for about 10. It should turn blue and show you a rough texture. I want to say (not certain) vanilla clothing is between 5-10 scale, usually. If you want more / less texture detail, just keep using edit > undo.

 

6. layer > transparency > add alpha channel

 

7. Double click the eraser tool, set the brush from fuzzy to solid, and slide the brush size until the circle covers the entire texture.

 

8. Set opacity, with the consideration that a lower number is shinier, a higher number is more dull, and above 95 has been said to cause problems. I can't remember which tutorial I read this on, but I have stayed below 95 and never had an issue with it. I generally do 94 for cloth, and somewhere around 80ish for shinier objects, depending on how shiny.

 

* If the texture needs to be shiny and dull, highlight the parts you want to be dull, erase those at about 94% and then select > invert, and move your opacity to however shiny you want it. I do it this way, so the thing I am more likely to change is closer to the last thing I've done, for undo purposes. You can also use the same method, if you want part of the normal map to have a rougher texture than another part. In that case, you would start highlighting right after you've removed the alpha channel, set your scale where you want it, invert, do the next, etc. When you have everything at the texture scale you want, re-add your alpha channel, then go to your eraser the same way, changing the opacity as you invert the selection. The 3x3, 5x5, 7x7, and 9x9 filters seem to make the item seem more embossed, but with less detail. You will want to play around with it some.

 

9. File > export as. Give it the same name as the diffuse, but with _n.dds behind it. Export, give it a compression (I usually use dxt3 but I don't do much high res stuff.) Make sure to check "Generate mipmaps" here.

 

It's a good idea to leave GIMP open and the map up, while testing, as you may need to tweak things, and undo is a lot easier than starting everything again.

 

Use either creation kit texture sets, or the corresponding mesh and diffuse map, in nifskope, to apply and test your new normals.

 

....................................................................................................................

 

This video, which I discovered after I made that original post, is how I do them now, and seems to give a lot better detail for mixed materials. Mind you, I am not an HD or photorealistic modder, and I just use the normal DXT1, 3, or 5, depending. I don't know much (anything) about bc7 yet.

 

 

 

Hope this helps. Sorry about Photoshop. Hopefully someone more knowledgeable will come in and have those answers.

Edited January 24, 2021 by thumbincubation

[bientje]

Thank you for the reply. I followed the GIMP tutorial, alas no improvement. I did notice that the normal map.dds was misaligned in relation to the regular .dds so this might be the source of the problem.

 

I also messed around with the mesh, changing the specular and glossiness strength, but this didn't make a difference.

 

I'll keep trying! I appreciate your help.

[thumbincubation]

I'm sorry it's not working. I wish I had better answers for Photoshop, but I've never done a .dds in it before.