SSE Model seeming to not use lighting

[LoonyMods]

Hello, I'm trying to replace the vanilla statues but the lighting is being a pain. I use 3DS Max to put it all together, and it seems to be fine in there, having the correct texture brightness and specular effects. Each material simply consists of the base color texture with no other maps. I went through and made sure the normals are all pointing out correctly. When opened up in Nifskope, however, the material seems to break, or at least not work how I want it to. The base color texture shows up, but no specular effects. In addition, with Do Lighting off (so no simulated lighting) the base texture is the correct brightness, but as soon as it's turned on to simulate any lighting the entire model gets noticeably darker (not pitch black, mind you). Finally, when I boot up the game with it all configured to replace the vanilla statue, it shows up fine, but the lighting is... weird... It's too dark in general, and shadows just aren't working. I tested some additional light with the Candlelight spell, and while it brightened up the rest of the room, the statue remained dark. But when I walked around to the back of the statue (what in Max and Nifskope would be the "front" direction pretty sure) it all suddenly started to get bright to match the rest of the room. Any ideas on what to do?

 

TL;DR - 3DS Max model has base color and specular and looks great, when moved to Nifskope no specular and no shadows or effect from lights, in game shadows are broke and lighting is wonky. What now?

[maxarturo]

1) Be sure that in NifSkope in the BSLightingShaderProperty of the mesh.

Shader Flags 1, the following are added.

SLSF1_Specular - SLSF1_Recieve_Shadows - SLSF1_Cast_Shadows - SLSF1_Own_Emit - SLSF1_Re,appable_Textures - SLSF1_ZBuffer_Test

Shader Flag 2, the following are added.

SLSF2_ZBuffer_White - SLSF2_Vertex_Color - SLSF2_EnvMap_Light_Fade

2) The "Area" + "Intensity" + "Range" of the "Shines/Glossiness" of where the light hits the mesh, is defined by the "ALPHA CHANNEL" of the Normal Map. With the Alpha Channel you can define yourself what parts or hieghts of the mesh should / should not have glossiness.

With NO Alpha channel in the normal map, according to what dds/texture compresion format you are using.

Either it will automaticly add a WHITE Alpha channel, so your mesh will have glossiness all over it ignoring the normal's map height levels structure.

Or it will add no Alpha channel, in this case the glossiness will be define just by the NifScope settings of the BSLightingShaderProperty, but it'll not use the normal's map height levels structure.

Edited October 3, 2019 by maxarturo

[LoonyMods]

All of the Shader Flags 1 were already applied, but I didn't have Vertex Color or EnvMap Light Fade applied for the Flags 2. However, when I turned on the vertex color flag on any piece, it turned the piece black (and I toggled Vertex Colors in the Render tab so it just wasn't working). In 3DS Max, the materials applied to all the parts have the Vertex Colors checkbox checked so they should be there. In addition, there weren't really any normal maps applied. I went back and just applied a "blank" normal map (just that light blue color across the entire texture), and it didn't fix the "turn to black" issue. So I guess it's narrowing down to Vertex Color not working. As for the whole specular thing, well, as I mentioned, no normal map before. Even after I plugged in my ghetto version of a normal map, nothing changed. So I messed with the specular settings in the BSLightingShaderProperty, turning the glossiness and specular strength crazy high and crazy low with no effect.

 

TL;DR - Shader Flags 1 all correct, turning on Vertex Colors in Flags 2 turns it completely black, and normal map should be fine but still no specular effects.

Edited October 5, 2019 by LoonyMods

[maxarturo]

In "NiTriShapeData" check to "YES" the "Has Vertex Colors", then in "Vertex Colors" either apply to each vertex the desirable color or right click "Vertex Colors" and set them all to white.

Note : if by default all the vertex colors are set to black then your model will appear darker than your initial construction.

Also check in NifSkope that your mesh isn't inside out or the maps/normals, this can also cause this darken effect.

Right click the mesh in the Render window and Mesh > Flip Faces & check FLIP NORMALS.

Edited October 5, 2019 by maxarturo

[LoonyMods]

None of the parts have NiTriShapeData anywhere in their hierarchies. In fact, they're all BSTriShape (probably because it's not Skyrim, it's Skyrim Special Edition). In addition, when I try to rightclick on any of the BSTriShapes, and then go to Block > Insert and add a NiTriShapeData, it is created completely outside of the model, not inside any of the hierarchies, so it doesn't apply to any of the parts pretty sure. The option doesn't even show up if I rightclick and look through the Nodes instead of the Blocks. Finally, I actually opened up the looooooong list of vertices in the BSTriShapes and poked around a bit and all of the vertex colors are pure white.

Edited October 5, 2019 by LoonyMods

[LoonyMods]

LIGHTING ISSUE SOLVED

 

It was the tangents, not vertex colors. I was looking through the vanilla statues in Nifskope, and gandered at the values in the vertices in their BSTriShapes, and noticed the vertices had tangent values, whereas mine did not. I originally overlooked tangents to the point where I just didn't even check the "Update Tangent Space" box when exporting the model from 3DS Max. Well, I exported it again just now with the box checked to see what would happen, opened up the new model in Nifskope, and it actually works. It lights up correctly, and when the light moves around the lighting of the model moves with it. I copy pasted it over the last version of the .nif in the Data\meshes\clutter\statues folder, booted up the game real quick, and checked - and it works in game too! So yeah, except for the missing specular effects, the statue now lights up correctly! No vertex colors required!

 

Settings for like the two people years from now who run into the same weird issue in their first modding experience:

The settings in the BSTriShapes - Pretty much everything default, in the Vertex Desc make sure Vertex/UVs/Normals/Tangents are checked.

The settings in the BSLightingShaderProperty - Shader Flags 1 are Specular/Receive_Shadows/Cast_Shadows/Own_Emit/Remappable_Textures/ZBuffer_Test like maxarturo said; Shader Flags 2 is just ZBuffer_Write

Just make sure when you export from Max that you check Normals and Update Tangent Space in the export settings.

[LoonyMods]

SPECULAR ISSUE SOLVED

 

The normal map I was using was actually mostly transparent but didn't look that was in 3DS Max... Opened it up in GIMP and realized it was almost seethrough. Made it 100% opaque and specular effects work.

 

Big takeaway from all of this is that models (at least, stationary decorative ones) need normals and tangents to receive lighting properly, and don't mess up your normal maps.

 

I'd consider this thread closed now, thank you for the help maxarturo!

[maxarturo]

Glad you solved it.

Some tips :

- Vertex colors are very useful / important when it comes for static mesh like statues, it can easily and without adding extra system pressure (consume system's resources) add visual FXs to the mesh surface, like small corner areas with moss (green tint FX that all exterior rock statues have), or shadow in the part where the feet meets the down side of your mesh base or inside cracks, etc...

This trick / technic is use a lot not only in Skyrim but in almost all games.

- To achieve a realistic statue visual result and not end up with a plastic mesh look, you need to have a well made normal map and especially its alpha channel, to dictate the statue's surface lighting area-range-strength-intensity.

The above are crucial to not end up like those meshes/models that i'm sure you've seen in game, that they look weird, plastic, fake, non realistic even if they have a nice HD texture.

Edited October 6, 2019 by maxarturo