In my quest to make a "dark space" in CK I have ended up with creating a new world space and pointed it to a custom Lightning Template (everything set to 0) and a custom weather and ticked No Sky in the world space.
As long as 'No Sky' is ticked I get this unwanted light as shown in the image 
If I uncheck the 'No Sky' flag I get the desired result until night time where the sky light is lighter.
Anyone who knows what I am doing wrong here?
 Image: https://imgur.com/a/NWTygRh