normal vs diffuse texture size

[nightscrawl]

I've noticed that when some modders (Gamwich is an example) provide different performance options for their textures, one option will be for half-size normals. That is, 2k diffuse with 1k normals. Is the diffuse more important for overall quality of the visual?

 

I typically use 2k everything, but am looking to eek out a few more fps on my aging system and was wondering if I could scale down some of my 2k normals for a performance increase without too much loss in quality. Perhaps I might be able to use some 4k diffuse as well, or increase my grass density, or some other performance-heavy things that I skimp out on, since I will have freed up more VRAM by doing that.