I really want to use some of the custom followers that people are producing, but they make the game so easy its pointless. No, decreasing the difficulty doesn't help. As a base line test I usually attack them until they fight back. Most followers one or two shot me. It's really not fun having them destroy every single enemy for you.