Alright after messing around with my mods for awhile I figured out what was causing the problem. It was the Skyrim HD-Misc file. I dont understand why out of all of the Skyrim HD files that it would be that one causing the problem. After re-reading over everything for Skyrim HD I saw this "Skyrim HD - Misc: This includes the rest; Reflections, Water, Sky, Clutter, Clothes and the fix for the textures in the HD DLC (also usable without HD DLC)." I have other mods that edit those textures and they have never caused this problem. I still havent figured out why this file would be the one increasing my load times. If anyone could help me on this that would be amazing. Disabling Skyrim HD- Misc fixed the problem for me, so if anyone else is having that problem disable it.