So far, I haven't uninstalled all the mods or enough mods to make a difference, because like I said, I'm trying to uninstall the ones that won't break my save. So I'm uninstalling in little batches or one by one, and if deleting one causes my save not to load, then I have to reload the mod it and try again with a different one, etc, so it's a real pain and slow process. The last time I tried it (yesterday I think) it was still crashing.
My brother's save, which doesn't have any mods, seems to run fine, the RAM goes up but not enough to peak and crash, he even managed to do an entire fight without a single crash. My save crashes after loading and moving my party 10 steps
I'll keep trying to remove mods and will update the game to Hotfix #23 to see if it helps.
Edit: I just read the Hotfix #23 changelog and one of the bullet points says "Optimised Poltergeist auras to reduce memory usage and prevent an out-of-memory crash." fingers crossed it helps with this issue