So for me, the culprit was the mod I'm Tired. After any siege using an army I created, that's when the problems started, but if I disband before a siege there was no problem. After deactivating the mod and loading a save from before the first siege with the army, everything worked fine. If you're using either I'm Tired or Space Time Continuum (that mod is integrated into I'm Tired) I'd advise disabling them and loading an earlier save.