Really, if you were wearing a suit of heavy armor, would you go anywhere near the water? Of course not! You'd sink like a rock.
Suggestion:
If you have heavy armor, you'll have to strip and leave your armor on the shore before getting in the water if you want to swim. Carrying heavy armor is no better than wearing heavy armor when it comes to swimming.
If you want to keep your armor, you can always walk across the bottom of the river or ocean. If you are a water breather, take all day for the crossing. If not, the crossing better be short enough or take some potions with you.
If you have the steed stone or a perk with leaves you unencumbered by heavy armor, the restriction on swimming is lifted.