True, in Daggerfall they had linguistic skills in Spriggan, Harpy, Orcish. I too find this a bit of a disappointing aspect they could have developed. But then again considering that geographically that none of the cultures are really separated from the dominant imperial culture apart from sumurset Isle, all the other cultures on Tamriel have been dealing with the Empire for millennia, so it isn't all that far-fetched that they all happen to speak the same language.