I just found this thread nearly three years on, and am very happy to have found it, as it addresses the very subject I wanted to start a conversation about.
Oblivion was my first TES game, then I waited for Skyrim with baited breath. One of the things that first drew me in and made Oblivion so wonderfully enjoyable was the AI characteristics being discussed here. When Todd Howard proclaimed that the AI had been greatly improved for Skyrim, I started counting the days. Then, the disappointment.
Some things in Skyrim have been improved. NPC's are less likely to stare at a wall for six hours. Sometimes they will annoyingly return something that you were trying to get rid of by dropping it, and they will continue working while talking to you. This is mostly good, but I really, really miss the excitement of thieves getting caught, chased, or killed as in Oblivion. Of course, that did show up in Skyrim too, but as a randomly occurring, tightly scripted event that happens only in Riften, and supposedly only if you are pursuing the Thieves Guild questline.
I think the posters here have also made very valid points about the game design in general being poor from a perspective of more developed AI, such as the way the cities have been implemented.
The Skyrim AI Overhaul mod continues to give me hope, however, even though the mod author quit modding and the game isn't really designed for it. On the mod page, the author states that:
"Branches and Procedures can all have Conditions, you can create very complex switches to route AI behavior through."
This means that the tools are available to give nearly every actor in the game a comprehensive AI, and I believe that for a person determined and dedicated enough, it will be possible to create something wonderful with the AI even IF the game has a poor design for it.
I would love to be able to gather a team of people to create a game with a World that's very large and with varied landscapes, large cities bustling with life as well as villages and backwater hovels, interesting races to play, each with their own well developed culture to start out in; folklore, gossip, hidden knowledge and misinformation, great game mechanics & animations, and a cutting edge AI that would thrill new players the way I was thrilled when I played Oblivion for the first time.
Oh, and I mildly resent the blame being put on console players like myself who have never had the money for a good PC, and who love these games in their vanilla version just as much as PC players who play them modded for years on end. I have played the vanilla versions of both games for years on end, because that's all I had; that, and my imagination.
I fully agree that the PC is the premier experience for playing these games, and applaud all those fortunate enough to do so, but it is very tiresome to constantly hear about how casual console players are ruining things for everyone else. People are people everywhere you go, and all of them have a full range of behavior, which unfortunately includes the scapegoating of groups outside one's daily experience.
I would like to thank the OP for bringing up this topic, which I believe to be one of the most important for RPG games. I enjoyed this thread tremendously.