I have had this problem again and again with "Missing in Action". After taking the book to prove that his brother is in jail, Avulstein asks me to leave even during the day. I have to reset is AI (resetai) or disable and enable him to be able to talk to him. Sometimes the Jarls ask me to leave even if nothing happens, no call for guards. Sometimes I have even guards asking me to leave in the Jarls house. One time I was attacked in the hold of Markarth by 2 guards (the ones near the Jarl) and all the Thalmors after one of the guards asked me to leave during the day, strange is after killing all of them, the others guards didn't do anything to me as for the Jarl and the bodyguard and the old man.