I think the most Annoying quest is the Bowerstone Book side quest. I give the teacher alot of books and he bloody gives me a stupid hat. All I want is the Key but I can't get it.
To beat the hobb cave I just brought two hitmen with me. The most annoying one for me was trying getting inside the bandit camp, but I know how to do it now.