and really, he's supposed to be a douche so I'm not sure what the big deal is.

Yeah it's generally characters that are presented as sympathetic or rootable who annoy me more, rather than someone where the writers clearly understand that he or she is being presented in villain mode. I used to really like Chuck's character even because I found him entertaining in the first season especially, but then the writers started piling on his manpain and encouraging us to see Chuck and Blair as some epic romantic endgame, at the same time as Chuck was physically lashing out around her and saying that "you're mine"