In a recent French interview, Marvel Studios President Kevin Feige had this comment of Ed Norton reprising the role of Bruce Banner in the first Avengers movie:
On negotiations with Edward Norton to reprise the role of Hulk for The Avengers, the part which Marvel chose to give to Mark Ruffalo instead, Feige said that “there never really was a negotiation period on that one … there were very, very brief discussions. But very soon after that, very early on the process, we identified Mark Ruffalo as who we wanted to play this character”.
Based on what I heard about Ed Norton and his conflicts with the director of the Incredible Hulk movie on how the movie was edited, I am kinda surprised they even had a conversation with him. Movie would have been a bit different if he did come back. I thought he did a good job on playing the role of Bruce Banner in the movie though.