I think a good consensus of people whom I know watched it agree with me that it would fall apart without Phoenix's performance. You just can't help but be enamored at how far he went to make the character work. As for Batman fans, I think one more thing that contributes to how great the film became is that how the writers twisted the usual Joker story into something refreshing and realistic and completely off the Nolan crazy train.