We never had to learn to process body language, facial expressions, and tone of voice. We evolved this capability...it's innate. But we had to spend years learning to read and write with any level of sophistication. The brain needs and expects these other--more significant--channels of information, and when they don't come... the brain suffers (and so does the communication). And the problem goes way beyond just an increased chance for misinterpretation.
From this wonderful post by Kathy Sierra on the huge advantage enjoyed by face-to-face communication over the other modes, such as e-mail, wikis, blogs and instant messaging. Much of the post is based on a talk by the neurobiologist Thomas B. Lewis.