I got an email from a friend this morning who is reading a book by Neil Gaiman and not enjoy it as much as one might. I really enjoyed my response to this email and thought I'd post it here.
It doesn't bother me if you don't like Neil Gaiman, and I hope you're not trying to like him on my account (or anyone else's other than your own); if nothing else, I do believe in personal taste.
My (rather obvious) response to your "Why fantasy?" question is "Why not fantasy?" I don't believe that fantasy and substance are mutually exclusive (and I don't think you're saying that they are, either, but I do want to elaborate on it). In most good sci-fi or fantasy (Kurt Vonnegut Jr. comes immediately to mind), we learn more about human beings than we do about any alien race, and we learn more about our own time than any future world; the contrast is just another way to examine ourselves.
Neil Gaiman's advice to people who want to write fantasy (and so I assume his own approach) is to use the "What if?" method: "What if...cats could talk?" "What if...my baby brother was a troll?" etc. Those are really banal examples, but I think that that method, which can certainly be called "fantasizing," can be applied to other realms of thought: "What if...war was always considered just?" "What if...adults really do always know best?" "What if...morality could be bought?"
Of, what you do with the "What if" is almost as important, if not more so, than the asking, and do feel like Neil Gaiman is making points. American Gods is making a point about mythology, for example. The specific question is something like, "What happens to old-world Gods when their cultures move/move on/disappear?" And that kind of question can be answered on many different levels. I feel like most fantasy at least tries to examine questions of love, society, politics, faith... No denying that some of it isn't just fun, of course.
I'm a little confused by a couple of your criticisms. You say both that you feel like Gaiman, as an author, is holding you at a distance and not letting you in, but also that you want him to challenge you as a reader. I having trouble reconciling your desires there. Perhaps you are looking for a way in (the challenge) and not finding one, and therefore assuming that the challenge does not exist, that there is just a wall to understanding with little to nothing behind it. And if that is true, then I agree that it's disappointing. I think for me, part of the challenge of the writing is having to imagine the things described...that's not a complete answer, just something that occurred to me.
I do feel comfortable saying that Coraline is a children's book, and that probably automatically means it won't be as deep as some other literature. I think that Coraline has as much "substance" as James and the Giant Peach, for example, which has important things to say about family, friendship, and being yourself, things that seem to be the general topics of children's literature (including Coraline). But, I don't necessarily put Coraline on the same shelf as American Gods or Sandman, and when I recommend it to people, it is usually by saying "This is a fun book."
Now, if the question is, "Why do I like fantasy?" at least a part of the answer has to do with the fact that I read a great deal besides fantasy, mostly for classes, and when I come home from school I want something a bit different. Not necessarily easier, but at least of a different character. I'm in a literature class right now that is focusing on very close readings of "perfect" short fiction. And I love it, but I also want a contrast from that in what I read for fun. I also read a lot of philosophy, and while I want philosophical ideas in everything I read, I want to spend my free time reading a style that differs from a philosophical text. This is a shallower point than most of the ones I'm trying to make, but it's definitely valid.
Finally, I think your actual question was about the need for fantasy in Neil Gaiman's writing. Personally, I think it's because a) he likes it and b) he's good at it. Generally, it seems to me that the kind of ideas that he gets it in his work are best examined, at least by him, in the worlds he chooses to operate it. To me, the question is kind of like asking about the need for music in Bizet's writings. Neil Gaiman is a fantasy writer, and I don't think he claims to be anything else.