Thestrals -- Why Didn't Harry See Them Until Cedric's Death?

Okay.

The first time Harry is recorded as seeing the thestrals pulling the carts is in Harry Potter and the Order of the Phoenix after witnessing the death of Cedric Diggory.
     I have no problem with this by any means. Cedric died and Harry saw it happen, thus he can see thestrals right now. It makes perfect sense.

HOWEVER, this is not the first time Harry saw death.  

In both the book and the movie, it appears that baby Harry watched Voldemort kill his mother.

      In the books, Voldemort showed up just as Lily was putting Harry to bed. James ordered her to take Harry and run upstairs to hide just before being hit by the Avada Kedavra curse. Lily runs upstairs, lays Harry in the crib, and is immediately hit by the killing curse as well. There is no time for Harry to fall asleep between his parents' deaths. Furthermore, Hagrid says that Harry fell asleep on the trip to Privet Drive, implying that Harry was awake when Hagrid picked him up.
     In the movie, the scene is portrayed the same way. We see Harry in his crib standing up, watching his mother die in front of him as she protected him from Voldemort.

So what the heck?

Of course he couldn't have seen the thestrals in Sorcerer's Stone (on boat) or in Chamber of Secrets (flying the car into the Whomping Willow), but what about the carriage rides in Prisoner of Azkaban and Goblet of Fire? Why couldn't he see the thestrals those years?

 Even Neville has the same problem. He saw his grandfather die, but the scene in Order of the Phoenix implies that he couldn't see them either.

Seriously, what is going on?

The only explanation I can think of is that the thestrals are ignored until this point to introduce Luna Lovegood -- the strange, odd girl who can see and believe in things that no one else does. When Harry starts asking about them, she tells him that he isn't mad because she can see them too. "You're just as sane as I am."     
      This plays into the attitude of Order of the Phoenix. Is Harry crazy like the Daily Prophet insinuates? After all, he's seeing what Luna does and Luna's nickname is "Loony Lovegood." Thus, one may guess that Harry is losing his mind as well.

But is waiting until Order of the Phoenix and Luna for the thestrals to appear worth a possible plot hole within the series? 
     I'm just not sure . . .