Well, I wouldn't say that there is zero risk of a potentially catastrophic NEO sneaking up on us and taking us by surprise, but the bigger they are, the easier it is for teams to spot and track. One thing I will say, though, is that currently if we do detect something and plot it as a threat, we still haven't quite figured out what exactly to do in response. I'm in the camp of "send up a nuke or two and deflect the rock", but there are a multitude of factors to consider with that approach.
EDIT: I did a bit of searching and there have actually been quite a few rocks that have passed by us closer than a lunar distance. They just never really got a lot of attention because they were pretty small and likely would have burnt up or exploded when they came into contact with the atmosphere (by the ay, that happens A LOT). A few ones ranged above 100 square metres in size, but most were your average small guys.
Although the sci-fi geek in me would like us to go one step further and direct the object into an orbit where we can study it relatively up-close and land some probes on it. In the end, it could be a practical method to mine these things for the potentially rare substances they comprise.
