All I'll say from that point of view is that we've got by for the last several million years without
anyone looking out for them

Cataclysmic asteroid collisions are not, I'm pleased to say, a common occurence among the inner planets. And, of course, we have a moon, take a look at the surface of that to find out where a lot of Earth-bound dangers ended up

Oh, and to put things into perspective, yes, we would see anything that was going to hit us, since, it order to be able to do damage it needs to have one of two things, mass or speed. Both of these create phenomenon that are detectable from long range over a wide are.
The Comet that hit Jupiter exploded with a force of
1 triliion megatonnes, it would have cracked the moon in two. However, something of that mass and speed would never get caught in the Earth's gravity well, and for a direct hit... well, imagine playing 3D zero-gravity, no-friction snooker on a table that is 200 km cubed with sawdust on the felt, and you'll have a rough idea of the chances
