It's indeed true that with lower orbital velocity, a smaller Δv is needed to change the orbit, but there's two other considerations to be made:
1: With lower orbital velocity, the resulting change in momentum from a collision is also less.
2: The volume taken up by the Oort Cloud is enormous, much more so than the asteroid belt. As a result, the average density is extremely low, thus making collisions extremely rare events.
Unless you have things orbiting in vastly different directions (possible in the case of the Oort Cloud, I wouldn't know) then collisions alone are not enough to cause a tremendous change in the orbit. I'd imagine that most interactions are gravitational rather than collisional in nature.