Well, thats the problem, isn't it? Human nature leads to the formation of ever larger and more powerful organizations, which become increasingly tyranical.
There's no way to keep government small and decentralized, cause people will just build it up again, like those novelty pencils that keep coming back up when you knock them down.
For me, and this is just me, losing control of my destiny to a large, omnipotent group (the government, when taken to the extreme) is far worse than worrying about who's going to come around with a bat and try to steal my food. At least that way, I am in control, and if I loose out, oh well. But with the type of resources that are at the government's disposal (everything from surveilance, to police to money), its nearly impossible to effectively fight back and maintain control.