...during the battle of Hoth era where only four force users exist 
There were more. If you count the three Skywalkers and Emperor, there's still Mara Jade and an unknown number of other Emperor's Hands.
If you also count force-sensitives, there's the Imperial Guard, of which at least two people exhibited moderate force sensitivity on later time (Kir Kanos and Carnor Jax) and presumably they all were selected based on their force sensitivity, though I wouldn't say they were force users at that point. Also on the list of force-sensitives are Maarek Stele, Tycho Celchu, Corran Horn, not to speak of Kyle Katarn.
Plus all the unknown shamans and wizards of worlds unknown to the Galactic Republic, worlds that never sent anyone to the Jedi order or had any contact with the rest of the Galaxy, who by all logic must have existed...
The problem is where you draw the line between force-user and force-sensitive. All A are B but not all B are A...
