PJs > anything else
This times a million.
Times a hundred million more.
Wait, who?
USAF Pararescue. They're called PJs from the old acronym "Para Jumpers", which is what they were called back in the days of the Army Air Force. These guys are beyond hardcore.
This doesn't apply to the US or any other country to my knowledge.
It's about 50/50 with the US. Some SF comes directly from basic training, some comes after a certain amount of time in service. All of the USAF ones I can think of are recruited specifically for SF, but they go through different levels in their careers.
I'll use SERE specialists as an example.
Right after their various courses, most SERE specialists become instructors at Fairchild for a few years. At the end of their initial instructor period, they can keep going as instructors, go on to additional training, or get PCS'd to another base to serve as one of that base's SERE specialists (to train and brief us regular folk). Some of the more senior (Staff Sergeants, Tech Sergeants, and Master Sergeants) can get deployed to actually paradrop with Army SF and basically act as a living breathing survival guide for that specific environment.
I don't know how the Army, Navy, or Marines do it though.