You can, I've done it several times due to defective googles or simply not having them. It stings less then you would think and the visual distortion is actually less than pool water usually (though I couldn't tell you why). However, you will do a lot of crying after you surface because the saltwater has, well, salt in it, and this will try to dry out your eyes. As with cholorinated water, you mainly pay later.
Although, from secondhand experience, do not attempt this below fifty feet, as the pressure may cause damage to your vision. (During my brief flirt with Scuba, my classroom instructor mentioned that he had to wear glasses as he had his googles crack when he was swept into a piece of hull while he was wreck diving at 120 feet or so, the pressure permanently damaged his eyesight.)