I just got myself an H320 a week ago, and it's great overall. There's a few nitpicks I do have, though:
First, the videos have to be a specific resolution, 10fps, and Xvid. Not that all that is bad, mind you (Cvid rocks, and the resolution is what the player's screen is, AFAIK), but the framerate is a bit annoying at times. Go get a proggie called "iriverter", which will convert just about any video into the proper format, and check how it looks to you.
Second, the interface is fine, really. Most things make sense, although not the iPod-level, intuitive sense. But learning what does what isn't hard, and isn't completely illogical, either, so it's not something you're going to have to return to the manual for time and again.
However, my main nitpick is with the navigation. Scrolling down a list of folders with the buttons is just annoyingly slow. So annoying, in fact, that it made me reorganize my music; I now have folders for each letter of the alphabet, and one for numbers, so I have a max of 27 folders to look through before I find a certain artist. Kinda annoying, granted, but not that bad.
But now here's the thing. iRiver is about to come out with a 20Gb version of it's beautiful H10 player - the one that's the size of the iPod Mini? The one with the touch strip?

If you have the software for it (it requires Windows XP / Media Player 10 AFAIK), then I'd definitely recommend holding off for a few weeks and going for that instead; the H3xx series
is a bit on the bulky side of things.