He was a deist, yes, and there's no basis to try to enforce either any particular religious standpoint or atheistic one with his quotes.
Einstein disliked his words being used that way.
EDIT: to add a quote to the message...
"I don't try to imagine a God; it suffices to stand in awe of the structure of the world, insofar as it allows our inadequate senses to appreciate it."
And while I personally don't see any reason to differentiate between divine and mundane (after all, everything that exists, just exists), so I don't use the same terminology as Einstein, but I tend to also think that universe is a magnificent, awesome place to be even without any sort of divinity in it. Or, if you want to see it that way, universe might be considered the highest possible divinity there can be. In other words, either everything is divine, or nothing is. But this thread is not of my views, but Einstein's.