what is the mathematical definition of "observer"?
From the new-mostly-abandoned Copenhagen interpretation, an explanation that it's simply a placeholder for an outside system interacting with the quantum state, whether a human being or a billiard ball (in fact it's the act of interaction that's important, not the interactor):
Of course the introduction of the observer must not be misunderstood to imply that some kind of subjective features are to be brought into the description of nature. The observer has, rather, only the function of registering decisions, i.e., processes in space and time, and it does not matter whether the observer is an apparatus or a human being; but the registration, i.e., the transition from the "possible" to the "actual," is absolutely necessary here and cannot be omitted from the interpretation of quantum theory.
—Heisenberg, Physics and Philosophy, p. 137
In modern QM the act of observation is no longer considered to be of any importance since the 'observer' - whether human or any other system - is also a quantum system and what results is simply a vast entangled quantum system.