Then I would say about optimization:
[Local algorithms]
- Brent's method (secant and parabolical fitting with some really involved mathematical tricks) since you don't always have gradient information and this involves both secant and parabolical fitting methods, both of them are recommendable.
- Newton's method for multivariate problems (important)
- Conjugate Gradients method
- Levenberg-Marquardt damped least squares optimization (important)
[Global Algorithms]
- Spall's SPSA (this is interesting from mathematical point of view because common sense would tell this doesn't work. I suspect it was discovered by an accident.)
- Simulated Annealing (modelling of the natural cooling processes)
- Differential Evolution algorithm (evolution is always interesting)
- And the newest, anthill / bee colony optimization strategies
It will take a long time to go through each of these and understand why them work as they do. But remember that quadratic convergence rocks.
Neural networks I would also say could be beneficial to understand, especially if you can combine neural network with a optimizer.
Sorting routines are also important. Matrix routines and image filtering functions are also important. Basically, the Numerical Recipes will give you a brief introduction into the calculation algorithms. But I'm only a semi-professional, and would not like to know how the professional programmers calculate stuff in C++. Might be radically different and use bitwise operations.
Mika
