Artificial neural networks

This chapter provides a general background on machine learning basics to a non-expert readership with a strong background in computational science. It rephrases the machine learning problem according to an information theory paradigm that highlights the tight entanglement between two perspectives on data science: the statistic versus the optimization ones. The chapter presents the sampling theory, including an introduction on the Fisher approach to statistical learning and an introduction to the principle of maximum entropy for conditional probability laws. It discusses the optimization framework in which major machine learning problems are cast, consisting of minimizing the empirical risk over a class of parametric statistical models. This framework is presented in connection with the Fisher information and the MaxEnt principle. First- and second-order (stochastic) gradient descent optimization methods are presented in detail, with meaningful examples and hands-on sessions.