A seminar of the A3SI team of the LIGM (joint research unit of the University Paris Est) will take place on Thursday, May 24 at 14h, in the meeting room B 412 of the group IMAGINE (ENPC - Bat Coriolis).
Abstract: We study a deep linear network expressed under the form of a matrix factorization problem. It takes as input a matrix X obtained by multiplying K matrices (called factors and corresponding to the action of a layer). Each factor is obtained by applying a fixed linear operator to a vector of parameters satisfying constraints. In machine learning, the error between the product of the estimated factors and X (i.e. the reconstruction error) relates to the statistical risk.
We first evaluate how the Segre embedding and its inverse distort distances. Then, we show that any deep matrix factorization can be cast in a generic multilinear problem (that uses the Segre embedding). We call this method tensorial lifting. Using the tensorial lifting, we provide necessary and sufficient conditions for the identifiability of the factors (up to a scale rearrangement). We also provide a necessary and sufficient condition called Deep Null Space Property (because of the analogy with the usual Null Space Property in the compressed sensing framework) which guarantees that even an inaccurate optimization algorithm for the factorization stably recovers the factors. In the machine learning context, the analysis provides sharp conditions on the network topology under which the error on the parameters defining the factors (i.e. the stability of the recovered parameters) scales linearly with the reconstruction error (i.e. the risk). Therefore, under these conditions on the network topology, any successful learning tasks leads to stably defined and therefore explainable layers.
We illustrate the theory with a practical example where the deep factorization is a convolutional linear network.