Issue No. 02 - Feb. (2017 vol. 39)
ISSN: 0162-8828
pp: 227-241
Tongliang Liu , Centre for Quantum Computation & Intelligent Systems and the Faculty of Engineering and Information Technology, University of Technology Sydney, 81 Broadway Street, Ultimo, NSW, Australia
Dacheng Tao , Centre for Quantum Computation & Intelligent Systems and the Faculty of Engineering and Information Technology, University of Technology Sydney, 81 Broadway Street, Ultimo, NSW, Australia
Mingli Song , College of Computer Science and Technology, Zhejiang University, Hangzhou, China
Stephen J. Maybank , Department of Computer Science and Information Systems, Birkbeck College, University of London, London, United Kingdom
ABSTRACT
Often, tasks are collected for multi-task learning (MTL) because they share similar feature structures. Based on this observation, in this paper, we present novel algorithm-dependent generalization bounds for MTL by exploiting the notion of algorithmic stability. We focus on the performance of one particular task and the average performance over multiple tasks by analyzing the generalization ability of a common parameter that is shared in MTL. When focusing on one particular task, with the help of a mild assumption on the feature structures, we interpret the function of the other tasks as a regularizer that produces a specific inductive bias. The algorithm for learning the common parameter, as well as the predictor, is thereby uniformly stable with respect to the domain of the particular task and has a generalization bound with a fast convergence rate of order $\mathcal {O}(1/n)$, where $n$ is the sample size of the particular task. When focusing on the average performance over multiple tasks, we prove that a similar inductive bias exists under certain conditions on the feature structures. Thus, the corresponding algorithm for learning the common parameter is also uniformly stable with respect to the domains of the multiple tasks, and its generalization bound is of the order $\mathcal {O}(1/T)$ , where $T$ is the number of tasks. These theoretical analyses naturally show that the similarity of feature structures in MTL will lead to specific regularizations for predicting, which enables the learning algorithms to generalize fast and correctly from a few examples.
INDEX TERMS
Algorithm design and analysis, Stability analysis, Complexity theory, Convergence, Prediction algorithms, Training, Electronic mail
CITATION

T. Liu, D. Tao, M. Song and S. J. Maybank, "Algorithm-Dependent Generalization Bounds for Multi-Task Learning," in IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 39, no. 2, pp. 227-241, 2017.
doi:10.1109/TPAMI.2016.2544314