This paper investigates the problem of matrix completion from corrupted data, when additional covariates are available. Despite being seldom considered in the matrix completion literature, these covariates often provide valuable information for completing the unobserved entries of the high-dimensional target matrix A. Given a covariate matrix $X$ with its rows representing the row covariates of A, we consider a column-space-decomposition model A=X \beta+B where \beta is a coefficient matrix and $B$ is a low-rank matrix orthogonal to X in terms of column space. This model facilitates a clear separation between the interpretable covariate effects and the flexible hidden factor effects. Besides, our work allows the probabilities of observation to depend on the covariate matrix, and hence a missing-at-random mechanism is permitted. We propose a novel penalized estimator for A by utilizing both Frobenius-norm and nuclear-norm regularizations with an efficient and scalable algorithm. Asymptotic convergence rates of the proposed estimators are studied. The empirical performance of the proposed methodology is illustrated via both numerical experiments and a real data application.