Research Summary

First-order optimization methods

On solving big-data problems, scalability and reliability are two very important factors in designing numerical approaches. For these problems, computing second or higher-order derivatives is often extremely expensive in terms of both time and space complexity. Hence, first-order methods that acquire only first and/or zero-th order information of a problem are popularly used mainly due to their nice scalability and fast convergence to medium-accurate solutions. First-order methods have been extensively studied for problems without constraints or with simple constraints, but few efforts have been made on solving problems with complicated constraints.

One of my current research interests is to develop efficient first-order methods and to explore their complexity for solving nonlinear-functional constrained problems with applications in machine learning, operations research, and engineering.

Stochastic optimization

If the data comes as a stream (e.g., in stochastic programs and online learning), and one wants to learn or extract important features from the data stream, storing all the data and then performing data reduction or mining may be impossible. At any time point, only partial samples of the data can be accessed. Even if one can wait until arrival of all the data and store it, accessing all the data for each update of the variables can be extremely expensive, and thus sampling small amount of the data is still beneficial and more efficient.

One of my current research interests is to develop accelerated stochastic approximation methods by adaptive learning and also to parallelize the methods in an effective way by mixing asynchronous computing and synchronous updates. Unconstrained problems and also functional constrained problems are targeted. Main applications include deep neural network training, distributed learning, and reinforcement learning.

Regularized matrix and tensor factorization

Even if a large or huge amount of data is completely acquired, storing all of it is often very expensive and may be wasteful due to possible data redundancy (e.g., face recognition). Data reduction is usually necessary to remove redundancy and maintain principal information. Matrix and tensor factorizations with regularization terms (e.g., non-negativity, sparsity, orthogonality) are efficient ways for dimensionality reduction and feature extraction.

One of current projects is to develop an efficient numerical approach to perform orthogonal low-rank tensor approximation. Without enforcing orthogonality, the rank-one tensors produced by solving a tensor approximation problem may not be orthogonal to each other. For certain applications (e.g., face recognition), orthogonality is important to achieve a good result.

Compressed sensing and low-rank matrix / tensor recovery

For large or huge-scale data, fully acquiring it can be very expensive (e.g., MRI), and to save acquisition time, the data is often partially sampled, or its very few measurements are taken. In some applications (e.g., movie-user rating by Netflix), it is even impossible to acquire the complete data. However, due to special structures (e.g., sparseness, smoothness, and low-rankness) of the data, it can be reliably reconstructed from its incomplete observations or under-determined measurements.