Research

Probabilistic modeling and optimization

Scalable machine learning with gradients and curvature

Gradients and curvature information characterize how functions and loss landscapes change locally. Gradient observations can sharpen probabilistic predictions and uncertainty estimates and guide sequential decisions in Bayesian optimization, while curvature can correct for anisotropy and improve conditioning in neural network training. The main obstacle is computational. In standard dense formulations, covariance and curvature matrices require quadratic storage and cubic-time factorization or inversion.

My research develops scalable representations that preserve the benefits of gradients and curvature on a prediction or update while avoiding this dense linear algebra.

Derivative Gaussian Processes and Bayesian Optimization Gradient-informed probabilistic learning for scalable inference and high-dimensional decision making
Zeroth-order LLM Fine-Tuning Memory-efficient, curvature-aware optimization without backpropagation
Second-order optimization Low-rank curvature approximations for scalable gradient preconditioning

Independent research program

Derivative-informed probabilistic modeling

Key idea. Predict the scalar response without carrying the entire gradient state.

Gradient observations can substantially improve Gaussian process (GP) surrogates, but standard derivative GPs treat all gradient coordinates at each location as separate responses. Their covariance blocks therefore scale with both sample size and input dimension.

For stationary kernels, the gradient components relevant to a target function value lie in the span of target-centered input differences. Within each Vecchia conditional, the full conditioning gradients can therefore be replaced by reduced directional derivatives without changing the corresponding full-gradient conditional density.

Because the reduction is exact within each Vecchia conditional, the Vecchia approximation is the only source of discrepancy from the full derivative GP. Direct construction of the reduced conditional factors avoids forming or factorizing full derivative covariance blocks. This makes derivative GP inference practical for large datasets and high-dimensional problems and enables its use in Bayesian optimization.

Representative work
Scalable Derivative Gaussian Processes via Exact Gradient Reduction Paper Code
Derivative Gaussian Processes on a Two-Direction Budget Paper Code

Foundational research program

Curvature-aware deep learning optimization

Key idea. Exploiting low-rank structure enables efficient approximations of dominant curvature geometry.

The dominant geometry of the layer-wise Fisher information in neural networks is well approximated by a rank-one structure defined by the mean activation. This yields a closed-form preconditioner that captures useful curvature without storing dense covariance matrices or performing costly matrix inversions.

The same structural principle shifts from preconditioning observed gradients to designing perturbation queries when gradients are unavailable. Each pair of function evaluations measures loss variation along a sampled direction, providing a finite-difference approximation to a directional derivative. A block-diagonal rank-one approximation to inverse curvature parameterizes an anisotropic Gaussian distribution over these directions, so that queries follow learned local geometry rather than isotropic perturbations. This yields curvature-aware zeroth-order optimization with only vector-sized auxiliary state and without explicit Hessian estimation.

Representative work
Low-Rank Curvature for Zeroth-Order Optimization in LLM Fine-Tuning Paper Code
MAC: An Efficient Gradient Preconditioning using Mean Activation Approximated Curvature Paper Code

My broader goal is to enable the practical use of gradients and curvature at modern scale by reducing computational cost while improving sample and optimization efficiency.