COPUS: Co-adaptive Parallelism and Batch Size Selection in Large Language Model Training
Jointly adapting batch size and parallelism to improve useful training progress per unit of time.
Research
Publications.
My work on efficient training, language model architectures, and training objectives reflects a growing focus on ML systems. Earlier work in NLP and computer vision is included below.
Citation information on Google Scholar
Predicting the Order of Upcoming Tokens Improves Language Modeling
An auxiliary training objective that learns the relative order of future tokens using a ranking loss.
Softpick: No Attention Sink, No Massive Activations with Rectified Softmax
Rethinking attention normalization to eliminate attention sinks and massive activations, with implications for quantization and sparsity.
LinguDistill: Recovering Linguistic Ability in Vision-Language Models via Selective Cross-Modal Distillation
Recovering language capabilities after multimodal adaptation through selective distillation from the original language model.
COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuances
Evaluating commonsense reasoning grounded in Indonesian culture, in both standard and colloquial Indonesian.
Gated Self-supervised Learning for Improving Supervised Learning
Combining self-supervised image transformations with a gating mechanism for supervised image classification.
Classification Comparison of Activity Distraction Detection in Car Driving
Comparing classification approaches for recognizing distracting activities during driving.