arXiv 2026
Beyond Language Modeling: An Exploration of Multimodal Pretraining
A systematic study of unified multimodal pretraining with representation autoencoders and Mixture-of-Experts, showing how visual data complements language, enables world modeling, and benefits both understanding and generation.



