Amir Zamir: "(Multimodal) Flexible-Length Tokenization"

I will discuss flexible-length tokenization and the intriguing structures that emerge as its byproduct. Most visual tokenizers map an image or video to a fixed number of tokens, regardless of its content. I will describe simple learning mechanisms, e.g., a latent PCA-like constraint, for developing flexible-length tokenization, where the same input can be represented by a variable number of tokens (FlexTok and VideoFlexTok). I will show the interesting structures that emerge in the representation as a result of the flexible-length compression – e.g., a coarse-to-fine semantic order in FlexTok and object-motion disentanglement in VideoFlexTok. I will then discuss multimodal learning (models like 4M and its extensions to 3D, videos, and decoder-only architectures), how we can move toward multimodal tokenization, and what we want from a multimodal tokenizer in the first place.