• ESM3 (1.4B, 7B, and 98B), a multimodal protein language model on sequence, structure, and function tokens, using MLM for representation learning and generation.
• Uses a MLM objective with diverse masking rates to predict missing tokens and thus enabling generation from partial inputs (unconditioned or conditioned on structure/sequence/function prompts)
• Tokenizes structures by VQ-VAE like model and handles atomic coordinates through geometric attention layers
• Generates a new fluorescent protein (esmGFP) significantly distant from known proteins