An equivariant pretrained transformer for unified 3D molecular representation learning

Summarize this content to 100 words: Eslami, M. et al. Artificial intelligence for synthetic biology. Commun. ACM 65, 88–97 (2022).
Google Scholar 
Blanco-Gonzalez, A. et al. The role of ai in drug discovery: challenges, opportunities, and strategies. Pharmaceuticals 16, 891 (2023).
Google Scholar 
Pyzer-Knapp, E. O. et al. Accelerating materials discovery using artificial intelligence, high performance computing and robotics. npj Comput. Mater. 8, 84 (2022).
Google Scholar 
Schütt, K. T., Sauceda, H. E., Kindermans, P.-J., Tkatchenko, A. & Müller, K.-R. Schnet–a deep learning architecture for molecules and materials. J. Chem. Phys. 148, 241722 (2018).Satorras, V. G., Hoogeboom, E. & Welling, M. E (n) equivariant graph neural networks. In International Conference on Machine Learning 9323–9332 (PMLR, 2021).Liao, Y.-L. & Smidt, T. Equiformer: equivariant graph attention transformer for 3D atomistic graphs. In International Conference on Learning Representations (2023).Thomas, N. et al. Tensor field networks: rotation-and translation-equivariant neural networks for 3D point clouds. Preprint at arXiv:1802.08219 (2018).Puny, O. et al. Frame averaging for invariant and equivariant network design. In International Conference on Learning Representations (2021).Zhou, G. et al. Uni-mol: a universal 3D molecular representation learning framework. In International Conference on Learning Representations (2023).Watson, J. L. et al. De novo design of protein structure and function with rfdiffusion. Nature 620, 1089–1100 (2023).
Google Scholar 
Xu, M. et al. Geodiff: a geometric diffusion model for molecular conformation generation. In International Conference on Learning Representations (2022).Hoogeboom, E., Satorras, V. G., Vignac, C. & Welling, M. Equivariant diffusion for molecule generation in 3D. In International Conference on Machine Learning 8867–8887 (PMLR, 2022).Kong, X., Huang, W. & Liu, Y. End-to-end full-atom antibody design. In International Conference on Machine Learning 17409-17429 (PMLR, 2023).Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 4171–4186 (Association for Computational Linguistics, 2019).Radford, A. et al. Improving language understanding by generative pre-training. OpenAI (2018).Liu, S. et al. Pre-training molecular graph representation with 3D geometry. In International Conference on Learning Representations (2021).Stärk, H. et al. 3D infomax improves GNNs for molecular property prediction. In International Conference on Machine Learning 20479–20502 (PMLR, 2022).Yu, Q. et al. Unified molecular modeling via modality blending. In International Conference on Learning Representations (2024).Zhang, Z. et al. Protein representation learning by geometric structure pretraining. In International Conference on Learning Representations (2023).Zaidi, S. et al. Pre-training via denoising for molecular property prediction. In International Conference on Learning Representations (2023).Liu, S., Guo, H. & Tang, J. Molecular geometry pretraining with se (3)-invariant denoising distance matching. In International Conference on Learning Representations (2023).Jiao, R., Han, J., Huang, W., Rong, Y. & Liu, Y. Energy-motivated equivariant pretraining for 3D molecular graphs. In Proc. AAAI Conference on Artificial Intelligence 8096–8104 (2023).Feng, S., Ni, Y., Lan, Y., Ma, Z.-M. & Ma, W.-Y. Fractional denoising for 3D molecular pre-training. In International Conference on Machine Learning 9938–9961 (PMLR, 2023).Jin, W., Sarkizova, S., Chen, X., Hacohen, N. & Uhler, C. Unsupervised protein-ligand binding energy prediction via neural Euler’s rotation equation. Adv. Neural Inf. Process. Syst. 36, 33514-33528 (2023).Luo, S. et al. One transformer can understand both 2D & 3D molecular data. In International Conference on Learning Representations (2023).Feng, S., Li, M., Jia, Y., Ma, W. & Lan, Y. Protein-ligand binding representation learning from fine-grained interactions. In International Conference on Learning Representations (2024).Radford, A. et al. Language models are unsupervised multitask learners. OpenAI (2019).Brown, T. et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33, 1877–1901 (2020).
Google Scholar 
Reed, S. et al. A generalist agent. Trans. Mach. Learn. Res. https://openreview.net/forum?id=1ikK0kHjvj (2022).Abramson, J. et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature 630, 493–500 (2024).
Google Scholar 
Axelrod, S. & Gomez-Bombarelli, R. Geom, energy-annotated molecular conformations for property prediction and molecular generation. Sci. Data 9, 185 (2022).
Google Scholar 
Hu, W. et al. OGB-LSC: a large-scale challenge for machine learning on graphs. Adv. Neural Inform. Process. Syst. Datasets and Benchmarks, 1 (2021).Berman, H. M. et al. The protein data bank. Nucleic Acids Res. 28, 235–242 (2000).
Google Scholar 
Wang, R., Fang, X., Lu, Y., Yang, C.-Y. & Wang, S. The pdbbind database: methodologies and updates. J. Med. Chem. 48, 4111–4119 (2005).
Google Scholar 
Townshend, Raphael JL, et al. Atom3d: Tasks on molecules in three dimensions. Adv. Neural Inform. Process. Syst. Datasets and Benchmarks, 1 (2021).Öztürk, H., Özgür, A. & Ozkirimli, E. Deepdta: deep drug–target binding affinity prediction. Bioinformatics 34, i821–i829 (2018).
Google Scholar 
Rao, R. et al. Evaluating protein transfer learning with TAPE. Adv. Neural Inform. Process. Syst. 32, 9689–9701 (2019).Elnaggar, A. et al. Prottrans: toward understanding the language of life through self-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell. 44, 7112–7127 (2021).
Google Scholar 
Gainza, P. et al. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nat. Methods 17, 184–192 (2020).
Google Scholar 
Hermosilla, P. et al. Intrinsic-extrinsic convolution and pooling for learning on 3D protein structures. In International Conference on Learning Representations (2020).Somnath, V. R., Bunne, C. & Krause, A. Multi-scale representation learning on proteins. Adv. Neural Inf. Process. Syst. 34, 25244–25255 (2021).
Google Scholar 
Wang, L., Liu, H., Liu, Y., Kurtin, J. & Ji, S. Learning hierarchical protein representations via complete 3D graph networks. In International Conference on Learning Representations (2023).Wu, F. et al. Discovering the representation bottleneck of graph neural networks from multi-order interactions. IEEE Trans. Knowl. Data Eng. 36, 7998-8008 (2024)Gao, B. et al. Self-supervised pocket pretraining via protein fragment-surroundings alignment. In International Conference on Learning Representations (2024).Jankauskaitė, J., Jiménez-García, B., Dapkūnas, J., Fernández-Recio, J. & Moal, I. H. Skempi 2.0: an updated benchmark of changes in protein–protein binding energy, kinetics and thermodynamics upon mutation. Bioinformatics 35, 462–469 (2019).
Google Scholar 
Jing, B., Eismann, S., Soni, P. N. & Dror, R. O. Equivariant graph neural networks for 3d macromolecular structure. Preprint at arXiv:2106.03843 (2021).Ramakrishnan, R., Dral, P. O., Rupp, M. & Von Lilienfeld, O. A. Quantum chemistry structures and properties of 134 kilo molecules. Sci. Data 1, 1–7 (2014).
Google Scholar 
Thölke, P. & De Fabritiis, G. Torchmd-net: equivariant transformers for neural network based molecular potentials. In International Conference on Learning Representations (2022).Gasteiger, J., Giri, S., Margraf, J. T. & Günnemann, S. Fast and uncertainty-aware directional message passing for non-equilibrium molecules. Preprint at arXiv:2011.14115 (2020).Schütt, K., Unke, O. & Gastegger, M. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning 9377–9388 (PMLR, 2021).Friesner, R. A. et al. Glide: a new approach for rapid, accurate docking and scoring. 1. Method and assessment of docking accuracy. J. Med. Chem. 47, 1739–1749 (2004).
Google Scholar 
Lin, Z. et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. Preprint at bioRxiv https://doi.org/10.1101/2022.07.20.500902 (2022).Abraham, M. et al. GROMACS 2024.2 manual (2024.2). Zenodo https://doi.org/10.5281/zenodo.11148638 (2024).Chiou, W.-C. et al. Repurposing existing drugs: identification of SARS-CoV-2 3C-like protease inhibitors. J. Enzym. Inhib. Med. Chem. 36, 147–153 (2021).
Google Scholar 
Zhu, Y. et al. A generalist cross-domain molecular learning framework for structure-based drug discovery. Preprint at arXiv:2503.04362 (2025).Joshi, C. K. et al. All-atom diffusion transformers: unified generative modelling of molecules and materials. In International Conference on Machine Learning 28393-28417 (PMLR, 2025).Luo, Y., Wang, J., Fan, S. & Nie, Z. Pharmolixfm: all-atom foundation models for molecular modeling and generation. Preprint at arXiv:2503.21788 (2025).Xia, Y. et al. Nature language model: deciphering the language of nature for scientific discovery. Preprint at arXiv:2502.07527v3 (2025).Zhang, G. et al. Unigenx: unified generation of sequence and structure with autoregressive diffusion. Preprint at arXiv:2503.06687 (2025).Lu, S. et al. Uni-3dar: Unified 3D generation and understanding via autoregression on compressed spatial tokens. Preprint at arXiv:2503.16278 (2025).Kong, X. et al. Unimomo: unified generative modeling of 3d molecules for de novo binder design. InInternational Conference on Machine Learning 31397-31418 (PMLR, 2025).Li, Z. et al. Large language-geometry model: when LLM meets equivariance. In Forty-second International Conference on Machine Learning (PMLR, 2025).Kong, X., Huang, W., Tan, Z. & Liu, Y. Molecule generation by principal subgraph mining and assembling. Adv. Neural Inf. Process. Syst. 35, 2550–2563 (2022).
Google Scholar 
Kong, X., Huang, W. & Liu, Y. Generalist equivariant transformer towards 3d molecular interaction learning. In International Conference on Machine Learning 25149-25175 (PMLR, 2024).Vaswani, A. et al. Attention is all you need. Adv. Neural Inform. Process. Syst. 30, 5998-6008 (2017).Leach, A., Schmon, S. M., Degiacomi, M. T. & Willcocks, C. G. Denoising diffusion probabilistic models on SO(3) for rotational alignment. In ICLR 2022 Workshop onf Geometrical and Topological Representation Learning (2022).Godwin, J. et al. Simple GNN regularisation for 3D molecular property prediction and beyond. In International Conference on Learning Representations (2022).Jiao, R. et al. An equivariant pretrained transformer for unified 3d molecular representation learning. https://github.com/jiaor17/EPT (2025).



Source link