RACE Reading Comprehension Dataset Leaderboard.MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models.Local Knowledge Powered Conversational Agents.Large Scale Multi-Actor Generative Dialog Modeling.End-to-End Training of Neural Retrievers for Open-Domain Question Answering.BioMegatron: Larger Biomedical Domain Language Model.We developed efficient, model-parallel ( tensor, sequence, and pipeline), and multi-node pre-training of transformer based models such as GPT, BERT, and T5 using mixed precision.īelow are some of the projects where we have directly used Megatron: This repository is for ongoing research on training large transformer language models at scale. Megatron ( 1, 2, and 3) is a large, powerful transformer developed by the Applied Deep Learning Research team at NVIDIA.
