Natural Language Processing

BioBERT is Bidirectional Encoder Representations from Transformers for Biomedical Text Mining. It is a domain-specific language representation model pre-trained on large-scale biomedical corpora. BioBERT significantly outperforms many text mining methods on the following three representative biomedical text mining tasks: biomedical named entity recognition, biomedical relation extraction and biomedical question answering.

Reference: PMID: 31501885

Contact: Lang Li

BERT is a transformer-based model that employs bidirectional training of Transformer encoders to achieve state-of-the-art performance in numerous natural language processing tasks at the time of its introduction. The development of BERT laid the groundwork for current generations of large language models, serving as a proof of concept for bidirectional transformer architectures. BERT is built upon the attention mechanism but deviates from the original encoder-decoder structure by focusing solely on the encoder. Its architecture consists of multiple layers of bidirectional self-attention and is pre-trained on a large corpus of text, including the entire Wikipedia and BookCorpus. BERT's key innovation is its ability to capture context from both directions, allowing for a deeper understanding of language semantics.

Reference: https://arxiv.org/abs/1810.04805

Contact: Lang Li

PubBERT is a variant of BERT specifically fine-tuned on a corpus of biomedical literature, such as PubMed articles, to enhance its performance in biomedical text processing tasks. This adaptation allows PubBERT to better understand and generate language pertinent to the biomedical field, improving upon the general BERT model for domain-specific applications.

Contact: Lang Li

Llama 2 is a series of large language models by Meta AI, designed for diverse NLP tasks, available in three sizes: 7 billion, 13 billion and 70 billion parameters.

Reference: https://arxiv.org/abs/2207.06947

Contact: Ping Zhang

Mistral is an advanced language model designed for various natural language processing tasks, available in two sizes: 7 billion and 8x7 billion parameters.

Reference: https://arxiv.org/abs/2307.02022

Contact: Steve Rust

Mixtral combines multiple 7 billion parameter models to enhance performance across diverse NLP tasks.

Reference: https://arxiv.org/abs/2307.02022

Contact: Steve Rust

Phi 2 is a powerful language model designed for natural language understanding and generation.

Reference: https://arxiv.org/abs/2306.11692

Contact: Steve Rust

Flan-T5 is a fine-tuned language model from Google, optimized for various NLP tasks, available in sizes Lg 780M and XXL 11B.

Reference: https://arxiv.org/abs/2210.11416

Contact: Steve Rust

Falcon is a high-performance language model optimized for speed and efficiency, available in sizes 7 billion and 40 billion parameters.

Reference: https://arxiv.org/abs/2208.13269

Contact: Steve Rust

GatorTron is a biomedical language model developed by the University of Florida, available in sizes S 345M, Medium 3.9B, and Large 9B.

Reference: https://arxiv.org/abs/2207.07177

Contact: Steve Rust

RoBERTa (Robustly optimized BERT approach) is a transformer-based model that improves on BERT by training with larger mini-batches and more data.

Reference: https://arxiv.org/abs/1907.11692

Contact: Steve Rust

Phi 3 is the latest iteration of the Phi series, designed for advanced natural language processing and understanding.

Reference: https://arxiv.org/abs/2308.11239

Contact: Steve Rust

MPT is a versatile transformer model available in sizes 7 billion and 30 billion parameters, designed for a wide range of NLP tasks.

Reference: https://arxiv.org/abs/2307.11590

Contact: Steve Rust

OPT (Open Pre-trained Transformer) is a series of transformer models designed for various NLP tasks, available in multiple sizes.

Reference: https://arxiv.org/abs/2205.01068

Contact: Steve Rust