README.md

---
library_name: transformers
license: mit
datasets:
  - chandar-lab/UR100P
language:
  - en
tags:
  - biology
---

> [!NOTE]
> This model has been optimized using NVIDIA's [TransformerEngine](https://github.com/NVIDIA/TransformerEngine)
> library. Slight numerical differences may be observed between the original model and the optimized
> model. For instructions on how to install TransformerEngine, please refer to the
> [official documentation](https://github.com/NVIDIA/TransformerEngine?tab=readme-ov-file#installation).

# AMPLIFY (TransformerEngine-Optimized) Overview

## Description:

AMPLIFY is an efficient, state-of-the-art protein language model (pLM). AMPLIFY can generate residue and protein
embeddings, suggest mutations, differentiate disordered proteins from non-protein sequences. AMPLIFY is available in two
sizes, 120M and 350M parameters.

This version of the AMPLIFY model is optimized with NVIDIA's
[TransformerEngine](https://github.com/NVIDIA/TransformerEngine) library. It is based on the original AMPLIFY model from
Chandar Research Lab (CRL), and (within numerical precision) has identical weights and outputs.

This model is ready for commercial/non-commercial use.

## Third-Party Community Consideration

This model is not owned or developed by NVIDIA. This model has been developed and built to a third-party's requirements
for this application and use case; see link to Non-NVIDIA [AMPLIFY Model
Card](https://huggingface.co/chandar-lab/AMPLIFY_350M).

### License/Terms of Use:

AMPLIFY is provided under the [MIT license](https://github.com/chandar-lab/AMPLIFY/blob/main/LICENSE).

### Deployment Geography:

Global

### Use Case:

Protein design, mutation prediction, and function analysis.

### Release Date:

Hugging Face 06/12/2025 via [https://huggingface.co/nvidia/AMPLIFY_350M](https://huggingface.co/nvidia/AMPLIFY_350M)

## References:

- [Protein Language Models: Is Scaling
  Necessary?](https://www.biorxiv.org/content/biorxiv/early/2024/09/23/2024.09.23.614603.full.pdf) - detailed
  information on the model architecture and training data.

## Model Architecture:

**Architecture Type:** Transformer
**Network Architecture:** ESM-2

**This model was developed based on:** [AMPLIFY](https://huggingface.co/chandar-lab/AMPLIFY_350M) <br>
**Number of model parameters:** 3.5 x 10^8

## Input:

**Input Type:** Text (Protein Sequences) <br>
**Input Format:** String <br>
**Input Parameters:** One-Dimensional (1D) <br>
**Other Properties Related to Input:** Protein sequence represented as a string of canonical amino acids. The maximum
context length is 2048 residues.

## Output:

**Output Type:** Embeddings (Amino acid and sequence-level) <br>
**Output Format:** Numeric vector <br>
**Output Parameters:** One-Dimensional (1D) <br>
**Other Properties Related to Output:** Numeric vector with floating-point values corresponding to an embedding for each
amino acid in the input protein sequence.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware
(e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times
compared to CPU-only solutions.

## Software Integration:

**Runtime Engines:**

- Hugging Face Transformers

**Supported Hardware Microarchitecture Compatibility:**

- NVIDIA Ampere
- NVIDIA Blackwell
- NVIDIA Hopper

**Preferred Operating System(s):**

- Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific
data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at
both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure
compliance with safety and ethical standards before deployment.

## Model and checkpoint versions are noted below:

- [AMPLIFY_350M](https://huggingface.co/nvidia/AMPLIFY_350M) <br>
- [AMPLIFY_120M](https://huggingface.co/nvidia/AMPLIFY_120M) <br>

**Get Started**

```python
from transformers import AutoModel
from transformers import AutoTokenizer
from datasets import load_dataset

# Load AMPLIFY and tokenizer
model = AutoModel.from_pretrained("nvidia/AMPLIFY_350M", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(
    "nvidia/AMPLIFY_350M", trust_remote_code=True
)

# Move the model to GPU (required due to Flash Attention)
model = model.to("cuda")

# Load the UniProt validation set
dataset = load_dataset("chandar-lab/UR100P", data_dir="UniProt", split="test")

for sample in dataset:
    # Protein
    print("Sample: ", sample["name"], sample["sequence"])

    # Tokenize the protein
    input = tokenizer.encode(sample["sequence"], return_tensors="pt")
    print("Input: ", input)

    # Move to the GPU and make a prediction
    input = input.to("cuda")
    output = model(input)
    print("Output: ", output)

    break
```

## Training and Evaluation Datasets:

## Training Datasets:

**Link:** [UniRef100](https://www.uniprot.org/uniref?query=identity%3A1.0)

**Data Modality:**

- Text (Protein Sequences)

**Text Training Data Size:**

- 1 Billion to 10 Trillion Tokens

**Data Collection Method:**

- Human

**Labeling Method:**

- N/A

**Properties (Quantity, Dataset Descriptions, Sensor(s)):** UniRef100 contains all records in the UniProt Knowledgebase
and selected UniParc records. In UniRef100, identical sequences and subfragments are placed into a single cluster using
the CD-HIT algorithm. The longest members of the cluster (seed sequences) are used to generate UniRef90. However, the
longest sequence is not always the most informative. There is often more biologically relevant information and
annotation (name, function, cross-references) available on other cluster members. All the proteins in each cluster are
ranked to facilitate the selection of a biologically relevant representative for the cluster.

**Link:** [Observed Antibody Space (OAS)](https://opig.stats.ox.ac.uk/webapps/oas/downloads_paired/)

**Data Modality:**

- Text (Protein Sequences)

**Text Training Data Size:**

- 1 Billion to 10 Trillion Tokens

**Data Collection Method:**

- Human

**Labeling Method:**

- Human

**Properties:** The Observed Antibody Space (OAS) database is a project to collect and annotate immune repertoires for
use in large-scale analysis. It currently contains over one billion sequences, from over 80 different studies. These
repertoires cover diverse immune states, organisms (primarily human and mouse), and individuals.

**Link:** [Structural Classification of Proteins (SCOP)](https://www.ebi.ac.uk/pdbe/scop/download)

**Data Modality:**

- Text (Protein Sequences)

**Text Training Data Size:**

- 1 Billion to 10 Trillion Tokens

**Data Collection Method:**

- Hybrid: Human, Automated

**Labeling Method:**

- Hybrid: Human, Automated

**Properties:** The main levels of classification in SCOP are:

- Class: Groups proteins based on their secondary structure content, such as all-alpha, all-beta, alpha/beta, and
  alpha+beta.
- Fold: Proteins within the same fold have the same major secondary structures arranged in the same way with the same
  topological connections.
- Superfamily: Groups protein domains with a probable common evolutionary ancestry based on shared structural and
  functional features, even if sequence similarity is low.
- Family: Groups closely related proteins with clear evidence of a common evolutionary origin, often detectable through
  sequence comparison methods.
- Species: Represents a distinct protein sequence.
- Protein: Groups similar sequences with the same function.

## Evaluation Datasets:

**Link:** [Continuous Automated Model EvaluatiOn (CAMEO)](https://pmc.ncbi.nlm.nih.gov/articles/PMC8673552/)

**Benchmark Score:** LR P@L of 20.9±15.7

**Data Collection Method:**

- Human

**Labeling Method:**

- N/A

**Properties:** The data is collected by taking sequences of protein structures that are about to be released weekly by
the Protein Data Bank (PDB). These sequences are sent as "blind targets" to participating protein structure prediction
servers, which then return their predictions.

**Link:** [CASP14 (Critical Assessment of Methods of Protein Structure
Prediction)](https://pubmed.ncbi.nlm.nih.gov/34533838/)

**Benchmark Score:** LR P@L of 16.6±13.6

**Data Collection Method:**

- Human

**Labeling Method:**

- N/A

**Properties:** The data for CASP14 targets is collected from protein structures that are newly solved by experimental
structural biologists. The CASP organizers receive the amino acid sequences of these proteins before their full,
three-dimensional structures are publicly released in the Protein Data Bank (PDB). They then provide these sequences to
participating research groups and servers, who must submit their predicted structures within a specific time frame.

**Link:** [CASP15 (Critical Assessment of Methods of Protein Structure
Prediction)](https://pubmed.ncbi.nlm.nih.gov/37920879/)

**Benchmark Score:** LR P@L of 20.0±14.6

**Data Collection Method:**

- Human

**Labeling Method:**

- N/A

**Properties:** The data for CASP15 targets is collected from protein structures that are newly solved by experimental
structural biologists. The CASP organizers receive the amino acid sequences of these proteins before their full,
three-dimensional structures are publicly released in the Protein Data Bank (PDB). They then provide these sequences to
participating research groups and servers, who must submit their predicted structures within a specific time frame.

## Inference:

**Acceleration Engine:**

- Hugging Face Transformers

**Test Hardware:**

- A100
- H100
- H200
- GB200

## Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable
development for a wide array of AI applications. When downloaded or used in accordance with our terms of service,
developers should work with their internal model team to ensure this model meets requirements for the relevant industry
and use case and addresses unforeseen product misuse.

Users are responsible for ensuring the physical properties of model-generated molecules are appropriately evaluated and
comply with applicable safety regulations and ethical standards.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns
[here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).