This repository contains AWS Inferentia2 and neuronx compatible checkpoints for Codestral-22B-v0.1. You can find detailed information about the base model on its Model Card.
This model has been exported to the neuron format using specific input_shapes and compiler parameters detailed in the paragraphs below.
It has been compiled to run on an inf2.24xlarge instance on AWS. Note that while the inf2.24xlarge has 12 cores, this compilation uses 12.
SEQUENCE_LENGTH = 4096 BATCH_SIZE = 4 NUM_CORES = 12 # each inferentia chip has 2 cores, e.g. inf2.48xlarge has 12 chips or 24 cores PRECISION = "bf16"