Snowflake
/

Llama-3.1-SwiftKV-405B-Instruct-FP8

aurick commited on 12 days ago

Commit

33d6612

verified ·

1 Parent(s): cabbe25

Set max_position_embeddings to 8192

Files changed (1) hide show

config.json CHANGED Viewed

@@ -17,7 +17,7 @@
   "initializer_range": 0.02,
   "intermediate_size": 53248,
   "key_value_group_size": 1,
-  "max_position_embeddings": 131072,
   "mlp_bias": false,
   "model_type": "llama_swiftkv",
   "num_attention_heads": 128,

   "initializer_range": 0.02,
   "intermediate_size": 53248,
   "key_value_group_size": 1,
+  "max_position_embeddings": 8192,
   "mlp_bias": false,
   "model_type": "llama_swiftkv",
   "num_attention_heads": 128,