Update README.md
Browse files
README.md
CHANGED
|
@@ -24,13 +24,37 @@ pipeline_tag: text-generation
|
|
| 24 |
---
|
| 25 |
|
| 26 |
## 13B-Ouroboros
|
| 27 |
-
Ouroboros is an experimental model based on Meta's LLaMA [v1] 13B base model
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
|
| 33 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
|
| 35 |
## Composition:
|
| 36 |
Ouroboros is comprised of 40 layers [LLaMAv1 13B standard] mixed at optimized
|
|
|
|
| 24 |
---
|
| 25 |
|
| 26 |
## 13B-Ouroboros
|
| 27 |
+
Ouroboros is an experimental model based on Meta's LLaMA [v1] 13B base model with a merging technique optimized per layer---
|
| 28 |
+
tags:
|
| 29 |
+
- llama
|
| 30 |
+
- alpaca
|
| 31 |
+
- vicuna
|
| 32 |
+
- uncensored
|
| 33 |
+
- merge
|
| 34 |
+
- mix
|
| 35 |
+
- airoboros
|
| 36 |
+
- openorca
|
| 37 |
+
- orcamini
|
| 38 |
+
- orca
|
| 39 |
+
- instruct
|
| 40 |
+
- mixtune
|
| 41 |
+
datasets:
|
| 42 |
+
- Open-Orca/OpenOrca
|
| 43 |
+
- anon8231489123/ShareGPT_Vicuna_unfiltered
|
| 44 |
+
- jondurbin/airoboros-uncensored
|
| 45 |
+
language:
|
| 46 |
+
- en
|
| 47 |
+
metrics:
|
| 48 |
+
- accuracy
|
| 49 |
+
pipeline_tag: text-generation
|
| 50 |
+
---
|
| 51 |
|
| 52 |
+
## 13B-Ouroboros
|
| 53 |
+
Ouroboros is an experimental model based on Meta's LLaMA [v1] 13B base model using a custom merging technique, tweaking
|
| 54 |
+
each layer's merge % based on internal tests against the PTB dataset, scoring ~26.31 according to internal evaluation
|
| 55 |
+
(6 samples, sequence length 1024; this testing is not empirical, it's a quick way to find near-optimum values). Testing,
|
| 56 |
+
evaluating, and remixing this model is absolutely permissible and even encouraged (within the bounds of Meta's LLaMAv1
|
| 57 |
+
license agreement); the more feedback the better we can tune our process! 😊
|
| 58 |
|
| 59 |
## Composition:
|
| 60 |
Ouroboros is comprised of 40 layers [LLaMAv1 13B standard] mixed at optimized
|