Transformers
PyTorch
code
English
custom_code
Dejiao Z commited on
Commit
ba0f032
·
1 Parent(s): fcf2699

updated readme

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -2,11 +2,12 @@
2
  license: apache-2.0
3
  datasets:
4
  - bigcode/the-stack-v2
5
- -tiiuae/falcon-refinedweb
6
 
7
  library_name: transformers
8
  language:
9
  - code
 
10
  ---
11
 
12
  ## SageLite-s
@@ -66,9 +67,8 @@ SageLite is a new family of open embedding models with an encoder architecture t
66
 
67
 
68
  ### Training Data
69
- This checkpoint is trained on both [The-Stack-v2](https://huggingface.co/datasets/bigcode/the-stack-v2) and [Falcon-refinedweb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb).
70
 
71
- Stack data (https://huggingface.co/datasets/bigcode/the-stack-dedup). Supported languages (15 in total) are as follows: english (for text-only task), c, c-sharp, go, java, javascript, typescript, php, python, ruby.
72
 
73
  ### Training procedure
74
  This checkpoint is first trained on code data via masked language modeling (MLM), followed by two-stage contrastive learning -- constrastive pre-finetuning on large amount of positive pairs mined from the internet and constrastive finetuning on a small amount of synthetic data.
 
2
  license: apache-2.0
3
  datasets:
4
  - bigcode/the-stack-v2
5
+ - tiiuae/falcon-refinedweb
6
 
7
  library_name: transformers
8
  language:
9
  - code
10
+ - text
11
  ---
12
 
13
  ## SageLite-s
 
67
 
68
 
69
  ### Training Data
70
+ This checkpoint is trained on both [The-Stack-v2](https://huggingface.co/datasets/bigcode/the-stack-v2) and [Falcon-refinedweb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb). Supported languages (15 in total) are as follows: english (for text-only task), c, c-sharp, go, java, javascript, typescript, php, python, ruby.
71
 
 
72
 
73
  ### Training procedure
74
  This checkpoint is first trained on code data via masked language modeling (MLM), followed by two-stage contrastive learning -- constrastive pre-finetuning on large amount of positive pairs mined from the internet and constrastive finetuning on a small amount of synthetic data.