<jupyter_start><jupyter_text>ELECTRALM可用的中文预训练参数：[`electra-small`](https://drive.google.com/file/d/1uab-9T1kR9HgD2NB0Kz1JB_TdSKgJIds/view?usp=drive_open),[`electra-small-ex`](https://drive.google.com/file/d/1LluPORc7xtFmCTFR4IF17q77ip82i7__/view),[`electra-base`](https://drive.google.com/file/d/1FMwrs2weFST-iAuZH3umMa6YZVeIP8wD/view),[`electra-large`](https://drive.google.com/file/d/1ny0NMLkEWG6rseDLiF_NujdHxDcIN51m/view)<jupyter_code>import uf

print(uf.__version__)
model = uf.ELECTRALM('../../demo/vocab.txt')
print(model)
X = [['天亮以前说再见，', '笑着泪流满面，', '去迎接应该你的，', '更好的明天。'],    # 每一条样本是一个doc，doc内可以由多个句子组成
     '他想知道那是谁, 为何总沉默寡言, 人群中也算抢眼, 抢眼的孤独难免']    # doc也可以是一个完整的文段<jupyter_output><empty_output><jupyter_text>训练<jupyter_code>model.fit(X, total_steps=20)<jupyter_output>INFO:tensorflow:Training on Generator and Discriminator. (Pass `electra_objective=False` to exclude Discriminator)
WARNING:tensorflow:From /Users/geyingli/Library/Python/3.8/lib/python/site-packages/tensorflow/python/util/dispatch.py:1096: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.<jupyter_text>如果希望如同论文所述，仅训练Generator，传递一个参数即可<jupyter_code>model.fit(X, total_steps=40, electra_objective=False)<jupyter_output>INFO:tensorflow:Training on Generator, with Discriminator frozen. (Pass `electra_objective=True` to include Discriminator)
INFO:tensorflow:Running training on 2 samples (step 20 -> 40)
INFO:tensorflow:step 21, MLM accuracy 0.3000, RTD accuracy 0.7910, MLM loss 2.196173, RTD loss 1.447132, 0.13 steps/sec, 0.25 examples/sec
INFO:tensorflow:step 22, MLM accuracy 0.3000, RTD accuracy 0.7910, MLM loss 2.158587, RTD loss 1.507510, 1.41 steps/sec, 2.83 examples/sec
INFO:tensorflow:step 23, MLM accuracy 0.3000, RTD accuracy 0.7910, MLM loss 2.112243, RTD loss 1.525882, 1.42 steps/sec, 2.83 examples/sec
INFO:tensorflow:step 24, MLM accuracy 0.3000, RTD accuracy 0.7910, MLM loss 2.100240, RTD loss 1.495942, 1.35 steps/sec, 2.69 examples/sec
INFO:tensorflow:step 25, MLM accuracy 0.3000, RTD accuracy 0.7910, MLM loss 2.057068, RTD loss 1.530674, 1.28 steps/sec, 2.55 examples/sec
INFO:tensorflow:step 26, MLM accuracy 0.3000, RTD accuracy 0.7910, MLM loss 2.014315, RTD loss 1.551594, 1.19 steps/sec,[...]<jupyter_text>推理<jupyter_code>X_tokenized = [['天', '亮', '以', '[MASK]', '说', '再', '见']]    # 可以手动赋予Mask
model.predict(X_tokenized=X_tokenized)<jupyter_output>INFO:tensorflow:Time usage 0m-2.59s, 0.39 steps/sec, 0.39 examples/sec
StarcoderdataJupyterStructuredCleanDedup