[Proposal] Migrate keras to tf.keras

I am proposing to change keras to tf.keras for better performance, better serving and add TPU support. Maybe we should re-write the whole project, clean up the code, add missing documents and so on.
Here are the features I am planning to add.

Multi-GPU/TPU support
Export model for Tensorflow Serving
Fine-tune Ability for W2V and BERT

@alexwwang @hahahu

I am not familiar with tf.keras yet so I read the document quickly.
After that I think we should first get clear with some questions and make an agreement:

Who would use kashgari and their purpose?

For this, I guess there may be two type, industrial and academic users.

What's the purpose for the quality of this project? Production level or toy level?

For this, I think a production level is the vision. However, a toy one would be more friendly for newbies to get involved in and make all kinds of experiments to prove their hypothesis.

So, what about keep the current one and make a new edition with tf.keras to give maximum choice freedom to users?

I think we should agree on those two questions before any change, too.

Who would use kashgari and their purpose?

Academic users Experiments to prove their hypothesis
NLP beginners Learn how to build an NLP project with production level code quality
NLP developers Build a production level classification/labeling model

What's the purpose of the quality of this project? Production level or toy level?

Product level is the vision. I think we should build a well organized and documented framework with high performance.

So, I think migrating to tf.keras is a good idea.

tf.keras provides the same API with keras
tf.keras provides better performance, easy to upgrade to tf2.0, easy to deploy the trained model on various platforms
Better training policy support

The only thing tf.keras not good at is switch keras backend to CNTK or Theano. But I think 99% of Kashgari's users don't need this feature. So I recommend maintaining one edition with tf.keras.

Mostly I agree with you, especially on your thoughts about users and vision
of this project, and advantages of TF.keras on technical level.
The only one thing I am not sure is if there're few nlp users that use
MXnet/CNTK as their main backend framework. So at least I suggest keeping
this edition which uses keras as backbone even we don't maintain it in near
future and the period of new TF.keras edition development. Maybe in future
there would be someone else who is interested in continuing develop it for
those two groups of users.
That's my opinion, say, never lock a door completely.
I think this is good to open source communities.
How do you think?
On Thu, 25 Apr 2019 at 19:05, Eliyar Eziz<EMAIL_ADDRESS>wrote:

I think we should agree on those two questions before any change, too.

Who would use kashgari and their purpose?

Academic users Experiments to prove their hypothesis
NLP beginners Learn how to build an NLP project with production
level code quality
NLP developers Build a production level classification/labeling
model

What's the purpose of the quality of this project? Production level
or toy level?

Product level is the vision. I think we should build a well
organized and documented framework with high performance.

So, I think migrating to tf.keras is a good idea.

tf.keras provides the same API with keras
tf.keras provides better performance, easy to upgrade to tf2.0, easy
to deploy the trained model on various platforms
Better training policy support

The only thing tf.keras not good at is switch keras backend to CNTK or
Theano. But I think 99% of Kashgari's users don't need this feature. So I
recommend maintaining one edition with tf.keras.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BrikerMan/Kashgari/issues/77#issuecomment-486627696,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGRFKS62J5PATIWBREA6XDPSGF6HANCNFSM4HIAZOGQ
.

Of course, will keep the keras version as the legacy version for those who want to use MXnet/CNTK.
Let's finish #76 and #69 and keep a legacy version (v0.1 ~ v0.3), Then start working on the new version from v0.4 with the tf.keras.

Ok. I've no problem.
On Thu, 25 Apr 2019 at 19:29, Eliyar Eziz<EMAIL_ADDRESS>wrote:

Of course, will keep the keras version as the legacy version for those who
want to use MXnet/CNTK.
Let's finish #76 https://github.com/BrikerMan/Kashgari/pull/76 and #69
https://github.com/BrikerMan/Kashgari/pull/69 and keep a legacy version
(v0.1 ~ v0.3), Then start working on the new version from v0.4 with the
tf.keras.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BrikerMan/Kashgari/issues/77#issuecomment-486634350,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGRFKXCT7KPVKMJHBGJ57TPSGIY7ANCNFSM4HIAZOGQ
.

In the future, tf.keras will be more closely integrated with Tensorflow2.
The improvement of computing performance and the convenient Tensorflow serving solution are of great significance to the engineering and landing of algorithm research.
We only need to keep the current version, and create a new project based on tf.keras. :D

I assume it would be easy to transfer most part of current models and layers code to support tf.keras justy by just modifying their import declaration sector.
Am I right?

I assume it would be easy to transfer most part of current models and layers code to support tf.keras justy by just modifying their import declaration sector.
Am I right?

It will be easy for most of the model. But we need to wait for keras-bert and keras-gpt-2 to support tf.keras.
Actually, I am hoping to fully rewrite && clean up our code, change model save function to this https://www.tensorflow.org/alpha/guide/saved_model .

Ok, I will check it later.
On Thu, 9 May 2019 at 19:27, Eliyar Eziz<EMAIL_ADDRESS>wrote:

I assume it would be easy to transfer most part of current models and
layers code to support tf.keras justy by just modifying their import
declaration sector.
Am I right?
It will be easy for most of the model. But we need to wait for keras-bert
and keras-gpt-2 to support tf.keras.
Actually, I am hoping to fully rewrite && clean up our code, change model
save function to this https://www.tensorflow.org/alpha/guide/saved_model .
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BrikerMan/Kashgari/issues/77#issuecomment-490864841,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGRFKSMCBEC7T36NRIY4SDPUQDARANCNFSM4HIAZOGQ
.

https://github.com/HighCWu/keras-bert-tpu
This project is a fork of CyberZHG/keras_bert which supports Keras BERT on TPU. I have tried it and found it's not perfectly compatible with the ours.

@HaoyuHu I have communicated with CyberZHG. He is planning to support tf.keras, I think we just need to focus on how to rewrite and clean up our code.
Here is the issue,  https://github.com/CyberZHG/keras-ordered-neurons/issues/1

Interesting. That will be fine if  he is planning to support tf.keras.
BTW: Instead of keras.utils.multi_gpu_model(), can we consider using distribution strategies(such as MirroredStrategy) which are recommended in TF2.0 to support Multi-GPU?
https://github.com/tensorflow/tensorflow/issues/26245#issuecomment-474611589

@HaoyuHu Yes, distribution strategies is a more elegant way to support multi-gpu/multi-device/ TPU training. We definitely should do that.

I have tried tf.keras for a classifier model, tf2.0 train tooks 9s for each epoch while tf1.13 need 68s. Prediction is 10x fast with tf2.0, too.

Exciting! For my use case, about 8 times faster.

Seems awesome!
On Mon, May 13, 2019, 01:01 hahahu<EMAIL_ADDRESS>wrote:

Exciting! For my use case, about 8 times faster.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BrikerMan/Kashgari/issues/77#issuecomment-491611921,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGRFKSFEZSCJ6AOA3ZT4U3PVBELZANCNFSM4HIAZOGQ
.

Added a new branch tf.keras-version for tf.keras version. Planning to start from beginning.

Good to go!

Roll on.

Clean up our code, add document with sphinx.
Add a preprocessor class to build dict for tokens and labels, which will able to dump index and numerized data to pickle file for reuse. It will be very handy when we need to compare several models' performance.
Set sequence length to None, Allow various length input.
Save the model as tf.saved_model format.

I think the training process for the professional user should like this.
x, y = Corpus.load_data()

embedding = XXEmbedding()
# Save numerized input output(without padding) data and dict to cache_path.
embedding.build_on_corpus(x, y, cache_path='./cached_data')

classifier = CNNClassifier(embedding)
classifier.build_model(cached_data_path)
# Change model to keras_model
keras_model = classifier.keras_model

classifier.fit()

And for beginners
x, y = Corpus.load_data()

embedding = XXEmbedding()

classifier = CNNClassifier(embedding)
classifier.fit(x, y)

Added document to readthedocs.io, here is the link https://kashgari.readthedocs.io/en/tf.keras-version

I have added a pre-processor class, added document, and test-cases. Feel free to check it out and tell me if anything needs to change. Let's make Kashgari better.

Two thoughts by far:

For benchmarking models, since we've decided building kash on tf.keras, maybe we could take advantage of tf.server for models benchmarking in the preprocessor class or make it a helper/utility. How does it sound like?

Refer to sequence length, my opinion is it would be safer that as a default value to set to 90% pctg of raw sequences length. As long as throw out a warning of using default value and complete the document, users may aware this rule. How do you think?

Two thoughts by far:

For benchmarking models, since we've decided building kash on tf.keras, maybe we could take advantage of tf.server for models benchmarking in the preprocessor class or make it a helper/utility. How does it sound like?
Refer to sequence length, my opinion is it would be safer that as a default value to set to 90% pctg of raw sequences length. As long as throw out a warning of using default value and complete the document, users may aware this rule. How do you think?

Yea. I agree with both Ideas. As far as sequence length as None. Maye we could make it an option for pro users.

One more thing, I think we should consider adding the text-generating task as our third task, supporting classification, labeling and generating. Any suggestions about this idea. @alexwwang @HaoyuHu

Just added a demo labeling model. checkt it out. https://github.com/BrikerMan/Kashgari/blob/8c560551a9bc9e8841138adac80c59c53e73ad7a/kashgari/tasks/labeling/models.py#L122
Remember to check the document too. Link for demo document

I'm not familiar with text-generating task, what's your opinion @alexwwang ?

Unfortunately, me neither. AFAIK, text-generating task relates to GAN, of some different models with classification or labeling. I should refer to articles in topic GAN and related corpus before diving into this, but I know little details about them by far.

@alexwwang I have added multi-input support for all the embedding layers for tf-keras version. User's can use the multi-input function without changing any code. And going to add a multi-output example so that user can easily change model to support their data.

@BrikerMan Great to hear that! Did you implement it by modifying data generator method? I cannot help to see how you do.

@BrikerMan Great to hear that! Did you implement it by modifying data generator method? I cannot help to see how you do.

I had added a new class called Processor. Each model has its own embedding, and each embedding contains a processor. We could replace the processor without changing any code.

I see, decoupling. Wait for your push!
On Tue, 21 May 2019 at 17:49, Eliyar Eziz<EMAIL_ADDRESS>wrote:

@BrikerMan https://github.com/BrikerMan Great to hear that! Did you
implement it by modifying data generator method? I cannot help to see how
you do.
I had added a new class called Processor. Each model has its own
embedding, and each embedding contains a processor. We could replace the
processor without changing any code.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKUAILQDW7RX53EZTC3PWPATBA5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV3MBNY#issuecomment-494321847,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGRFKR4KWKYPH66O232EMLPWPATBANCNFSM4HIAZOGQ
.

@alexwwang Pushed, check it out.

Now we could finalize the tf.keras version's release.
Todo list:

[ ] add all classification models & tests.
[ ] add CRF labeling model & tests .
[ ] add document for basic classification and labeling.
[ ] add document for advance use-case [optional].
[ ] add docstring to public function.

Which part you guys want to work on? @alexwwang @HaoyuHu

One more thing, should we simplify our basic models by decrease the neuron count?

Try best to do some part I'm familiar with. 😄

I think we could learn from keras-bert project, to build a middle layer to hold all backend's dependents.

Try best to do some part I'm familiar with. 😄

How about add classification models, test and doc?
Here is the labeling tutorial doc

I could dive in from here.
On Wed, Jun 12, 2019 at 4:01 PM Eliyar Eziz<EMAIL_ADDRESS>wrote:

Try best to do some part I'm familiar with. 😄
How about add classification models, test and doc?
Here is the labeling tutorial doc
https://kashgari.readthedocs.io/en/tf.keras-version/tutorial/sequence_labeling_model.html
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKV4KEMWH7LYCOOEHZDP2CUL3A5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXPSRHA#issuecomment-501164188,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGRFKSPNJTYLATXTW37Y23P2CUL3ANCNFSM4HIAZOGQ
.

@BrikerMan Sorry for the late reply. I'm interested in advance use-case. At the same time, I can participate in some other documentation work.

@HaoyuHu Here is several advance use-case examples, maybe you could start from here, fell free to update documents about these use-cases.

Entity Relation Extraction
Customize Multi Output Model
Handle Numeric features

@alexwwang @HaoyuHu I am thinking about rename all models name for readability,
maybe BLSTMCRFModel to Bi_LSTM_CRF_Model. Any idea?

@HaoyuHu Here is several advance use-case examples, maybe you could start from here, fell free to update documents about these use-cases.

Entity Relation Extraction
Customize Multi Output Model
Handle Numeric features

I will delve into these. :smile:

@alexwwang @HaoyuHu I am thinking about rename all models name for readability, Any idea?
Here are several styles I found. For example, renaming BLSTMCRFModel:
Bi_LSTM_CRF_Model
BiLSTM_CRF_Model
Bi_LSTM_CRF                  # I prefer this one
BiLSTM_CRF

I prefer the last two.

I think the second and the last is the most concise.
The question is whether _Model suffix  is necessary.

I think the second and the last is the most concise.
The question is whether _Model suffix  is necessary.

I think you are right, let’s keep the Mode. Use the Second one. Camel Case + underline between uppercase names.

Camel case for a structure name of a main unit in model and underline for combination of all units a model utilized and last model to indicate it's a model since similar naming rule is used for layer part but without _Model suffix.
It's my thought. Any better idea? @BrikerMan @HaoyuHu

Camel case for a structure name of a main unit in model and underline for combination of all units a model utilized and last model to indicate it's a model since similar naming rule is used for layer part but without _Model suffix.
It's my thought. Any better idea? @BrikerMan @HaoyuHu

Could you give several samples?

Models:
BiLSTM_Model
CNN_LSTM_Model
AVCNN_Model
KMaxCNN_Model
RNN_CNN_Model
AVRNN_Model
DropoutBiGRU_Model
DPCNN_Model
Layers:
AttentionWeightedAverage
KMaxPooling
CRF

Models:
BiLSTM_Model
CNN_LSTM_Model
AVCNN_Model
KMaxCNN_Model
RNN_CNN_Model
AVRNN_Model
DropoutBiGRU_Model
DPCNN_Model
Layers:
AttentionWeightedAverage
KMaxPooling
CRF

Look good to me. Let's use this.

@alexwwang Are u working on classification models, If not I am planning to work on classifications.

Yes. And you may write test for the new models. Thanks in advance.
On Mon, 24 Jun 2019 at 16:42, Eliyar Eziz<EMAIL_ADDRESS>wrote:

@alexwwang https://github.com/alexwwang Are u working on classification
models, If not I am planning to work on classifications.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKX3IDENH6JDMAEQTZ3P4CCHLA5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYMGXTY#issuecomment-504916943,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGRFKUPWNSCHIZ35C647TLP4CCHLANCNFSM4HIAZOGQ
.

@alexwwang @HaoyuHu How about releasing a new version with package name kashgari-tf, and update keras version with name kashgari. I tried that and it works, check it out. https://pypi.org/project/kashgari-tf/

@BrikerMan I have fix a bug in https://github.com/BrikerMan/Kashgari/pull/108, but that part of the code was modified in 9e72bbb6060449cd4562b030b2996a4a70eed4f6. Then the problem reappears.
reproduce:
import kashgari
from kashgari.corpus import ChineseDailyNerCorpus
from kashgari.tasks.labeling import BiLSTM_Model

train_x, train_y = ChineseDailyNerCorpus.load_data('train', shuffle=False)
model = BiLSTM_Model()

model.build_model(train_x, train_y)

model = kashgari.utils.convert_to_multi_gpu_model(model, gpus=2)
model.fit(train_x, train_y)

@HaoyuHu Sorry man, it must be a mistake.

So that means to keep developing original Keras edition with future
features?
Any routine plan or consideration to share?
Let me give out a brick to bring golds.
I think the better idea is to make an abstract up-level program to define
embeddings, layers, models even optimizers and data processors, and build a
midware to stick keras backend, and tf-keras to the up-level  program.
Thus we don't have to maintain two projects, to do duplicate works in each
branch.
How do you think?
On Wed, Jun 26, 2019, 20:36 Eliyar Eziz<EMAIL_ADDRESS>wrote:

@alexwwang https://github.com/alexwwang @HaoyuHu
https://github.com/HaoyuHu How about releasing a new version with
package name kashgari-tf, and update keras version with name kashgari. I
tried that and it works, check it out.
https://pypi.org/project/kashgari-tf/
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKTBIC6XJOFKOVJPF2DP4NPFFA5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYTMAEI#issuecomment-505856017,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGRFKV3LF5SZIMTDEIN4CTP4NPFFANCNFSM4HIAZOGQ
.

@alexwwang No, I think we should keep the old version so that we will not break the user's code. keras version will be maintained for a while without new features.
Supporting both keras and tf.keras sounds good. But I am worried it will cost lots of time to maintain.

@BrikerMan I agree with you now. The proposal I drew this afternoon appears to be too ideal and I'm afraid it won't ever come true in that simple way.

@BrikerMan I agree with you now. The proposal I drew this afternoon appears to be too ideal and I'm afraid it won't ever come true in that simple way.

Yes, now the only thing we should discuss is where to use the package name kashgari-tf.

I am not sure yet.
I need some time to figure out the reasons to those known and unknown
errors first.
On Thu, 27 Jun 2019 at 19:00, Eliyar Eziz<EMAIL_ADDRESS>wrote:

@BrikerMan https://github.com/BrikerMan I agree with you now. The
proposal I drew this afternoon appears to be too ideal and I'm afraid it
won't ever come true in that simple way.
Yes, now the only thing we should discuss is where to use the package name
kashgari-tf.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKSOWMGKXSWLFZEPCH3P4SMV3A5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYWYMRA#issuecomment-506299972,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGRFKV446E5TXGFQKBCXETP4SMV3ANCNFSM4HIAZOGQ
.

@BrikerMan I have fix a bug in #108, but that part of the code was modified in 9e72bbb. Then the problem reappears.
reproduce:
import kashgari
from kashgari.corpus import ChineseDailyNerCorpus
from kashgari.tasks.labeling import BiLSTM_Model

train_x, train_y = ChineseDailyNerCorpus.load_data('train', shuffle=False)
model = BiLSTM_Model()

model.build_model(train_x, train_y)

model = kashgari.utils.convert_to_multi_gpu_model(model, gpus=2)
model.fit(train_x, train_y)

Could you help me fix this bug again, I have no access to multi-gpu device now. I can't reproduce this issue.

@alexwwang Maybe we just release tf.keras version with less classification model, Then consider how to add more models.

@BrikerMan I have fix a bug in #108, but that part of the code was modified in 9e72bbb. Then the problem reappears.
reproduce:
import kashgari
from kashgari.corpus import ChineseDailyNerCorpus
from kashgari.tasks.labeling import BiLSTM_Model

train_x, train_y = ChineseDailyNerCorpus.load_data('train', shuffle=False)
model = BiLSTM_Model()

model.build_model(train_x, train_y)

model = kashgari.utils.convert_to_multi_gpu_model(model, gpus=2)
model.fit(train_x, train_y)

Could you help me fix this bug again, I have no access to multi-gpu device now. I can't reproduce this issue.

I prefer to add a member function BaseModel.build_multi_gpu_model(...) instead of current utils.convert_to_multi_gpu_model(...). This will make it easier to solve this problem.

@alexwwang Maybe we just release tf.keras version with less classification model, Then consider how to add more models.

I may pass the test for those models refactored today. Then the new edition could be launched, I think. Pls remember to help me clarify the trouble I met during the #118 work.

problem

Ok, if so let's add BaseModel.build_multi_gpu_model(...) and BaseModel.build_tpu_model(...)

@alexwwang will do, one more thing, customize layer issue may be related to tensorflow version. My CRF layer works fine at first, but now it will crash #120 . I need to fix this. Let's work on tensorflow==1.14.0

@BrikerMan Yes, under tf1.14 and I've solved the customize layer issues related yesterday. They're related to type conversion and difference of api between tf.keras and Keras. Maybe we met different situations.

My CRF layer work on 1.13.1, but will crash at 1.14... Let's specify a tf-version for every Kashgari releases. Or we will have to deal with endless problems. This version lets work on 1.14.0.

I decide to make a qr first, and you will know where the problem may lay on.
Are you sure all the weights you add in CRF without setting trainable?

@alexwwang Here is still 4 error with classification models, please check it out.

@BrikerMan where could I see the error info?

@BrikerMan where could I see the error info?

https://travis-ci.com/BrikerMan/Kashgari

Thanks.
Yes, an error occurred when refactor the struct codes. And it has been corrected.
I'll make a pr right now.

@alexwwang @HaoyuHu All issue fixed, test-case passed. Now it is time to finally finalize this release. Here is the new todo list.

[ ] Add convert model to multi-gpu/TPU function to base-model class
[ ] Clean up the code
[ ] Add classification tutorial
[ ] Add embedding tutorial
[ ] choose pip package name, kashgari or kashgari-tf

If we use kashgari, it will break the existing code. But if we use kashgari-tf, we need to remind users to migrate to kashgari-tf.

Or a whole new name, say, shouwen, named as a Chinese monk in 9th century?
On Fri, 28 Jun 2019 at 22:36, Eliyar Eziz<EMAIL_ADDRESS>wrote:

@alexwwang https://github.com/alexwwang @HaoyuHu
https://github.com/HaoyuHu All issue fixed, test-case passed. Now it is
time to finally finalize this release. Here is the new todo list.

Add convert model to multi-gpu/TPU function to base-model class
Clean up the code
Add classification tutorial
Add embedding tutorial
choose pip package name, kashgari or kashgari-tf

If we use kashgari, it will break the existing code. But if we use
kashgari-tf, we need to remind users to migrate to kashgari-tf.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKTMUWMZ5M3I52YGKILP4YOYHA5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY2IBWQ#issuecomment-506757338,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGRFKQA2OZJ3QDJSNJQSNLP4YOYHANCNFSM4HIAZOGQ
.

@alexwwang @HaoyuHu All issue fixed, test-case passed. Now it is time to finally finalize this release. Here is the new todo list.

[ ]  Clean up the code
[ ]  Add classification tutorial
[ ]  Add embedding tutorial
[ ]  choose pip package name, kashgari or kashgari-tf

If we use kashgari, it will break the existing code. But if we use kashgari-tf, we need to remind users to migrate to kashgari-tf.

Nice job guys. Don't forget also to add tutorials about multilabel and multioutput. Thanks.

@alexwwang I prefer stick to the name Kashgari.

Well then, -tf suffix may be better.
On Sat, 29 Jun 2019 at 10:17, Eliyar Eziz<EMAIL_ADDRESS>wrote:

@alexwwang https://github.com/alexwwang I prefer stick to the name
Kashgari.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKXFO35HFXL6264GDF3P43A4ZA5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY3PQYA#issuecomment-506919008,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGRFKVZ2FWO7D2SU22B3YDP43A4ZANCNFSM4HIAZOGQ
.

But don't you think it's a little long?
How about kash-tf?
On Sat, 29 Jun 2019 at 11:50, Alex Wang<EMAIL_ADDRESS>wrote:

Well then, -tf suffix may be better.
On Sat, 29 Jun 2019 at 10:17, Eliyar Eziz<EMAIL_ADDRESS>wrote:

@alexwwang https://github.com/alexwwang I prefer stick to the name
Kashgari.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKXFO35HFXL6264GDF3P43A4ZA5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY3PQYA#issuecomment-506919008,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGRFKVZ2FWO7D2SU22B3YDP43A4ZANCNFSM4HIAZOGQ
.

@alexwwang We could use kashgari-tf as package name. But still use import Kashgari in the code. Just like tensorflow-gpu.

@alexwwang @HaoyuHu Guys. Do you have any time this weekend to work on the document. I am planning to release the first tf.keras version this weekend. With kashgari-tf package name with 0.5.0 version.

Sorry but I could only have time working on it on week days.
And I have to check the program carefully to make it clear how to use it,
especially on stacked-embedding, hyper parameter settings, the optimizor
part, and model save with customized layers. These parts are not covered by
unit test yet.
On Sat, 29 Jun 2019 at 11:59, Eliyar Eziz<EMAIL_ADDRESS>wrote:

@alexwwang https://github.com/alexwwang @HaoyuHu
https://github.com/HaoyuHu Guys. Do you have any time this weekend to
work on the document. I am planning to release the first tf.keras version
this weekend. With kashgari-tf package name with 0.5.0 version.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKQPNO2643K72DLAW3TP43M3LA5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY3QYNQ#issuecomment-506924086,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGRFKTR2YCI75SARO5CQ23P43M3LANCNFSM4HIAZOGQ
.

Sorry but I could only have time working on it on week days.
And I have to check the program carefully to make it clear how to use it,
especially on stacked-embedding, hyper parameter settings, the optimizor
part, and model save with customized layers. These parts are not covered by
unit test yet.

Yes. I am a little bit in rush. Let’s cool down. I will finish embedding tutorial this weekend. Take your time, I will try add more test case too. Let’s release a little bit later.

Thank you for your understanding.
I plan to implement the DPCNN model next week. I think it's interesting and
do fast and well in certain classification tasks.
On Sat, 29 Jun 2019 at 20:36, Eliyar Eziz<EMAIL_ADDRESS>wrote:

Sorry but I could only have time working on it on week days.
And I have to check the program carefully to make it clear how to use it,
especially on stacked-embedding, hyper parameter settings, the optimizor
part, and model save with customized layers. These parts are not covered by
unit test yet.
Yes. I am a little bit in rush. Let’s cool down. I will finish embedding
tutorial this weekend. Take your time, I will try add more test case too.
Let’s release a little bit later.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKS7XGDB3O4T4KLLVRTP45JK7A5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY3YA4Y#issuecomment-506953843,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGRFKSF2VTPVBH7KBKUFULP45JK7ANCNFSM4HIAZOGQ
.

@alexwwang @HaoyuHu Guys. Do you have any time this weekend to work on the document. I am planning to release the first tf.keras version this weekend. With kashgari-tf package name with 0.5.0 version.

I am afraid that I don't have a lot of time to work on the document due to my busy work. I am so sorry. 😿
BTW, I am very much looking forward to the release of the tf.keras version.

Thank you for your understanding. I plan to implement the DPCNN model next week. I think it's interesting and do fast and well in certain classification tasks.
…
On Sat, 29 Jun 2019 at 20:36, Eliyar Eziz @.***> wrote: Sorry but I could only have time working on it on week days. And I have to check the program carefully to make it clear how to use it, especially on stacked-embedding, hyper parameter settings, the optimizor part, and model save with customized layers. These parts are not covered by unit test yet. Yes. I am a little bit in rush. Let’s cool down. I will finish embedding tutorial this weekend. Take your time, I will try add more test case too. Let’s release a little bit later. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#77?email_source=notifications&email_token=AAGRFKS7XGDB3O4T4KLLVRTP45JK7A5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY3YA4Y#issuecomment-506953843>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGRFKSF2VTPVBH7KBKUFULP45JK7ANCNFSM4HIAZOGQ .

I am also studying DPCNN recently. This is very nice if we can use DPCNN in a future release. That's sweet of you!

Anything else needs to check? finished our first checklist. Going to finalize the 0.5.0 release.

[x] Clean up the code
[x] Add classification tutorial
[x] Add embedding tutorial
[x] choose pip package name, kashgari-tf

@alexwwang Maybe we just get rid of those codes, since we have already broken our user's code and they need to start with the whole new version.
BLSTMModel = BiLSTM_Model
BGRUModel = BiGRU_Model
CNNModel = CNN_Model
... ...

🎉🎉🎉🎉 Released 0.5.0 🎉🎉🎉🎉

Any idea to support some short alias for those model names? I think that
would be friendly to users.
On Thu, Jul 11, 2019, 13:33 Eliyar Eziz<EMAIL_ADDRESS>wrote:

@alexwwang https://github.com/alexwwang Maybe we just get rid of those
codes, since we have already broken our user's code and they need to start
with the whole new version.
BLSTMModel = BiLSTM_Model
BGRUModel = BiGRU_Model
CNNModel = CNN_Model... ...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKXRWGUVRYXEZHD6GLDP63A2ZA5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZVRW4Y#issuecomment-510335859,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGRFKUXWVLK34CIYL6AMGTP63A2ZANCNFSM4HIAZOGQ
.

Any idea to support some short alias for those model names? I think that would be friendly to users.

Sounds good, any specific idea? Maybe we should open a new issue for this matter.

Ok. Let me see how to abridge them.
On Sat, Jul 13, 2019 at 16:57 Eliyar Eziz<EMAIL_ADDRESS>wrote:

Any idea to support some short alias for those model names? I think that
would be friendly to users.
Sounds good, any specific idea? Maybe we should open a new issue for this
matter.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKRXYLO4QRNPWR5OYQLP7GKGDA5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ3NKAY#issuecomment-511104259,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGRFKUGLCXRDJ3S7ZQ4NCTP7GKGDANCNFSM4HIAZOGQ
.


Encoding: object has no 'ShipmentOrder' property

Hi,
in first place I got an error with the dev-master version saying:
Array (     [0] => SECURITY_VIOLATION )
So i read https://github.com/Petschko/dhl-php-sdk/issues/64#issuecomment-600298840 and installed the petschko/dhl-php-sdk:dev-dev
After that I got (with the same credentials) this error:
Array (     [0] => SOAP-ERROR: Encoding: object has no 'ShipmentOrder' property )
Maybe therefore: https://github.com/Petschko/dhl-php-sdk/issues/64#issuecomment-615859975
The DHL Business Account is brand new, so I guess I have to use version 3 of the API.
The App is activated, and also ALL services are activated, so I retrieved an AppID and an App-Token.
When testing with sandbox credentials in the dev-master version everything was fine.
What to do?

I think the best way to find out whats the issue is when you send that code here or to me via E-Mail. Some error-messages of DHL are not that self explanatory
Yes you are right, you have to use the Dev-Branch when your DHL-Account is new (then you use Version 3), so the issue must be something different

Bump to see if this is still relevant

Bump to see if this is still relevant


Avoid using ref modifier in form

addresses #1181 by avoiding the ref helper

Thanks @basz!

This test is failing:

Yes, also for all other (dependabot) PRs, so unrelated. Needs to be investigated...


Adding Coinbase app

Displays your current Coinbase balance and holdings.

I have read the CLA Document and I hereby sign the CLA

This is live now. Thanks for the contribution!


Custom commit message before publish it to Github (feature request)

Option to edit the commit message before publish it from stackedit.io editor. The current default commit message is "Uploaded by https://stackedit.io/"

anyone working on this?

Stackedit is just great !
Whether or not it's a prenium feature, if it does not exist, it just does not make sense for Githubers.
Regarding its author, either it is deliberate or under prenium account.
+1

Stackedit is very cool indeed!
But I feel like custom github commit message would very very useful addition to the current version.


Enable services configuration tests

Purpose
$subject.

Codecov Report

Merging #15563 into http+io will increase coverage by 0.05%.
The diff coverage is 100%.

@@              Coverage Diff              @@
##             http+io   #15563      +/-   ##
=============================================
+ Coverage       5.91%    5.96%   +0.05%
- Complexity      1574     1603      +29
=============================================
  Files           1560     1560
  Lines          77762    77762
  Branches       10210    10210
=============================================
+ Hits            4596     4642      +46
+ Misses         72611    72574      -37
+ Partials         555      546       -9

Impacted Files
Coverage Δ
Complexity Δ

...main/java/org/ballerinalang/net/http/HttpUtil.java
31.63% <100%> (+1.78%)
78 <0> (+11)
:arrow_up:

...n/java/org/ballerinalang/net/http/HttpService.java
53.84% <0%> (ø)
32% <0%> (+1%)
:arrow_up:

...allerinalang/net/http/HttpResourceDataElement.java
28.24% <0%> (+0.76%)
14% <0%> (+1%)
:arrow_up:

...c/main/java/org/ballerinalang/net/uri/URIUtil.java
19.69% <0%> (+3.03%)
4% <0%> (+1%)
:arrow_up:

...g/ballerinalang/net/http/HTTPServicesRegistry.java
62.5% <0%> (+3.12%)
14% <0%> (+1%)
:arrow_up:

.../net/http/compiler/ResourceSignatureValidator.java
75% <0%> (+5%)
13% <0%> (+1%)
:arrow_up:

...allerinalang/net/uri/parser/URITemplateParser.java
44.82% <0%> (+5.17%)
10% <0%> (+2%)
:arrow_up:

...ballerinalang/net/http/HttpResourceDispatcher.java
39.47% <0%> (+5.26%)
4% <0%> (+1%)
:arrow_up:

...in/java/org/ballerinalang/net/uri/URITemplate.java
69.23% <0%> (+7.69%)
7% <0%> (+1%)
:arrow_up:

... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d975cd9...20842fe. Read the comment docs.


[BUG] deepspeed.init_inference() is Giving two Responses on 2 GPUs

Describe the bug
I am running the below-given code after putting it in model.py using deepspeed --num_gpus 2 model.py, the aim is to make the inference rate faster using two GPUs with the help of DeepSpeed inference. It runs well as expected but I am getting two same responses instead of one. I expect it to give me only one response.
To Reproduce
Steps to reproduce the behavior:

put the below code in model.py file:

import os
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
import deepspeed
import time

local_rank = int(os.getenv('LOCAL_RANK', '0'))
world_size = int(os.getenv('WORLD_SIZE', '2'))

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf", use_fast=True)
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf", torch_dtype=torch.float16, low_cpu_mem_usage=True,device_map=local_rank)

ds_model = deepspeed.init_inference(
         model=model, mp_size=world_size, dtype=torch.float16, replace_method="auto", replace_with_kernel_inject=True)

start = time.time()
prompt = 'Write a detailed note on AI'
inputs = tokenizer.encode(f"<human>: {prompt} \n<bot>:", return_tensors='pt').to(model.device)
outputs = ds_model.generate(inputs, max_new_tokens=2000)
output_str = tokenizer.decode(outputs[0])
print(output_str)
end = time.time()
print('Inference Time is:', end - start)

run deepspeed --num_gpus 2 model.py

ds_report output
[2023-08-15 20:30:08,150] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
async_io ............... [NO] ....... [OKAY]
fused_adam ............. [YES] ...... [OKAY]
cpu_adam ............... [YES] ...... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.0
 [WARNING]  using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/usr/local/lib/python3.10/dist-packages/torch']
torch version .................... 2.0.1+cu118
deepspeed install path ........... ['/usr/local/lib/python3.10/dist-packages/deepspeed']
deepspeed info ................... 0.10.1+9d79cfd1, 9d79cfd1, master
torch cuda version ............... 11.8
torch hip version ................ None
nvcc version ..................... 11.8
deepspeed wheel compiled w. ...... torch 2.0, cuda 11.8
shared memory (/dev/shm) size .... 125.00 GB

System info (please complete the following information):

OS: Ubuntu 22.04.2
GPU count and types: 2xA6000
DeepSpeed: DeepSpeed v0.10.0
Transformers: 4.31.0
Python version: 3.10.6

@asifehmad running with deepspeed --num_gpus 2 is effectively launching 2 python processes, each running your script. This means that each will output the print line. To avoid this, you can do the following:
if local_rank==0:
    print(output_str)
    print('Inference Time is:', end - start)

The model is being split across 2 GPUs! The response from the model is returned to both processes. Try running with a single GPU (using --num_gpus 1) while monitoring the memory usage using watch -n 1 nvidia-smi. Then do the same with 2 GPU. You should see that with the single GPU setup the memory usage on GPU 0 is about 2x that as with the 2 GPU setup.

The model is being split across 2 GPUs! The response from the model is returned to both processes. Try running with a single GPU (using --num_gpus 1) while monitoring the memory usage using watch -n 1 nvidia-smi. Then do the same with 2 GPU. You should see that with the single GPU setup the memory usage on GPU 0 is about 2x that as with the 2 GPU setup.

Thanks! So then the inferences are performed using the two GPUs parallelly, right? Or one after each?

The inference is happening across 2 GPUs. Your model is being sharded across those 2 GPUs (with tensor parallelism via DeepSpeed-Inference)! This means you can run inference on models that are too large for a single GPU and in many cases you will see lower latency too!

Unfortunately, it is not like what you said, I just tested with --num_gpus=2 which is using 28% of each GPU, and for --num_gpus=1 it is also using 28% of a single GPU. The only difference I see is I get faster inferences with the 2 processes.
Is there any way to run this as a single process on the two GPUs? I don't load the model and other processes two times with --num_gpus=2.

@asifehmad the memory usage you are seeing with 2 GPUs is due to the loading the model with this line:
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf", torch_dtype=torch.float16, low_cpu_mem_usage=True,device_map=local_rank)

If you switch the device to CPU, then you will see the maximum GPU memory usage after calling deepspeed.init_inference reflects that the model is sharded across the GPUs. Also I provided bad advice earlier when I suggested to use nvidia-smi to watch the memory usage. Often this will show cached memory as part of used memory. The most accurate memory usage information can be obtained via deepspeed.runtime.utils.see_memory_usage:
https://github.com/microsoft/DeepSpeed/blob/19e9a7c0284cc886d84afd7ec58721af5e045811/deepspeed/runtime/utils.py#L793
Try the following:
import os
import torch
import deepspeed
import time

from transformers import AutoTokenizer, AutoModelForCausalLM
from deepspeed.runtime.utils import see_memory_usage

local_rank = int(os.getenv("LOCAL_RANK", "0"))
world_size = int(os.getenv("WORLD_SIZE", "1"))
hf_token = "<your_token>"
max_tokens = 2000
prompt = "Write a detailed note on AI"

tokenizer = AutoTokenizer.from_pretrained(
    "meta-llama/Llama-2-7b-chat-hf",
    use_fast=True,
    token=hf_token,
)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-chat-hf",
    torch_dtype=torch.float16,
    device_map="cpu",
    token=hf_token,
)
see_memory_usage("After load model", force=True)

ds_model = deepspeed.init_inference(
    model=model,
    mp_size=world_size,
    dtype=torch.float16,
    replace_method="auto",
    replace_with_kernel_inject=True,
    max_out_tokens=max_tokens,
)
see_memory_usage("After DS-inference init", force=True)

torch.cuda.synchronize()
start = time.time()
inputs = tokenizer.encode(f"<human>: {prompt} \n<bot>:", return_tensors="pt").to(
    f"cuda:{local_rank}"
)
outputs = ds_model.generate(inputs, max_new_tokens=max_tokens)
output_str = tokenizer.decode(outputs[0])
torch.cuda.synchronize()
end = time.time()
see_memory_usage("After forward", force=True)

print("output:", output_str)
print("Inference Time:", end - start)

When I run this with 1 GPU, I see:
[2023-08-18 09:35:58,970] [INFO] [utils.py:803:see_memory_usage] After load model
[2023-08-18 09:35:58,970] [INFO] [utils.py:804:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB
[2023-08-18 09:36:13,513] [INFO] [utils.py:803:see_memory_usage] After DS-inference init
[2023-08-18 09:36:13,513] [INFO] [utils.py:804:see_memory_usage] MA 12.81 GB         Max_MA 12.81 GB         CA 13.45 GB         Max_CA 13 GB
[2023-08-18 09:36:35,368] [INFO] [utils.py:803:see_memory_usage] After forward
[2023-08-18 09:36:35,368] [INFO] [utils.py:804:see_memory_usage] MA 12.82 GB         Max_MA 12.82 GB         CA 13.45 GB         Max_CA 13 GB

and with 2 GPU, I see:
[2023-08-18 09:37:26,176] [INFO] [utils.py:803:see_memory_usage] After load model
[2023-08-18 09:37:26,176] [INFO] [utils.py:804:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB
[2023-08-18 09:37:44,499] [INFO] [utils.py:803:see_memory_usage] After DS-inference init
[2023-08-18 09:37:44,499] [INFO] [utils.py:804:see_memory_usage] MA 6.65 GB         Max_MA 6.65 GB         CA 7.71 GB         Max_CA 8 GB
[2023-08-18 09:37:57,657] [INFO] [utils.py:803:see_memory_usage] After forward
[2023-08-18 09:37:57,658] [INFO] [utils.py:804:see_memory_usage] MA 6.66 GB         Max_MA 6.66 GB         CA 7.71 GB         Max_CA 8 GB

Note the Memory Allocated (MA) is 1/2 for the 2 GPU case compared to the single GPU case.
However, it is still true that initially the model must be loaded by both processes. You can avoid this by loading the model using meta tensors. See the example where we do this here:
https://github.com/microsoft/DeepSpeedExamples/blob/918634750a2bc22d93c83f3444d99ae4ed4e71f4/inference/huggingface/text-generation/utils.py#L44
Note that we currently do not support loading Llama2 models with meta tensor, but there is a recently submitted PR that adds support if you would like to test that. Otherwise, we will be reviewing and merging this work soon: #4147

Thanks a lot @mrwyattii for the detailed information. I will these soon.

@asifehmad the memory usage you are seeing with 2 GPUs is due to the loading the model with this line:
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf", torch_dtype=torch.float16, low_cpu_mem_usage=True,device_map=local_rank)

If you switch the device to CPU, then you will see the maximum GPU memory usage after calling deepspeed.init_inference reflects that the model is sharded across the GPUs. Also I provided bad advice earlier when I suggested to use nvidia-smi to watch the memory usage. Often this will show cached memory as part of used memory. The most accurate memory usage information can be obtained via deepspeed.runtime.utils.see_memory_usage:
https://github.com/microsoft/DeepSpeed/blob/19e9a7c0284cc886d84afd7ec58721af5e045811/deepspeed/runtime/utils.py#L793
Try the following:
import os
import torch
import deepspeed
import time

from transformers import AutoTokenizer, AutoModelForCausalLM
from deepspeed.runtime.utils import see_memory_usage

local_rank = int(os.getenv("LOCAL_RANK", "0"))
world_size = int(os.getenv("WORLD_SIZE", "1"))
hf_token = "<your_token>"
max_tokens = 2000
prompt = "Write a detailed note on AI"

tokenizer = AutoTokenizer.from_pretrained(
    "meta-llama/Llama-2-7b-chat-hf",
    use_fast=True,
    token=hf_token,
)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-chat-hf",
    torch_dtype=torch.float16,
    device_map="cpu",
    token=hf_token,
)
see_memory_usage("After load model", force=True)

ds_model = deepspeed.init_inference(
    model=model,
    mp_size=world_size,
    dtype=torch.float16,
    replace_method="auto",
    replace_with_kernel_inject=True,
    max_out_tokens=max_tokens,
)
see_memory_usage("After DS-inference init", force=True)

torch.cuda.synchronize()
start = time.time()
inputs = tokenizer.encode(f"<human>: {prompt} \n<bot>:", return_tensors="pt").to(
    f"cuda:{local_rank}"
)
outputs = ds_model.generate(inputs, max_new_tokens=max_tokens)
output_str = tokenizer.decode(outputs[0])
torch.cuda.synchronize()
end = time.time()
see_memory_usage("After forward", force=True)

print("output:", output_str)
print("Inference Time:", end - start)

When I run this with 1 GPU, I see:
[2023-08-18 09:35:58,970] [INFO] [utils.py:803:see_memory_usage] After load model
[2023-08-18 09:35:58,970] [INFO] [utils.py:804:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB
[2023-08-18 09:36:13,513] [INFO] [utils.py:803:see_memory_usage] After DS-inference init
[2023-08-18 09:36:13,513] [INFO] [utils.py:804:see_memory_usage] MA 12.81 GB         Max_MA 12.81 GB         CA 13.45 GB         Max_CA 13 GB
[2023-08-18 09:36:35,368] [INFO] [utils.py:803:see_memory_usage] After forward
[2023-08-18 09:36:35,368] [INFO] [utils.py:804:see_memory_usage] MA 12.82 GB         Max_MA 12.82 GB         CA 13.45 GB         Max_CA 13 GB

and with 2 GPU, I see:
[2023-08-18 09:37:26,176] [INFO] [utils.py:803:see_memory_usage] After load model
[2023-08-18 09:37:26,176] [INFO] [utils.py:804:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB
[2023-08-18 09:37:44,499] [INFO] [utils.py:803:see_memory_usage] After DS-inference init
[2023-08-18 09:37:44,499] [INFO] [utils.py:804:see_memory_usage] MA 6.65 GB         Max_MA 6.65 GB         CA 7.71 GB         Max_CA 8 GB
[2023-08-18 09:37:57,657] [INFO] [utils.py:803:see_memory_usage] After forward
[2023-08-18 09:37:57,658] [INFO] [utils.py:804:see_memory_usage] MA 6.66 GB         Max_MA 6.66 GB         CA 7.71 GB         Max_CA 8 GB

Note the Memory Allocated (MA) is 1/2 for the 2 GPU case compared to the single GPU case.
However, it is still true that initially the model must be loaded by both processes. You can avoid this by loading the model using meta tensors. See the example where we do this here: https://github.com/microsoft/DeepSpeedExamples/blob/918634750a2bc22d93c83f3444d99ae4ed4e71f4/inference/huggingface/text-generation/utils.py#L44
Note that we currently do not support loading Llama2 models with meta tensor, but there is a recently submitted PR that adds support if you would like to test that. Otherwise, we will be reviewing and merging this work soon: #4147

Hi @mrwyattii ! I tried this multiple times but getting errors related to meta tensors, and deepspeed for multiple GPUs works but I don't want multiple processes. So I have tried another solution using device_map='balalnced, I think that way I can find a solution without multiple processes, but I couldn't be able to use the GPUs for inferences, the model splits equally into the GPUs, but for inferences, it uses only one GPU.
The code is now:
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
import time

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf", use_fast=True)
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf", torch_dtype=torch.half,low_cpu_mem_usage=True,device_map='balanced')

start = time.time()
prompt = 'Write a detailed note on AI'
inputs = tokenizer.encode(f"<human>: {prompt} \n<bot>:", return_tensors='pt').to(model.device)
outputs = model.generate(inputs, max_new_tokens=256)
output_str = tokenizer.decode(outputs[0])

print(output_str)
end = time.time()
print('Inference Time is:', end-start)

Couldn't you please help me to run the inferences on the two GPUs as well?

Can you please share what errors you are seeing related to meta tensor? The example code I provided does not load the model with meta tensor so I'm confused what might cause that.
I don't think you will be able to run on multiple GPUs without multiple processes. Multiple processes are necessary so that we can send commands to each GPU in parallel and achieve far great performance. For example, running with the example that you shared and generated 50 tokens on 2xA6000 GPU I see an average time of 1.24s. On a single GPU with DeepSpeed the average time is 1.09s and with 2 GPU it's 0.65s.
If you are interested in having a single process to interface with, then perhaps checkout our other project, DeepSpeed-MII. This would allow you to stand up an inference server across multiple GPUs (and processes) and run a query from a single process:
# File: mii-example.py
# Usage: python mii-example.py

import mii

mii_configs = {"tensor_parallel": 2, "dtype": "fp16"}

# Deploy will launch multiple processes from this single process
mii.deploy(task='text-generation',
           model="meta-llama/Llama-2-7b-chat-hf",
           deployment_name="llama_deployment",
           mii_config=mii_configs)

# Then you can leverage MII to run a query on the sharded model from a single process
generator = mii.mii_query_handle("llama_deployment")
result = generator.query({'query': ["DeepSpeed is the", "Seattle is"]})

mii.terminate("llama_deployment")

Can you please share what errors you are seeing related to meta tensor? The example code I provided does not load the model with meta tensor so I'm confused what might cause that.
I don't think you will be able to run on multiple GPUs without multiple processes. Multiple processes are necessary so that we can send commands to each GPU in parallel and achieve far great performance. For example, running with the example that you shared and generated 50 tokens on 2xA6000 GPU I see an average time of 1.24s. On a single GPU with DeepSpeed the average time is 1.09s and with 2 GPU it's 0.65s.
If you are interested in having a single process to interface with, then perhaps checkout our other project, DeepSpeed-MII. This would allow you to stand up an inference server across multiple GPUs (and processes) and run a query from a single process:
# File: mii-example.py
# Usage: python mii-example.py

import mii

mii_configs = {"tensor_parallel": 2, "dtype": "fp16"}

# Deploy will launch multiple processes from this single process
mii.deploy(task='text-generation',
           model="meta-llama/Llama-2-7b-chat-hf",
           deployment_name="llama_deployment",
           mii_config=mii_configs)

# Then you can leverage MII to run a query on the sharded model from a single process
generator = mii.mii_query_handle("llama_deployment")
result = generator.query({'query': ["DeepSpeed is the", "Seattle is"]})

mii.terminate("llama_deployment")

Hi @mrwyattii ! As you have mentioned earlier However, it is still true that initially the model must be loaded by both processes. You can avoid this by loading the model using meta tensors. following this I tried meta tensor as:
#####deepspeed meta tensor#####
import torch
import deepspeed
from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig
from transformers import LlamaForCausalLM, LlamaTokenizer
import transformers
import os
import torch.distributed as dist
import io
import json
from pathlib import Path

local_rank = int(os.getenv("LOCAL_RANK", "0"))
world_size = int(os.getenv("WORLD_SIZE", "2"))

deepspeed.init_distributed()
rank = dist.get_rank()
print(f"========Current randk:{rank} || lcoal rank:{local_rank} || world size:{world_size}==========")

model_path = 'meta-llama/Llama-2-7b-chat-hf'

config = AutoConfig.from_pretrained(model_path)

tokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=True)
with deepspeed.OnDevice(dtype=torch.float16, device="meta"):
  model = AutoModelForCausalLM.from_config(config, torch_dtype=torch.float16)

model = model.eval()
ds_model = deepspeed.init_inference(
model,
mp_size=world_size,
dtype=torch.float16, replace_method="auto", replace_with_kernel_inject=True, max_out_tokens=12000
)

prompt = "Write a detailed note on AI"

encoding = tokenizer(prompt, return_tensors="pt")
generation_config = transformers.GenerationConfig(
temperature=0.0,
top_k=20,
repetition_penalty=1.2,
)

input_ids = encoding["input_ids"].to(model.device)

result = ds_model.generate(
input_ids=input_ids,
generation_config=generation_config,
return_dict_in_generate=False,
output_scores=False,
max_new_tokens=512,
)

output = tokenizer.decode(result[0][len(input_ids[0]):])
if rank == 0:
  print(output)

and I got this error:
AssertionError: Meta tensors are not supported for this model currently.

Hi @mrwyattii ! As you have mentioned earlier However, it is still true that initially the model must be loaded by both processes. You can avoid this by loading the model using meta tensors. following this I tried meta tensor as:
and I got this error: AssertionError: Meta tensors are not supported for this model currently.

I see. My apologies, that advice would have worked for other model families (e.g., OPT, BLOOM, etc.) but we do not currently support meta tensor loading with the Llama models. This feature should be added in the near future!

No problem! I really appreciate what you do.
So it is not possible to run inferences on the two GPUs in the case of device_map='balanced'?

No problem! I really appreciate what you do. So it is not possible to run inferences on the two GPUs in the case of device_map='balanced'?

I know that device_map='balanced' splits the model weights across the available GPUs, but I don't think that enables running inference in parallel (I may be incorrect here, please reach out to HuggingFace developers for more accurate information). If I'm not mistaken, you would still need to use accelerate and launch multiple processes:
accelerate launch my_script.py


Add a govulncheck job.

Description
https://go.dev/blog/vuln
Checklist

[ ] Investigated and inspected CI test results
[x] ~Unit test and regression tests added~
[x] ~Evaluated and added CHANGELOG entry if required~
[x] ~Determined and documented upgrade steps~
[x] ~Documented user facing changes (create PR based on openshift/openshift-docs and merge into rhacs-docs)~

Testing Performed
Relying on CI.

So golangci will not integrate this check

https://github.com/golangci/golangci-lint/issues/3094

I've created a PR that scans binaries not code and it's working

https://github.com/stackrox/stackrox/pull/8653

FTR it's currently crashing when scanning operator source.
https://github.com/tasks/tasks/issues/2638

@porridge can we close this? I believe we have it now :)

@RTann govulncheck has at least two functionalities - source scanning and binary scanning, I believe the job we have is only using one of them (the latter). This is about the former one.


Not compiling on Linux

After commit 30e2079 xfiles does not compile:
$ make cc -std=c99 -pedantic -D_POSIX_C_SOURCE=200809L -D_BSD_SOURCE -D_DEFAULT_SOURCE -I/usr/local/include -I/usr/X11R6/include -I/usr/include/freetype2 -I/usr/X11R6/include/freetype2   -o xfiles.o -c xfiles.c xfiles.c: In function 'checkicon': xfiles.c:343:25: error: 'FNM_CASEFOLD' undeclared (first use in this function)   343 |                 flags = FNM_CASEFOLD | FNM_PATHNAME;       |                         ^~~~~~~~~~~~ xfiles.c:343:25: note: each undeclared identifier is reported only once for each function it appears in xfiles.c: In function 'runxfilesctl': xfiles.c:659:13: warning: implicit declaration of function 'pipe2'; did you mean 'pipe'? [-Wimplicit-function-declaration]   659 |         if (pipe2(pipefds, O_CLOEXEC) == RETURN_FAILURE)       |             ^~~~~       |             pipe make: *** [Makefile:74: xfiles.o] Error 1

Commit 97e8a90 should fix that.
I do not have a Linux around to test it right now, but according to the
manuals it was only a matter of using the right feature_test_macros(7).

Tried the #52 PR and it fixes the compile issue.
97e8a90 still has an INFTIM undefined error.

Fixed again by ecf8e4b
I am gonna check compilation on Linux tomorrow if it complains of any other undefined symbols.


Update ruby and rails

I think I got all the places that need updating

If it passes, :ship:.
TODO:

[ ] ruby 2.2 -> 2.3
[ ] rails 4 -> 5 (maybe)

Isn't rails 5 still beta?

Yeah, but it's on its third beta :stuck_out_tongue:

hmmm, try changing rake <task> to bundle exec rake <task> ?

Every single failure has to do with methods defined in "shoulda-matchers". Looking at Gemfile.lock, it had a major version bump. We should either pin it to ~> 2.0, or figure out what the upgrade process is

They all passed :) +1s welcome

:+1: (even though I have a commit on here)

Also, I opened an issue for the shoulda business.


Fix issue 772; Use existing iframe DOM element instead of document.ge…

I'm not able to test with your automated tests.
I tried with jasmine and with qunit.
qunit in Chrome work, but shadowRoot not seems to be supported with PhantomJS.
with jasmine, even with the master code, the tests passed.

So are the tests in this PR doing anything?

I kept the tests in case of you was willing to trying to fix it.
But I think it's ok to not make any other tests, because your tests assure that all is working again.
I don't added a new feature but I make a bug fix when getElementById is impossible.
I'm actualy trying to test an invalid getElementById instead of trying to make shadowDom works.

Ok can you take the tests out then if they are not doing anything, the change is small enough that I am happy if the current test pass with this ok

Ok can you take the tests out then if they are not doing anything, the change is small enough that I am happy if the current test pass with this ok

Done


dataset path

Thank you very much for receiving your dataset. I encountered a problem when I replaced the data set path. Your default path is "/media/imed/data/mayuhui/TMI_OCTA_vseg/data/OCTA/CRIA", and I replaced it with "../../data/ROSE-2".and get infomation “IndexError: list index out of range”in  code\OCTA-Net\train.py, line 17, in train_first_stage  thin_gt = sample[2].to(device)。
That is, sampmle out of range.Hope you can answer, thank you very much！

We are sorry to forget to clarify that for ROSE-2, changes should be made as follows:
thin_gt = sample[1].to(device)
thick_gt = sample[1].to(device)
if there are any bugs else, please contact us. Good luck!

你好，你能给我分享一下数据集吗？下载数据集的链接失效了


Search reindex job fails when environment synchronizer is triggered automatically

I have configured the environment synchronizer job to trigger the search reindex job after a database sync (e.g. prod => preprod). But the search reindex job fails (as shown in the image) when the environment synchronizer is triggered automatically. However, when I manually run the environment sync job everything works correctly.

Hi @conorhepi . Could you possible share the stack trace of the error that you are getting from the logs?

Here's the stack trace:

https://marisks.net/2019/04/23/episerver-starting-scheduled-job-programmatically/
Could be a solution...

Solved by using custom handler. Search and navigation reindex job using HttpContext and is not available when run autorun. Read more: https://github.com/Epinova/Addon.Episerver.EnvironmentSynchronizer/blob/main/documentation/SearchReindexSynchronizer.md


Wiki Update

Today I set down with a colleague of mine who uses a Mac and we tried flashing a P1-dongle. I updated the Flashing Instructions on the Wiki accordingly, but for some reason Github doesn't allow you to make a PR for wikis.
So here it is: https://github.com/plan-d-io/P1-dongle/compare/main...JrtPec:P1-dongle:master?expand=1
@Epyon01P could you update the wiki?

Thanks for the additions. Merged!


Storage data from RTS-GMLC

Hi,
It seems that the data imported from the RTS-GMLC data set is wrong for HydroEnergyReservoir and GenericBattery instances. I could identify two issues but there may be also the third one (see my inflow/initial-storage related constraints).

Scale: the storage.csv lists volumes in GWh. After import, the capacity values appear to be too small by a factor of 1000.
Base power: This issue affects mostly the hydro instances. On import, the storage.csv>Rating MVA is used to do the per-unit conversion for attributes coming from the storage file. For attributes coming from the gen.csv, the gen.csv>Base MVA column is used. When building the JuMP model the later is used for all. The result is that all the parameters that originate from the storage.csv are wrong in the JuMP model, by about 6% as the Rating MVA is usually 50 while the Base MVA 53.

Maybe an example can better demonstrate:
The 222_Hydro_1 plant has a max power of 50MW and a reservoir capacity of 1 GWh
The two constraints now look like:
P__HydroEnergyReservoir_{222_HYDRO_1, 1} ≤ 0.5 This seems to be correct
E__HydroEnergyReservoir_{222_HYDRO_1, 1} ≤ 0.0106 The right side here should be 10. The magnitude problem is what I listed as issue 1 the inaccuracy problem seems to be issue 2
E__HydroEnergyReservoir_{222_HYDRO_1, 2} - E__HydroEnergyReservoir_{222_HYDRO_1, 1} + Sp__HydroEnergyReservoir_{222_HYDRO_1, 2} + P__HydroEnergyReservoir_{222_HYDRO_1, 2} = 0.0007238423518714864 The right side here should be the inflow (0.093) but I haven't figured it out where this goes wrong.
It would be great if your could look into these problems. Please let me know if I can help with anything.
Thanks,
Tamas

Hi,
Thanks for your quick response!
I rely on the RTS-GMLC repository much like how you use it in the SIIP_Examples.
This attached script (rename it to .jl) Hydro_storage_RTS.txt should allow you reproduce the issue.
Thanks again,
Tamas

@tamas-borbath thanks. I'm working on improving the hydro and storage parsing now. I'll have a fix soon.

@tamas-borbath I made some changes to address what I think are the problems, see #594
A more complete summary with descriptions of the modeling implications follows

EditURL = "<unknown>/../../../Downloads/Hydro_storage_RTS.jl"

My Env
using Pkg
Pkg.status()

Project SIIPExamples v0.0.1
Status `~/Documents/repos/SIIPExamples.jl/Project.toml`
  [336ed68f] CSV v0.7.7
  [9961bab8] Cbc v0.7.0
  [41994980] D3TypeTrees v0.1.1
  [a93c6f00] DataFrames v0.21.6
  [becb17da] Feather v0.5.6
  [60bf3e95] GLPK v0.13.0
  [2cd47ed4] InfrastructureSystems v0.15.0
  [b6b21f68] Ipopt v0.6.3
  [2535ab7d] JSON2 v0.3.2
  [4076af6c] JuMP v0.21.3
  [98b081ad] Literate v2.5.1
  [47be7bcc] ORCA v0.4.0
  [f0f68f2c] PlotlyJS v0.13.1
  [91a5bcdd] Plots v1.5.8
  [5f7eddb3] PowerGraphics v0.5.2
  [e690365d] PowerSimulations v0.6.3 [`../PowerSimulations.jl`]
  [bcd98974] PowerSystems v0.26.0 [`../PowerSystems.jl`]
  [9e3dc215] TimeSeries v0.18.0
  [f269a46b] TimeZones v1.3.2
  [0f1e0344] WebIO v0.8.14
  [ade2ca70] Dates
  [56ddb016] Logging
  [44cfe95a] Pkg
  [9a3f8284] Random
  [10745b16] Statistics
  [8dfed614] Test

Example from Tamas
using SIIPExamples

using PowerSystems
using PowerSimulations
using JuMP
using Dates
using DataFrames

using JuMP
using Cbc #solver
const PSI = PowerSimulations
solver = optimizer_with_attributes(Cbc.Optimizer, "logLevel" => 1, "ratioGap" => 0.5)

rts_dir = SIIPExamples.download("https://github.com/GridMod/RTS-GMLC")
rts_src_dir = joinpath(rts_dir, "RTS_Data", "SourceData")
rts_siip_dir = joinpath(rts_dir, "RTS_Data", "FormattedData", "SIIP");
system_base_power = 100.0

rawsys = PowerSystems.PowerSystemTableData(
    rts_src_dir,
    system_base_power,
    joinpath(rts_siip_dir, "user_descriptors.yaml"),
    timeseries_metadata_file = joinpath(rts_siip_dir, "timeseries_pointers.json"),
    generator_mapping_file = joinpath(rts_siip_dir, "generator_mapping.yaml"),
);

sys = System(rawsys; forecast_resolution = Dates.Hour(1), time_series_in_memory=true);

for gen in get_components(HydroEnergyReservoir, sys)
    gen.available=false
end
get_component(HydroEnergyReservoir, sys, "222_HYDRO_1").available=true

branches = Dict{Symbol,DeviceModel}()
devices = Dict(:Hydro => DeviceModel(HydroEnergyReservoir, HydroDispatchReservoirStorage),:Bat => DeviceModel(GenericBattery, BookKeeping))
services = Dict()

template = OperationsProblemTemplate(CopperPlatePowerModel, devices, branches, services);
op_problem = OperationsProblem(GenericOpProblem, template, sys; optimizer = solver, horizon = 2)

PSI.export_op_model(op_problem, "my_op_prob.json")

lets look at this reservoir
reservoir = get_component(HydroEnergyReservoir, sys, "222_HYDRO_1")

222_HYDRO_1 (PowerSystems.HydroEnergyReservoir):
   name: 222_HYDRO_1
   available: true
   bus: Bell (PowerSystems.Bus)
   active_power: 0.9433962264150944
   reactive_power: -0.13150943396226414
   rating: 0.9905211130872972
   prime_mover: HY
   active_power_limits: (min = 0.0, max = 0.9433962264150944)
   reactive_power_limits: (min = -0.18867924528301888, max = 0.3018867924528302)
   ramp_limits: (up = 0.9433962264150944, down = 0.9433962264150944)
   time_limits: (up = 0.0, down = 0.0)
   operation_cost: PowerSystems.TwoPartCost
   base_power: 53.0
   storage_capacity: 20.0
   inflow: 4.0
   initial_storage: 10.0
   storage_target: 1.0
   conversion_factor: 1.0
   services: 0-element Array{PowerSystems.Service,1}
   dynamic_injector: nothing
   ext: Dict{String,Any}()
   forecasts: InfrastructureSystems.Forecasts: 3

the energy limit constraint for period 1 in constraint c_3 in "my_op_problem.json" has an rhs of  0.006008603565516297.
this corresponds to the value of initial_storage + the forecasted inflow:
inflow_ts = values(get_forecast_values(Deterministic, reservoir, Dates.DateTime("2020-01-01T00:00:00"), "get_inflow"))[1:2]
@show get_initial_storage(reservoir) + inflow_ts[1]

# the energy limit constraint for period 2 in constraint c_4 in "my_op_problem.json" had an rhs of  0.0007238423518714864, now it's 0.7238423518714864

6.008603565516298

this corresponds to the value of  the forecasted inflow:
@show inflow_ts[2]

# the upper bound for the energy variable in period 2 in constraint c10_2 in "my_op_problem.json" had an rhs of 0.0106 now it's 10.6.

0.7238423518714864

this corresponds to the upper limit of the storage capacity of the reservoir:
@show get_storage_capacity(reservoir)

10.600000000000001

Notes from Tamas
Mismatch on magnitude as the import treates GWh as MWh
The extra 6% inacuarcy becase in user_descriptors.yaml the Rating MVA is defined as base_power. The reservoir.storage_capacity becomes 1(Max Volume column)/50 (Rating column)= 0.02 It should be: 1000/53 = 18.86792
print("The capacity for 222_HYDRO_1 should be: ",1000/system_base_power," now it's: ",reservoir.storage_capacity*reservoir.base_power/system_base_power)

The capacity for 222_HYDRO_1 should be: 10.0 now it's: 10.6

Same thing happens for reservoir.initial_storage
print("The initial volume for 222_HYDRO_1 should be: ",500/system_base_power," now it's: ",reservoir.initial_storage*reservoir.base_power/system_base_power)

The initial volume for 222_HYDRO_1 should be: 5.0 now it's: 5.3

The P max however is correct
print("The P max for 222_HYDRO_1 should be: ",50/system_base_power," and it's correctly: ",reservoir.active_power_limits.max*reservoir.base_power/system_base_power)

The P max for 222_HYDRO_1 should be: 0.5 and it's correctly: 0.5

Changes made
I implemented unit conversions in the tabular parser to convert GW(h) and kW(h) to MW(h). This requires that the unit_descriptors.yaml file has the correct origin units defined. So, I made those changes to RTS-GMLC and the PowerSystemsTestData to correctly reflect the storage column headers.

This page was generated using Literate.jl.

see https://github.com/NREL-SIIP/PowerSystemsTestData/pull/36 for an example of how to make the required changes to the user_descriptors.yaml file to enable unit conversion

@claytonpbarrows can we close this?

I think so. @tamas-borbath if you need further resolution, feel free to re-open.


66 - Projectile-Motion_Simulator

I have added my own JS project which is a simple simulator to the projectile motions, it was created for an educational purposes and gained a lot of respect from my mentors. Project number is 66.

What is the number of my project to  change it

"-prompt(8)-"

'-prompt(8)-'

What is the number of my project to change it

84

but 84 is already occupied, what about 87 ?

but 84 is already occupied, what about 87 ?

88 ✅

Done bro

I have made another pull request


DNM: canary PR 20220112

Signed-off-by: Francesco Romani<EMAIL_ADDRESS>

/hold
please never merge canary PRs.

no longer needed


Trying out 3.0.0-beta.1

I'm not sure how ready it is but since I'm migrating to react-native-notifications I figured I should give it a go (particularly given it looks like the APIs have been merged). I've found a few issues that I think I've managed to fix by patching the package - some I know how to fix, some I don't. Let me know if you want me to put together a PR with these changes (should I PR to the v3 branch?). Hopefully this is useful to you, apologies if it's just wasting time!
Documentation change
In the new documentation it says to call Notifications.requestPermissions() but Notifications.registerRemoteNotifications() seems to be the correct call.
Use of "Proxy"
The hermes JS engine doesn't support proxies, and might not in the future (https://github.com/facebook/hermes/issues/33). Since it's only used for a relatively minor feature (in NotificationsAndroid.ts and NotificationsIOS.ts) in this package a simple fix is to do something like:
if (Platform.OS !== "android") {
    Object.getOwnPropertyNames(NotificationsAndroid.prototype).forEach(propertyName => {
        if (propertyName !== "constructor") {
            this[propertyName] = () => {};
        }
    });
}

Android building
One of the paths in lib/android/app/build.gradle is wrong:
-    def reactNativePackageJson = packageSlurper.parse file('../../../node_modules/react-native/package.json')
+    def reactNativePackageJson = packageSlurper.parse file('../../../../../node_modules/react-native/package.json')

And I couldn't compile without changing:
-    implementation "com.google.firebase:firebase-messaging:17.3.0"
+    implementation "com.google.firebase:firebase-messaging:20.0.0"

I was able to get auto linking working by adding:
    dependencies: {
        "react-native-notifications": {
            platforms: {
                android: {
                    sourceDir: path.resolve("./node_modules/react-native-notifications/lib/android/app"),
                    folder: path.resolve("./node_modules/react-native-notifications/lib"),
                    packageImportPath: "import com.wix.reactnativenotifications.RNNotificationsPackage;",
                    packageInstance: "new RNNotificationsPackage(this.getApplication())"
                }
            }
        }
    }

to my react-native.config.js. Note: the use of MainApplication.this suggested in the docs did not work but changing to this.getApplication() seemed to fix that. As far as I can tell there's no way to include this config with the package. Possibly using rnpm in package.json but that is being deprecated.
Platform difference
Lastly Notifications.registerRemoteNotificationsRegistered provides {deviceToken: string} on iOS and just a string on Android.

Some more additional things I've found trying to get notifications working on Android:
In NativeEventsReceiver.ts on Android it should be new Notification(response) - I haven't tried iOS yet but I'm assuming that's correct as new Notification(response.notification) given the previous difference I found.
Also I've found that I need to include a "pushNotification" key in my remote push notification data for it to be passed to the javascript when opening from the notification (because of the way NotificationIntentAdapter.canHandleIntent is used on onNewIntent). I'm not sure if that's ideal - maybe checking for the google.message_id key would be better? It looks like the "pushNotification" key is related to local notifications.

Also it is confusing that it seems the only way to avoid notifications showing up (a blank notification in my case) when the app is in the foreground is to comment postNotification(null); in onReceived() in PushNotification.java.

Continuing to add to this issue with my findings - now I'm moving on to iOS!
There's inconsistency in the library about using setBadgeCount and setBadgesCount (same with getBadgeCount and getBadgesCount). I think probably the correct solution is to rename the native iOS code to use singular "Badge" rather than "Badges".

@sdcooke Thank you for the detailed steps! I'm also struggling with the same stuff you mention in this thread.
I could get the Android device token and also receive a notification on device's notification center (I'm using firebase-admin library from the server to send notifications)
Anyway I'm not being able to get the registerRemoteNotificationReceived nor the registerRemoteNotificationOpened callbacks from the app.
Did you also follow the steps described here to receive the notification callbacks in the app or it just worked with the stuff auto-linking does?
BTW for IOS I could get almost the flow working but the notification actions (NotificationAction / NotificationCategory) Did you find a workaround for this?

+1 for not using «Proxy» as it is also not supported on iOS 9 and below.

Thanks to you @sdcooke, I was able to build for android and iOS. But I still encounter some issues, for example I cannot make getInitialNotification work... It always returns undefined for me.
Did you also face this problem ?

@sdcooke Thank you for sharing!!
I should definitely see your documents.


Can't download violent/graphic video from Facebook

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm reporting a new site support request
[X] I've verified that I'm running yt-dlp version 2023.07.06 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that none of provided URLs violate any copyrights or contain any DRM to the best of my knowledge
[X] I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[X] I've read about sharing account credentials and am willing to share it if required

Region
Bulgaria
Example URLs
https://www.facebook.com/ivon.mladenova/posts/7266440750037713/
Provide a description that is worded well enough to be understood
I tried to download that has this:

I  clicked Learn more and then "See video":

I'm able to see the video, but I'm unable to download it with yt-dlp.
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] If using API, add 'verbose': True to YoutubeDL params instead
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
[debug] Command-line config: ['-vU', '--cookies-from-browser', 'chrome', 'https://www.facebook.com/ivon.mladenova/posts/7266440750037713/']
[debug] User config "C:\Users\johnl\AppData\Roaming\yt-dlp\config.txt": ['-f', 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best', '-i', '-o', 'D:/Courses/%(title)s.%(ext)s', '--no-mtime']
[debug] Encodings: locale cp1251, fs utf-8, pref cp1251, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version<EMAIL_ADDRESS>[b532a3481] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg 2021-02-02-git-2367affc2c-full_build-www.gyan.dev, ffprobe 2021-02-02-git-2367affc2c-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.05.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-11.0.3
[Cookies] Extracting cookies from chrome
[debug] Extracting cookies from: "C:\Users\johnl\AppData\Local\Google\Chrome\User Data\Default\Network\Cookies"
[debug] Found local state file at "C:\Users\johnl\AppData\Local\Google\Chrome\User Data\Local State"
[Cookies] Extracted 5966 cookies from chrome
[debug] cookie version breakdown: {'v10': 5970, 'other': 0, 'unencrypted': 0}
[debug] Proxy map: {}
[debug] Loaded 1855 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version<EMAIL_ADDRESS>Current version<EMAIL_ADDRESS>Current Build Hash: 5ff3e702171a50175c34397494e2d18ce35d771c2110b1e59bd173ec2fb352aa
yt-dlp is up to date<EMAIL_ADDRESS>[facebook] Extracting URL: https://www.facebook.com/ivon.mladenova/posts/7266440750037713/
[facebook]<PHONE_NUMBER>037713: Downloading webpage
[facebook]<PHONE_NUMBER>037713: Downloading webpage
ERROR: [facebook]<PHONE_NUMBER>037713: Cannot parse data; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp\extractor\common.py", line 710, in extract
  File "yt_dlp\extractor\facebook.py", line 707, in _real_extract
  File "yt_dlp\extractor\facebook.py", line 654, in _extract_from_url

Probable duplicate of #4311
The error is being caused by yt-dlp's facebook extractor not working with cookies (#4311), but I suppose it's possible that even if the linked issue is resolved, the extractor may need an additional fix/improvement to be able to get graphic content


BrowserFetcher is unable to download stable (and other) revisions because they are missing from the download site

The stable and other chromium snapshots often cannot be downloaded via puppeteer's BrowserFetcher because the snapshots for the relevant revision numbers are missing. This makes it very difficult to programmatically obtain up-to-date revision info and download those revisions without manual intervention. I would expect that stable revisions at least should always be available for download.
Is there an alternate source for these / can these revisions be published consistently? Otherwise is there an alternate data source to find revisions that are definitely available in the snapshots?
Steps to reproduce

Create a BrowserFetcher using puppeteer.createBrowserFetcher.
Use https://omahaproxy.appspot.com/all.json?os=mac to get the latest revision info for stable (or other channels)
Attempt to download stable (etc.) using BrowserFetcher.download({STABLE_REVISION})

Tell us about your environment:

Puppeteer version: 1.4.0
Platform / OS version: Mac OS 10.13.3
URLs (if applicable): https://omahaproxy.appspot.com/all.json?os=mac, https://storage.googleapis.com/chromium-browser-snapshots
Node.js version: 8.9.1

What is the expected result?
The specified stable (etc.) chrome revision is downloaded by BrowserFetcher.
What happens instead?
Download fails. The download URL provided by BrowserFetcher.getRevisionInfo 404s and visiting https://storage.googleapis.com/chromium-browser-snapshots/index.html?prefix=Mac/ confirms that the revision is not available.

I encountered the same issue and I believe there is actually more to it: From what I understand, the revision numbers will just get you the base/snapshot versions of Chromium. So, even if there would be a working download for the current stable revision (e.g revision 540276 at this moment), you'll just get Chromium 66.0.3359.0 and not the actual stable version 66.0.3359.181.
This means a whole lot of bugfixes and more importantly security fixes will be missing. That's worth keeping in mind if you plan to load untrusted sites with that browser...
As for finding a revision where the snapshot exists, you'd probably be best off using the revision in the LAST_CHANGE file e.g https://storage.googleapis.com/chromium-browser-snapshots/Mac/LAST_CHANGE. It looks to me like this is what https://download-chromium.appspot.com/ uses.

Download fails. The download URL provided by BrowserFetcher.getRevisionInfo 404s and visiting https://storage.googleapis.com/chromium-browser-snapshots/index.html?prefix=Mac/ confirms that the revision is not available.

@casey-speer Yes, chromium buildbots are not building every revision - this would take too long. And buildbots are not synchronized between each other since there's no big need.
To deal with this, we have a small script that helps us figure which revisions are downloadable: utils/check_availability.js.  Is there a new home for it?
Thanks!


Small dependency optimization for ModelLib

ModelLib already has a dependency on ExecutionSession and CompilerUtils. By making the dependency public, it can propagate to other targets simplifying the dependency chain.

Jenkins Linux ppc64le Build #2780 [push] Small dependency optimiz... started at 08:43

Jenkins Linux s390x Build #3664 [push] Small dependency optimiz... started at 08:36

Jenkins Linux amd64 Build #3673 [push] Small dependency optimiz... started at 07:36

Jenkins Linux s390x Build #3664 [push] Small dependency optimiz... passed after 45 min

Jenkins Linux ppc64le Build #2780 [push] Small dependency optimiz... passed after 55 min

Jenkins Linux amd64 Build #3673 [push] Small dependency optimiz... passed after 1 hr 0 min


Create flakey decorator for retrying flaky tests and apply it to #143 and #148

A test that is failing intermittently (a flakey test) either is badly written and needs to be fixed or it is it is exposing a bug in the router. I would rather fix the test bug or the router bug than marking the test as flakey.
There is usually already an issue associated with a flakey test. If a test is marked as flakey and say the test does not fail for a month or so, do we consider that issue fixed  and close it ? Or, are we going to come back to the issue and fix it when we have some time ?

all good points; I knew, going in, that @flakey is controversial

If a test is marked as flakey and say the test does not fail for a month or so, do we consider that issue fixed and close it?

That's how it worked in Dispatch, I'd say; Cannot reproduce -> Closed

mark a test as flakey and forget about it

We should not forget. But we should not be nagged by them all the time. I think the idea is to spend less time on CI review revisiting tests we know they are unreliable, again and again. Having the unreliable tests tracked in the Issues should be sufficient. Contributors should not be forced to investigate random flaky failures when they want to propose a PR.
Doing a grep '@flakey' -A3 will quickly show what tests need fixing.

The end result is we are going to ignore fixing the test which for me has the same effect as skipping a test.

Largely yes, but not quite. A @flakey-decorated test still has to be capable of passing at least sometimes, because the CI checks for that. So we know it is either flakey, or possibly fixed. It may never become completely broken (=incapable of passing).
Furthermore, the @flakey decorator only catches AssertionError, so if the test starts failing somehow unexpectedly, with some other exception, that will be also revealed in CI.
The router has been disabling tests in the past. Marking them as flakey would've been better solution, IMO. (Fixing is of course best, but that's matter of prioritization. The @flakey decorator gives you that flexibility of postponing the investigation/fix and still having usable CI that reveals problems in all the other reliable tests, and does not waste people's time having to deal with the flaky fails over and over.
(The last time I suggested doing something like this (autoretry) the folks did mostly dislike that; lets see what other people think now, ~3 years later)

Closed due to the lack of enthusiasm.


Adding relations to time-series task with <Text valueType="url"> - e._spans is undefined

Describe the bug
I have a problem when using the Relation Extraction template AND using <Text ... valueType="url"> (time series data). Adding a single relation, then submitting the task, and trying to open it again results in:
Uncaught (in promise) TypeError: can't access property 0, e._spans is undefined
To Reproduce
Steps to reproduce the behavior:

Create a new project
Set the templae to this:

<View>
  <Relations>
    <Relation value="org:founded_by"/>
    <Relation value="org:founded"/>
  </Relations>
  <Labels name="label" toName="text">
    <Label value="Organization" background="orange"/>
    <Label value="Person" background="green"/>
    <Label value="Datetime" background="blue"/>
  </Labels>
  <Text name="text" value="$text" valueType="url"/>
</View>
<!-- {"data": { "text": "Microsoft was founded by Bill Gates and Paul Allen on April 4, 1975, to develop and sell BASIC interpreters for the Altair 8800." }, "completions": [{"result": [ { "value": { "start": 0, "end": 9, "text": "Microsoft", "labels": [ "Organization" ] }, "id": "a-ubmF-dP2", "from_name": "label", "to_name": "text", "type": "labels" }, { "value": { "start": 25, "end": 35, "text": "Bill Gates", "labels": [ "Person" ] }, "id": "dQ_pmsrWsm", "from_name": "label", "to_name": "text", "type": "labels" }, { "value": { "start": 40, "end": 50, "text": "Paul Allen", "labels": [ "Person" ] }, "id": "IpmpOMj0vw", "from_name": "label", "to_name": "text", "type": "labels" }, { "value": { "start": 54, "end": 67, "text": "April 4, 1975", "labels": [ "Datetime" ] }, "id": "Q-zXcdacte", "from_name": "label", "to_name": "text", "type": "labels" }, { "from_id": "a-ubmF-dP2", "to_id": "dQ_pmsrWsm", "type": "relation", "direction": "right", "labels": [ "org:founded_by" ] }, { "from_id": "a-ubmF-dP2", "to_id": "IpmpOMj0vw", "type": "relation", "direction": "right", "labels": [] }, { "from_id": "a-ubmF-dP2", "to_id": "Q-zXcdacte", "type": "relation", "direction": "right", "labels": [ "org:founded" ] } ] }]
} -->

Import a .txt file for labelling as time series
Enter the .txt file in task view
Annotate two entities and submit (in top right) - at this point everything is still ok - I can re-enter the tasks view for this task just fine
Enter the task again, click an annotation, press "r" (to enter relationship mode) and click on the other annotation to create a relation
Click "Update" in top right
The window will reload, but you won't be able to see the task view anymore. You'll get console errors like (I'm on Firefox, in Chrome the wording is a bit different):

Uncaught (in promise) TypeError: can't access property 0, e._spans is undefined

Expected behavior
It should be possible to add relations to time series, where Text has valueType=url
Screenshots

This is the state after you complete step 5 from the reproduction: (where everything is still ok)

This is the state from step 6 from the reproduction:
Right now, if I press "Update" in top right, I won't be able to see the task view anymore (after the view reloads)

This is after doing step 7 from the repro

Environment (please complete the following information):

OS:

label-studio is hosted on Debian 10, using Docker
I access it from my Arch Linux laptop with Firefox (but the error also happens on Chromium)

Label Studio Version [1.0.2]

Additional context
When you already have a "broken" task, if you:

export it to json
manually remove all the "results" objects where "type: "relation""
re-import it again

It'll be possible to open the task again (so "revert the task" to the state from step 5 in the reproduction)

For context, I'm uploading a .txt file as a whole task because I'm annotating a whole legal document (that was parsed from PDF). For more see https://github.com/heartexlabs/label-studio/issues/954#issuecomment-851019824

thank you for a detailed description. we'll need some time to reproduce the bug and pull out a fix. I'll get back to you as soon as it's fixed

@sethidden please check this in the version 1.3.0 release, it should be fixed!

@smoreface Can confirm the reported issue is fixed on the latest release 1.3.0post0 :) (1.3.0post0 running inside Docker). Thanks a lot!


Blurry Fonts

I'm using Ferdi in the ff. pc:
Win 10 Home SL 64x
Nvidia GeForce GTX 770
Samsung SyncMaster S23B300 monitor
1920 x 1080 resolution
in dark mode. I don't know why, but the fonts in Twitter and Facebook Messenger look blurry in contrast to the browser (Firefox) and other apps in the system. I tried it with GPU acceleration off and troubleshoot capacity (it suggested Win 8) and the results are the same.
I also tried compatibility settings like high DPI settings.

Hey @monkeylove, can you still reproduce this?


dlib: 19.9 -> 19.13

Motivation for this change
This fixes the latest dlib build for nixpkgs compliancy:

Patched setup.py to use number of jobs defined in $NIX_BUILD_CORES
rather than using all available cores.

Bumped dlib to latest version (v19.13 ATM).

Dropped openblas build input, cblas which actually works lives in
dlib/external. Otherwise the test suite runs into segfaults (see
https://github.com/NixOS/nixpkgs/pull/39255#issuecomment-384535129 for
further reference).

Added myself as maintainer in case of any further breakage in the
future.

Closes #39255
/cc @dotlambda @ryantm
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

nox-review is still pending, I'll notify you when it's ready :)

Success on x86_64-linux (full log)
Attempted: dlib
Partial log (click to expand)
-- Installing: /nix/store/ciry2d3b4ar3hyjc0nydbdpsqf3qf2d0-dlib-19.13/lib/cmake/dlib/dlib-release.cmake
-- Installing: /nix/store/ciry2d3b4ar3hyjc0nydbdpsqf3qf2d0-dlib-19.13/lib/cmake/dlib/dlibConfig.cmake
-- Installing: /nix/store/ciry2d3b4ar3hyjc0nydbdpsqf3qf2d0-dlib-19.13/lib/cmake/dlib/dlibConfigVersion.cmake
-- Installing: /nix/store/ciry2d3b4ar3hyjc0nydbdpsqf3qf2d0-dlib-19.13/lib/pkgconfig/dlib-1.pc
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/ciry2d3b4ar3hyjc0nydbdpsqf3qf2d0-dlib-19.13
strip is /nix/store/21ymadblbmsbb2bk4q7gl4kjasp8zmgd-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/ciry2d3b4ar3hyjc0nydbdpsqf3qf2d0-dlib-19.13/lib
patching script interpreter paths in /nix/store/ciry2d3b4ar3hyjc0nydbdpsqf3qf2d0-dlib-19.13
checking for references to /build in /nix/store/ciry2d3b4ar3hyjc0nydbdpsqf3qf2d0-dlib-19.13...

Success on aarch64-linux (full log)
Attempted: dlib
Partial log (click to expand)
-- Installing: /nix/store/xhxmyz8yybvj3qmf24g1sz3sgxl19g7m-dlib-19.13/lib/cmake/dlib/dlib-release.cmake
-- Installing: /nix/store/xhxmyz8yybvj3qmf24g1sz3sgxl19g7m-dlib-19.13/lib/cmake/dlib/dlibConfig.cmake
-- Installing: /nix/store/xhxmyz8yybvj3qmf24g1sz3sgxl19g7m-dlib-19.13/lib/cmake/dlib/dlibConfigVersion.cmake
-- Installing: /nix/store/xhxmyz8yybvj3qmf24g1sz3sgxl19g7m-dlib-19.13/lib/pkgconfig/dlib-1.pc
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/xhxmyz8yybvj3qmf24g1sz3sgxl19g7m-dlib-19.13
strip is /nix/store/qg2agrqkf240s656d207zqhipl0bc2id-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/xhxmyz8yybvj3qmf24g1sz3sgxl19g7m-dlib-19.13/lib
patching script interpreter paths in /nix/store/xhxmyz8yybvj3qmf24g1sz3sgxl19g7m-dlib-19.13
checking for references to /build in /nix/store/xhxmyz8yybvj3qmf24g1sz3sgxl19g7m-dlib-19.13...

wanted to catch soe sleep last night, but nox-review succeeded then :)
lrwxrwxrwx 1 ma27 users 76 Jun 14 03:57 result -> /nix/store/2z0hqczr8xg5kmnw4fwi330527nq2bc8-python3.6-face_recognition-1.2.2
lrwxrwxrwx 1 ma27 users 64 Jun 14 03:57 result-2 -> /nix/store/a9f1gvy134xdjgl8wag667jalxhrrlc8-python2.7-dlib-19.13
lrwxrwxrwx 1 ma27 users 54 Jun 14 03:57 result-3 -> /nix/store/ciry2d3b4ar3hyjc0nydbdpsqf3qf2d0-dlib-19.13
lrwxrwxrwx 1 ma27 users 64 Jun 14 03:57 result-4 -> /nix/store/5vin2fysxxzhmyjkhwi0r5srzbw4m2d1-python3.6-dlib-19.13
lrwxrwxrwx 1 ma27 users 76 Jun 14 03:57 result-5 -> /nix/store/xldyv3b57xniz07f1azxgckd72qqjfhm-python2.7-face_recognition-1.2.2


Params injection into statistics broken

This can easily be replicated by trying to inject params into a brand new actor.
Say for a fighter with double slice and trying to inject on the first "weapon" action non-agile weapon.
await token.actor.system.actions[0].attack({options: ["double-slice-second"]});
See https://discord.com/channels/880968862240239708/880969304365994034/1082500271793508373
Also skipDialog: true doesn't work either.

The root case is that roll options injected in the attack() or roll() call will get deleted if a RollOption RE exists for the same option and its value is false.
RollOption REs have a beforeRoll method, which adds the option to the set if the RE is true and deletes it if it's false.
Maybe this should be changed to just add the option on true and do nothing on false?  I.e., let injected options override the toggles.

This isn't a Statistic, and the method you're using doesn't accept that parameter. Feel free to make separate feature requests for each item mentioned.


⚠ ClamAV has degraded performance

In 557ca42, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 592 ms

Resolved: ClamAV performance has improved in 6e000f1.