Training Electra on 2 phases like Bert

Bert could be trained in 2 phases. The first phase with shorter length (128) then the second phase with longer length (512). The first phase accelerate training while the second phase makes the positional encoding learn longer sentences.
This does work as long as the "max_position_embeddings" is 512.
Can Electra trained on the same way or since it has a final layer for classifying each token it will not work ?
@clarkkev @stefan-it @mrm8488 @michelole , Your feedback is highly appreciated.

ok, so I tested it and it does work.
Electra could be trained on 2 phases.
1st phase, shorter sequence length.
2nd phase, longer sequence length.
This should accelerate training.

Hi, @agemagician
I pretrained my model (electra base) on 2 phases but the performance seems worse :/
my max_position_embeddings was also 512. and below is my training details
phase1: trained 1M steps with max_seq_length: 128
phase2: trained 0.1M - 0.3M steps with max_seq_length: 512
I would appreciate it a lot if you share more information about yours 🤝
thank you

Hi, did you finally improved your results?
El mar., 9 jun. 2020 a las 6:20, santaonchair<EMAIL_ADDRESS>escribió:

Hi, @agemagician https://github.com/agemagician
I pretrained my model (electra base) on 2 phases but the performance seems
worse :/
my max_position_embeddings was also 512. and below is my training
details
phase1: trained 1M steps with max_seq_length: 128
phase2: trained 0.1M - 0.3M steps with max_seq_length: 512
I would appreciate it a lot if you share more information about yours 🤝
thank you
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/google-research/electra/issues/66#issuecomment-641018489,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA34BHNRZYL7H2U3H3N7H33RVWZ7BANCNFSM4NQV7X7A
.

Hi :) Sorry for the late reply.
I finally improved my result with max_seq 512 but when I pretrained my model with max_seq 1024. it seems not working.
max_position_embeddings: 1024

phase1: trained 1M steps with max_seq 128, batch_size 400
phase2: trained 0.1M - 0.2M steps with max_seq 512, batch_size 200
phase3: trained 0.1M - 0.2M steps with max_seq 1024, batch_size 80 (phase3 result is way lower than phase2)


Support for multiple streams at a time

ERROR Error: Uncaught (in promise): RuntimeError: abort(Module.arguments has been replaced with plain arguments_ (the initial value can be provided on Module, but after startup the value is only looked for on a local variable of that name)) at Error
at jsStackTrace (https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation_solution_simd_wasm_bin.js:9:69361)
at stackTrace (https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation_solution_simd_wasm_bin.js:9:69537)
at abort (https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation_solution_simd_wasm_bin.js:9:43094)
at Object.get (https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation_solution_simd_wasm_bin.js:9:24082)
at https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation_solution_simd_wasm_bin.js:9:23905
at qa.h (http://localhost:4200/vendor.js:29137:414)
at ta (http://localhost:4200/vendor.js:29101:299)
at ua.next (http://localhost:4200/vendor.js:29102:91)
at http://localhost:4200/vendor.js:29127:619
at new ZoneAwarePromise (http://localhost:4200/polyfills.js:4544:29)
RuntimeError: abort(Module.arguments has been replaced with plain arguments_ (the initial value can be provided on Module, but after startup the value is only looked for on a local variable of that name)) at Error
at jsStackTrace (https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation_solution_simd_wasm_bin.js:9:69361)
at stackTrace (https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation_solution_simd_wasm_bin.js:9:69537)
at abort (https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation_solution_simd_wasm_bin.js:9:43094)
at Object.get (https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation_solution_simd_wasm_bin.js:9:24082)
at https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation_solution_simd_wasm_bin.js:9:23905
at qa.h (http://localhost:4200/vendor.js:29137:414)
at ta (http://localhost:4200/vendor.js:29101:299)
at ua.next (http://localhost:4200/vendor.js:29102:91)
at http://localhost:4200/vendor.js:29127:619
at new ZoneAwarePromise (http://localhost:4200/polyfills.js:4544:29)
at abort (https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation_solution_simd_wasm_bin.js:9:43125)
at Object.get (https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation_solution_simd_wasm_bin.js:9:24082)
at https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation_solution_simd_wasm_bin.js:9:23905
at qa.h (http://localhost:4200/vendor.js:29137:414)
at ta (http://localhost:4200/vendor.js:29101:299)
at ua.next (http://localhost:4200/vendor.js:29102:91)
at http://localhost:4200/vendor.js:29127:619
at new ZoneAwarePromise (http://localhost:4200/polyfills.js:4544:29)
at Y (http://localhost:4200/vendor.js:29127:389)
at http://localhost:4200/vendor.js:29137:254
at resolvePromise (http://localhost:4200/polyfills.js:4328:31)
at http://localhost:4200/polyfills.js:4235:17
at rejected (http://localhost:4200/vendor.js:378694:89)
at _ZoneDelegate.invoke (http://localhost:4200/polyfills.js:3489:26)
at Object.onInvoke (http://localhost:4200/vendor.js:440180:33)
at _ZoneDelegate.invoke (http://localhost:4200/polyfills.js:3488:52)
at Zone.run (http://localhost:4200/polyfills.js:3251:43)
at http://localhost:4200/polyfills.js:4392:36
at _ZoneDelegate.invokeTask (http://localhost:4200/polyfills.js:3523:31)
at Object.onInvokeTask (http://localhost:4200/vendor.js:440167:33)

ok ,cloning file and work around is for multiple streams?

ok sure

I have tried by cloning below files and still i am facing same issue
selfie_segmentation_landscape.tflite
selfie_segmentation_solution_simd_wasm_bin.data
selfie_segmentation_solution_simd_wasm_bin.js
selfie_segmentation_solution_simd_wasm_bin.wasm
selfie_segmentation_solution_wasm_bin.js
selfie_segmentation_solution_wasm_bin.wasm
selfie_segmentation.binarypb
selfie_segmentation.js
selfie_segmentation.tflite
Please let me know anyother i have to clone.
Thanks.


Make Pipecat Flows an independent package

Please describe the changes in your PR. If it is addressing an issue, please reference that as well.
Pipecat Flows is now a standalone project which serves as an add-on to Pipecat:
https://github.com/pipecat-ai/pipecat-flows
Examples and installation instructions are moved to the pipecat-flows repo.

LGTM!


kele-get: Namespace filtering unnecessary

kele-get currently asks user to select a namespace to filter for if the requested resource kind is namespaced. The completion candidate set for the resource names of that given kind are then filtered for those that belong to that specific namespace.
This is largely a manufactured roadblock that is feasible to remove. The DeploymentList output, for example, returns the manifests of all Deployments across all namespaces -- which kele--fetch-resource-names then itself filters candidates from based on namespace.
Rather than up-front ask users to select the namespace to filter by, here's an alternative:

We present all the resource name candidates to user in a single list;
That list's entries are group by namespace, using group-function;
Then, when user selects an entry, the corresponding entry's manifest (which is already returned in full as part of the List) is forwarded directly to kele--render-object

As example, consider the following:

(defun jjin/foo-group (completion transform)
  (if transform
      completion
    (car (s-split " " completion))))

(defun jjin/foo-complete (str pred action)
  (if (eq action 'metadata)
      '(metadata (group-function . jjin/foo-group))
    (complete-with-action action '("foo bar" "foo baz" "what 1" "what 2") str pred)))

(completing-read "Select thing: " #'jjin/foo-complete)

This would output a completion buffer like follows:

Segmenting resource candidates by namespace in this way seems promising.

consult--multi-group could be useful reference here.


doc: updated readme so key isn't optional

Goal
Updated README to reflect the fact that a Google Maps API key is no longer optional.

Coverage remained the same at 100.0% when pulling cbc797ac77dd5313d5c1ed5bd41216a2e4297fa8 on maps-key-doc into 5e56d6b788196585db1e8d350818a21b7a1e0af4 on master.


chore: add test branch as workflow trigger

:tada: This PR is included in version 1.1.0-test.1 :tada:
The release is available on:

npm package (@test dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.1.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Deprecate /masternode/list -/validate endpoints

responding with a message explaining the deprecation.
We'll add the new endpoints in a later PR

Please do not comment code. Just remove it. If we need it we always can use git history.

done!


Mysql: Entity __ has database generated fields that don't contain a primary key.

Hello, I was attempting to use your repository with a Mysql database. I have an entity which has a composite primary key of which none of the keys are database generated. There is however a ModifiedTimestamp, which is calculated on insert and update. Sample Here:
    public class AggregatedRow
    {
        [Key]
        public int Id1{ get; set; }
        [Key]
        public string Id2{ get; set; }
        [Key]
        public string Id3 { get; set; }
        public int AggregateValue { get; set; }
        [DatabaseGenerated(DatabaseGeneratedOption.Computed)]
        public System.DateTime ModifiedTimestamp { get; set; }
    }

Which is intended to correlate to the following table:
CREATE TABLE `AggregatedRow` (
  `Id1` int(11) NOT NULL,
  `Id2` varchar(255) NOT NULL,
  `Id3` varchar(55) NOT NULL,
  `AggregateValue` int(11) NOT NULL,
  `ModifiedTimestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`FileCollectionCycleId`,`StatusName`,`EventTimestamp`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

When I attempt to insert into this table with FastCRUD, I get:
Entity 'AggregatedRow' has database generated fields that don't contain a primary key.

Which looks like it is getting thrown from MySqlBuilder.ConstructFullInsertStatementInternal. It looks like my entity has a value of 0 for: InsertKeyDatabaseGeneratedProperties. Looking at the following code, it looks like those checks could be against KeyProperties == 0 instead. I'd make the pull request myself, but I'm not entirely sure if there's something I am missing.
If there's anything I can do to help, please let me know.
Also, thanks for the great work on this repo!

Interesting. Can you check if it works without the DatabaseGenerated property?

Given the right conditional statement, could something like this work to construct the select statement?
sql += $"SELECT {databaseGeneratedColumnSelection} FROM {this.GetTableName()} WHERE" + this.ConstructKeysWhereClause(this.GetTableName())

I don't think ConstructKeysWhereClause requires an alias but other than that it looks good. Could you create a PR? We're in the middle of the beta for 2.4.0 so this one couldn't have come at a better time.

Probably should have mentioned that I already started on it tonight: https://github.com/MoonStorm/Dapper.FastCRUD/pull/35.

Thanks for providing a fix.  The existing tests all passed and a preview package is available on nuget.


Fixed #6296 -- Removed localization for pk values in toolbar js (#6294)

https://github.com/divio/django-cms/issues/5970

Coverage remained the same at 82.794% when pulling b91f29ca27aab8a6fae43f1fb32ad6c573f9c7df on vxsx:bugfix/localized-pks-3.4.x into f292f8abef5a7ab3ff296fa45aade9550b3c62d6 on divio:release/3.4.x.


issue #1881 - add logic to allow reading of provenance resources

For search, skip compartment checking; direct Provenance search is already scoped to the patient compartment
and for revincluded Provenance we will let the access to the targeted resources carry the day.
For direct access (read/vread/history) we will read the targets of the Provenance and allow the access if the user
has access to one or more of them.
Additionally, I beefed up the javadoc for
ReferenceUtil.createReferenceValueFrom and ReferenceValue from
fhir-search since I had to study them to use them.
Finally, I added a variant of TestUtil.getMinimalResource which doesn't
force the Format argument (which is almost never relevant).
Signed-off-by: Lee Surprenant<EMAIL_ADDRESS>

LGTM - just one comment on an error msg

LGTM - just one comment on an error msg


GraphQL support

Evaluate adding GraphQL support to Helidon.
Justification:
The concept of Back End Services For Front end(BFF pattern) has compelled us think beyond REST and GraphQL offers excellent consolidation reducing network traffic as the number of HTTP calls are consolidated in a single POST operation.
Please let me know should you have subsequent questions, I have highlighted the areas relevant for our requirement below
Detailed explanation:
In the traditional frontend and backend design, usually the Mobile App or Website directly accesses the backend service, the backend microservices call each other, and then return the final result to the frontend client.
For clients (especially mobile), too many HTTP requests are expensive.
Therefore, in order to minimize the number of requests, the frontend generally tends to obtain associated data through a single API.
It means that sometimes the backend will do some UI-related logic processing to meet the needs of the client .
The biggest difference in BFF’s frontend and backend architectures is that the frontend (Mobile, Web) no longer directly accesses the backend microservices, but rather accesses it through the BFF layer. And each client will have a BFF service.
With BFF, there are fewer mutual calls between microservices. This is because some UI logic is processed at the BFF level.
Advantages of GraphQL API:

On demand data model definition
Get many resources in a single request

GraphQL vs REST:
For the comparison between GraphQL and REST, there are mainly the following differences:

Data Acquisition: REST lacks scalability and GraphQL can be accessed on demand. The payload can be extended when the GraphQL API is called.

API calls: REST’s operation for each resource is an endpoint, and GraphQL only needs a single endpoint, but the post body is not the same.

Complex data requests: REST requires multiple calls for nested complex data, GraphQL calls once, reducing network overhead.

Error code processing: REST can accurately return HTTP error code, GraphQL returns 200 uniformly, and wraps error information.

Version number: REST is implemented via v1/v2, and GraphQL is implemented through the Schema extension.

https://github.com/eclipse/microprofile-graphql

FYI the plan is for initial version to work with the soon to be release microprofile-graphql 1.0.3 spec.


Better example of how to use ProviderLogin

For instance callbackUri={${window.location.origin}/welcome} seems to be required? Also there's a typo 'inputPlaholder'

Good catch - the callbackUri is used to pass into solid-auth-client (so we know where to go on-return). The docs can be updated to reflect this.

Issues have been fixed and will be included in the next release.


Prevent "go back to sleep" messages to be added to the wakeup queue

Describe the bug
I just noticed this in the log:
23:55:54.785 CNTRLR » [Node 015] Sending node back to sleep...
23:55:54.804 CNTRLR   Failed to execute controller command after 1/3 attempts. Scheduling next try i
                      n 100 ms.
23:56:13.556 CNTRLR   [Node 015] The node did not respond after 1 attempts.
                      It is probably asleep, moving its messages to the wakeup queue.
23:56:13.556 CNTRLR   [Node 015] The node is now asleep.

It feels weird that a "go back to sleep" message gets added to the wakeup queue. If these are already being filtered out, I think it would be great to slightly change the logging here. If not, then I think we should not send a "go to sleep" immediately when a device wakes up.
In "normal" situations, I see this:
23:39:54.980 CNTRLR « [Node 015] received wakeup notification
23:39:54.981 CNTRLR   [Node 015] The node is now awake.
23:39:56.211 CNTRLR » [Node 015] Sending node back to sleep...
23:39:56.306 CNTRLR   [Node 015] The node is now asleep.

I think a potential fix would be to move the logging at https://github.com/zwave-js/node-zwave-js/blob/910155c88ffb2f8ab6ddce6adc2c0dbb57a135f6/packages/zwave-js/src/lib/driver/Driver.ts#L1511 to https://github.com/zwave-js/node-zwave-js/blob/910155c88ffb2f8ab6ddce6adc2c0dbb57a135f6/packages/zwave-js/src/lib/driver/Driver.ts#L1521
Device information
It's a recessed door sensor 5 from Aeotec. Node ID 15.
Last Known Working Configuration

[ ] New device

[ ] Previously working device (node-zwave-js)

Which library version/docker image/adapter version?
Have you made any recent configuration changes to the device? Describe.

[ ] Previously working device (other platform)

Which platform?
Have you made any recent configuration changes to the device? Describe.

Installation information
How did you install node-zwave-js?

[ ] zwavejs2mqtt (latest) docker image
[ ] zwavejs2mqtt (dev) docker image
[ ] ioBroker.zwave2 adapter
[ ] Pkg
[ ] Manual Docker build

node-zwave-js branch:
zwavejs2mqtt branch:

[ ] Manually built (as described in the docs)
[ ] Other:

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '...'
Scroll down to '...'
See error

Additional context
Add any other context about the problem here.
Logfile:

The logging is not optimal here, but I can assure you that these kinds of messages are not moved to the wakeup queue. The same message would also be logged if there was no message left in the queue.
https://github.com/zwave-js/node-zwave-js/blob/910155c88ffb2f8ab6ddce6adc2c0dbb57a135f6/packages/zwave-js/src/lib/driver/Driver.ts#L2309-L2327


Request body is not valid JSON

Description

After i use swagger2.0 swagger-codegen generate the flask for python service, when posting data from web, the error turns out :
{
  "detail": "Request body is not valid JSON",
  "status": 400,
  "title": "Bad Request",
  "type": "about:blank"
}

what can i do now ?
Expected behaviour
Actual behaviour
Steps to reproduce
Additional info:
Output of the commands:

python --version
pip show connexion | grep "^Version\:"

well, the error message is pretty self explanatory. other json validators are also not working with this:

if you somehow want to make this work, then i suggest to serialize those line breaks.

@drummerwolli is right, closing.


Netstandard20

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@41fe571). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master       #1   +/-   ##
=========================================
  Coverage          ?   95.55%
=========================================
  Files             ?       18
  Lines             ?      810
  Branches          ?        0
=========================================
  Hits              ?      774
  Misses            ?       36
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 41fe571...2e6d4a4. Read the comment docs.


CSRF development wss

Add wss (cable) to the CSRF policy in development. Otherwise cable (Hotwire) won't work in safari, at least in development mode.

Thanks.

Yes, you are right, thanks a lot!


Remove Docker Compose make targets

We were only making two references to the services-start target in the README.md.
Now we explictely tell developers to use docker-compose up <service> if they want to use it 😄

🦀 Requesting reviewers for this pull request:

@thermech (contributor with 0 commits in the last 90 days and 8 commits overall)
@olivierbuffon (expert reviewer for the elixir stack)


update: go

Name: go
CVEs: CVE-2023-39325
CVSSs: n/a
Action Needed: update to >= 1.20.10
Summary: net/http: rapid stream resets can cause excessive work. A malicious HTTP/2 client which rapidly creates requests and immediately resets them can cause excessive server resource consumption. While the total number of requests is bounded to the http2.Server.MaxConcurrentStreams setting, resetting an in-progress request allows the attacker to create a new request while the existing one is still executing.
HTTP/2 servers now bound the number of simultaneously executing handler goroutines to the stream concurrency limit. New requests arriving when at the limit (which can only happen after the client has reset an existing, in-flight request) will be queued until a handler exits. If the request queue grows too large, the server will terminate the connection.
This issue is also fixed in golang.org/x/net/http2 v0.17.0, for users manually configuring HTTP/2.
The default stream concurrency limit is 250 streams (requests) per HTTP/2 connection. This value may be adjusted using the golang.org/x/net/http2 package; see the Server.MaxConcurrentStreams setting and the ConfigureServer function.
This is CVE-2023-39325 and Go issue https://go.dev/issue/63417. This is also tracked by CVE-2023-44487.
See also https://groups.google.com/g/golang-announce/c/iNNxDTCjZvo.
refmap.gentoo: https://bugs.gentoo.org/915555

Fix will be in next alpha release.


Using [filewords] in

Describe the bug
trying to use file words in separate text files for each image results in an error, am I setting it up wrong or is this something that is just not supported with this colab
concepts_list = [
{
"instance_prompt":      "[filewords]",
"class_prompt":         "a illustration in a style",
"instance_data_dir":    "/content/data/gotothemoon",
"class_data_dir":       "/content/data/illustration"
},
]
Reproduction
add a text file in the data directory that matches an image filename and use the [filewords] keyword in the instance prompt
Logs
Traceback (most recent call last):
  File "train_dreambooth.py", line 822, in <module>
    main(args)
  File "train_dreambooth.py", line 613, in main
    for batch in tqdm(train_dataloader, desc="Caching latents"):
  File "/usr/local/lib/python3.7/dist-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 681, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 721, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "train_dreambooth.py", line 323, in __getitem__
    instance_image = Image.open(instance_path)
  File "/usr/local/lib/python3.7/dist-packages/PIL/Image.py", line 2896, in open
    "cannot identify image file %r" % (filename if filename else fp)
PIL.UnidentifiedImageError: **cannot identify image file** '/content/data/gotothemoon/00008-0-1959YouWillGoToTheMoon11.txt'
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/accelerate_cli.py", line 43, in main
    args.func(args)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 837, in launch_command
    simple_launcher(args)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 354, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', 'train_dreambooth.py', '--pretrained_model_name_or_path=runwayml/stable-diffusion-v1-5', '--pretrained_vae_name_or_path=stabilityai/sd-vae-ft-mse', '--output_dir=', '--revision=fp16', '--with_prior_preservation', '--prior_loss_weight=1.0', '--seed=1337', '--resolution=512', '--train_batch_size=1', '--train_text_encoder', '--mixed_precision=fp16', '--use_8bit_adam', '--gradient_accumulation_steps=1', '--learning_rate=1e-6', '--lr_scheduler=constant', '--lr_warmup_steps=0', '--num_class_images=260', '--sample_batch_size=4', '--max_train_steps=4000', '--save_interval=10000', "--save_sample_prompt=a children's book illustration in the style of gotothemoon", '--concepts_list=concepts_list.json']' returned non-zero exit status 1.

System Info
Tesla T4, 15109 MiB, 15109 MiB on google colab

Yeah I was wondering if this repo allows us to use filewords/captions on our images and in the bat/shell file so we can have more customization with separating pictures within an overall concept. Would love to know how to do this as well

I don't see anything in train_dreambooth.py that would suggest this feature exists. The term "filewords" does not appear in the script.
Would be a nice addition, although I remember hearing from folks on Discord that the careful labeling of input images may not have a positive effect on Dreambooth training.

[filewords] is implemented in the "Automatic1111 extension" fork for this repo:
https://github.com/d8ahazard/sd_dreambooth_extension


Adds http2 test against Suricata bug fixes

Requires https://github.com/OISF/suricata/pull/5424

https://github.com/OISF/suricata/pull/5460 got merged, so tests should now pass

Merged in https://github.com/OISF/suricata-verify/commit/f87e26d06004621174150eb61c4f15adcc7939e8


[ISSUE#4263]Delete topic route info based on cluster when delete topic.

#4263

Coverage increased (+0.04%) to 52.161% when pulling c74e3a7d60395424eabcc62f513c5c83bbe132a2 on sunxi92:delete-topic-route-based-on-cluster into 5d2bf3573c683eb08f028b3a6496d1cbfe1af33b on apache:develop.

Is the purpose of this pr to support multiple clusterNames?
otherwise, the code is very similar to the original, I think, the same parts should be reused as much as possible.

In DeleteTopicSubCommand, The parameter 'clusterName' is mandatory. So deleting a topic with the parameters 'topic' and 'clusterName' is clear.
DeleteTopicSubCommand  should delete topic route info based on clusterName in nameserver. But the processing logic of  nameserver is  deleting all the topic route info in topicQueueTable before updating the code.  @lwclover

eleteTopicSubCommand should delete topic route info based on clusterNa

In my opion, You don't have to write compatibility code.

Codecov Report

Merging #4268 (1a96c2c) into develop (4574d59) will increase coverage by 0.02%.
The diff coverage is 23.07%.

@@              Coverage Diff              @@
##             develop    #4268      +/-   ##
=============================================
+ Coverage      47.90%   47.92%   +0.02%
- Complexity      5001     5046      +45
=============================================
  Files            633      642       +9
  Lines          42523    42803     +280
  Branches        5573     5604      +31
=============================================
+ Hits           20371    20515     +144
- Misses         19654    19771     +117
- Partials        2498     2517      +19

Impacted Files
Coverage Δ

...g/apache/rocketmq/client/impl/MQClientAPIImpl.java
13.96% <ø> (+0.10%)
:arrow_up:

...r/namesrv/DeleteTopicFromNamesrvRequestHeader.java
0.00% <0.00%> (ø)

...apache/rocketmq/tools/admin/DefaultMQAdminExt.java
41.40% <0.00%> (ø)

...he/rocketmq/tools/admin/DefaultMQAdminExtImpl.java
39.83% <0.00%> (ø)

...tmq/tools/command/topic/DeleteTopicSubCommand.java
27.27% <0.00%> (ø)

...e/rocketmq/namesrv/routeinfo/RouteInfoManager.java
78.83% <26.31%> (-2.69%)
:arrow_down:

...tmq/namesrv/processor/DefaultRequestProcessor.java
74.28% <100.00%> (ø)

...va/org/apache/rocketmq/store/FlushDiskWatcher.java
81.25% <0.00%> (-9.38%)
:arrow_down:

...broker/processor/AbstractSendMessageProcessor.java
40.62% <0.00%> (-8.91%)
:arrow_down:

...he/rocketmq/remoting/protocol/RemotingCommand.java
72.45% <0.00%> (-6.18%)
:arrow_down:

... and 81 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4574d59...1a96c2c. Read the comment docs.

the method deleteTopic(final String topic) is useless, The new method need testcase.

@sunxi92  I'm sorry I told you removed the compatibility code.

Hi, @sunxi92 , IMO, compatibility needs to be considered here. If clusterName exists, delete it based on the clusterName. Otherwise, delete the whole topic. Otherwise, there will be issues with older versions of the admin tool or console calls.

done!


Cleanup removed benchmarks

Which problem is this PR solving?
This resolved a bit of confusing documentation, stating that benchmarks need to be checked, when those don't exist anymore.
See:

Removal of benchmarks: https://github.com/open-telemetry/opentelemetry-js/pull/2462
https://github.com/open-telemetry/opentelemetry-js/pull/2462

Short description of the changes
Type of change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?
This is tested through the existing test suite.
Checklist:

[x] Followed the style guidelines of this project
[x] Unit tests have been added
[x] Documentation has been updated

Thanks for cleaning this up


A possible null pointer dereference

Hi, can fdopen return null value? the value should be checked before passing to fgets? Thank you for checking.
https://github.com/openssl/openssl/blob/9710d72b95f4fc218ed613f42dc90ad0d263b14f/apps/speed.c#L3511-L3512

I think that if fdopen somehow fails, the fgets call will return NULL, so the while guard check will fail. The check seems redundant.

fgets crashing when passed a NULL stream pointer is permitted behaviour I believe.
That some libc's check and fail gracefully, doesn't mean all do.

Should be closed, this issue has been resolved by the PR #19651.


Implementation of the Parser

[ ] Implement the parser 🎉
[ ] Create a dxf document from the result of the parser 🎉
[ ] Link the document to the dxf writer to update or save the dxf 🎉
[ ] Adding tests if possible 🤬

See @dxf/parser


Run compatibility check based on PR base branch

Description
Run compatibility check based on PR base branch rather than main branch. See pull_request_target and pull_request
Related Issues
Resolves #9371

Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Commits are signed per the DCO using --signoff
[ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@gaiksaya Any progress here?

@gaiksaya Any progress here?

Sorry for the delay. Updated the PR with the changes. Here is sample comment: https://github.com/gaiksaya/OpenSearch/pull/8


kernel: s390 configs

The patch 0003-serial-forbid-8250-on-s390.patch fixes a conflict between
the ttysclp0 and serial 8250 console. The patch is already upstream and
it has been introduce in version v4.16-rc1.
However, it is not backported. See https://lore.kernel.org/patchwork/patch/861679/
The s390_kata_kvm_4.14.x configs have been obtained by applying the patch and
the combination:
make defconfig kvmconfig localyesconfig
Fixes: #246
Signed-off-by: Alice Frosi<EMAIL_ADDRESS>

Also, before the CI runs, it's going to fail as every kernel config file (to any architecture) needs to bump the version number in:
kernel/kata_config_version

@grahamwhaley thanks I'll split the PR in 2 commits and the large difference is intentional. The actual s390 configs have been taken from linuxkit. Principally, they work but they're tune for that project and they're difficult to reproduce if you want to support another kernel version. I just notice that the defconfig doesn't activate the vsock options. I still need to test this feature. We can leave this PR open until I verified or merge it and I'll update the configs later on. @jodh-intel thanks I'll update the kata version!

lgtm

Ping @kata-containers/packaging.

lgtm
/test


Add support for ItemID overrides (re: reverting to old divine potion appearances)

Is your feature request related to a problem? Please describe.
Currently, it is not possible to override ItemIDs the same way that we can with overriding SpriteIDs or WidgetIDs. This means that it is not possible to -- as an example -- create a plugin in the Plugin Hub where one could revert the appearance of divine potions (and egniol potions) to their pre-July 2020 look.
Describe the solution you'd like
Ideally, I'd like for developers to have the ability to override ItemID sprites freely with custom images.
Describe alternatives you've considered
AFAIK there is no alternative currently for the proposed behavior (asked in the RL Discord and it was confirmed to not be possible with the current limitations).
Additional context
Here is my current implementation of such a plugin (just trying it out): https://github.com/parasailer/old-divine-potions
I realized that the sprite overriding didn't work, which is when I turned to Github/Discord for more answers.

duplicate #7954


Resolve errors for import paths fs and path

With Deno v1.25.0, run:
deno info<EMAIL_ADDRESS>
And you can see in the output that there are resolve errors for the import paths fs and path…
Relative import path "fs" not prefixed with / or ./ or ../ and not in import map from<EMAIL_ADDRESS>(resolve error)

Source: https://github.com/lucacasonato/deno-puppeteer/blob/45b3162585b98ad8d54abeb56f48ecbb17c614eb/vendor/puppeteer-core/puppeteer/common/IsolatedWorld.js#L393
Relative import path "path" not prefixed with / or ./ or ../ and not in import map from<EMAIL_ADDRESS>(resolve error)

Source: https://github.com/lucacasonato/deno-puppeteer/blob/45b3162585b98ad8d54abeb56f48ecbb17c614eb/vendor/puppeteer-core/puppeteer/common/ElementHandle.js#L665

I'm also facing this problem.


MAE6286 project: Compact finite difference schemes

Hi Prof. Barba
This is my submission for the course project. Please let me know if you have any recommendations/ suggestions so I can improve this work before the deadline.
Thank you,
Shantanu.

Hey Shantanu, I've read you project, it is really interesting. Here are some suggestions:

In this report ---->  In this notebook (to be consistent with previous notebook)

for some reason nbviewer is not rendering this line:
where: \begin{equation} 0 \leq x, y \leq 1 \end{equation}
try using $ $

Is there a reason why you put an image of the countour plots of u and laplacian
of u instead of the code that generates them? I think it would be better to have the code.

How you obtain the Pade scheme for f_xx ? I'm not following how you get it from the
explanation in the section "Conducting a Pade scheme", because you explained for
the first derivative. If you get it from another place, can you explain a little bit more about it?

It will help a lot if you add docstrings in the functions, as we did in the notebooks for the class.

typo :  conamination ---> contamination

Hey Naty,
Thanks for the feedback. Shall I make the changes and update the document on github? Do you want the changes today or can I make them over the weekend?
Shantanu

You can make updates whenever you want. I won't be merging any pull requests until I go over all the projects, and it will take a while!

Hi Naty, Prof. Barba:
I've made the changes that you suggested earlier this week. Particularly I included the derivation for the second order derivative formulation. Could you go check it out and let me know what you think of it? Thank you.


How to exclude node_modules and other directories from the scan?

Describe the bug
I have a react project, and I would like to exclude node_modules and other specific directories
The Azure Pipeline task is below
- task: Checkmarx AST@2
  displayName: 'Checkmarx AST'
  condition: and(always(), eq('${{ parameters.ShouldRunCheckMarx }}', true))
  continueOnError: true
  inputs:
    CheckmarxService: 'CheckMarx-Connection'
    projectName: $(Build.Repository.Name)
    branchName: $(Build.SourceBranchName)
    tenantName: ${{ parameters.tenantName }}
    additionalParams: |
      --file-filter '!node_modules'

This is not working.

@ausshadu thanks for the bug report, we will get back to you asap. This should indeed work.
FYI @pedrompflopes

Hi @jbrotsos
This is working now. I looked at pipeline configurations and found a small typo. Apologies for the confusion.
I had written
--file-filter: '!node_modules'

Instead of
--file-filter '!node_modules'

Now this is scanning excluding the node_modules directory.


Decouple from Guzzle

To completely decouple from Guzzle I've used the HTTPlug abstraction. This change does not break BC.
The benefits of this is users stuck with Guzzle5 can still use this library.
FYI: other libraries using HTTPlug are Geocoder, Mailgun, FosCache, KnpGithubApi and more.

FYI there is a similar and related conversation in the oauth2-client

Yes, thank you for the link.
What do you think of this PR @stevenmaguire?

Hi @Nyholm,
Firstly, thank you very much for this PR! Wow. I love it.
I'm going to look further into the code to see if I can provide any feedback and I'll checkout the discussion that @stevenmaguire referred to.
Watch this space :)

Thanks for the PR @Nyholm
:+1: for me

Hi @Nyholm, so obviously this discussion is way stale. But I have an update.
In the new version, we have completely decoupled from any particular HTTP implementation. We're using PSR-7, PSR-17 and PSR-18 only. We recommend the use of Guzzle, but we don't require it.
Thank you for opening this discussion years ago, it's definitely been factored in with the rewrite.


The language is useless.

In your implementations, the [CLS] token of the BertEmbedding is used. Without any attention operation, the [CLS] token of the BertEmbedding cannot take any information about the input description. In other words, the language modal is useless in the current implementations. Do I have any misunderstanding here?

We just use BertEmbedding as the tokenizer. During the training phase, the tokenizer can be updated and the mean token can be used. We also found that using only the trained visual branch for inference can achieve good results, due to the large amount of ambiguity in language prompts.

In your implementations, whatever the input description is, the output of the [CLS] will be the same. Therefore, I don`t think the performance improvement is related to the adding of the language description.


Bug Report - ESLZ Deployment Using AzOps in Azure US Government

Describe the bug
I was able to deploy ESLZ to US Government cloud with GitHub integration. However, two issues I found so far after deployment  ....
(1) Right after deployment when AzOps try to fetch current state of Azure, it fails. What I found is that pull.yml file does not use "-environment AzureUSGovernment" as part of Connet-AzAccount command.  So, pull.yml file need to be updated to correct this. (Side note - similar changes are needed for push.yml and validate.yml file as well)
(2) Whenever I tried to deploy new policy, it by default targets location as northequrope. In json template location is defined as [deployment().location]. I did not find any solution to resolve this issue.
How can I resolve this?
Steps to reproduce

Screenshots

Thank you @jtracey93 and @njpatel54.
AzOps is indeed supported in the sovereign clouds such as US Gov and Azure China.
To solve the issue with deployment, you need to change the setting Core.DefaultDeploymentRegion that Jack pointed out to one of the USGov supported regions, such as usgovvirginia.
We will use this issue to track creation of guidance for this. :)

Update -
I just tested everything end-to-end works in Azure US Government. Couple of changes and it all works now. Changes made are
(1) In "Action Secrets", I added "ARM_ENVIRONMENT" with value "AzureUSGovernment".
(2) Edited pull.yml, push.yml and validate.yml file to include ARM_ENVIRONMENT: ${{ secrets.ARM_ENVIRONMENT }} and then modified Connect-AzConnect commandline to include -Environment $env:ARM_ENVIRONMENT.
(3) Edited settings.json file to include "Core.DefaultDeploymentRegion": "usgovvirginia",.

Thank you for the update!

I just test the new updates to resolve this issue and I found one minor change that needs to be made ....
(1) After fresh ESLZ deployment in AzureUSGovernment environment, when initial pull (Enterprise-Scale Deployment) occurs, it fails with following error.

(2) I tracked down the action.yml file under .github/actions/sharedSteps. Looks like when following command carried out, it fails.
(ConvertTo-SecureString -String ${{env.ARM_CLIENT_SECRET}} -AsPlainText -Force)
What it seems that if this en.ARM_CLIENT_SECRET variable is passed on with quotes then it succeeds.


difference between weave, wandb projector and vega-lite with wandb?

Hi!
I'm so excited to meet such great tool and thanks for contributors! I'm recently trying different tools like:  wandb projector and vega-lite to visualize on wandb. Is it a little bit easy to being confused about these tools, isn't it?
I'd love to ask some questions:

I want to know the differences between weave and them, and in which situation i should use waeve and not to use.
Should I move on to Weave? My usecase mainly falls into drawing fantastic research graph and embeddings visualizing.

bests,
JC

Hello - thank you for your issue. As part of our effort to cleanup issues in the Weave repository, I am closing this due to inactivity. We can reopen if applicable in the future.


Make artisan command to up Laravel Sail

Details of Improvement
Some Laravel users are dissatisfied with typing the command to up the Laravel Sail every time.
So, create an artisan command to up the Laravel Sail and support users.
Expected behavior

Enter artisan command
(ex: php artisan sail:up)

Up the Sail

You can start using Laravel Sail

No plans for this sorry


mobile: Fix missing logging output in Swift integration tests

In Swift, print goes to stdout buffering. The fix is to flush the stdout and stderr before the end of the test.
Risk Level: low (tests only)
Testing: CI
Docs Changes: n/a
Release Notes: n/a
Platform Specific Features: n/a

/assign @abeyad


[Xamarin Shell] Using images that are larger than icon space scale tab bar significantly on iOS, Droid it works fine. [Xamarin Shell]

[Xamarin Shell]
The typical size of an icon is around 30 x 30 px.
Lets say for some reason I wanted to use a 512 x 512 px .png for an icon.
On iOS this scales the tab bar to the size of the image, NOT the other way around.
On Droid, works as desired. 512px image is scaled to fit the area provided for the tab icon.

In this image you can clearly see the icons are larger than the tab bar. The size of the icon is 60 x 60px.
Below is an example of using a 512x 512px .png

**[Xamarin Shell]**

Hi @XXcharlosXX
Thanks for logging this issue, but it really belongs in the engineering repo.


SSL Certificate

For non-enterprise projects you can request an App Service Managed Certificates from Microsoft/GeoTrust for free and then bind it to your existing custom domain; however, this feature is currently in preview so a terraform provider might not exist yet.

they do have a provider for managed certificates!
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/app_service_managed_certificate

they do have a provider for managed certificates!
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/app_service_managed_certificate


Using exclusive lock instead of shared locks in logical host address write/read routines

Title: Using exclusive lock instead of shared locks in logical host address write/read routines
Description:
Currently, the logical host write and read methods are using shared locks to avoid race:
Write:
https://github.com/envoyproxy/envoy/blob/10a10039b3a82d43ff47c319e0ef4faf229f3327/source/extensions/clusters/common/logical_host.h#L45
Read:
https://github.com/envoyproxy/envoy/blob/10a10039b3a82d43ff47c319e0ef4faf229f3327/source/extensions/clusters/common/logical_host.h#L65
Should we change this to just use exclusive lock for performance reasons?

/assign @jmarantz

@jmarantz is recommending making this change. Do we need some load test results before we making this change or we can directly make the change and send the PR?

The change is committed by: https://github.com/envoyproxy/envoy/pull/32919


Add recaptcha to login

This should address #707

I don't think we want a captcha on every single login, that would be somewhat annoying I think. It would be better if it only required a captcha after N failed login attempts for that IP address or for that user in the last Y minutes (rolling).
It's also not a complete solution because we also have the basic auth which is used in automated systems, so we probably want the two systems to share attempt counters, and have it so that for basic auth if you get too many failed attempts it just rejects your login, and for regular auth it shows a captcha.

Yep, missing basic auth throws a bit of a wrench into this for sure. Captcha doesn't help much in that scenario and I imagine that's the path that would most likely be targeted by attackers.
Per-IP is going to be an issue with IPv6 (noted here). Also, I'm not sure how fond I am of per-IP blocks on auth given it has the potential to DoS legitimate users. Is this approach one used in PyPI?
So, I'll keep what's done so far, but I'll also add global system, per user and potentially per-IP sliding windows that will either trigger recaptcha in the FE or lock out accounts in the BE.
Sound reasonable?

Right now in PyPI we have separate counters for user and for IP address. I think it's probably OK to drop the per IP address if we replace it with a global threshold that starts to trigger a captcha for all logins if the global failed logins reach some sort of threshold. We would (probably?) ignore that in the basic auth, but that's OK because longer term I plan to deprecate the basic auth and move to something generated by the system with high entropy (API key, Client TLS Certificate? Something Else?!) that shouldn't be vulnerable to that and at that point the via-basic-auth bypass for the global failed logins should be closed.

To be clear, I'd suggest that we only ignore the global threshold when authenticating with basic auth, the per user block should not be ignored.

#707 was fixed in #1530 so this can be closed.
Thanks for taking a shot at this @demianbrecht!


Avoid unnecessary send_replace calls in seqwait

The notifications need to be sent whenever the waiters heap changes, per the comment in update_status. But if 'advance' is called when there are no waiters, or the new LSN is lower than the waiters so that no one needs to be woken up, there's no need to send notifications. This saves some CPU cycles in the common case that there are no waiters.

@erikgrinaker , I noticed this some time ago while working on some other refactorings, but forgot about it back then. I was reminded by the pageserver CPU usage issues now. I'm not sure how much of a difference this makes, but I remember I saw at least some CPU cycles being used on the Mutex in the status_channel, and the patch seems simple enough.
Please do verify my thinking that this is safe. And do you have a setup now to do profiling and verify if this makes an impact?

do you have a setup now to do profiling and verify if this makes an impact?

I'll check when I have some time.


Fix svg not being saved as optimized

Also checked "Convert Colors With Editor" property for all icon resources.


Run build step in Docker build process

Fixes #288
change-type: patch

@hedss
Only the files necessary to run Procbots end up in the final image.
I've kept package.json as it is useful for checking versions when one is debugging.
The working directory looks like this
drwxr-xr-x   8 root root  4096 Oct 13 11:38 .
drwxr-xr-x   8 root root  4096 Oct 13 11:22 ..
drwxr-xr-x   2 root root  4096 Oct 13 11:38 bin
drwxr-xr-x   6 root root  4096 Oct 13 11:38 build
drwxr-xr-x 628 root root 20480 Oct 13 11:39 node_modules
-rw-rw-r--   1 root root  2220 Oct 13 11:38 package.json


MSBuild.exe /? still shows /debug option

We removed support for /debug in #163, but we didn't remove the help text that mentions it. We shouldn't tease people with a cut feature.

@rainersigwald, is it reasonable to remove enum value ParameterlessSwitch.Debugger from the set?

@nikson My first inclination is to leave the enum value (and the string resource) while we have the code #ifed out. When we commit to removal we can clean it all up together.

Fixed via #430. Thanks, @nikson.


added eslint & prettier

Added a basic eslint and prettier config. I tried to keep the style of the repo as much as possible, but if there are any changes you'd like to me make please let me know!
By adding eslint, I also found one bug that I didn't know how to proceed with. In the following code in processors.js, frameWidth is never defined:
https://github.com/xenova/transformers.js/blob/e628c0a957d4b40810e73ea50a786142327998a2/src/processors.js#L416-L418

Thanks for the PR! The repo has changed quite a bit since you made the PR, so, a lot of the files are quite outdated. However, it is still important to get a linter and prettier included as it will help keep the repo organized and structured.
If you'd like, you could maybe try update your branch to have all the latest changes, then we can review the linter and prettier options.

I will take a look at this later tonight and clean things up 😄

I'm still keen to add this, so, if you're able to fix the conflicts, I'll be able to merge it more easily 👍


Validating a new Babel run

I've just completed my first run of Babel on Sterling (on a container with 500GB of memory!) using the changes in draft PR https://github.com/TranslatorSRI/Babel/pull/37. The results I've obtained (on Hatteras at /scratch/gaurav/babel-outputs/2022apr4) has lots of differences from the 2022-01-01 run, but I haven't come up with a good way of summarizing the changes or figuring out if it's working "correctly".
I've tried using diff/diffstat, but there are tons of changes, so it's not easy to see how signficant the changes are. I tried diffing some files individually, and was able to find a few patterns: for example, the polypeptide LSM-37009 in synonyms/Polypeptide.txt is referred to as CHEBI:125504 in the new run and INCHIKEY:GGLDQJNBYFODOM-RDCMKPLUSA-N in the previous run.
Diffstat comparison of Jan 1 and Apr 4 Babel runs
 compendia/AnatomicalEntity.txt         |284873
 compendia/BiologicalProcess.txt        |55258
 compendia/Cell.txt                     |15690
 compendia/CellularComponent.txt        |24855
 compendia/ChemicalEntity.txt           |6976071
 compendia/ChemicalMixture.txt          |  889
 compendia/ComplexMolecularMixture.txt  |  296
 compendia/Disease.txt                  |654029
 compendia/Gene.txt                     |77898179 ++---
 compendia/GeneFamily.txt               |55418
 compendia/GrossAnatomicalStructure.txt |20397
 compendia/MolecularActivity.txt        |294143
 compendia/MolecularMixture.txt         |16366879 -
 compendia/OrganismTaxon.txt            |4783919
 compendia/Pathway.txt                  |104290
 compendia/PhenotypicFeature.txt        |700283
 compendia/Polypeptide.txt              |  753
 compendia/Protein.txt                  |456484451 ++++++++++++++++-----------------
 compendia/SmallMolecule.txt            |204804339 +++++++-------
 conflation/GeneProtein.txt             |16857753 -
 reports/AnatomicalEntity.txt           |  100
 reports/BiologicalProcess.txt          |   17
 reports/Cell.txt                       |   73
 reports/CellularComponent.txt          |   60
 reports/ChemicalEntity.txt             | 1451
 reports/ChemicalMixture.txt            |   20
 reports/ComplexMolecularMixture.txt    |   23
 reports/Disease.txt                    | 8278
 reports/Gene.txt                       |   72
 reports/GeneFamily.txt                 |    8
 reports/GrossAnatomicalStructure.txt   |   80
 reports/MolecularActivity.txt          |   70
 reports/MolecularMixture.txt           | 2310
 reports/OrganismTaxon.txt              |   12
 reports/Pathway.txt                    |    8
 reports/PhenotypicFeature.txt          | 1175
 reports/Polypeptide.txt                |   30
 reports/Protein.txt                    |  445
 reports/SmallMolecule.txt              | 8790
 reports/disease_completeness.txt       |   69
 reports/process_completeness.txt       |    4
 synonyms/AnatomicalEntity.txt          |624380
 synonyms/BiologicalProcess.txt         |224432
 synonyms/Cell.txt                      |43534
 synonyms/CellularComponent.txt         |57426
 synonyms/ChemicalEntity.txt            |1428975
 synonyms/ChemicalMixture.txt           | 3566
 synonyms/ComplexMolecularMixture.txt   | 1672
 synonyms/Disease.txt                   |2407347
 synonyms/Gene.txt                      |1060645
 synonyms/GeneFamily.txt                |55418
 synonyms/GrossAnatomicalStructure.txt  |105021
 synonyms/MolecularActivity.txt         |393102
 synonyms/MolecularMixture.txt          |16811698 -
 synonyms/OrganismTaxon.txt             |139483
 synonyms/Pathway.txt                   |109508
 synonyms/PhenotypicFeature.txt         |1712740
 synonyms/Polypeptide.txt               | 3133
 synonyms/Protein.txt                   |2871694
 synonyms/SmallMolecule.txt             |215593042 +++++++--------
 60 files changed, 525618409 insertions(+), 504434267 deletions(-)

Probably the best way to compare the changes is by comparing line counts, which shows that most files are pretty similarly sized, except for compendia/ChemicalEntity.txt (which is 1577.58% bigger), compendia/MolecularMixture.txt (58.43% bigger) and synonyms/MolecularMixture.txt (56.51% bigger).
Does anybody have suggestions for comparing/validating the new Babel output before we try to move it to the dev server? We could for instance dump all the IDs alphabetically and run a massive diff on that. Having some method to do this would help with #36 as well.
| January 1, 2022 | April 4, 2022 | Percentage change
-- | -- | -- | --
reports/chemical_completeness.txt | 1 | 1 | 0.00%
reports/disease_completeness.txt | 60 | 123 | 105.00%
reports/taxon_done | 1 | 1 | 0.00%
reports/process_done | 1 | 1 | 0.00%
reports/ChemicalEntity.txt | 741 | 732 | -1.21%
reports/MolecularMixture.txt | 1144 | 1182 | 3.32%
reports/gene_done | 1 | 1 | 0.00%
reports/ChemicalMixture.txt | 15 | 17 | 13.33%
reports/protein_done | 1 | 1 | 0.00%
reports/anatomy_done | 1 | 1 | 0.00%
reports/MolecularActivity.txt | 39 | 41 | 5.13%
reports/Disease.txt | 4154 | 4174 | 0.48%
reports/OrganismTaxon.txt | 11 | 11 | 0.00%
reports/Protein.txt | 197 | 274 | 39.09%
reports/Cell.txt | 42 | 43 | 2.38%
reports/genefamily_done | 1 | 1 | 0.00%
reports/CellularComponent.txt | 36 | 40 | 11.11%
reports/process_completeness.txt | 3 | 1 | -66.67%
reports/ComplexMolecularMixture.txt | 18 | 15 | -16.67%
reports/taxon_completeness.txt | 1 | 1 | 0.00%
reports/anatomy_completeness.txt | 1 | 1 | 0.00%
reports/PhenotypicFeature.txt | 603 | 626 | 3.81%
reports/GrossAnatomicalStructure.txt | 43 | 45 | 4.65%
reports/Polypeptide.txt | 20 | 22 | 10.00%
reports/BiologicalProcess.txt | 15 | 14 | -6.67%
reports/disease_done | 1 | 1 | 0.00%
reports/gene_completeness.txt | 1 | 1 | 0.00%
reports/Pathway.txt | 11 | 11 | 0.00%
reports/genefamily_completeness.txt | 1 | 1 | 0.00%
reports/AnatomicalEntity.txt | 52 | 62 | 19.23%
reports/SmallMolecule.txt | 4384 | 4432 | 1.09%
reports/protein_completeness.txt | 1 | 1 | 0.00%
reports/chemicals_done | 1 | 1 | 0.00%
reports/GeneFamily.txt | 9 | 9 | 0.00%
reports/Gene.txt | 45 | 47 | 4.44%
compendia/ChemicalEntity.txt | 392499 | 6584478 | 1577.58%
compendia/MolecularMixture.txt | 6334558 | 10035657 | 58.43%
compendia/ChemicalMixture.txt | 475 | 482 | 1.47%
compendia/MolecularActivity.txt | 145925 | 149030 | 2.13%
compendia/Disease.txt | 322229 | 332754 | 3.27%
compendia/OrganismTaxon.txt | 2375027 | 2412122 | 1.56%
compendia/Protein.txt | 223676217 | 232834484 | 4.09%
compendia/Cell.txt | 7678 | 8210 | 6.93%
compendia/CellularComponent.txt | 12510 | 12623 | 0.90%
compendia/ComplexMolecularMixture.txt | 165 | 169 | 2.42%
compendia/PhenotypicFeature.txt | 355408 | 345793 | -2.71%
compendia/GrossAnatomicalStructure.txt | 10379 | 10238 | -1.36%
compendia/Polypeptide.txt | 408 | 409 | 0.25%
compendia/BiologicalProcess.txt | 27790 | 27714 | -0.27%
compendia/Pathway.txt | 52370 | 52452 | 0.16%
compendia/AnatomicalEntity.txt | 142269 | 143562 | 0.91%
compendia/SmallMolecule.txt | 104226454 | 100590131 | -3.49%
compendia/GeneFamily.txt | 27892 | 27770 | -0.44%
compendia/Gene.txt | 37802616 | 40108195 | 6.10%
synonyms/ChemicalEntity.txt | 698121 | 732464 | 4.92%
synonyms/MolecularMixture.txt | 6555269 | 10259687 | 56.51%
synonyms/ChemicalMixture.txt | 1856 | 1870 | 0.75%
synonyms/MolecularActivity.txt | 195416 | 198534 | 1.60%
synonyms/Disease.txt | 1189024 | 1219429 | 2.56%
synonyms/OrganismTaxon.txt | 69926 | 69993 | 0.10%
synonyms/Protein.txt | 1421157 | 1451959 | 2.17%
synonyms/Cell.txt | 20674 | 23034 | 11.42%
synonyms/CellularComponent.txt | 28577 | 29027 | 1.57%
synonyms/ComplexMolecularMixture.txt | 878 | 890 | 1.37%
synonyms/PhenotypicFeature.txt | 858920 | 855136 | -0.44%
synonyms/GrossAnatomicalStructure.txt | 52860 | 52553 | -0.58%
synonyms/Polypeptide.txt | 1641 | 1628 | -0.79%
synonyms/BiologicalProcess.txt | 112432 | 112364 | -0.06%
synonyms/Pathway.txt | 54941 | 55021 | 0.15%
synonyms/AnatomicalEntity.txt | 309911 | 315239 | 1.72%
synonyms/SmallMolecule.txt | 108292041 | 107313775 | -0.90%
synonyms/GeneFamily.txt | 27892 | 27770 | -0.44%
synonyms/Gene.txt | 497027 | 564344 | 13.54%
conflation/GeneProtein.txt | 8168582 | 8692887 | 6.42%

Yes, this is a question I have struggled with.
Looking at the sizes of the compendia themselves is a good way to get some idea what's going on.  The big changes in ChemicalEntity and MolecularMixture are both concerning.  To me, that suggests that both are not merging on something, so we have more smaller cliques than fewer larger ones.  And of course, the new way may be better; it will require a bit of digging into.
The stuff in reports is meant as a partial answer to this problem.
The reports that are *done are not interesting, they are just artifacts for snakemake to know that the process has completed.  Probably they should be written into another place?  Maybe they can be dispensed with entirely with a bit of thought.
The _completeness files are there to check for a particular kind of error: making sure that we're not losing identifiers.  At times past we have sometimes had input identifiers that didn't end up in any cliques.  So these are lists of identifiers that didn't end up in cliques.   Really, you want all these to just have the header, and you don't want them to increase in size.  We previously have had a long running issue in disease: https://github.com/TranslatorSRI/NodeNormalization/issues/64 .  It looks like that one got worse; if it's a question of the same cause then so be it but maybe something else has changed.
The most useful (?) files are the ones called e.g. reports/MolecularMixture.txt.  First you get a cluster size distribution.  It's often good to review the biggest cluster and see if anything can be done about them.  Usually if there's not a bulge at the end of the distribution though, there's not much that can be done.  The other useful thing is the prefix list.  So for instance in that MM.txt, you see this:
frozenset({('UNII', 2), ('CAS', 1), ('PUBCHEM.COMPOUND', 2), ('INCHIKEY', 1)})  1
frozenset({('MESH', 1), ('PUBCHEM.COMPOUND', 1), ('UNII', 2), ('CHEMBL.COMPOUND', 1), ('UMLS', 3), ('INCHIKEY', 1)})    1
frozenset({('MESH', 1), ('PUBCHEM.COMPOUND', 1), ('HMDB', 1), ('UNII', 1), ('CHEBI', 1), ('CHEMBL.COMPOUND', 1), ('INCHIKEY', 1)})      1
...
frozenset({('INCHIKEY', 1), ('PUBCHEM.COMPOUND', 1), ('CHEMBL.COMPOUND', 1)})   87120
frozenset({('CAS', 1), ('PUBCHEM.COMPOUND', 1), ('INCHIKEY', 1)})       196331
frozenset({('INCHIKEY', 1), ('PUBCHEM.COMPOUND', 1)})   5863401

So the most common clique/cluster consists of a pubchem.compound id and an inchikey, and there are 5.8 M like that.  The next biggest group is that plus a CAS, with almost 200k, and so on down to clusters that only happen once, like some compund that has a mesh, a pubchem, an hmdb, a unii, a chebi, a chembl, and an inchikey.
So to figure out what's going on with the big % change in the compendia, I would look at the differences in these reports and see what is getting merged/split differently.  With those big % changes I think they'll be easy to find?   And then it's a matter of digging around and looking at examples and seeing if it's an improvement or not.
I've made half hearted attempts to compare directly two compendium runs to see what changed, but have never found anything I liked very much.  Happy to try other ideas though.

I've gotten some working diff-Babel-runs code that took around 9.5 hours (and over 500GB of memory!) to run a comparison on the entire Babel run between Jan 1 and May 21, 2022.
This comparison works by making a list of all the identifiers used in either the previous version of the compendium or the current version of the compendium -- for example, OrganismTaxon.txt has 2,494,604 such identifiers. For each identifier, it then collects all the cliques/clusters from the previous compendium and the current compendium, and assigns each identifier to one of a few general categories (detailed in https://github.com/gaurav/babel-validation/pull/5) -- for example, DELETED means that the identifier was present in one or more clusters/cliques in the previous compendium and not present in new compendium. My plan is to come up with additional ways of clustering identifiers based on measures that are useful to determine if we've lost data or started overly grouping clusters/cliques.
Some highlights:

All but seven compendia have over 80% unchanged identifiers, which is what we'd expect to see. In some cases, such as , this seems to be because there are a lot of new identifiers (ChemicalEntity.txt: 12,449,396 or 95.97% of the identifiers from prev and current compendia, and Polypeptide.txt has 362 or 35.84% of the identifier union).
SmallMolecule.txt (55.68%), MolecularMixture.txt (37.35%), Polypeptide.txt (14.06%), ComplexMolecularMixture.txt (7.81%) and Protein.txt (5.95%) all have high percentages of "changed with identical identifiers", i.e. where an identifier has the same set of identifiers in both previous and current compendia. Unless I've got a bug in that part of my code, this should only happen when the labels on identifiers change, which is likely just a source database being improved.
Wherever identifiers have been deleted, it comprises less than 6% of the combined identifier count except for GrossAnatomicalStructure.txt (1630 or 10.91%). There's no obvious reasons why so many identifiers has been deleted here -- they point to a variety of sources (mostly NCIT, some UMLS, MESH, UBERON) and the labels for these identifiers appear to be correct (see /scratch/gaurav/babel-outputs/2022may21/babel_outputs/reports/diff_to_2022jan1/GrossAnatomicalStructure.txt for all the changes grouped by change type), so this looks to me like we're losing information for some reason. I wonder if maybe Uberon or another data source removed lots of cross-references to NCIT and some other data sources a little over enthusiastically.
The following compendia have over 5% of combined identifiers changed for no clear reason: , Disease.txt (9.18%), ComplexMolecularMixture.txt (5.71%), GrossAnatomicalStructure.txt (13.02%), Cell.txt (5.57%), MolecularActivity.txt (4.28%) and Gene.txt (3.84%). This is probably too many to look through manually and figure out any kinds of patterns, so I'll try to look at them and figure out if I can find any patterns of interest there -- for example, some identifiers (such as ORPHANET:2055) are being removed from a clique of 6 IDs into its own identifiers, which seems... not great. I'll make a separate category for some of those patterns and then will rerun my diffing tool.

Also some practical notes:

OrganismTaxon.txt and smaller (2,494,604 identifiers between the two runs) files run pretty quickly (< 1 hour, I think), but the larger files take up most of the remaining 8.5 hours.
We don't currently report compendia not present in both runs, which isn't an issue at the moment, but will be in the future once we integrate unused UMLS identifiers into Babel.
This currently reports "Changed" when the "information content" (IC) of a cluster/clique changes, but doesn't do any additional grouping based on that.
This doesn't include any kind of validation on either the conflation or the synonyms, but those might be easier to validate by just comparing the size of the files as I did in a previous comment? I'll look into this in more detail.

The full lists of all cluster diffs by identifier are on Hatteras at /scratch/gaurav/babel-outputs/2022may21/babel_outputs/reports/diff_to_2022jan1.

This is excellent progress.   I wonder if we should be versioning the intermediate products.   For instance, in the GrossAnatomicalStructure stuff, if we had the previous relationship files, we could see that e.g. the one built off of source X is suddenly smaller...
I would like for us to target having a dev release of this build by the end of August.

This is excellent progress. I wonder if we should be versioning the intermediate products. For instance, in the GrossAnatomicalStructure stuff, if we had the previous relationship files, we could see that e.g. the one built off of source X is suddenly smaller...

Implemented in PR https://github.com/TranslatorSRI/Babel/pull/57.


FIX: Use mris_convert --to-scanner, and update normalization step

Following exploration in #292, this should get us to a point where we're producing files that are at least less broken from the perspective of connectome workbench and compatible with FreeSurfer.
I initially set the VolGeom affine to an LIA affine, but FreeSurfer will set that as the default. Removing it seemed more honest in that it indicates that this is, in effect, not a FreeSurfer-generated GIFTI but one edited by sMRIPrep. I can revert this to LIA if people think that would be better.
Closes #292
Closes #293

Codecov Report

Merging #295 (bae250a) into master (03bf056) will decrease coverage by 3.14%.
The diff coverage is 78.43%.

@@            Coverage Diff             @@
##           master     #295      +/-   ##
==========================================
- Coverage   69.54%   66.40%   -3.15%
==========================================
  Files          17       18       +1
  Lines        1087     1137      +50
  Branches      175      189      +14
==========================================
- Hits          756      755       -1
- Misses        275      322      +47
- Partials       56       60       +4

Flag
Coverage Δ

ds005
60.68% <78.43%> (+1.62%)
:arrow_up:

ds054
51.36% <33.33%> (-0.90%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

smriprep/interfaces/surf.py
78.00% <78.00%> (ø)

smriprep/workflows/surfaces.py
76.59% <100.00%> (ø)

smriprep/conftest.py
0.00% <0.00%> (-100.00%)
:arrow_down:

smriprep/conf/__init__.py
0.00% <0.00%> (-100.00%)
:arrow_down:

smriprep/utils/misc.py
8.57% <0.00%> (-37.15%)
:arrow_down:

smriprep/utils/bids.py
29.05% <0.00%> (-17.95%)
:arrow_down:

smriprep/interfaces/templateflow.py
76.08% <0.00%> (-15.22%)
:arrow_down:

smriprep/cli/run.py
77.46% <0.00%> (+0.46%)
:arrow_up:

smriprep/interfaces/reports.py
92.40% <0.00%> (+6.32%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 03bf056...bae250a. Read the comment docs.


Cd working v1.0

Solutions to Issues 21,23,24
🗣 Description

Solve issue# 21
Solve issue# 23
Solve issue# 24

💭 Motivation and context

Issue 21 Initializes variable values, so the software will continue without errors if there are no values returned
Issue 23 There are some related Bandit issues PyCQA/Bandit#455 and PyCQA/bandit#333 that justify module 'subprocess' usage.  This issue may be reinstated after further review.
Issue 24 There is a security concern. The eval() has been replaced with ast.literal_eval().

✅ Checklist

[ ] This PR has an informative and human-readable title.
[ ] Changes are limited to a single goal - eschew scope creep!
[ ] All relevant type-of-change labels have been added.
[ ] I have read the CONTRIBUTING document.
[ ] These code changes follow cisagov code standards.
[ ] All relevant repo and/or project documentation has been updated
to reflect the changes in this PR.
[ ] All new and existing tests pass.

@cduhn17, can you take a look at these two items:

In loading in the new code and stepping through its clean-up, can we carry over the work in done in "develop" that includes the appropriete references of "TODOs" to Issues. For example - this commit. Other than the issues addressed in this PR most will still be outstanding and can be addressed as we go.
Tests - we should adjust the following test to handle the new command-line arguments. This should allow us to pass all checks.

def test_log_levels(level):
    """Validate commandline log-level arguments."""
    with patch.object(
        sys,
        "argv",
        [
            "pe_reports",
            "2021-01-01",
            "input/",
            "output/",
            f"--log-level={level}",
        ],
    ):

@cduhn17, I referenced PR https://github.com/cisagov/pe-reports/pull/33, as a quick resolve to the tests issues presented in this PR.

@cduhn17 when you get a moment can you update the title/description with brief explanation to issues. Also looks like we can link Issue #23 with PR  https://github.com/cisagov/pe-reports/pull/33 and remove it from this PR.

As a note to all involved, the PRs #31 and #33 are extremely old and only valid for our LookingGlass data pulls for which the contract ends on 9/30 and they will be OBE.  Is there any possibility we can have them completed before I have to redirect the team to continue other work?

Closing pull-request as efforts to update the main branch have been canceled. Routines used in version 0.1.1 have been sunsetted with no future plans to update.


Validate required property

If we add a tag to the yaml schema required to each property, there is some validation we can do to be friendlier. Most directly, we can emit a better error message when the property is not set, so the recipient knows the error is not their fault, and knows who to go to to address it.

Carefully consider this in the context of established rules in our GUIDELINES

I'm going to close this, since it reverses current guidelines, and the proposed solution wouldn't work anyway.


Animating text character by character

How can I animate text character by character with this library

There are a bunch of ways that I could think of to animate text in one letter at a time — how you'd do it would depend on the effect of course. Let's say you want to animate text in a sort of typewriter effect:

Create an array containing each character of your string
Create a YapAnimator<Double> that you'll use to toggle from 0 to 1
The animator's eachFrame closure would take the value (ranging from 0 to 1) and re-map it to the number of characters in your array.
Create a string from your array (up to the index of the remapped value) and apply it to your label.

Now you can simply set the animator to 0 or 1 (or any value in between) and have the text transition in or out.
Ultimately, the best way of using YapAnimator is to think, "How can I control yourproblemhere with a number". Sometimes it is very direct (like setting opacity), but often times the more creative use cases will involve mapping/interpolating to something else.


Normalizing struct with signal reference causes RuntimeError

Hi all,
I was experimenting with some more exotic ways to parametrize a register and ran into some issues here. I did not get any parser errors so I suppose that it should be correct syntactically. It's not really a big issue for me but since I did not find this on your limitations page I thought I might as well open an issue in case this is unknown to you.
And maybe I just misread the RDL spec, since value_normalization.py says that I
# Should never get [t]here

🙂
This is the RDL I tried to compile:
signal {activelow; async;} rst_async_n;
signal {activehigh;}       rst_sync;

struct rst_struct {
    signal rst;
};

reg myReg #(rst_struct RST_SIGNAL = rst_struct'{rst: rst_async_n}) {
    field {
        resetsignal = RST_SIGNAL.rst;
    } data[0:0] = 0;
};

addrmap paremeters {
    myReg                                            reg0;
    myReg #(.RST_SIGNAL(rst_struct'{rst: rst_sync})) reg1;
};

This is the runtime error I am seeing:
  File "/usr/lib/python3.9/site-packages/systemrdl_compiler-1.20.0-py3.9-linux-x86_64.egg/systemrdl/compiler.py", line 385, in elaborate
    walker.RDLWalker(skip_not_present=False).walk(
  File "/usr/lib/python3.9/site-packages/systemrdl_compiler-1.20.0-py3.9-linux-x86_64.egg/systemrdl/walker.py", line 121, in walk
    self.walk(child, *listeners)
  File "/usr/lib/python3.9/site-packages/systemrdl_compiler-1.20.0-py3.9-linux-x86_64.egg/systemrdl/walker.py", line 121, in walk
    self.walk(child, *listeners)
  File "/usr/lib/python3.9/site-packages/systemrdl_compiler-1.20.0-py3.9-linux-x86_64.egg/systemrdl/walker.py", line 123, in walk
    self.do_exit(node, listener)
  File "/usr/lib/python3.9/site-packages/systemrdl_compiler-1.20.0-py3.9-linux-x86_64.egg/systemrdl/walker.py", line 172, in do_exit
    listener.exit_Component(node)
  File "/usr/lib/python3.9/site-packages/systemrdl_compiler-1.20.0-py3.9-linux-x86_64.egg/systemrdl/core/elaborate.py", line 578, in exit_Component
    extra_type_name_segments.append(inst_parameter.get_normalized_parameter())
  File "/usr/lib/python3.9/site-packages/systemrdl_compiler-1.20.0-py3.9-linux-x86_64.egg/systemrdl/core/parameter.py", line 65, in get_normalized_parameter
    return self.name + "_" + normalize(self.get_value())
  File "/usr/lib/python3.9/site-packages/systemrdl_compiler-1.20.0-py3.9-linux-x86_64.egg/systemrdl/core/value_normalization.py", line 24, in normalize
    return normalize_struct(value)
  File "/usr/lib/python3.9/site-packages/systemrdl_compiler-1.20.0-py3.9-linux-x86_64.egg/systemrdl/core/value_normalization.py", line 112, in normalize_struct
    norm_elements.append("%s_%s" % (member_name, normalize(member_value)))
  File "/usr/lib/python3.9/site-packages/systemrdl_compiler-1.20.0-py3.9-linux-x86_64.egg/systemrdl/core/value_normalization.py", line 33, in normalize
    raise RuntimeError(value)
RuntimeError: <systemrdl.rdltypes.ComponentRef object at 0x7f91bdd93520>

The problem seems to be related to the fact that a signal is being used in the struct. When replacing the signal with a string, everything works fine:
signal {activelow; async;} rst_async_n;
signal {activehigh;}       rst_sync;

struct rst_struct {
    string rst;
};

reg myReg #(rst_struct RST_SIGNAL = rst_struct'{rst: "test"}) {
    field {
        //resetsignal = RST_SIGNAL.rst;
        desc = RST_SIGNAL.rst;
    } data[0:0] = 0;
};

addrmap paremeters {
    myReg                                            reg0;
    myReg #(.RST_SIGNAL(rst_struct'{rst: "test2"}))  reg1;
};

I do think that it is allowed to reference signals in a struct since Section <IP_ADDRESS>.2 of the SystemRDL 2.0 book says that

Structs may include all of the types defined in Table 7.

and Table 7 lists instance references.

Good find!
This is technically not supported. The SystemRDL spec has a habit of contradicting itself in a few sections. I write about this specific situation here: https://systemrdl-compiler.readthedocs.io/en/latest/dev_notes/rdl_spec_errata.html#inconsistent-definition-of-the-ref-type-keyword
Even though the ref type is listed in Table 7 and allows use of references in structs, the spec contradicts this later. When discussing parameters, clause <IP_ADDRESS>-e explicitly states that references of any type cannot be used, even inside a struct:

e) Component instance references shall not be used as parameter values (either directly or as part of an
aggregate type).

Regardless of any of that, you definitely found a bug! The compiler should not have thrown an exception like that. I'll look into cleaning up the error handling here. It might be easier for me to fix the underlying issue, and therefore ignore the super odd restriction that clause <IP_ADDRESS>-e imposes. I never really understood why they did that.

Cool, thanks for elaborating on why the compiler behaves like it does, pretty clear.
Next time before filing an issue I will first try finding the reason for that particular behavior in your Spec Errata; I remember reading it in the past but kind of forgot about its existence again. I did indeed notice that there are unfortunately a couple of contradictions and/or vague definitions in the spec.

Finally got around to fixing this one. v1.25.5.
Thanks again for reporting this one. Despite it being a simple error case cleanup, it is worth fixing.


Suggestions & Considerations for Script Outcomes

Had a couple thoughts on enhancements / suggestions /considerations to check with the script.
1. Changing the % count to 100% when all top line checks pass:

e.g - I've tested on a couple VPS's of mine which get a pass in all stages but I have skips in the child elements of the SSH Security section. This is due to me using my VPS providers FW UI to block any traffic on port 22 aside from connections from my home static IP. As well, I also sometimes setup a VPN tunnel (CloudFlare) to my VPS's, in which case ssh (and all other ports) for inbound are blocked. I think its fine to show the child elements as skips but in this case it shouldn't hurt the % score stat, seeing as if you can't hit port 22, it really doesn't matter what is setup. Only case(s) i can see this not handling is if someone has mapped the ssh service to a random or different port. (see 2 for handling that)

2. Related to 1, adding in a port map scan to tag open ports.
If your are after a zero deps or install approach, you could use the ss (Socket Statistics) command which is built-in on debian/ubuntu systems. Or nmap if installing a tool as part of the script is ok. This might be more of an FYI section as we can't infer intent or purpose for a port being open. But highlighting common exploited ports, DB ports and ports outside the normal collections (80, 433, 22, etc) or highlighting inherently insecure version of certain protocols, like 21, 25, etc.
3. Suggestion/Check for Ubuntu Pro subscription (Ubuntu only)
Ubuntu Pro is free for up to 5 machines and when installed it empowers your apt update with an expanded repository list of CVE update fixes that are not part of the normal (non-Pro) updates out of the box. All of the current script checks and ones listed above can all be for not if a bad actor can get in due to a exploited CVE that is bypassing said guards.
I think thats all for now. If I think of anything else I will try to update!

Hey @KyTechInc thank you for opening the ticket!

Changing the % count to 100% when all top line checks pass

Totally. Fixing this tonight 🙏

Adding in a port map scan to tag open ports

I like this idea, but it's a tricky one to implement. Like you said, we cannot infer intent. I think this would serve best as an INFO state, which currently does not exist, but I'd like to add.

highlighting inherently insecure version of certain protocols, like 21, 25, etc

Can you elaborate on which protocols are insecure? These can be a WARN at the very least (another state I'd like to add!)

Suggestion/Check for Ubuntu Pro subscription

Didn't know this. What would be the check in this case?

I like those states WARN and INFO.
Cause ya we can't infer use but I find often times people will cover security related to their use case but may not be fully aware of all ports that are open. Plus, sometimes some VPS providers leave other ports open for their infa use tools or services that may not be within the normal scope.
So I think a INFO > List of open ports and let the user do with that info as needed.
As for the WARN > Insecure ports. I was thinking of stuff like they have 21/FTP or 25/SMTP open. Regardless of intent, these are the insecure (non-HTTPS/TLS) versions of the protocol OR there is a replacement protocol that is secure. 22 (SFTP) or 990 FTPS and 465/587 Secure SMTP.
I think its safe to say that even with a purposeful use case for using 21/25 over these secure options (thinking about legacy support), that still puts the system at risk of attacks without further measures in place.
As for the Ubuntu Pro, there is a CLI built into Ubuntu (16.04 LTS, 18.04 LTS, 20.04 LTS or 22.04 LTS). You could do a check for those distro versions because frankly, if your doing a security audit and running in "PROD" you should be using the LTS distro version.
You can run a check for the Pro version in use and if its not installed/enabled it will return an error:
pro --version
34~24.04

You could also check for the status of the installed tool set that makes up the Pro suite:
pro status
SERVICE          ENTITLED  STATUS       DESCRIPTION
anbox-cloud      yes       disabled     Scalable Android in the cloud
esm-apps         yes       enabled      Expanded Security Maintenance for Applications
esm-infra        yes       enabled      Expanded Security Maintenance for Infrastructure
landscape        yes       disabled     Management and administration tool for Ubuntu
livepatch        yes       enabled      Canonical Livepatch service
realtime-kernel* yes       disabled     Ubuntu kernel with PREEMPT_RT patches integrated

 * Service has variants

For a list of all Ubuntu Pro services and variants, run 'pro status --all'
Enable services with: pro enable <service>

     Account<EMAIL_ADDRESS>Subscription: Ubuntu Pro - free personal subscription

***Note: This command does output PII with the account email at the bottom
esm-apps, esm-infra and livepatch are the bare minimum required to have the expanded CVE package installed work from the Pro repo.
You can also do cool things too like check the package count on the machine and see how many have an update available that would be covered by the Pro repo. (I believe this can be done without registering):
pro security-status --esm-apps
1919 packages installed:
	10 packages from Ubuntu Universe/Multiverse repository

Ubuntu Pro with 'esm-apps' enabled provides security updates for
Universe/Multiverse packages until 2030 and has 1 pending security update.

Run 'pro help esm-apps' to learn more

Package names in bold currently have an available update
with 'esm-apps' enabled
Packages:
ansible python3-argcomplete python3-kerberos python3-libcloud
python3-ntlm-auth python3-requests-kerberos python3-requests-ntlm
python3-selinux python3-winrm python3-xmltodict

Also, if you were to put together some sort of opinionated list of the top or most exploited CVE's, think like Wordpress, NGINX, SSL, etc. You could run a "Top 10 CVE" check using the Pro CLI:
pro fix CVE-2021-3583
CVE-2021-3583:
A flaw was found in Ansible, where a user's controller is vulnerable to
template injection. This issue can occur through facts used in the template
if the user is trying to put templates in multi-line YAML strings and the
facts being handled do not routinely include special template characters.
This flaw allows attackers to perform command injection, which discloses
sensitive information. The highest threat from this vulnerability is to
confidentiality and integrity.
https://ubuntu.com/security/CVE-2021-3583
1 affected source package is installed: ansible
(1/1) ansible:
A fix is available in Ubuntu Pro with `esm-apps` enabled.
The update is not installed because this system is not attached to a subscription.

And if this system has the fix applied already:
pro fix CVE-2021-3583
CVE-2021-3583: Ansible vulnerabilities
 - https://ubuntu.com/security/CVE-2021-3583

No affected source packages are installed.

✔ CVE-2021-3583 does not affect your system.

The Ubuntu Pro case is def a cool tool but obviously only applies to Ubuntu based systems and also I would still probably suggest it being a WARN or some sort of special category as it is an additional "product" to install and not core to all VPS's setups.
I would however suggest the top 10 CVE list (or whatever number you come to) being part of the scoring system. No reason to have top tier exploited holes in a PROD system.

Great explanation Ky, thank you. I think the ports suggestion is fantastic, I'll work on adding it in the near future!
Interestingly, I don't have Pro, yet the pro commands are working:
~|⇒ pro --version
34~22.04
~|⇒ pro status
SERVICE          AVAILABLE  DESCRIPTION
anbox-cloud      yes        Scalable Android in the cloud
esm-apps         yes        Expanded Security Maintenance for Applications
esm-infra        yes        Expanded Security Maintenance for Infrastructure
fips-preview     yes        Preview of FIPS crypto packages undergoing certification with NIST
fips-updates     yes        FIPS compliant crypto packages with stable security updates
livepatch        yes        Canonical Livepatch service
realtime-kernel  yes        Ubuntu kernel with PREEMPT_RT patches integrated
usg              yes        Security compliance and audit tools

For a list of all Ubuntu Pro services, run 'pro status --all'

This machine is not attached to an Ubuntu Pro subscription.
See https://ubuntu.com/pro
~|⇒ pro fix CVE-2021-3583
CVE-2021-3583: Ansible vulnerabilities
 - https://ubuntu.com/security/CVE-2021-3583

No affected source packages are installed.

✔ CVE-2021-3583 does not affect your system.

Any idea why this might be?

Sorry, stepped away for family time during the holidays.
I think since the Pro CLI comes pre installed on the system a lot of the commands will return info or results but to take action you need to have an active subscription for the machine. And again, this is free for 5 machines.
Your return shows all the features as "AVAILABLE", where as mine shows with "ENTITLED" and STATUS":
pro status
SERVICE          ENTITLED  STATUS       DESCRIPTION
anbox-cloud      yes       disabled     Scalable Android in the cloud
esm-apps         yes       enabled      Expanded Security Maintenance for Applications
esm-infra        yes       enabled      Expanded Security Maintenance for Infrastructure
landscape        yes       disabled     Management and administration tool for Ubuntu
livepatch        yes       enabled      Canonical Livepatch service
realtime-kernel* yes       disabled     Ubuntu kernel with PREEMPT_RT patches integrated

 * Service has variants

For a list of all Ubuntu Pro services and variants, run 'pro status --all'
Enable services with: pro enable <service>

This is a good thing though cause you it seems you could get results back from those CVE checks to provide warn statuses without the user having an active Pro subscription.


'Invalid Credentials'

I get an invalid credentials notification when I try to log into Memrise through the extension, but I know it's the right username and password.
If I try to enter my information a second time I get an error message.
Debug info:
Anki 2.1.49 (dc80804a) Python 3.8.6 Qt 5.14.2 PyQt 5.14.2
Platform: Mac 10.14.6
Flags: frz=True ao=True sv=2
Add-ons, last update check: 2022-03-21 20:38:51
Add-ons possibly involved: memrise2anki
Caught exception:
Traceback (most recent call last):
File "/Users//Library/Application Support/Anki2/addons21/2071525018/importer.py", line 210, in accept
if self.memriseService.login(self.usernameLineEdit.text(),self.passwordLineEdit.text()):
File "/Users//Library/Application Support/Anki2/addons21/2071525018/memrise.py", line 773, in login
for field in form.find_all("input"):
AttributeError: 'NoneType' object has no attribute 'find_all'

You are using an old version of the addon, please try the 1.2.7 release.


change the range loop in visit to scan in sorted order so that Max-flow produces deterministic output

I got non-deterministic output from max-flow, so I try to avoid this by making the edge iteration in sorted order.

Thanks for your patch!
Currently the MaxFlow function only promises to return "a maximum flow". If you need this function to be deterministic it's seems more natural to change the MaxFlow function directly rather than the Visit function in Mutable. However, this will make the code a bit inefficient since there can be multiple calls to the residualFlow function. Also, it seems like a rather special case where you need to call the MaxFlow function repeatedly for the same graph. My guess is that a deterministic behavior isn't necessary for most use cases. Therefor I choose to leave the code as is.


eCarePlanner 3.1.1 - Care Team data not populated

in eCarePlanner 3.1.1, we see the following under Care Team:

whereas this is what's displayed in MyCarePlanner:

of note, there are no browser console logs that contain the term "careteam" which makes me think the app is just missing code to populate this section, as you can see by my browser console logs, filtered by the term "careteam":

@drdavec can you specify the logic of what's pulled in the care team tab on MyCarePlanner?
@swmuir to investigate current logic in eCarePlanner.

MyCarePlanner reads all CareTeam resources for a patient, where category = 'longitudinal'
CareTeam?category=longitudinal
For Epic, this is described here as their USCDI conformance support for CareTeam:
https://fhir.epic.com/Specifications?api=1069
MyCarePlanner then displays all members of the returned CareTeam(s)
In addition, MyCarePlanner reads Patient.generalPractitioner, and if populated by the EHR, will display this Practitioner name as the patient's Primary Care Provider, shown at the top of the Care Team tab.

@swmuir remove all the crossed out fields and everything "under the fold". Move Race/Ethnicity into the place where Patient ID currently is.


Nested Python venvs lead to wrong path order

Describe the bug
If I have a .mise.toml, say in folder ~/dev/project1 file that defines a python venv with
[env]
_.python.venv = '.venv'

and then have another .mise.toml in some subfolder of ~/dev/project1, say ~/dev/project1/src/notebooks that also defines a venv, then there are a few weird behaviors:

I cannot easily create a venv with the specified python version directly, because the venv of the parent folder is still active
So python -m venv .venv would actually create the venv using the Python binary from the parent venv
If I manage to correctly do (1), e.g. using the full path to the correct python version, then, when the new venv is activated, the PATH is not correct.
I have the bin of the venv of the parent folder before the bin of the current venv, so actually the wrong python binaries come first.

I know this is a bit of a weird edge case, but I have an actual use case for it.
To Reproduce
Described above: create two .mise.tomls specifying a Python venv in two nested folders.
Enter the lower folder.
Investigate $PATH
Expected behavior
About 1.: Actually, I would expect the parent venv to not be active when another venv is specified in .mise.toml, but doesn't exist.
So that if I am in the nested folder, python would refer to the python version specified in tools, and not the parent venv
(This might need a bit more thought / discussion)
About 2.: Here I'm more confident. If the nested venv gets activated, the parent venv should not be on the path anymore at all.
It definitely should not come before the nested venv.
mise doctor output
version: 2024.3.1 macos-arm64 (2024-03-04)
activated: yes
shims_on_path: yes

build_info:
  Target: aarch64-apple-darwin
  Features: DEFAULT, NATIVE_TLS
  Built: Mon, 4 Mar 2024 22:28:40 +0000
  Rust Version: rustc 1.76.0 (07dca489a 2024-02-04) (Homebrew)
  Profile: release

shell:
  /bin/zsh
  zsh 5.9 (x86_64-apple-darwin23.0)

dirs:
  data: ~/.local/share/mise
  config: ~/.config/mise
  cache: ~/Library/Caches/mise
  state: ~/.local/state/mise
  shims: ~/.local/share/mise/shims

config_files:
  ~/.config/mise/config.toml
  ~/dev/project1/.mise.local.toml # redacted, but maintaining correct structure
  ~/dev/project1/src/notebooks/.mise.local.toml # redacted, but maintaining correct structure

plugins:
  bun     (core)
  deno    (core)
  erlang  (core)
  go      (core)
  java    (core)
  node    (core)
  python  (core)
  ruby    (core)
  usage   https://github.com/jdx/mise-usage.git#fe3888a

toolset:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
env_vars:
  MISE_SHELL=zsh

settings:
  activate_aggressive = false
  all_compile = false
  always_keep_download = false
  always_keep_install = false
  asdf_compat = false
  cargo_binstall = true
  color = true
  disable_default_shorthands = false
  disable_tools = []
  experimental = true
  go_default_packages_file = "~/.default-go-packages"
  go_download_mirror = "https://dl.google.com/go"
  go_repo = "https://github.com/golang/go"
  go_set_gopath = false
  go_set_goroot = true
  go_skip_checksum = false
  jobs = 4
  legacy_version_file = true
  legacy_version_file_disable_tools = []
  node_compile = false
  not_found_auto_install = true
  paranoid = false
  plugin_autoupdate_last_check_duration = "7d"
  python_compile = false
  python_default_packages_file = "/Users/daniel/.default-python-packages"
  python_pyenv_repo = "https://github.com/pyenv/pyenv.git"
  raw = false
  trusted_config_paths = []
  quiet = false
  verbose = false
  yes = false
  ci = false
  debug = false
  trace = false
  log_level = "info"
  python_venv_auto_create = false

  [status]
  missing_tools = "if_other_versions_installed"
  show_env = false
  show_tools = false

It seems to be a duplicate of #1724

https://github.com/jdx/mise/pull/1751


[Feat]: Adding contact us option also

What feature?
Adding contact us/ rate us option also so that user can interact
please assign this isuue to me under gssoc ext and hacktoberfest
Add screenshots
Add screenshots
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Start working on this features

Start working on this features
but before that add your profile on Devdisplay. It's is the first priority of contribution in DevDisplay
if you don't assign in this you will not be able to contribute on this project

ok

📢 Invitation to Join the DevDisplay WhatsApp Community!
We are excited to invite you to join the WhatsApp community of DevDisplay, where developers and contributors come together to collaborate and share their ideas.
Collaborative Support: If you are interested in working together with extensive support, we encourage you to join our community.
Open Source Contribution: If you would like to contribute to the DevDisplay open-source project, your participation is welcomed in our WhatsApp group.
By joining, you will have the opportunity to connect with like-minded individuals, discuss project developments, and contribute to our shared goals.
For more information and to join the community, please visit: Join DevDisplay WhatsApp Community.
We look forward to your participation!


extra space in svg class attribute

Package

[X] lucide
[ ] lucide-angular
[ ] lucide-flutter
[ ] lucide-preact
[X] lucide-react
[ ] lucide-react-native
[ ] lucide-solid
[ ] lucide-svelte
[ ] lucide-vue
[ ] lucide-vue-next
[ ] Figma plugin
[ ] source/main
[ ] other/not relevant

Version
0.293.0
Browser

[ ] Chrome/Chromium
[ ] Firefox
[ ] Safari
[ ] Edge
[ ] iOS Safari
[ ] Opera
[ ] Other/not relevant

Operating system

[ ] Windows
[ ] Linux
[ ] macOS
[ ] Other/not relevant

Description
we have some snapshot tests and they started to show extra space at the end on this classname:

not a big problem, but if anyone could easily remove the space please do trim that
Steps to reproduce
just render and see the html
Checklist

[X] I have searched if someone has submitted a similar issue before and there weren't any. (Please make sure to also search closed issues, as this issue might already have been resolved.)

This issue still exists in version 0.331.0
https://github.com/lucide-icons/lucide/blob/42c1faed7597021baff4aed7f807483beb8971c6/packages/lucide-react/src/createLucideIcon.ts#L61