[BUG] Profiler records values that differ from transformers

Describe the bug
When comparing zero-1 and zero-2, I noticed discrepancies between the results in the DeepSpeed Flops Profiler and the training speed metrics in transformers, and the conclusions drawn from the two are completely opposite. Therefore, I would like to understand why this phenomenon exists.
Below is my statistical table:

method
samples /sec(deepspeed)
samples /sec(hf transformers)
throughput (hf transformers)

zero-1
33.68
39.669
13540

zero-2
33.77
35.527
12170

To Reproduce
pass
Expected behavior
The value of samples/s for zero-1 should be higher than that of zero-2, and close to the transformers.
ds_report output
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
async_io ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_lion ............... [NO] ....... [OKAY]
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
evoformer_attn ......... [NO] ....... [NO]
fused_lamb ............. [NO] ....... [OKAY]
fused_lion ............. [NO] ....... [OKAY]
inference_core_ops ..... [NO] ....... [OKAY]
cutlass_ops ............ [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
ragged_device_ops ...... [NO] ....... [OKAY]
ragged_ops ............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.0
 [WARNING]  using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
torch version .................... 2.0.1
deepspeed info ................... 0.13.1, unknown, unknown
torch cuda version ............... 11.8
nvcc version ..................... 11.8
deepspeed wheel compiled w. ...... torch 2.0, cuda 11.8

Screenshots
pass
System info (please complete the following information):

OS: debian 12
x8 A100s
Python 3.10

Launcher context
torchrun $DISTRIBUTED_ARGS pretraining.py \
    --model_config_path $MODEL \
    --data_dir $DATA \
    --do_train true \
    --do_eval false \
    --fp16 true \
    --output_dir output_test \
    --num_train_epochs 1 \
    --per_device_train_batch_size 32 \
    --gradient_accumulation_steps 4 \
    --evaluation_strategy "no" \
    --save_strategy "epoch" \
    --save_total_limit 10 \
    --learning_rate 1e-3 \
    --weight_decay 0.1 \
    --adam_beta1 0.9 \
    --adam_beta2 0.98 \
    --logging_steps 1 \
    --report_to "none" \
    --model_max_length 2048 \
    --include_tokens_per_second true \
    --include_num_input_tokens_seen true \
    --gradient_checkpointing \
    --deepspeed ${DS_CONFIG_PATH}

Docker context
pass
Additional context
here is the deepspeed flops profiler summary text,
zero-1
world size:                                                             6
data parallel size:                                                     6
model parallel size:                                                    1
batch size per GPU:                                                     32
params per GPU:                                                         1.34 B
params of model = params per GPU * mp_size:                             1.34 B
fwd MACs per GPU:                                                       78.96 TMACs
fwd flops per GPU:                                                      157.93 T
fwd flops of model = fwd flops per GPU * mp_size:                       157.93 T
fwd latency:                                                            1.2 s
fwd FLOPS per GPU = fwd flops per GPU / fwd latency:                    131.12 TFLOPS
bwd latency:                                                            3.98 s
bwd FLOPS per GPU = 2 * fwd flops per GPU / bwd latency:                79.35 TFLOPS
fwd+bwd FLOPS per GPU = 3 * fwd flops per GPU / (fwd+bwd latency):      91.38 TFLOPS
step latency:                                                           515.57 ms
iter latency:                                                           5.7 s
FLOPS per GPU = 3 * fwd flops per GPU / iter latency:                   83.11 TFLOPS
samples/second:                                                         33.68

zero-2
world size:                                                             6
data parallel size:                                                     6
model parallel size:                                                    1
batch size per GPU:                                                     32
params per GPU:                                                         1.34 B
params of model = params per GPU * mp_size:                             1.34 B
fwd MACs per GPU:                                                       78.96 TMACs
fwd flops per GPU:                                                      157.93 T
fwd flops of model = fwd flops per GPU * mp_size:                       157.93 T
fwd latency:                                                            1.2 s
fwd FLOPS per GPU = fwd flops per GPU / fwd latency:                    131.35 TFLOPS
bwd latency:                                                            3.97 s
bwd FLOPS per GPU = 2 * fwd flops per GPU / bwd latency:                79.56 TFLOPS
fwd+bwd FLOPS per GPU = 3 * fwd flops per GPU / (fwd+bwd latency):      91.6 TFLOPS
step latency:                                                           512.26 ms
iter latency:                                                           5.68 s
FLOPS per GPU = 3 * fwd flops per GPU / iter latency:                   83.34 TFLOPS
samples/second:                                                         33.77

Hi @ziliwang - sorry for getting to this so late, are you still hitting this with the latest DeepSpeed release?

Hi @ziliwang, just wondering, could you please share your change to apply tflops profile?
According to the official specs, we can avoid user code changes only if we apply DeepSpeed runtime.
However, I figured out that in /usr/local/lib/python3.10/dist-packages/deepspeed/runtime/engine.py, even though the self.flops_profiler.start_profile(ignore_list=None) and self.flops_profiler.stop_profile() are indeed invoked in forward(), there are no tflops profiler summary text.
The root cause is that the step(self, lr_kwargs=None), which includes self.flops_profiler.print_model_profile(), is not invoked, thus no flops profiler summary text. Thanks for any suggestions or hints on how to trigger the step() function


Add flink logs

What does this PR do?
Adds documentation on how to enable flink logs.
Motivation

Additional Notes

Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] PR title must be written as a CHANGELOG entry (see why)
[ ] Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
[ ] PR must have changelog/ and integration/ labels attached

logs-backend: https://github.com/DataDog/logs-backend/pull/7523


test: export schemas in testing_libdir_fixture

When constructing a minimal environment for osbuild the exported dirs lacked "schemas" so the test_exports.py test failed on RHEL8.
This commit adds it (and also "assemblers" for good measure). With that the test will pass.

So this should address:
test_exports_normal FAILED
test_exports_with_force_no_preserve_owner FAILED
Right?

I am rechecking latest version on RHEL pipeline.

Verified with the Centos 8/9 pipelines. LGTM.


Rearrange OAuth doc

:+1:

Note that codeship is building master just fine. Travis however is having issues, for some reason. The build process gets killed. I am looking into it.


Some icons showing up as completely black on Carbon site

"Recording--Filled" and "Recording--Alt" icons are showing up as black circles on the Carbon site
SVGs on the design end seem to be fine, @joshblack mentioned it could be the transparent rectangle causing the issue.

Looking into this more, seems like the inner path is having its fill stripped by the build process. Will need to investigate what's the best way to resolve this 🤔

cc @chrisconnors-ibm I think this is the same two-tone issue, do we want to add this to the current sprint?

I can fix these in pr #375 if you want

Let's do ittt @chrisconnors-ibm

Closed in #375


Lonewolff

Keybase proof
I hereby claim:

I am lonewolff2020 on github.
I am abrahamlink (https://keybase.io/abrahamlink) on keybase.
I have a public key ASDvQMYfQSZiAJvOAOFajPTMOhVjHe9lUHxxArlupD7kjAo

To claim this, I am signing this object:
{
  "body": {
    "key": {
      "eldest_kid": "0120ef40c61f412662009bce00e15a8cf4cc3a15631def65507c7102b96ea43ee48c0a",
      "host": "keybase.io",
      "kid": "0120ef40c61f412662009bce00e15a8cf4cc3a15631def65507c7102b96ea43ee48c0a",
      "uid": "76b192617288989624d732dd629de719",
      "username": "abrahamlink"
    },
    "merkle_root": {
      "ctime":<PHONE_NUMBER>,
      "hash": "5dc4f4b43c60c7950fa6cdc08338c895e73956d894887208ab01b6acce7b840e730d2e49af14e0c18cc0791a96f43fbf5d1800a36b349c909043a76e1235141e",
      "hash_meta": "86c9f69f198a5081bc3860aa146ef88fe5499635862a62a4f59c93c101f46f34",
      "seqno": 18572596
    },
    "service": {
      "entropy": "XEbPmrNW1rr9LK3ffJe3rCJL",
      "name": "github",
      "username": "lonewolff2020"
    },
    "type": "web_service_binding",
    "version": 2
  },
  "client": {
    "name": "keybase.io go client",
    "version": "5.5.0"
  },
  "ctime":<PHONE_NUMBER>,
  "expire_in": 504576000,
  "prev": "ebbee03dfe022e5e5945a78e734a074b3385cb619d138b85acaf84cfba871e69",
  "seqno": 4,
  "tag": "signature"
}

with the key ASDvQMYfQSZiAJvOAOFajPTMOhVjHe9lUHxxArlupD7kjAo, yielding the signature:
hKRib2R5hqhkZXRhY2hlZMOpaGFzaF90eXBlCqNrZXnEIwEg70DGH0EmYgCbzgDhWoz0zDoVYx3vZVB8cQK5bqQ+5IwKp3BheWxvYWTESpcCBMQg677gPf4CLl5ZRaeOc0oHSzOFy2GdE4uFrK+Ez7qHHmnEIOx8xogE1xDSPz6IVsOr8ZEV74bjXizSiYGj84UwCKxdAgHCo3NpZ8RAvAkUbVtZJ//1EHSthR6loKDcviIUqRjxhlEe49Nt4kl599CyfypcgTE1M8atx8aPDodco3ZgJl/reYRM2v3MC6hzaWdfdHlwZSCkaGFzaIKkdHlwZQildmFsdWXEIPXi52IqjK9L7p+YERhlh49Xehbi1mySWm0kB4Jh/d2ro3RhZ80CAqd2ZXJzaW9uAQ==

And finally, I am proving ownership of the github account by posting this as a gist.
My publicly-auditable identity:
https://keybase.io/abrahamlink
From the command line:
Consider the keybase command line program.
# look me up
keybase id abrahamlink

I believe this belongs in a Gist and not as an issue.

I believe this belongs in a Gist and not as an issue.

Ya my bad I'm a newbie at this lol

Ya my bad I'm a newbie at this lol

@Lonewolff2020 please close this issue, as it was indeed meant to be a gist :D

@Lonewolff2020 please close this issue, as it was indeed meant to be a gist :D

it seems nobody cleans issues anymore.
that's to be worried

it seems nobody cleans issues anymore.
that's to be worried


Replaced FieldInfo with FieldRefAccess wherever possible

Additionally, updated Choice Of Psycasts and Path Avoid (as I noticed they weren't up to date while working on them).
Using FieldRefAccess is more convienient than using FieldInfo, as well as it's significantly more performant. It might be negligible in most of those cases as they're very rarely used - although there's no point in not doing it either. Unfortunately, I don't think we can do similar to MethodInfo, unless we reference the .dll and call the methods directly.
Relies on #143 for a couple of small changes to Vanilla Furniture Expanded - Security.

Significant performance increase compared to keeping the reflected field around? I'll have to see it in action because it's hard to believe.


Make directory creation idempotent

Description of changes:
Add -Force param to new dir creation to allow future runs to be error-free.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Just to make sure I am clear on this one, the -Force is where the Idempotency comes in? Should a If statement or something added to Check? Should this be put in a Custom Script Block in DSC?  Look forward to hearing your feedback and thoughts.

Yep, the -Force ignores the error if the directory already exists and doesn't mess with anything previously created. e.g. if you create a dir and even add items to it, re-running New-Item ... -Force leaves things intact.
Perhaps not the best named flag! ;-)
IMHO, I don't think you need an if statement since the New-Item cmdlet already has this built-in mechanism. I also don't think you need it in the custom script block either for the same reason.
HTH!

Cool Thanks I am testing this and the other PR, it failed so I need to determine cause. I will throw up errors in bit.


Squarespace: Update WFH policy

Adds or updates the following events or companies:
Squarespace
Checklist
Company updates

[x ] I have put the most recent relevant date in the "Last Update" column
[ ] I have linked to the article about the change, not the company's website (Please put N/A if there is no public post)

Event updates

[ ] I have linked to the article about the change, not the event's website

There's another PR made earlier for the same change.


Always respond with go-import metadata when go-get=1

Description
When requesting a private repo with authorization credentials (eg. Authorization: Basic ...) it would be nice if gogs would respond with the relevant meta tags instead of redirecting via 302.
Alternatively, gogs could serve the meta tags regardless of whether the repo actually exists thereby mitigating the security concerns discussed in #816

I agree that if the same behavior is present for all repositories, even when the repository does not even exist, there would technically be no data leak. My proposal is if ?go-get=1 then always serve a page with go-import meta tags and return a 200 status code, even if the user is unauthenticated, and even if the repository does not exist

When requesting a private repo with authorization credentials (eg. Authorization: Basic ...) it would be nice if gogs would respond with the relevant meta tags instead of redirecting via 302.

@rschmukler  Somehow, I think basic auth already works: https://github.com/gogits/gogs/blob/master/modules/auth/auth.go#L122

My proposal is if ?go-get=1 then always serve a page with go-import meta tags and return a 200 status code, even if the user is unauthenticated, and even if the repository does not exist

@bradrydzewski any use cases?

We give the ability (with Drone) to authenticate with a netrc file. This enables cloning multiple private repositories over git+https without prompting the end user for a username / password. This works well because git uses curl under the covers which respects the authentication profiles defined by netrc files.
While there are multiple approaches to git authentication, the use of a netrc is pretty common and documented by most major hosting providers:
https://confluence.atlassian.com/bitbucketserver/permanently-authenticating-with-git-repositories-776639846.html

Somehow, I think basic auth already works: https://github.com/gogits/gogs/blob/master/modules/auth/auth.go#L122

Does this work with when using {token}:x-oauth-basic?
I am able to clone a private repository using the netrc, however, it seems to fail authentication when attempting the go get meta-data request with ?go-get=1. This would lead me to believe that go get does not respect the netrc file, or that gogs does not support basic authentication at this endpoint using the oauth2 token (I am probably totally wrong here, but just trying to guess some possible root causes).
This is not an issue with private GitHub or Bitbucket repositories because the go tool does not make outbound http requests to these systems to fetch the  go-import metadata. This is because this information is already hard-coded into the go tool.
The workaround would be to use git clone instead of  go get to ensure the netrc file is used as seen below. This works for simple projects with a small number of dependencies, however, this may be onerous for a project with a large number of private repositories compared to using go get ./...
- go get gogs.foo.com/bar/baz
+ git clone gogs.foo.com/bar/baz.git

@bradrydzewski thanks for the detail!

Does this work with when using {token}:x-oauth-basic?

I think this is not supported, only standard format Authorization: Basic xxx.

The workaround would be to use git clone instead of go get to ensure the netrc file is used as seen below. This works for simple projects with a small number of dependencies, however, this may be onerous for a project with a large number of private repositories compared to using go get ./...

OK make sense, let's wait for @rschmukler 's response.

Ah, I actually had only also tested it with Drone's injected netrc file (which uses x-oauth-basic). I have the same use-case as @bradrydzewski mentioned (as I actually ran into this while trying to get drone + gogs working with private gogs repos).
Another reason that this would be nice is that some of the tooling around vendoring also expect go get to work so relying on git clone <repo>.git can rule out using some of the tools, or at the very least require additional configuration (for example glide can be tricked into working with manually specifying VCS, but still logs warnings)

OK, maybe you want to change issue title to Always response go-import metadata when go-get=1?

@Unknwon done :+1:

Patch has pushed to fix this issue, please test on develop branch or https://try.gogs.io.

Hey, is this already on master? How it's supposed to work?
I'm asking this because on private repositories it seems do not work on a docker container of Gogs, and at least on release notes of version v0.9.71 there is a mention of this issue as an improvement ("Always response with go-import metadata when ?go-get=1 #2825"), however..in this docker container, when trying with Authorization: Basic .... and without that header on a repository page it simply returns Found. without adding any go-import metadata...
Thanks for the great software, anyway. :)

Test on demo site as required.

@fjorgemota You can't normally go get a private repository (for obvious reasons)
See this: https://gist.github.com/shurcooL/6927554

Just in case this might be useful for someone else.
All our repositories are private, but we wanted to use the go toolchain with as little friction as possible (and without adding .git at the end of all our imports). Since we already proxy Gogs with Nginx, I modified the config to return the "go-import"  content if the go-get query parameter is set.
Tested with go get and it worked :)
Example Config
Just replace git.domain.com with your domain. This assumes your using git over ssh.
location ~ "(/[^/]+/[^/]+)(/.*)?" {
  if ($arg_go-get = "1") {
    return 200 '\n\n<meta name="go-import" content="git.domain.com$1 git<EMAIL_ADDRESS>get git.domain.com$1\n\n';
  }
  proxy_pass http://localhost:3000;
}

location / {
  proxy_pass http://localhost:3000;
}

@wgriffiths I think it is working by itself? https://try.gogs.io/unknwon/404?go-get=1

@Unknwon  I think the reason it doesn't work for us is that we have "Require Sign In View" enabled.
If we run the following curl command that goes directly to gogs.
curl -i http://localhost:3000/org/repo\?go-get\=1

We get this response
HTTP/1.1 302 Found
Location: /user/login
Set-Cookie: lang=en-US; Path=/; Max-Age=2147483647
Set-Cookie: i_like_gogits=fdae2c13d4d64a97; Path=/; HttpOnly
Date: Wed, 05 Apr 2017 21:27:25 GMT
Content-Length: 34
Content-Type: text/html; charset=utf-8

Found.

@wgriffiths I see... that is indeed complicated situation...


Added the possibility to change the ARB path

I added an optional setting for the yaml file to change the path to the ARB files from lib/l10n to a custom path. It should be fully compatible with any previous version and won't affect any project as long as the arb_path is not set manually.
To keep it compatible with your service, I also added a optional argument for the localizely download and upload which defaults to the standard folder lib/l10n.

Hi @Tyxz
Thanks for your contribution!
We will check it out within few days

Hi @Tyxz,
Your PR is merged now. We will try to publish intl_utils v1.7.0 today.
We still need to adopt Flutter Intl IDE plugins to use configurable ARB directory paths, and we will try to deliver that soon.
Thank you for the great work!


Disable mouse interactions (or allow selecting text)

Is there a way to disable mouse interactions? It makes it impossible to select the test displayed on the screen (iTerm2, OSX), e.g. I cannot perform simple operations, such as copying a URL.

Hi there. Are you using a recent checkout from master (i.e. one from the last day)? You can select text with mouse interaction enabled by holding down a modifier key (different key for different platforms - command on my Mac). Yes, this is slightly obscure, but as of yesterday we detect attempts to select and actually give users a hint in the interface. I want to avoid including a special flag to turn off mouse interaction, if this is clear enough.

I don't that mouse interactions should be part of a CLI program to begin with. Most people running CLI program will never lift their hands from keyboard just to select something that would take few keyboard strokes.
That said, if there is a new build that at least allows to select screen on text using a modifier key, that at least is going to work. Nevertheless, I expect many people reporting this as an inconvenience.
Thank you

You can use the modifier keys on any recent build - we've just added a hint to let you know that you need to use the modifier keys in the release that's on the way out the door. This release also includes things like being able to scroll with the scrollbar, click to select flows, and so forth - that is, some of the features we think justifies the inconvenience.
As for wether a CLI program should include mouse interaction - I'm not so sure. Believe me, I'm as fanatical as anyone about keyboard interaction (to the extent that I wrote qtile.org specifically so I never needed to touch a mouse), but here I think it pays dividends. The truth is that most people use mitmproxy on their workstations, and being able to do things like scroll through flows with the scrollwheel is very convenient. Never fear - mitmproxy's keyboard interaction will always remain very strong.

This is surprisingly troublesome. I think that the right approach should have been to break the mitmproxy into two separate programs:

One acting as a mere API that encompasses all the mitmproxy functionality
Second a CLI tool as it has been using the underlying API

Then, suppose you insist to keep the mouse interactions, have a separate program using the latter API that provides UI. Added benefit is that it would allow community to build custom interfaces to mitmproxy. Now you have a quasi CLI tool and no proper UI for people who prefer pointer interactions. Both camps lose.

Do you mean it's surprisingly troublesome that you have to press a modifier key to select, or have you found an issue that means it doesn't work?
The architecture you describe is in fact exactly the one mitmproxy has - mitmproxy, mitmdump and the soon-to-be-released mitmweb are all just front-ends onto the same proxying back-end.

It's quite annoyning, beside the fact that I'm on OS X El capitain and no modifier key allows me to select :( don't know what could be?

Same for me, on OS X none of the listed modifiers work.
Instead you have to press the 'fn' button to be able to select and copy.

In case anybody hit here from Google, you could disable mouse interaction temporarily via (#735):
mitmproxy --no-mouse

You could also write the option to the configuration file to permanently disable it.
echo no-mouse >> ~/.mitmproxy/mitmproxy.conf

cc #647.

--no-mouse is deprecated, can use --set console_mouse=false inline

Also, in the latest version (after https://github.com/mitmproxy/mitmproxy/pull/5393), the name of config file has been changed to config.yaml or config.yml, and you may set console_mouse to false accordingly:
console_mouse: false


Remapping media keys to Spotify ?

Have you found any fixes to do this? Best I've tried so far is just disabling iTunes with this config. But it also disables the button for spotfity
Wouldn't it be nice to start Spotify with f8?
    # Stop iTunes from responding to the keyboard media keys
    launchctl unload -w /System/Library/LaunchAgents/com.apple.rcd.plist 2> /dev/null

Wouldn't it be nice to start Spotify with f8?

Would be nice for sure 👍
I remember when macOS High Sierra caused F7-F9 to stop working for Spotify, I added highsierramediakeyenabler in my global packages f1e5d9782d1359332b92c4437dfd42c5198cceb6
But then when Mojave came and I upgraded, the keys worked "correctly" again, and I removed it 72b8e49f62b1f2302ce90f10b4d6290e504baa77
So I no longer can reproduce the issue neither in Mojave nor in Catalina 🤔


fix: Escape elements swallowed in markdown

@nschonni : Thanks for your contribution! The author, @goraco, has been notified to review your proposed change.

This request is in review.

@GitHubber17 : Thanks for your contribution! The author, @goraco, has been notified to review your proposed change.

@GitHubber17 : Thanks for your contribution! The author, @goraco, has been notified to review your proposed change.

All corrections verified. Thank you


Task #50 set mandatory constraints

With this PR I want to:

Set new tab metadata as a required field (only from a UI perspective not from a code perspective)
add a placeholder in rndt/layers/forms.py
add extension JS for geonode
remove DISABLE_LAYER_CONSTRAINTS as a variable, now are disabled by default

NOTE: this PR included also the part related to UI_REQUIRED_FIELDS, this part is going to be introduced in GeoNode soon

@mattiagiupponi can you pls link in a comment the related PR in geonode?


Journal Entry: Event Ingestion

See #184 for context.

[ ] common requirements
[ ] Fabrikam's requirements
[ ] logical solutions
[ ] technology options
[ ] security concerns
[ ] learnings

#236, #245


Update clipboard icon to latest version

Is your feature request related to a specific problem?
The design team has released the latest version of the copy/clipboard icon. We will need to update all pages that use it.
Media icons can be found in the "Fides UI library" in Figma.

We'll save this for Chris Calhoun's first day

@adriaaaa I'm not seeing the copy/clipboard icon in figma. Mind showing me specifically where it is?

@conceptualshark this is the ticket where we are updating the clipboard icon. Not a high priority but in case you have any images in the docs using this icon, then we should update our docs.

@adriaaaa I'm not seeing the copy/clipboard icon in figma. Mind showing me specifically where it is?

@eastandwestwind did my slack tag help answer this question?

@adriaaaa yup I got it, thanks!


Cannot start service netbox-housekeeping

Hi. I can not deploy netbox-plugin config-officer. I received an error trying to create netbox-docker_netbox-housekeeping_1_2da30c.
Could you help me with this? I'm following step-by-step, from github and youtube.
I'm using Ubuntu: 20.04.3 LTS with vagrant/virtualbox.
Creating netbox-docker_redis_1_eda484f67269       ... done
Creating netbox-docker_postgres_1_d9596c0d647a    ... done
Creating netbox-docker_redis-cache_1_bbd69de6aef9           ... done
Recreating netbox-docker_netbox-housekeeping_1_2da30cc7e86e ... error
Creating netbox-docker_netbox-worker_1_269284fe7c9d         ...
ERROR: for netbox-docker_netbox-housekeeping_1_2da30cc7e86e  Cannot start service netbox-housekeeping: driver failed programming external connectivity on endpoint netbox-docker_netbox-housekeeping_1_2da30cc7e86e (Creating netbox-docker_netbox-worker_1_269284fe7c9d         ... done
Creating netbox-docker_netbox_1_28722ab27218                ... error
ERROR: for netbox-docker_netbox_1_28722ab27218  Cannot start service netbox: driver failed programming external connectivity on endpoint netbox-docker_netbox_1_5cb5c8adec32 (d5516554e621d0452f9d64074071c81e5f64ef273582c0540b1a5c33f535a116): Bind for <IP_ADDRESS>:8080 failed: port is already allocated

Hello @pslencinas .
Thank you for the iisue.
First of all I have to warn you, my plugin is only supported by Netbox 2.7 and earlier. I was out of coding life for a while and didn't have time to fix it. So you could not use it with new NetBox.
Regarding error - looks strange, but need more investigation.
Don't you have other containers that listen 8080 port?
Anyway, my plan is make some changes in my plugin's code and make it available for netbox 3.x.


Start your bot instructions are lacking.

"Start your bot
Change directories to the one created for your bot, and start it. At this point, your bot is running locally."
These instructions are lacking.
Something like this would be better:
In Powershell/Bash, change directories to the one created for your bot.
Run:
npm start
[now start the Bot Emulator, etc.]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: dfaad3f2-75c5-e958-4734-8925d03d95c6
Version Independent ID: 8d98b390-5368-1fa4-8cce-3b0cff252eaa
Content: Create a bot using Bot Builder SDK for JavaScript - Bot Service
Content Source: articles/javascript/bot-builder-javascript-quickstart.md
Product: bot-framework
GitHub Login: @JonathanFingold
Microsoft Alias: jonathanfingold

This will be resolved in the next publish.


[BUG] GpuInMemoryTableScanExec doesn't wrap it's children to GPU versions

Describe the bug
The GpuInMemoryTableScanExec doesn't convert it's children to GPU which causes tests to fail if the InMemoryTableScanExec has children. We have been getting away with this for quite sometime

Can you give more details on the test failures that necessitate this change?

We are not testing for this. I saw this while working on adding tests for #10518. I can create a test for this and add to this issue.

Here is the code I am referring to, we can see that we are passing the parameters without converting them to the GPU versions.
I think this is not OK but I have to write a test for it before I can prove it.


feature request: msys2-runtime build with -gdwarf-2 -gstrict-dwarf for easier debugging on Wine

Dear MSYS2 folks,
I'm continue working on msys2 support on Wine.
Due to limitation of Wine Debugger, dwarf3 and dwarf4 is not supported yet, as a result, it was difficult to debug MSYS2 on Wine since modern gcc use dwarf4 by default.
Wine itself is compiled with -gdwarf-2 -gstrict-dwarf in order to be compatible with Wine Debugger (winedbg).
In the past, when I debug 32 bit msys2 on Wine, I tried several workarounds, one is to rebuilt 32 bit msys2 on Windows with -gdwarf-2 -gstrict-dwarf, the other one is to use another working debugger to debug msys2 on Wine. None of them is very convenient, sometimes we need some mix solution.
Now i'm starting to debug 64 bit msys2 on Wine, however, I don't have a 64 bit Windows, and before we are able to bootstrap 64 bit msys2 on Wine, I'm not able to rebuild msys2-runtime by myself, so it become a chicken and egg problem for me: in order to debug 64 bit msys2, I need to rebuild 64 bit msys2; in order to rebuild 64 bit msys2, I need to fix 64 bit msys2 bugs :)
Could you ship a separate msys2-runtime binary and debugging symbol compiled with -gdwarf-2 -gstrict-dwarf so I can download and easily debug them on Wine? This won't be a request forever, I just need a timing window to fix Wine bugs, and once I'm able to bootstrap 64bit msys2 on Wine, then I'll close this feature request, and we can stop shipping dwarf2 msys2-runtime.
And once I'm free from urgent task like bootstrap 64 bit msys2-runtime on Wine, I might have time to work on improving dwarf4 support in Wine debugger, or someone else might have interesting to do that, who knows :)
Thank you!

I applied the following changes to the msys2-runtime package:
--- a/msys2-runtime/PKGBUILD
+++ b/msys2-runtime/PKGBUILD
@@ -23,7 +23,7 @@ makedepends=('cocom'
              'gettext-devel'
              'libiconv-devel'
              'diffutils')
-# options=('debug' '!strip')
+options=('debug' '!strip')
 source=('msys2-runtime'::'git+https://github.com/Alexpux/Cygwin.git#branch=msys2-master'
         '0001-dcrt0.cc-Untangle-allow_glob-from-winshell.patch'
         '0002-dcrt0.cc-globify-Don-t-quote-literal-strings-differe.patch'
@@ -46,6 +46,8 @@ pkgver() {

 prepare() {
   cd "${srcdir}"/msys2-runtime
+  git config user.email private@private
+  git config user.name private
   git am "${srcdir}"/0001-dcrt0.cc-Untangle-allow_glob-from-winshell.patch
   git am "${srcdir}"/0002-dcrt0.cc-globify-Don-t-quote-literal-strings-differe.patch
   git am "${srcdir}"/0003-strace.cc-Don-t-set-MSYS-noglob.patch
@@ -61,7 +63,7 @@ build() {
   local -a extra_config
   if check_option "debug" "y"; then
     export CCWRAP_VERBOSE=1
-    OPTIM="-O0"
+    OPTIM="-O0 -gdwarf-2 -gstrict-dwarf"
     extra_config+=(--enable-debugging)
   else
     OPTIM="-O2"

Then I attempted to build msys2-runtime, but I hit an error in the build() step, and the error did not go away when I tried again:
/d/builds/msys2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/speclib --cpu=x86_64 --ar=ar --as=as --nm=nm --objcopy=objcopy --exclude='cygwin' --exclude='msys' --exclude='(?i:dll)' --exclude='reloc' --exclude='^main$' --exclude='^_main$' libmsys-2.0.a minires.o libresolv.a
/d/builds/msys2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/speclib --cpu=x86_64 --ar=ar --as=as --nm=nm --objcopy=objcopy --exclude='cygwin' --exclude='msys' --exclude='(?i:dll)' --exclude='reloc' --exclude='^main$' --exclude='^_main$' libmsys-2.0.a posix_ipc.o librt.a
/d/builds/msys2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/speclib --cpu=x86_64 --ar=ar --as=as --nm=nm --objcopy=objcopy --exclude='cygwin' --exclude='msys' --exclude='(?i:dll)' --exclude='reloc' --exclude='^main$' --exclude='^_main$' libmsys-2.0.a sec_posixacl.o libacl.a
perl -p -e 'BEGIN{binmode(STDIN); binmode(STDOUT);}; s/msys-2.0/msys0/g' < libmsys-2.0.a > libmsys0.a
/d/builds/msys2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/speclib --cpu=x86_64 --ar=ar --as=as --nm=nm --objcopy=objcopy --exclude='cygwin' --exclude='msys' --exclude='(?i:dll)' --exclude='reloc' --exclude='^main$' --exclude='^_main$' libmsys-2.0.a /d/builds/msys2-packages/msys2-runtime/src/build-x86_64-pc-msys/x86_64-pc-msys/winsup/cygwin/libm.a libpthread.a libutil.a -v libc.a
make[3]: Leaving directory '/d/builds/msys2-packages/msys2-runtime/src/build-x86_64-pc-msys/x86_64-pc-msys/winsup/cygwin'
make[2]: Leaving directory '/d/builds/msys2-packages/msys2-runtime/src/build-x86_64-pc-msys/x86_64-pc-msys/winsup'
Makefile:8381: recipe for target 'all-target-winsup' failed
make[1]: *** [all-target-winsup] Error 2
make[1]: Leaving directory '/d/builds/msys2-packages/msys2-runtime/src/build-x86_64-pc-msys'
Makefile:833: recipe for target 'all' failed
make: *** [all] Error 2
==> ERROR: A failure occurred in build().

Here's a full build log in case it helps:
http://tmphax.com/hold/msys2-runtime-build.log

I'm not able to build 32 bit of msys2-runtime for a while as well.
 Here I don't see what is wrong

Thanks, @AlexPux.  I posted my full build log here: http://tmphax.com/hold/msys2-runtime-build.log
Yeah, there doesn't seem to be an error message, but I would assume speclib returns a non-zero error code.

Error here:
/opt/lib/gcc/x86_64-w64-mingw32/4.9.2/../../../../x86_64-w64-mingw32/lib/../lib/libadvapi32.a: error adding symbols: File in wrong format
collect2: error: ld returned 1 exit status
Makefile:76: recipe for target 'cyglsa64.dll' failed
make[3]: *** [cyglsa64.dll] Error 1

Hi @DavidEGrayson , did you finally success to build msys2-runtime? Recently Alexpux push a msys2-runtime update but it fails to build on CI: http://wine-ci.org/Alexpux/MSYS2-packages/109
Makefile:70: recipe for target 'cyglsa.dll' failed

I still don't have time to look deeper, if someone else can reproduce that would be great.

Looking into last build failure I can say only that seems configure can't find mingw-w64 g++ from mingw-w64-cross-gcc package

@fracting: Nope, I tried again and I am getting an error.
    Aborting...

I ran which perl and verified that it is /usr/bin/perl.  This error looks pretty similar to my last one but I assure you that I updated all my MSYS2 packages and I updated the MSYS2-packages github repo before trying.  I also reapplied the changes described in my earlier comment to enable DWARF-2 debugging.

Now I guess I know what's wrong: It's because mingw-w64-cross-crt-clang-git-<IP_ADDRESS>92.90b8472-1 is used instead of  mingw-w64-cross-crt-git.

Hi @DavidEGrayson , alexey updated mingw-w64-cross-crt-clang-git and it won't be selected as default anymore, could you manually install mingw-w64-cross-crt-git and try again? Thanks!

@fracting  OK, I tried again and now it's a different error.  Apparently ../stdio/lib.a is not found.
gcc -L/d/builds/msys2-packages/msys2-runtime/src/build-x86_64-pc-msys/x86_64-pc-msys/winsup/cygwin -isystem /d/builds/msys2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/include -B/d/builds/msys2-packages/msys2-runtime/src/build-x86_64-pc-msys/x86_64-pc-msys/newlib/ -isystem /d/builds/msys2-packages/msys2-runtime/src/build-x86_64-pc-msys/x86_64-pc-msys/newlib/targ-include -isystem /d/builds/msys2-packages/msys2-runtime/src/msys2-runtime/newlib/libc/include    -I/d/builds/msys2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/include -DPACKAGE_NAME=\"newlib\" -DPACKAGE_TARNAME=\"newlib\" -DPACKAGE_VERSION=\"2.3.0\" -DPACKAGE_STRING=\"newlib\ 2.3.0\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -I. -I/d/builds/msys2-packages/msys2-runtime/src/msys2-runtime/newlib/libc/xdr -DHAVE_OPENDIR -DHAVE_RENAME -DSIGNAL_PROVIDED -D_COMPILING_NEWLIB -DHAVE_BLKSIZE -DHAVE_FCNTL -DMALLOC_PROVIDED -fno-builtin      -g -O2 -O0 -gdwarf-2 -gstrict-dwarf -pipe -ggdb -c -o lib_a-xdr_sizeof.o `test -f 'xdr_sizeof.c' || echo '/d/builds/msys2-packages/msys2-runtime/src/msys2-runtime/newlib/libc/xdr/'`xdr_sizeof.c
gcc -L/d/builds/msys2-packages/msys2-runtime/src/build-x86_64-pc-msys/x86_64-pc-msys/winsup/cygwin -isystem /d/builds/msys2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/include -B/d/builds/msys2-packages/msys2-runtime/src/build-x86_64-pc-msys/x86_64-pc-msys/newlib/ -isystem /d/builds/msys2-packages/msys2-runtime/src/build-x86_64-pc-msys/x86_64-pc-msys/newlib/targ-include -isystem /d/builds/msys2-packages/msys2-runtime/src/msys2-runtime/newlib/libc/include    -I/d/builds/msys2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/include -DPACKAGE_NAME=\"newlib\" -DPACKAGE_TARNAME=\"newlib\" -DPACKAGE_VERSION=\"2.3.0\" -DPACKAGE_STRING=\"newlib\ 2.3.0\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -I.
    Aborting...

Hi @DavidEGrayson now I have a Win64 installation, unfortunately I have the same error with you. @Alexpux any ideas?

gcc -L/home/fracting/MSYS2-packages/msys2-runtime/src/build-x86_64-pc-msys/x86_64-pc-msys/winsup/cygwin -isystem /home/fracting/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/include -B/home/fracting/MSYS2-packages/msys2-runtime/src/build-x86_64-pc-msys/x86_64-pc-msys/newlib/ -isystem /home/fracting/MSYS2-packages/msys2-runtime/src/build-x86_64-pc-msys/x86_64-pc-msys/newlib/targ-include -isystem /home/fracting/MSYS2-packages/msys2-runtime/src/msys2-runtime/newlib/libc/include    -I/home/fracting/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/include -DPACKAGE_NAME=\"newlib\" -DPACKAGE_TARNAME=\"newlib\" -DPACKAGE_VERSION=\"2.3.0\" -DPACKAGE_STRING=\"newlib\ 2.3.0\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -I. -I/home/fracting/MSYS2-packages/msys2-runtime/src/msys2-runtime/newlib/libc/stdio -DHAVE_OPENDIR -DHAVE_RENAME -DSIGNAL_PROVIDED -D_COMPILING_NEWLIB -DHAVE_BLKSIZE -DHAVE_FCNTL -DMALLOC_PROVIDED -fno-builtin      -g -O2 -O0 -pipe -gdwarf-2 -gstrict-dwarf -c -o lib_a-putc.o `test -f 'putc.c' || echo '/home/fracting/MSYS2-packages/msys2-runtime/src/msys2-runtime/newlib/libc/stdio/'`putc.c
gcc -L/home/fracting/MSYS2-packages/msys2-runtime/src/build-x86_64-pc-msys/x86_64-pc-msys/winsup/cygwin -isystem /home/fracting/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/include -B/home/fracting/MSYS2-packages/msys2-runtime/src/build-x86_64-pc-msys/x86_64-pc-msys/newlib/ -isystem /home/fracting/MSYS2-packages/msys2-runtime/src/build-x86_64-pc-msys/x86_64-pc-msys/newlib/targ-include -isystem /home/fracting/MSYS2-packages/msys2-runtime/src/msys2-runtime/newlib/libc/include    -I/home/fracting/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/include -DPACKAGE_NAME=\"newlib\" -DPACKAGE_TARNAME=\"newlib\" -DPACKAGE_VERSION=\"2.3.0\" -DPACKAGE_STRING=\"newlib\ 2.3.0\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -I. -I/home/fracting/MSYS2-packages/msys2-runtime/src/msys2-runtime/newlib/libc/stdio -DHAVE_OPENDIR -DHAVE_RENAME -DSIGNAL_PROVIDED -D_COMPILING_NEWLIB -DHAVE_BLKSIZE -DHAVE_FCNTL -DMALLOC_PROVIDED -fno-builtin      -g -O2 -O0 -pipe -gdwarf-2 -gstrict-dwarf -c -o lib_a-putchar.o `test -f 'putchar.c' || echo '/home/fracting/MSYS2-packages/msys2-runtime/src/msys2-runtime/newlib/libc/stdio/'`putchar.c
In file included from /home/fracting/MSYS2-packages/msys2-runtime/src/msys2-runtime/newlib/libc/stdio/putc.c:83:0:
/home/fracting/MSYS2-packages/msys2-runtime/src/msys2-runtime/newlib/libc/include/stdio.h: In function '_putc_r':
/home/fracting/MSYS2-packages/msys2-runtime/src/msys2-runtime/newlib/libc/include/stdio.h:655:22: error: inlining failed in call to always_inline '__sputc_r': function not considered for inlining
 _ELIDABLE_INLINE int __sputc_r(struct _reent *_ptr, int _c, FILE *_p) {
                      ^
/home/fracting/MSYS2-packages/msys2-runtime/src/msys2-runtime/newlib/libc/include/stdio.h:658:4: error: called from here
    __sputc_r (_ptr, '\r', _p);
    ^
Makefile:1104: recipe for target 'lib_a-putc.o' failed
make[5]: *** [lib_a-putc.o] Error 1
make[5]: *** Waiting for unfinished jobs....
make[5]: Leaving directory '/home/fracting/MSYS2-packages/msys2-runtime/src/build-x86_64-pc-msys/x86_64-pc-msys/newlib/libc/stdio'

DavidEGrayson, did you succeed in building? For me https://github.com/Alexpux/MSYS2-packages/issues/421#issuecomment-199046874 was the solution. The question now is how to obtain backtraces. For now wine-staging crashes and there are still no line numbers. @fracting, what should I check?

I don't remember.


POST Queries?

A simple question:
Openapi-to-Graphql sees some of the endpoints as mutations. However, these are not really mutations, they are queries.  All these require a POST request to be sent instead of a GET, since they have several parameters and it's easier to transmit those in the request body.
Can I make Openapi-to-Graphql to handle these as Queries?
Do I have to edit the generated GraphQL schema manually to achieve this?

By the way, how exactly OtG map the REST op to GraphQL type? I mean, how does it know if the op is a Query or Mutation? Is this determined by the request method? All GETs are queries and  POST, PUT, DELETE..etc are mutations?
It would be great to find way to use non-GET endpoints as Queries. Using POST method doesn't necessarily  mean the endpoint will mutate data.

@gtamas OpenAPI-to-GraphQL does indeed rely on the HTTP method to determine whether to create a query or mutation field. OtG will create query fields only for GET operations, which are both safe and idempotent. POST operations in particular should, according to HTTP, neither be safe nor idempotent (see for example https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/POST). I think the current behavior is sound, and should remain the default to ensure most users get an expected outcome.
Using POST for a safe, idempotent operation is a bit an anti-pattern, so I am also skeptical of enabling it via configuration. Still, to think this through, how would you propose an option to overwrite the default behavior to look like?

@ErikWittern
Yes it's true, using POST as query is not recommended, it's actually a sign of bad API design, so therefore the current default behaviour should remain unchanged.
However, we have to accept that people still use such patterns for some reason, so it should be possible to handle these somehow.
I think, normally all POST ops should be mutations, but it should be possible to "mark" certain ops, so OtG knows these are actually queries and handles them as such. I see 2 possibilities:

Handle this on the spec level. For example, Operation objects have a "tags" field which is an array of string. So one could add the "query" tag to let OtG (and everybody else :) know this is a query.

2 it should be at least possible to pass some command line args to OtG like:
opeanapi-to-graphql  --qurey someOperation --query someOp2
... where someOperation and someOp2 are  OpenAPI 3 operationIds.
What do you think?

@ErikWittern
Actually, the using the tags field is not ideal, but I didn't find any other suitable field in the spec.
However, maybe there is a better way. OtG can already save the generated the GraohQL schema right?. How about adding the ability to load custom GraphQL schema? Then one could simply generate a schema, modify it and then load it in which case OtG could skip the conversion and use the passed schema directly.
Of course this only works if changing the generated schema is easy, like moving the given operation to Query { ... } block from Mutation { ... }. If complex changes are required then this method is not good either.
Maybe this would be cleaner solution.
Anyway, some kinda solution should be provided to support not only REST but also REST-like APIs.
Unfortunately, many APIs don't follow the REST spec. This quite common with old, legacy systems which cannot be changed easily (or at all).
What do you think?

@gtamas Interesting proposal! Perhaps instead of using the tags field, the user can provide an object that points to a number of operations (or perhaps just the path) similar to the customResolvers option.
On loading and modifying a schema, I think this idea might be outside the scope of OtG. It sounds like it should be a separate tool.

I am also not sure about loading modified schemas. OtG currently associates created resolver functions with the data types returned by the corresponding REST operations. Modifying the schema easily breaks that relationship. I currently think this is out of scope.
If certain operations should be treated differently than expected, I would propose to make that an option of OtG. I don't like the idea of extending an OpenAPI description with non-standard things / conventions just for the sake of using that specification with OtG. I.e., I prefer a --query and --mutation option followed by the operation id that overwrites how OtG decides to treat that operation.

@ErikWittern
OK I see, cool. Adding the --query / --mutation flags would be great, I also proposed this. Maybe this would be the best solution.
And it should be also possible to do this when OtG is used as a lib. Something  like @Alan-Cha  proposes would be nice.

@wtrocki @ErikWittern @Alan-Cha
Actually, this sounds like a really cool idea but unfortunately it doesn't seem to work.
Whenever I mutate the schema this way, all queries return something like this:
{
  "data": {
    "someType": null
  }
}

In fact, this happens even I don't return any value in leave() / enter() methods,  in which case nodes won't be changed at all, so the schema should be unchanged.
I guess, this is because buildASTSchema() returns something with no resolvers and thus it cannot be used to execute queries, as stated here:
https://graphql.org/graphql-js/utilities/#buildastschema
Do I understand this correctly? Or am I doing something wrong? It seems to me that the schema created by OtG's createGraphQlSchema() cannot be modified with graphqlJS?

Any . other ideas to try? Anyone?
Is there any way to modify (post-process) the generated GQL Schema and make it work with OtG?
I really don't want to fork this and change the code.
It should be possible to make changes to the schema without losing functionality... As @wtrocki  pointed out, there are several use cases. A solution like the one he proposed would give us lots of flexibility.

@gtamas I'm so sorry. I let this slip by! I've been super busy and it's been difficult to keep up with everything.
Actually, in regards to forking and changing the code. For us, that is highly encouraged if those changes can lead to some kind of PR. That is the reason why we open-sourced, so we could work with a community to improve OtG.
I think we need to implement some kind of customResolvers option like I proposed earlier.
If you want to begin that effort, I would be happy to help you!

@Alan-Cha
OK thanks. I'm also pretty busy right now, but will check the code. :) Maybe we could fork it.
But just for clarification: Should @wtrocki's solution work or not?
import { printSchema, buildASTSchema, parse } from 'graphql';
import { createGraphQlSchema } from 'openapi-to-graphql';
...
  let { schema: graphQLSchema, report: graphQLReport } = await createGraphQlSchema(openAPISchema, options);

graphQLSchema = buildASTSchema(parse(printSchema(graphQLSchema)))

Executing this produces a schema which is valid, but none of the queries return any meaningful results. Without the  last line, however, it works fine.
So should the schema generated by buildASTSchema() be compatible with OtG?
Another question: are the resolvers part of the GraphQL schema OtG creates, or a resolver map has to be added separately?

Regarding adding an object like customResolvers, that would be cool!
However, there could other similar issues as well.
For example, how about adding custom links?
Use case: I'm working here with several REST APIs whose OAS does not have link objects, so OtG cannot generate links.
So maybe adding an option that allows one to pass link definitions (something like customResolvers) would be also useful.
That's why I tried @wtrocki's idea. A mechanism like this would be able to solve several such issues.

@Alan-Cha
OK so I'm not familiar with the code yet. Could you help me get started? First I would look into adding this object we discussed (custom operation mapping). But some basic info on how the relevant part of the code works would be appreciated. :)
Thanks in advance!

@gtamas I'm actually not too sure if @wtrocki's solution will work for this case. It can modify the schema itself but I don't think it can modify the resolvers. You would have to generate a new set of resolvers some other way.
To answer your question, the resolvers are contained within the schema (GraphQLSchemaObject) object that OtG creates.

I see what you mean about your use case but is it not possible to update them so that they do have link objects? I'm guessing they may be auto-generated? On one hand, I would like to direct users towards a single option, an option that is naturally provided by OAS, instead of externalizing it, but on the other hand, this might make OtG work better with auto-generated OASs. I would love to get the opinion of @ErikWittern and @wtrocki again if that's possible!

I am very happy to give you some pointers!
We preprocess (extracting the data) the OAS and create the data object here.
We create the GraphQL schema by iterating through all the operations stored in data here.
There is an if statement here that will determine whether an operation will be turned into a Query field or a Mutation field.
I think this should just be a matter of adding some additional logic to that if statement. We just need to check if the operation is mentioned in the option, maybe just call it selectQueryOrMutationField for now, and if it is, respond appropriately.
We will need to add the new option here and here. We process the options here and here.
I don't think it's that much work. Along with that, we will need to update the documentation and add a test case which I am happy to help you with when we reach that point.

@gtamas The selectQueryOrMutationField option should have a similar shape as customResolvers. You can see how the latter is defined here but instead of containing a ResolveFunction, I think it should point to an enum (which will we also need to define) containing just Query and Mutation.

@Alan-Cha
Thanks for the detailed info! I tried to follow your instructions and just opened a PR.
Now it is possible to make any operation a query or mutation.
https://github.com/IBM/openapi-to-graphql/pull/279
Please check it out! I'm not entirely sure if it's all OK, since this is my first PR here and I'm not that familiar with the code yet :)
Tests and docs are yet to be added. If you could provide some help with that, it would be great. :)
Thanks in advance

@gtamas This is a great start and thanks for taking a deep breath and diving in! I will try to give you some input later today! Honestly, this is fantastic work. We really appreciate your help!

@Alan-Cha
OK thank you very much! :)

@Alan-Cha
Hi, any news about this :)?

@gtamas I made some changes! Take a look and let me know what you think!

Closed with PR #282
Thanks @gtamas for your contributions!!!

@Alan-Cha
Thank you very much for merging and all the help! I think this feature will be very useful.


Using a temporary file in function downloadImage:

Fix the bugs that may occur after the download is accidentally interrupted

@z372183629 Sweet! Thanks :)


Paper Retraction

Hi,
Thank you for your work.
I saw the updated README, showing that the superior performance in your paper was because of a dataset leakage. Shouldn't that warrant a retraction of the paper, given that the results would have played a good role in the acceptance?

Hi,
Thanks for your interest. The updated results of our method are still the best template-free results at the time of submission. Our proposed method is still novel and effective.


Add spam memo exception for valid payers

Description
This PR builds on #415 and this comment about valid transactions that could come in below the 'memo spam threshold'
Approach
To solve this, @dolcalmi helped me to identify that the user with the "funder" role is usually the one sending these valid low-sat transactions.
Our approach was to add an additional step if a transaction is flagged below the memo threshold. We do a db call to lookup the username of the user that has the role property of "funder, and then compare this retrieved username to the from user's username property gotten from the transaction` object.
Implementation details
An additional db call is added asynchronously in the getTransactions method on the UserWallet object to retrieve the user with the role: "funder" property.
An earlier approach was tried where the additional lookup was included in the memoSpamFilter function itself only and only triggered (and then cached for that run of the method) if a txn amount was found below the memo sats threshold to reduce the number of times we would hit the db.
This added more complexity and seemed to be mixing concerns to much, so I abandoned it given our upcoming work to clean things up and separate different concerns.
Potential future improvements

To avoid this extra db call, we can also alternatively add an appropriate additional property (like user role) to transactions when they are created. This would require a migration.

While implementing I got stuck with a circular dependency bug I introduced here that was fairly obfuscated for quite a while and didn't cause tests to fail until quite late in the run.
Are there any linting or check we can add to catch circular deps at compile time?  (cc @samerbuna)
For e.g. I saw some folks use things like madge for this

Questions

Is the username property unique for users? (I assume that it is)

Is the username property unique for users? (I assume that it is)
@vindard Yes, it is https://github.com/GaloyMoney/galoy/blob/main/src/services/mongoose/schema.ts#L161

This PR will be closed and the new functionality should be implemented based on this proposal: https://github.com/GaloyMoney/galoy/issues/363#issuecomment-900305025


Early stopping with mmdetection

Dear all,
Thank again for the amazing library. It's a real pleasure to work with it.
I would like to introduce early stopping conditions in my training schedule, is it something which is already implement ? If not, could you point me to the modules that would need to be updated for such features ?
Best

Hi @EtienneDavid ,
We have not implemented it because it is a little bit rarely used in the detection field.
Since it is about the training pipeline, it is worth trying to implement it as a hook or something used by the runner in mmcv.

OK thank you for your answer. Why is it rarely used in detection field ? Is there an habit or there are more theorical reasons to have a fixed schedule ?

Practically, the overfitting phenomenon with the COCO dataset is not so apparent in common schedules (1x, 20e, 2x) since the dataset is large and augmentations are already enough.

OK thank you very much :)

But it becomes useful when training to customize a small number of data sets

But it becomes useful when training to customize a small number of data sets

@EtienneDavid @xincyu Hello Guys i am trying to implement early stopping mmdetection. it would be of great help if you can provide any insights apart from already mentioned. Since i am using very small dataset its needed. As of now i am running workflow with [(train,1)(val,1)] to capture losses on val set. Any help would be very much appreciated :)

Same problem, early stopping is a must when finetuning a model. I've tried to think how to implement it in mmdetection but it is so hard to extend the system and fetch the metric that it may be impossible without some heavy hacks

Early stopping would be really useful when someone wants to finetune a model to the small dataset.

EarlyStoppingHook is available since mmengine 0.7.0. (MMDetection >= 3.0)
You can call the hook it in your config under the "default_hooks" dictionary:
default_hooks = dict(     early_stopping=dict(type='EarlyStoppingHook',         monitor="mAP",         patience=10,         min_delta=0.005),     checkpoint=dict(         type="CheckpointHook",         interval=interval,         save_begin=100,         max_keep_ckpts=max_keep_ckpts,         save_best=save_best) )

Just expanding on @leetoffolo answer, is there a way to have dynamic learning rate schedules? Ie if coco/bbox_mAP does not increase for 10 epochs by min delta, drop the learning rate by a factor of 10 rather than finish training. I would like to do this twice, ie 2 learning rate drops, and the next time min delta change is not achieved finish the training.
I am comparing pretraining weights, ie COCO vs Imagenet. However, I have noticed that models pre-trained on COCO converge faster than Imagenet (this is expected from literature and transfer learning). However, models pre-trained on Imagenet are performing better than their COCO counterparts, I am attributing this to the Imagenet models running long enough to utilize the learning rate scheduler, however, the COCO models converge and finish training before the learning rate is dropped.
I guess a simple solution may be to decrease min delta & increase patience to ensure the models train long enough to hit the learning rate schedules, but this wouldn't be the greatest approach.


Implement MUC in Chat Server

Please consider implementing XEP-0045 in the Chat Server. http://xmpp.org/extensions/xep-0045.html

+1

A won't fix then? Good to have that sorted out.

@jessereynolds sounds like a bit of a dead project to me - check out https://github.com/sdelements/lets-chat
Yes it's nodejs shiver and mongodb vomit but it does work really well, supports XMPP, OTR, MUC, Message archiving etc...

@sammcj lets-chat looks good, thanks for that!

While we're on the Node topic, there's also https://github.com/node-xmpp/node-xmpp-server


Default port not set for MariaDB

When the URL for MariaDB doesn't contain the port we don't set the default value.
I also noticed that the build get stuck if a test fails during the setup.
I have a fix almost ready.

🙏


Add Podman + Wasm + GPU section

Most parts/steps are duplicate with the Docker version, so this is a simplified version of Podman, providing only the necessary instructions and minimum version requirements.
Explanation
Add podman execution version.
Related issue
None, internal request.
What type of PR is this
Documentation.
Proposed Changes
Add other container manager clients support is more user-friendly, especially since support for CDI toolchain is not yet fully mature. This also records a tested and runnable combination for end user.

Hello, I am a code review bot on flows.network.
It could take a few minutes for me to analyze this PR. Relax, grab a cup of coffee and check back later. Thanks!

We should have a complete guide, not a simplified one.

Ok, I will add those part.


@conda-forge-admin, please add bot automerge

@conda-forge-admin, please add bot automerge

Hi! This is the friendly automated conda-forge-webservice.
I just wanted to let you know that I added bot automerge in conda-forge/docker-feedstock#7.


Add Framed::into_parts_and_codec as temporary solution for #53

Needs to be removed once codec becomes a part of FramedParts.  As discussed in #53.

Thanks!

@alexcrichton Any chance we could see a new tokio-io release with this soon? tokio-cdp depends on it and currently has to use a patch entry in its Cargo.toml.

Sure thing, done!

Thanks very much!


Add test for file change events.

Followup to rakudo/rakudo#3100

I think this is to be merged when the related ticket is resolved.


pathological naming schemes can create clashing structs

Problem
suppose you have a struct endpoints in the schema with properties:

metrics_relabelings  (generating EndpointsMetricsRelabelings struct)
metrics (generating EndpointsMetrics struct)

suppose further that the metrics property in the schema has further properties:

relabelings (generating EndpointsMetricsRelabelings struct - a second time!)

I.e. it is possible to have multiple ways to reach the same struct names, and kopium does not handle this (it will emit multiple structs with the same name).
Potential workarounds

use smaller modules (mod Endpoints { mod Metrics {... to create a namespace for each struct level
naming disambiguation at insert time (we have the full list of OutputStruct available when adding another)

Leaning towards the latter, even though it will likely just be us suffixing numbers (EndpointsMetricsRelabelings2) to the struct and hoping the users do manual disambiguation where possible.

no one has actually complained about this happening yet, but if it does happen, it can be solved by us inside the output module by creating an Output wide analogue of Container::rename: https://github.com/kube-rs/kopium/blob/561adaea444e94689eaeb342af6629c7c0a4dbad/src/output.rs#L83-L128


ffmpeg: switch to the built-in RTMP implementation

Motivation for this change
See #142260 (cc @sersorrel)
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

With under a 100 rebuilds we can also target master here.
@ofborg eval
@ofborg build


🐛 intelliJ plugin 1.0.0 doesn't lint and output the CLI help when formatting

Environment information
CLI:
  Version:                      1.6.1
  Color support:                true

Platform:
  CPU Architecture:             x86_64
  OS:                           linux

Environment:
  BIOME_LOG_DIR:                unset
  NO_COLOR:                     unset
  TERM:                         "xterm-256color"
  JS_RUNTIME_VERSION:           "v20.10.0"
  JS_RUNTIME_NAME:              "node"
  NODE_PACKAGE_MANAGER:         "npm/10.2.5"

Biome Configuration:
  Status:                       Loaded successfully
  Formatter disabled:           false
  Linter disabled:              false
  Organize imports disabled:    false
  VCS disabled:                 true

Workspace:
  Open Documents:               0

What happened?
First, Using intelliJ on Windows with a project loaded via WSL or directly Windows, the Biome plugin seems to load successfully (LSP server says it is running) but it reports no linting issues (I saw them on VSCode for the same file)
Secondly, when I "format on save", it pastes the CLI output on the file. We can see that the formatting is done first, but then replaced by the help CLI output.
Expected result
lint and format properly ;-)
Code of Conduct

[X] I agree to follow Biome's Code of Conduct

Looks like the v1.0.2 fixed the issue. We'll test a bit more and I close the ticket if everything is fine.

lineWidth from the config is not taking into account (if not the whole config file?) We set it explicitly in IntelliJ.

java.lang.Throwable: Failed to run Biome check with (format, safefixes) for file Whatever.tsx - \wsl.localhost\Ubuntu-22.04\home[..]/rome.json internalError/fs ━━━━━━━━━━━
× Biome couldn't read the file
× Biome couldn't read the following file, maybe for permissions reasons or it doesn't exists: \wsl.localhost\Ubuntu-22.04\home[..]/rome.json
PS: [..] being folder path here of the project (where package.json is).

Also reordering of import doesn't seem to work neither with 1.0.2

java.lang.Throwable: Failed to run Biome check with (format, safefixes) for file Whatever.tsx - \wsl.localhost\Ubuntu-22.04\home[..]/rome.json internalError/fs ━━━━━━━━━━━
× Biome couldn't read the file
× Biome couldn't read the following file, maybe for permissions reasons or it doesn't exists: \wsl.localhost\Ubuntu-22.04\home[..]/rome.json
PS: [..] being folder path here of the project (where package.json is).

I reopened the config and the link the biome.json path where changed to the folder of the app (where package.json is)
We have a workspace config if it may be the issue to test.

Would you mind updating Biome to the latest version (v1.7.0)?

I just did now, it's well detected regarding the status bar, but it's not better for the formatting.
Formatting also often times out btw.


dep: pin anstyle to 1.65 compat

Motivation
anstyle at 1.0.3 has a 1.70 MSRV
Solution
Pin dep to compatible version
PR Checklist

[ ] Added Tests
[ ] Added Documentation
[ ] Updated the changelog

closed in favor of #320


feat: upgrade to Next.js 15 and Nextra 3

Closes: #85, #84, #25
Upgrades the docs template to Next.js v15 and Nextra v3.

If you can provide me with the Vercel logs I can take a closer look at whatever is failing in the deployment. The dev server is successful locally and the docs site builds with pnpm build.

hey @sean-perkins , great work. Just came across your updates. For me locally using your sean-perkins:main branch gave me the following error:
❯ pnpm build

><EMAIL_ADDRESS>build /Users/johndoe/developer/nextra-docs-template
> next build

(node:20762) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
(Use `node --trace-warnings ...` to show where the warning was created)
 ⨯ Failed to load next.config.js, see more info here https://nextjs.org/docs/messages/next-config-error

> Build error occurred
SyntaxError: Cannot use import statement outside a module
    at <unknown> (next.config.js:1)
 ELIFECYCLE  Command failed with exit code 1.

Checking the docs for nextra and then renaming next.config.js -> next.config.mjs fixed the dev and build. Maybe that will also fix the vercel build.

node --version: v20.18.1
pnpm --version: 9.15.0

@mavogel thanks for the suggestion! Not sure how I overlooked that warning in the terminal.
Enabling ESModules seems to have fixed the preview deployment.


sensors mapping

Hi, good project. I have a question. There is a way to map an IFC model with external sensors. I explain, i need to define sensors inseide ifc model, load it in viewer and then map sensors in ifc model with real sensors (es. raspberry) in build. How can i do this?
Thanks a lot!

Exactly the same way it is done here.
If you look into https://github.com/cyrillef/Pier9.IoT/blob/master/www/data/poi-reduced.json
you'll notice there is a dbId
"sensors": {
"id": "",
"dbid": 3184,
which represent the ID of the object associated the sensor in the scene. RCT, IDC, or any other format do not make any difference.


[Feature Request] VK_EXT_descriptor_indexing example?

Hello,
I wanted to first thank you for all the hard work you have put into this repo. It has been instrumental in my Vulkan learning experience!
I was hoping I could request an example using the VK_EXT_descriptor_indexing extension. I've been looking all over the web for anyone that had example code of using it, but couldn't seem to find any.
This extension is probably the most useful one I've seen since it allows you to bind all your textures at once as a unsized array, then simply index into whatever texture you need. Maybe you could convert ArrayTextures example to use descriptor indexing?
Thanks again!

Sure, been thinking about doing a sample for that but haven't had the time do so. I'll try to whip up something.

Closing this for now. If you still want an example for this, feel free to open up an issue at the Khronos Vulkan Samples Repository.


Fixed agent config and logging

@sbaogang @jinruh

Not to pick on this specific diff but can we please add more detail to the diff on what the problem being solved is and how this diff fixes it? I think this is a good practice for a few reasons:
a) This is good code development and git hygine. That way looking at the history it would be clear what has changed and why we changed it.
b) A random observer like me would have no context on weather this change is important when I look at the history or the code.
c) Now that we are open source, including this info would also inform external reviewers why this change was made.

Per discussion, we will treat this as an exception - once we open sourced, we will follow a more strict process.

This is an urgent fix. I'll provide more context next time.


19042.789 please!

19042.789 please!

Please upload the file termsrv.dll

termsrv.zip
This is termsrv.dll from C:/Windows/System32. Thanks!

https://github.com/anhkgg/SuperRDP/releases/tag/v1.0.3


docs: rename slinky to connect in our docs [BLO-1456]

In This PR

I rename all references of slinky to connect in our docs.
Any references to the binary / code / commands are unchanged

Overall the biggest thing we should figure out is aligning on what we're changing to what: So far isolated mentions of Slinky → Connect are all good. The biggest question comes from how we're changing side car mentions. There are some mentions of Slinky Sidecar changing to "Connect Instance", others to "Connect", and others to "Connect Sidecar"
I definitely trust however you want to do it, just want to make sure we're aligned across the docs.

Discussed offline, we'll merge this, and handle the semantics between connect, connect sidecar and connect instance later


Intermittent test failures in ManagedLedgerErrorsTest.recoverLongTimeAfterMultipleWriteErrors

Tests run: 219, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 81.903 sec <<< FAILURE! - in TestSuite
recoverLongTimeAfterMultipleWriteErrors(org.apache.bookkeeper.mledger.impl.ManagedLedgerErrorsTest)  Time elapsed: 0.014 sec  <<< FAILURE!
java.lang.AssertionError: expected [1] but found [2]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerErrorsTest.recoverLongTimeAfterMultipleWriteErrors(ManagedLedgerErrorsTest.java:470)
Results :
Failed tests:
  ManagedLedgerErrorsTest.recoverLongTimeAfterMultipleWriteErrors:470 expected [1] but found [2]

I repro'd on a checkout from when this was reported, and it looks like a legit bug and potentially serious.
However, I haven't been able to get a direct repro on master, though it looks like the bug still exists.
The bug occurs as follows:

A managed ledger, with current ledger as 3
The managed ledger attempts to write two entries to 3
First entry write to 3 fails, 3 is closed, deleted, and replaced by 5 in the managed ledger, the writes are retried
Second entry write fails, 3 is closed, deleted from the ledger list (it's not there) and ledger 6 is created and added to the ledger list.

The solution should be to make sure the ledgerClosed() checks whether the passed in ledger is the current ledger and only try to add a new one if so. However, I would like a reliable test for this first, but we'll need to modify the mocks first.
(cc @merlimat )

Closed by #2573

It becomes a flaky test again.
Example failure
https://github.com/apache/pulsar/actions/runs/4329269822/jobs/7559697736
Exception stacktrace
Error:  recoverLongTimeAfterMultipleWriteErrors(org.apache.bookkeeper.mledger.impl.ManagedLedgerErrorsTest)  Time elapsed: 0.025 s  <<< FAILURE!
java.lang.AssertionError: expected [null] but found [org.apache.bookkeeper.mledger.ManagedLedgerException$ManagedLedgerAlreadyClosedException: Waiting to recover from failure]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotSame(Assert.java:1033)
	at org.testng.Assert.assertNull(Assert.java:965)
	at org.testng.Assert.assertNull(Assert.java:953)
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerErrorsTest.recoverLongTimeAfterMultipleWriteErrors(ManagedLedgerErrorsTest.java:541)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

Replace with a new issue so that we have better visibility https://github.com/apache/pulsar/issues/20364.


feat: deprecate http record handlers

closing #121

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 80.921%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
4780

Relevant Lines:
5907

💛  - Coveralls


Party movement no longer syncing

The party movement on both host and client are not in sync. I think this is due to a state machine issue.

This bug was caused by #75. Unpausing fixed issue. Closing


Warning: React does not recognize the tinaForm prop on a DOM element

Description
When using TinaCMS with either Gatsby or Next.js, I am seeing the following error:

Steps to reproduce
I followed both the Gatsby and Next.js tutorials.
Expected result
No error should occur.
Actual result
The error occurred, though things seem to work fine still. The error seems to be more of a warning.
Environment
Tinacms Package Versions

├─┬<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├─┬<EMAIL_ADDRESS>├─┬<EMAIL_ADDRESS>│ ├──<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>│ ├──<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>│ ├──<EMAIL_ADDRESS>

Tools

Tool
Version

browser
Chrome 79.0.3945.130 (Official Build) (64-bit)

os
macOS Version 10.15.1 (Build 19B88

node
12.14.1

npm
6.13.4

yarn
1.21.1

@all-contributors please add @chaddjohnson for bugs

@ncphillips Thanks! Nice of you.

Reproducable on the demo-next index page.

The issue is the tinaForm prop in the fields builder. If we change tinaForm to tinaform, the warning goes away but I'd have to look further into where else we would need to make changes if we refactored this prop name.


Careers 2.0: Add formatting options for dates

Adds formatting option for dates when they are displayed as text rather than headers (recommended by user testing on Careers pages & currently only applied in that section.) This PR is based on @anselmbradford's work in #3240 & review suggestions
Additions

strftime wrapper to allow additional patterns (custom month abbreviations, day values without leading zeros)
date filter that adds option of formatting dates for text contexts in addition to the default header context

Testing

Check out branch
Navigate to careers landing page, current openings page, and a job opening page and verify the date formatting
Verify that other instances of dates on the site (like on the Blog page) retain their existing styles

Screenshots
Careers landing page

Current openings page

Job opening page

Existing date format

Checklist

[ ] PR has an informative and human-readable title
[ ] Changes are limited to a single goal (no scope creep)
[ ] Code can be automatically merged (no conflicts)
[ ] Code follows the standards laid out in the development playbook
[ ] Passes all existing automated tests
[ ] Any change in functionality is tested
[ ] New functions are documented (with a description, list of inputs, and expected output)
[ ] Placeholder code is flagged
[ ] Visually tested in supported browsers and devices
[ ] Project documentation has been updated
[ ] Reviewers requested with the Reviewer tool :arrow_right:

@willbarton I think all the ordering/spacing issues should be fixed in https://github.com/cfpb/cfgov-refresh/pull/3680/commits/7d6beec5178c53e43a43d38c743ab11b2f736f91 -- thanks!

@virginiacc thank you!

Now that this is in, what should I do in https://github.com/cfpb/cfgov-refresh/pull/3680? Should the changes here be applied to other date occurrences?

@anselmbradford That's a good question! We restricted the new formatting to the Careers section in this PR because we had specific recommendations about that use case. Do you think we should create an issue for discussion about applying it more generally?


CMYK issues

This doesn't bother me very much, but if I ever get into a situation where I need CMYK to be used with my favorite colorPicker of all time, well....
Here's the thing: I set an input
<input class="colorpicker" data-colormode="cmyk" type="text" value="cmyk(15,25,30,40)">

var pickers = new jsColorPicker('input.colorpicker', {});

Issues:

my data-colormode attribute gets changed to cmy, weird
Everytime I click my input, the color gets brighter and changes all values in all color modes instead of just opening the current colorpicker instance.

Hi @thednp The issue is not related with a bug but with wrong expectations. The initiator of colorPicker accepts color formats that are supported by the browser which are RGB, rgb, rgba, hsl and hsla. RGBA is also supported nowerdays, but I guess colorPicker doesn't (I have to look it up).
You might want to write an input that takes cmyc and convert it to rgb and feed a hidden input with the value so the hidden input triggers the colorPicker but it shows values from the cmyc input field.
If you wrap both inside a <label> tag, the click event will be transferred to the hidden input and the colorPicker should show.
Hope this helped.
Greetings.

Thanks for taking the time to reply.
I'm not using the colorpicker to use it's CMYK value as color for web elements, I know it's not supported, but I use it as a way for people to input a color in any format they have, perhaps some folk has come from print with CMYK values and wants to use those colors into web apps, converted to supported formats like rgb/hex or hsl in the near future.
I suggest you take a look, it won't take much.


Modules, routing - lazy loaded modules with webpack

There is no explanation in documentation that you have to use angular2-router-loader in order to use lazy loaded modules configured in routing with webpack.
This should be at least on webpack page. I think link could be also on modules page and FAQ.
Are there any more requirements to get lazy loaded modules working with webpack?

I will document that eventually. We need some other pieces to go first.


Prestwich/replica enroll

Solution
Foundry script to output gov actions to re-enroll replicas on a domain
Break down script into logic and entrypoint
PR Checklist

[ ] Added Tests
[ ] Updated Documentation
[x] Updated CHANGELOG.md for the appropriate package

I rebased onto main so we can get this merged - I want to inherit the logic in forks


Clarification regarding Googlebot

I've read https://github.com/GoogleChrome/rendertron/pull/258#issuecomment-453104073, but my understanding matches that of @barroudjo (see https://github.com/GoogleChrome/rendertron/pull/258#issuecomment-534944916). As far as I'm aware, JavaScript indexing occurs in a "second wave", and that can be delayed by a non-trivial amount of time. So, by default, content will not be indexed as fast as it could be if it was pre-rendered. This doesn't seem ideal?
What is the benefit of not pre-rendering for Googlebot? What is a "possibly richer version"? Aren't pages supposed to include the same data?

There's not really a second wave with a significant delay in this way, AFAIK. All pages go through the renderer, and the median time for that is apparently around the 5 second mark See Martin's Twitter Comment.
So IMHO in this respect, prerendering for googlebot doesn't introduce benefit by default, just fragility by the nature of doing something different, caches not being cleared, additional infrastructure to debug / remember about when debugging, and potentially more overhead for you catering for it.
I guess there are circumstances where you might be doing something googlebot doesn't quite get along with, where rendertron might make sense, but I would imagine these cases are getting fewer and further apart.

@dwsmart,
Thanks for linking to that Twitter comment, that makes me feel better about leaving Googlebot out of the list.
It seems that my understanding (based on this YouTube video) is outdated.
Perhaps there could be an FAQ entry like "Why isn't Googlebot included in the default list of user agents?".
Otherwise, feel free to close this issue (anyone with the permission to do so). 🙂

It kind of blows my mind that Googlebot isn't included by default if that truly is the case now? I've been using Rendertron for approximately 2 years now and I've always had Googlebot in my middleware list. Furthermore I pre-render ALL my pages (and invalidate and re-render when content changes) so that when Googlebot finally hits me the json is already ready to be served in a flash, I don't wait for the search engines to require a page before I pre-render it. And I have ~50k pages, all pre-rendered on an unlimited filesystem cache
According to Google Search Console I'm doing swimmingly well, have tons of my content indexed and all is good. I should note that my content is all loaded through AJAX queries. All is great in my world right now.. it seems like removing Googlebot would be completely contrary to the purpose of Rendertron o_O

Googlebot renders JavaScript, so adding it to the bot list is not necessary unless your JavaScript causes problems in certain scenarios.
In general, as we're deprecating the project, you should look into alternative approaches to rendering on the web.


Taxonomy classification with different DB

Dear emu team,
I am currently using your pipeline to classify ONT fastq. I played a bit with the DB to compare emu-db and SILVA. I found several times, different results in term of number of classified taxa.
Here is an example of an output of the same sample:
from SILVA:
8903	0.3333333333333333	Bacteria;Planctomycetota;Phycisphaerae;Phycisphaerales;Phycisphaeraceae;SM1A02;	1.0
9107	0.3333333333333333	Bacteria;Proteobacteria;Alphaproteobacteria;Rhizobiales;Labraceae;Labrys;	1.0
285	0.3333333333333333	Bacteria;Actinobacteriota;Actinobacteria;Propionibacteriales;Nocardioidaceae;Nocardioides;	1.0
from EMU-db:
tax_id	abundance	species	genus	family	order	class	phylum	clade	superkingdom	subspecies	species subgroup	species group	estimated counts
443157	1.0	Nocardioides hankookensis	Nocardioides	Nocardioidaceae	Propionibacteriales	Actinobacteria	Actinobacteria	Terrabacteria group	Bacteria		3.0
unassigned	0.0
I can imagine that SILVA db is larger than emu-db and could identify more taxa but here I am not it would explain this.
Cheers,
Nico

The SILVA database is larger and also less curated. You can use --keep-files to see the minimap2 alignments between your reads and the database


Ensure UseNode Task satisfies npm requirement

Fix for UseNodeV1 does not satisfy npm demand like NodeToolV0 did #11012

This change is

Nit - build task so that task.loc.json gets updated as well. Otherwise, looks good, thanks for catching this!

@damccorm bumped task.loc.json version as well

Yeah, you should take 1.156.2. That will ensure it gets published - we only publish tasks that have new version numbers

@damccorm done :)


Ckeditor

This PR adds:

[x] ckeditor
[x] bug fix for model filter function
[x] model getQuery and getObject functions with the ability to overwrite them in local model

@smebberson This is ready for review!

Hey @sandytrinh, this looks good. When we finish up the plugin integration, we'll rip this stuff out to keep Linz core nice and lean.

@sandytrinh, please merge!


JerseyClientConfiguration timeout not properly set on Client object from yaml

Hi all,
AFAICT, dropwizard isn't actually setting the timeout values for jersey clients when they're specified in a config file. Here's my quick example (CatConfiguration is cut down for brevity):
public class CatConfiguration extends Configuration {
  @Valid
  @NotNull
  @JsonProperty("jerseyClient")
  private JerseyClientConfiguration jerseyClientConfiguration;

  public JerseyClientConfiguration getJerseyClientConfiguration() {
    return jerseyClientConfiguration;
  }

  public void setJerseyClientConfiguration(JerseyClientConfiguration jcc) {
    this.jerseyClientConfiguration = jcc;
  }
}

Config file contains:
jerseyClient:
  timeout: 3s

And when I build it (I've tried this with and without the "using" method)
final Client jerseyClient = new JerseyClientBuilder(environment)
        .using(catCfg.getJerseyClientConfiguration()).build(getName());
System.out.println("Timeout:");
System.out.println(jerseyClient.getConfiguration().getProperty(ClientProperties.READ_TIMEOUT));
System.out.println(catCfg.getJerseyClientConfiguration().getTimeout());

This prints
Timeout:
null
3 seconds

So seems like it's read from the config just fine, but not actually set on the client object.
I fairly sure I haven't messed this up - is the timeout simply failing to be set? Right now I'm just manually setting the timeout as a temporary workaround. Let me know if I've done something wrong, otherwise hopefully this is an easy fix. Thanks!

We use our own Jersey Connector implementation, that's backed by the Apache HTTP Client. The timeout is actually is applied to it. See there.
Also our connectors ignores Jersey properties, so the call jerseyClient.getConfiguration().getProperty(ClientProperties.READ_TIMEOUT) will always return null.

Ah, ok. That said, I get read timeouts from something that takes more than 500ms to respond without my manual workaround, but I don't timeout with the workaround -- so something still seems fishy.

That's certainly strange. It would be great, if you could isolate your problem and provide a test case that reproduce it. AFAIK, we have a test for this case and it works as expected.
If you find a flaw in the code or in the test, please report.

Shot in the dark, do you also have to set the connectionTimeout?
jerseyClient:
  timeout: 3s
  connectionTimeout: 3s

Is this resolved by https://github.com/dropwizard/dropwizard/pull/1137?

I am closing this issue. Without a test case, we can't do much.
Feel free to open a new issue against Dropwizard 0.8.5/0.9.1, if the problem still exists


There's no way to get the tweet's full text with timelineOperations

only if it's length is shorter than 140 chars

can u give some examples

I'm presuming the OP is referring to these upcoming changes https://dev.twitter.com/overview/api/upcoming-changes-to-tweets


fix(MSH): add support for internal components

:tada: This PR is included in version 5.6.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


api: Use the console_scripts entry point to point to zulip-send.

Instead of using the scripts keyword, we now use the
console_scripts entry point to point to the zulip-send script
to be installed. This is what the Python Packaging User Guide
recommends for better cross-platform compatibility.

Merged, thanks @eeshangarg!


feat: capitalize locale names

Some translated locale names were not capitalized, for some reason.
This PR capitalizes ensures that all the values of the JSON files under data folder are properly capitalized.
It is recommended to just review the generate.ts file (the first commit), since all the other files are just the output of the modified yarn generate command.

Hi, thank you for your PR.
Have you also considered submitting your changes to the official ICU module (https://github.com/unicode-org/icu), which is the source of the data for this repo?
I noticed that for my language, the correct spelling should be written with a lowercase character instead of the updated uppercase character. Specifically, čeština is correct, while Čeština is incorrect. It's worth noting that country names in Czech language should still begin with a capital letter.
Therefore I am not sure whether I should merge it, as it only corrects a few languages while potentially producing incorrect translations for others.
What do you think?

@ihmpavel Didn't know about that in other languages! Everyday you learn something new :), thank you very much for the feedback!
And yes, I guess I should contribute there instead. I'd need to carefully find how, since the repo seems to be a source of a lot of other localization libraries.
I'm closing this, hopefully some day I can report back!

Sure, looking forward to your report someday! 🙌