Synthesis failed for nodejs-redis

Hello! Autosynth couldn't regenerate nodejs-redis. :broken_heart:
Here's the output from running synth.py:
f25494a4219a73928
2020-12-11 21:38:08,864 autosynth [DEBUG] > Running: git log -1 --pretty=%at 3559dd8cb634347737632776ef4089443eacdb4f
2020-12-11 21:38:08,867 autosynth [DEBUG] > Running: git log -1 --pretty=%at 7d1ee3207371ef72a1ca931a0964c4007e687280
2020-12-11 21:38:08,870 autosynth [DEBUG] > Running: git log -1 --pretty=%at 352f25621c28d2fa0784984ea0ed5ea697aa1525
2020-12-11 21:38:08,873 autosynth [DEBUG] > Running: git log -1 --pretty=%at 996775eca5fd934edac3c2ae34b80ff0395b1717
2020-12-11 21:38:08,876 autosynth [DEBUG] > Running: git checkout 78d14cd0b9ba66c8319ec540ea050606b7ec7a6c
Note: checking out '78d14cd0b9ba66c8319ec540ea050606b7ec7a6c'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 78d14cd chore: generate GAPIC metadata JSON file (#452)
2020-12-11 21:38:08,880 autosynth [DEBUG] > Running: git checkout 996775eca5fd934edac3c2ae34b80ff0395b1717
Note: checking out '996775eca5fd934edac3c2ae34b80ff0395b1717'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 996775e fix(python): skip samples readmegen in synthtool (#875)
2020-12-11 21:38:08,885 autosynth [DEBUG] > Running: git checkout dd372aa22ded7a8ba6f0e03a80e06358a3fa0907
Note: checking out 'dd372aa22ded7a8ba6f0e03a80e06358a3fa0907'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at dd372aa2 fix: remove client recv msg limit fix: add enums to `types/__init__.py`
2020-12-11 21:38:08,902 autosynth [DEBUG] > Running: git branch -f autosynth-60
2020-12-11 21:38:08,905 autosynth [DEBUG] > Running: git checkout autosynth-60
Switched to branch 'autosynth-60'
2020-12-11 21:38:08,910 autosynth [INFO] > Running synthtool
2020-12-11 21:38:08,910 autosynth [INFO] > ['/usr/local/bin/python', '-m', 'synthtool', '--metadata', 'synth.metadata', 'synth.py', '--']
2020-12-11 21:38:08,910 autosynth [DEBUG] > log_file_path: /kokoro/artifacts/logs/nodejs-redis/60/sponge_log.log
2020-12-11 21:38:08,912 autosynth [DEBUG] > Running: /usr/local/bin/python -m synthtool --metadata synth.metadata synth.py --
2020-12-11 21:38:09,119 synthtool [DEBUG] > Executing /root/.cache/synthtool/nodejs-redis/synth.py.
On branch autosynth-60
nothing to commit, working tree clean
2020-12-11 21:38:09,249 synthtool [DEBUG] > Ensuring dependencies.
DEBUG:synthtool:Ensuring dependencies.
2020-12-11 21:38:09,259 synthtool [DEBUG] > Cloning googleapis.
DEBUG:synthtool:Cloning googleapis.
2020-12-11 21:38:09,260 synthtool [DEBUG] > Using precloned repo /root/.cache/synthtool/googleapis
DEBUG:synthtool:Using precloned repo /root/.cache/synthtool/googleapis
2020-12-11 21:38:09,264 synthtool [DEBUG] > Generating code for: //google/cloud/redis/v1:redis-v1-nodejs.
DEBUG:synthtool:Generating code for: //google/cloud/redis/v1:redis-v1-nodejs.
2020-12-11 21:38:14,490 synthtool [SUCCESS] > Generated code into /tmp/tmpm6y7qzr_.
SUCCESS:synthtool:Generated code into /tmp/tmpm6y7qzr_.
2020-12-11 21:38:14,516 synthtool [DEBUG] > Generating code for: //google/cloud/redis/v1beta1:redis-v1beta1-nodejs.
DEBUG:synthtool:Generating code for: //google/cloud/redis/v1beta1:redis-v1beta1-nodejs.
2020-12-11 21:38:20,500 synthtool [SUCCESS] > Generated code into /tmp/tmpj6yj6m9m.
SUCCESS:synthtool:Generated code into /tmp/tmpj6yj6m9m.
2020-12-11 21:38:20,526 synthtool [DEBUG] > Using precloned repo /root/.cache/synthtool/synthtool
DEBUG:synthtool:Using precloned repo /root/.cache/synthtool/synthtool
2020-12-11 21:38:20,543 synthtool [INFO] > successfully generate `src/index.ts`
INFO:synthtool:successfully generate `src/index.ts`
.eslintignore
.eslintrc.json
.gitattributes
.github/ISSUE_TEMPLATE/bug_report.md
.github/ISSUE_TEMPLATE/feature_request.md
.github/ISSUE_TEMPLATE/support_request.md
.github/PULL_REQUEST_TEMPLATE.md
.github/release-please.yml
.github/workflows/ci.yaml
.kokoro/.gitattributes
.kokoro/common.cfg
.kokoro/continuous/node10/common.cfg
.kokoro/continuous/node10/docs.cfg
.kokoro/continuous/node10/test.cfg
.kokoro/continuous/node12/common.cfg
.kokoro/continuous/node12/lint.cfg
.kokoro/continuous/node12/samples-test.cfg
.kokoro/continuous/node12/system-test.cfg
.kokoro/continuous/node12/test.cfg
.kokoro/docs.sh
.kokoro/lint.sh
.kokoro/populate-secrets.sh
.kokoro/presubmit/node10/common.cfg
.kokoro/presubmit/node12/common.cfg
.kokoro/presubmit/node12/samples-test.cfg
.kokoro/presubmit/node12/system-test.cfg
.kokoro/presubmit/node12/test.cfg
.kokoro/publish.sh
.kokoro/release/docs-devsite.cfg
.kokoro/release/docs-devsite.sh
.kokoro/release/docs.cfg
.kokoro/release/docs.sh
.kokoro/release/publish.cfg
.kokoro/samples-test.sh
.kokoro/system-test.sh
.kokoro/test.bat
.kokoro/test.sh
.kokoro/trampoline.sh
.kokoro/trampoline_v2.sh
.mocharc.js
.nycrc
.prettierignore
.prettierrc.js
.trampolinerc
CODE_OF_CONDUCT.md
CONTRIBUTING.md
LICENSE
README.md
api-extractor.json
renovate.json
samples/README.md
2020-12-11 21:38:20,693 synthtool [WARNING] > No replacements made in src/v*/*_client.ts for pattern /compute/docs/, maybe replacement is no longer needed?
WARNING:synthtool:No replacements made in src/v*/*_client.ts for pattern /compute/docs/, maybe replacement is no longer needed?
2020-12-11 21:38:20,693 synthtool [DEBUG] > Post-processing GAPIC library...
DEBUG:synthtool:Post-processing GAPIC library...
2020-12-11 21:38:20,693 synthtool [DEBUG] > Installing dependencies...
DEBUG:synthtool:Installing dependencies...
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/synthtool/synthtool/__main__.py", line 102, in <module>
    main()
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/synthtool/synthtool/__main__.py", line 94, in main
    spec.loader.exec_module(synth_module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/root/.cache/synthtool/nodejs-redis/synth.py", line 31, in <module>
    node.postprocess_gapic_library()
  File "/synthtool/synthtool/languages/node.py", line 193, in postprocess_gapic_library
    install(hide_output=hide_output)
  File "/synthtool/synthtool/languages/node.py", line 167, in install
    shell.run(["npm", "install"], hide_output=hide_output)
  File "/synthtool/synthtool/shell.py", line 33, in run
    encoding="utf-8",
  File "/usr/local/lib/python3.6/subprocess.py", line 423, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/local/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'npm': 'npm'
2020-12-11 21:38:20,757 autosynth [ERROR] > Synthesis failed
2020-12-11 21:38:20,758 autosynth [DEBUG] > Running: git reset --hard HEAD
HEAD is now at 78d14cd chore: generate GAPIC metadata JSON file (#452)
2020-12-11 21:38:20,770 autosynth [DEBUG] > Running: git checkout autosynth
Switched to branch 'autosynth'
2020-12-11 21:38:20,778 autosynth [DEBUG] > Running: git clean -fdx
Removing __pycache__/
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/synthtool/autosynth/synth.py", line 354, in <module>
    main()
  File "/synthtool/autosynth/synth.py", line 189, in main
    return _inner_main(temp_dir)
  File "/synthtool/autosynth/synth.py", line 334, in _inner_main
    commit_count = synthesize_loop(x, multiple_prs, change_pusher, synthesizer)
  File "/synthtool/autosynth/synth.py", line 65, in synthesize_loop
    has_changes = toolbox.synthesize_version_in_new_branch(synthesizer, youngest)
  File "/synthtool/autosynth/synth_toolbox.py", line 259, in synthesize_version_in_new_branch
    synthesizer.synthesize(synth_log_path, self.environ)
  File "/synthtool/autosynth/synthesizer.py", line 120, in synthesize
    synth_proc.check_returncode()  # Raise an exception.
  File "/usr/local/lib/python3.6/subprocess.py", line 389, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-m', 'synthtool', '--metadata', 'synth.metadata', 'synth.py', '--']' returned non-zero exit status 1.

Google internal developers can see the full log here.

Autosynth is still having trouble generating nodejs-redis. :sob:
Here's the output from running synth.py:
f25494a4219a73928
2020-12-11 21:51:50,469 autosynth [DEBUG] > Running: git log -1 --pretty=%at 3559dd8cb634347737632776ef4089443eacdb4f
2020-12-11 21:51:50,472 autosynth [DEBUG] > Running: git log -1 --pretty=%at 7d1ee3207371ef72a1ca931a0964c4007e687280
2020-12-11 21:51:50,475 autosynth [DEBUG] > Running: git log -1 --pretty=%at 352f25621c28d2fa0784984ea0ed5ea697aa1525
2020-12-11 21:51:50,478 autosynth [DEBUG] > Running: git log -1 --pretty=%at 996775eca5fd934edac3c2ae34b80ff0395b1717
2020-12-11 21:51:50,481 autosynth [DEBUG] > Running: git checkout 78d14cd0b9ba66c8319ec540ea050606b7ec7a6c
Note: checking out '78d14cd0b9ba66c8319ec540ea050606b7ec7a6c'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 78d14cd chore: generate GAPIC metadata JSON file (#452)
2020-12-11 21:51:50,500 autosynth [DEBUG] > Running: git checkout 996775eca5fd934edac3c2ae34b80ff0395b1717
Note: checking out '996775eca5fd934edac3c2ae34b80ff0395b1717'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 996775e fix(python): skip samples readmegen in synthtool (#875)
2020-12-11 21:51:50,506 autosynth [DEBUG] > Running: git checkout dd372aa22ded7a8ba6f0e03a80e06358a3fa0907
Note: checking out 'dd372aa22ded7a8ba6f0e03a80e06358a3fa0907'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at dd372aa2 fix: remove client recv msg limit fix: add enums to `types/__init__.py`
2020-12-11 21:51:50,523 autosynth [DEBUG] > Running: git branch -f autosynth-60
2020-12-11 21:51:50,527 autosynth [DEBUG] > Running: git checkout autosynth-60
Switched to branch 'autosynth-60'
2020-12-11 21:51:50,531 autosynth [INFO] > Running synthtool
2020-12-11 21:51:50,531 autosynth [INFO] > ['/usr/local/bin/python', '-m', 'synthtool', '--metadata', 'synth.metadata', 'synth.py', '--']
2020-12-11 21:51:50,532 autosynth [DEBUG] > log_file_path: /kokoro/artifacts/logs/nodejs-redis/60/sponge_log.log
2020-12-11 21:51:50,533 autosynth [DEBUG] > Running: /usr/local/bin/python -m synthtool --metadata synth.metadata synth.py --
2020-12-11 21:51:50,740 synthtool [DEBUG] > Executing /root/.cache/synthtool/nodejs-redis/synth.py.
On branch autosynth-60
nothing to commit, working tree clean
2020-12-11 21:51:50,865 synthtool [DEBUG] > Ensuring dependencies.
DEBUG:synthtool:Ensuring dependencies.
2020-12-11 21:51:50,875 synthtool [DEBUG] > Cloning googleapis.
DEBUG:synthtool:Cloning googleapis.
2020-12-11 21:51:50,876 synthtool [DEBUG] > Using precloned repo /root/.cache/synthtool/googleapis
DEBUG:synthtool:Using precloned repo /root/.cache/synthtool/googleapis
2020-12-11 21:51:50,880 synthtool [DEBUG] > Generating code for: //google/cloud/redis/v1:redis-v1-nodejs.
DEBUG:synthtool:Generating code for: //google/cloud/redis/v1:redis-v1-nodejs.
2020-12-11 21:51:56,105 synthtool [SUCCESS] > Generated code into /tmp/tmpklvu5fsy.
SUCCESS:synthtool:Generated code into /tmp/tmpklvu5fsy.
2020-12-11 21:51:56,132 synthtool [DEBUG] > Generating code for: //google/cloud/redis/v1beta1:redis-v1beta1-nodejs.
DEBUG:synthtool:Generating code for: //google/cloud/redis/v1beta1:redis-v1beta1-nodejs.
2020-12-11 21:52:02,167 synthtool [SUCCESS] > Generated code into /tmp/tmpl1eqfyf2.
SUCCESS:synthtool:Generated code into /tmp/tmpl1eqfyf2.
2020-12-11 21:52:02,196 synthtool [DEBUG] > Using precloned repo /root/.cache/synthtool/synthtool
DEBUG:synthtool:Using precloned repo /root/.cache/synthtool/synthtool
2020-12-11 21:52:02,212 synthtool [INFO] > successfully generate `src/index.ts`
INFO:synthtool:successfully generate `src/index.ts`
.eslintignore
.eslintrc.json
.gitattributes
.github/ISSUE_TEMPLATE/bug_report.md
.github/ISSUE_TEMPLATE/feature_request.md
.github/ISSUE_TEMPLATE/support_request.md
.github/PULL_REQUEST_TEMPLATE.md
.github/release-please.yml
.github/workflows/ci.yaml
.kokoro/.gitattributes
.kokoro/common.cfg
.kokoro/continuous/node10/common.cfg
.kokoro/continuous/node10/docs.cfg
.kokoro/continuous/node10/test.cfg
.kokoro/continuous/node12/common.cfg
.kokoro/continuous/node12/lint.cfg
.kokoro/continuous/node12/samples-test.cfg
.kokoro/continuous/node12/system-test.cfg
.kokoro/continuous/node12/test.cfg
.kokoro/docs.sh
.kokoro/lint.sh
.kokoro/populate-secrets.sh
.kokoro/presubmit/node10/common.cfg
.kokoro/presubmit/node12/common.cfg
.kokoro/presubmit/node12/samples-test.cfg
.kokoro/presubmit/node12/system-test.cfg
.kokoro/presubmit/node12/test.cfg
.kokoro/publish.sh
.kokoro/release/docs-devsite.cfg
.kokoro/release/docs-devsite.sh
.kokoro/release/docs.cfg
.kokoro/release/docs.sh
.kokoro/release/publish.cfg
.kokoro/samples-test.sh
.kokoro/system-test.sh
.kokoro/test.bat
.kokoro/test.sh
.kokoro/trampoline.sh
.kokoro/trampoline_v2.sh
.mocharc.js
.nycrc
.prettierignore
.prettierrc.js
.trampolinerc
CODE_OF_CONDUCT.md
CONTRIBUTING.md
LICENSE
README.md
api-extractor.json
renovate.json
samples/README.md
2020-12-11 21:52:02,357 synthtool [WARNING] > No replacements made in src/v*/*_client.ts for pattern /compute/docs/, maybe replacement is no longer needed?
WARNING:synthtool:No replacements made in src/v*/*_client.ts for pattern /compute/docs/, maybe replacement is no longer needed?
2020-12-11 21:52:02,357 synthtool [DEBUG] > Post-processing GAPIC library...
DEBUG:synthtool:Post-processing GAPIC library...
2020-12-11 21:52:02,357 synthtool [DEBUG] > Installing dependencies...
DEBUG:synthtool:Installing dependencies...
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/synthtool/synthtool/__main__.py", line 102, in <module>
    main()
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/synthtool/synthtool/__main__.py", line 94, in main
    spec.loader.exec_module(synth_module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/root/.cache/synthtool/nodejs-redis/synth.py", line 31, in <module>
    node.postprocess_gapic_library()
  File "/synthtool/synthtool/languages/node.py", line 193, in postprocess_gapic_library
    install(hide_output=hide_output)
  File "/synthtool/synthtool/languages/node.py", line 167, in install
    shell.run(["npm", "install"], hide_output=hide_output)
  File "/synthtool/synthtool/shell.py", line 33, in run
    encoding="utf-8",
  File "/usr/local/lib/python3.6/subprocess.py", line 423, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/local/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'npm': 'npm'
2020-12-11 21:52:02,415 autosynth [ERROR] > Synthesis failed
2020-12-11 21:52:02,415 autosynth [DEBUG] > Running: git reset --hard HEAD
HEAD is now at 78d14cd chore: generate GAPIC metadata JSON file (#452)
2020-12-11 21:52:02,427 autosynth [DEBUG] > Running: git checkout autosynth
Switched to branch 'autosynth'
2020-12-11 21:52:02,434 autosynth [DEBUG] > Running: git clean -fdx
Removing __pycache__/
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/synthtool/autosynth/synth.py", line 354, in <module>
    main()
  File "/synthtool/autosynth/synth.py", line 189, in main
    return _inner_main(temp_dir)
  File "/synthtool/autosynth/synth.py", line 334, in _inner_main
    commit_count = synthesize_loop(x, multiple_prs, change_pusher, synthesizer)
  File "/synthtool/autosynth/synth.py", line 65, in synthesize_loop
    has_changes = toolbox.synthesize_version_in_new_branch(synthesizer, youngest)
  File "/synthtool/autosynth/synth_toolbox.py", line 259, in synthesize_version_in_new_branch
    synthesizer.synthesize(synth_log_path, self.environ)
  File "/synthtool/autosynth/synthesizer.py", line 120, in synthesize
    synth_proc.check_returncode()  # Raise an exception.
  File "/usr/local/lib/python3.6/subprocess.py", line 389, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-m', 'synthtool', '--metadata', 'synth.metadata', 'synth.py', '--']' returned non-zero exit status 1.

Google internal developers can see the full log here.

Autosynth is still having trouble generating nodejs-redis. :sob:
Here's the output from running synth.py:

.github/ISSUE_TEMPLATE/support_request.md
.github/PULL_REQUEST_TEMPLATE.md
.github/release-please.yml
.github/workflows/ci.yaml
.kokoro/.gitattributes
.kokoro/common.cfg
.kokoro/continuous/node10/common.cfg
.kokoro/continuous/node10/docs.cfg
.kokoro/continuous/node10/test.cfg
.kokoro/continuous/node12/common.cfg
.kokoro/continuous/node12/lint.cfg
.kokoro/continuous/node12/samples-test.cfg
.kokoro/continuous/node12/system-test.cfg
.kokoro/continuous/node12/test.cfg
.kokoro/docs.sh
.kokoro/lint.sh
.kokoro/populate-secrets.sh
.kokoro/presubmit/node10/common.cfg
.kokoro/presubmit/node12/common.cfg
.kokoro/presubmit/node12/samples-test.cfg
.kokoro/presubmit/node12/system-test.cfg
.kokoro/presubmit/node12/test.cfg
.kokoro/publish.sh
.kokoro/release/docs-devsite.cfg
.kokoro/release/docs-devsite.sh
.kokoro/release/docs.cfg
.kokoro/release/docs.sh
.kokoro/release/publish.cfg
.kokoro/samples-test.sh
.kokoro/system-test.sh
.kokoro/test.bat
.kokoro/test.sh
.kokoro/trampoline.sh
.kokoro/trampoline_v2.sh
.mocharc.js
.nycrc
.prettierignore
.prettierrc.js
.trampolinerc
CODE_OF_CONDUCT.md
CONTRIBUTING.md
LICENSE
README.md
api-extractor.json
renovate.json
samples/README.md
2020-12-11 22:12:42,860 synthtool [WARNING] > No replacements made in src/v*/*_client.ts for pattern /compute/docs/, maybe replacement is no longer needed?
WARNING:synthtool:No replacements made in src/v*/*_client.ts for pattern /compute/docs/, maybe replacement is no longer needed?
2020-12-11 22:12:42,860 synthtool [DEBUG] > Post-processing GAPIC library...
DEBUG:synthtool:Post-processing GAPIC library...
2020-12-11 22:12:42,860 synthtool [DEBUG] > Installing dependencies...
DEBUG:synthtool:Installing dependencies...
npm WARN deprecated<EMAIL_ADDRESS>Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)

><EMAIL_ADDRESS>postinstall /root/.cache/synthtool/nodejs-redis/node_modules/protobufjs
> node scripts/postinstall

npm WARN lifecycle<EMAIL_ADDRESS>cannot run in wd<EMAIL_ADDRESS>npm run compile-protos && npm run compile (wd=/root/.cache/synthtool/nodejs-redis)
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY<EMAIL_ADDRESS>(node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for<EMAIL_ADDRESS>wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

added 687 packages from 781 contributors and audited 689 packages in 16.052s

58 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

2020-12-11 22:12:59,654 synthtool [DEBUG] > Running prelint...
DEBUG:synthtool:Running prelint...

><EMAIL_ADDRESS>prelint /root/.cache/synthtool/nodejs-redis
> cd samples; npm link ../; npm install

npm WARN lifecycle<EMAIL_ADDRESS>cannot run in wd<EMAIL_ADDRESS>npm run compile-protos && npm run compile (wd=/root/.cache/synthtool/nodejs-redis)
npm WARN optional SKIPPING OPTIONAL DEPENDENCY<EMAIL_ADDRESS>(node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for<EMAIL_ADDRESS>wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

audited 689 packages in 3.904s

58 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

/root/.nvm/versions/node/v14.15.1/lib/node_modules/@google-cloud/redis -> /root/.cache/synthtool/nodejs-redis
/root/.cache/synthtool/nodejs-redis/samples/node_modules/@google-cloud/redis -> /root/.nvm/versions/node/v14.15.1/lib/node_modules/@google-cloud/redis -> /root/.cache/synthtool/nodejs-redis
npm WARN deprecated<EMAIL_ADDRESS>Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY<EMAIL_ADDRESS>(node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for<EMAIL_ADDRESS>wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

added 131 packages from 98 contributors and audited 219 packages in 4.701s

58 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

2020-12-11 22:13:10,559 synthtool [DEBUG] > Running fix...
DEBUG:synthtool:Running fix...

><EMAIL_ADDRESS>fix /root/.cache/synthtool/nodejs-redis
> gts fix

version: 14

/root/.cache/synthtool/nodejs-redis/samples/quickstart.js
  24:38  error  "@google-cloud/redis" is not found  node/no-missing-require

✖ 1 problem (1 error, 0 warnings)

Error: Command failed with exit code 1: node ./node_modules/eslint/bin/eslint --fix **/*.ts **/*.js **/*.tsx **/*.jsx --no-error-on-unmatched-pattern
    at makeError (/root/.cache/synthtool/nodejs-redis/node_modules/execa/lib/error.js:59:11)
    at handlePromise (/root/.cache/synthtool/nodejs-redis/node_modules/execa/index.js:114:26)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async run (/root/.cache/synthtool/nodejs-redis/node_modules/gts/build/src/cli.js:120:17) {
  shortMessage: 'Command failed with exit code 1: node ./node_modules/eslint/bin/eslint --fix **/*.ts **/*.js **/*.tsx **/*.jsx --no-error-on-unmatched-pattern',
  command: 'node ./node_modules/eslint/bin/eslint --fix **/*.ts **/*.js **/*.tsx **/*.jsx --no-error-on-unmatched-pattern',
  exitCode: 1,
  signal: undefined,
  signalDescription: undefined,
  stdout: undefined,
  stderr: undefined,
  failed: true,
  timedOut: false,
  isCanceled: false,
  killed: false
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>fix: `gts fix`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>fix script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2020-12-11T22_13_26_282Z-debug.log
2020-12-11 22:13:26,639 synthtool [ERROR] > Failed executing npm run fix:

None
ERROR:synthtool:Failed executing npm run fix:

None
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/synthtool/synthtool/__main__.py", line 102, in <module>
    main()
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/synthtool/synthtool/__main__.py", line 94, in main
    spec.loader.exec_module(synth_module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/root/.cache/synthtool/nodejs-redis/synth.py", line 31, in <module>
    node.postprocess_gapic_library()
  File "/synthtool/synthtool/languages/node.py", line 194, in postprocess_gapic_library
    fix(hide_output=hide_output)
  File "/synthtool/synthtool/languages/node.py", line 179, in fix
    shell.run(["npm", "run", "fix"], hide_output=hide_output)
  File "/synthtool/synthtool/shell.py", line 39, in run
    raise exc
  File "/synthtool/synthtool/shell.py", line 33, in run
    encoding="utf-8",
  File "/usr/local/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['npm', 'run', 'fix']' returned non-zero exit status 1.
2020-12-11 22:13:26,718 autosynth [ERROR] > Synthesis failed
2020-12-11 22:13:26,718 autosynth [DEBUG] > Running: git reset --hard HEAD
HEAD is now at 78d14cd chore: generate GAPIC metadata JSON file (#452)
2020-12-11 22:13:26,745 autosynth [DEBUG] > Running: git checkout autosynth
Switched to branch 'autosynth'
2020-12-11 22:13:26,764 autosynth [DEBUG] > Running: git clean -fdx
Removing __pycache__/
Removing node_modules/
Removing package-lock.json
Removing samples/node_modules/
Removing samples/package-lock.json
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/synthtool/autosynth/synth.py", line 354, in <module>
    main()
  File "/synthtool/autosynth/synth.py", line 189, in main
    return _inner_main(temp_dir)
  File "/synthtool/autosynth/synth.py", line 334, in _inner_main
    commit_count = synthesize_loop(x, multiple_prs, change_pusher, synthesizer)
  File "/synthtool/autosynth/synth.py", line 65, in synthesize_loop
    has_changes = toolbox.synthesize_version_in_new_branch(synthesizer, youngest)
  File "/synthtool/autosynth/synth_toolbox.py", line 259, in synthesize_version_in_new_branch
    synthesizer.synthesize(synth_log_path, self.environ)
  File "/synthtool/autosynth/synthesizer.py", line 120, in synthesize
    synth_proc.check_returncode()  # Raise an exception.
  File "/usr/local/lib/python3.6/subprocess.py", line 389, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-m', 'synthtool', '--metadata', 'synth.metadata', 'synth.py', '--']' returned non-zero exit status 1.

Google internal developers can see the full log here.


Test::Unit missing with Ruby 2.2.0

Hi,
I was trying to update a Rails app (4.1.6) to Ruby 2.2.0 but Test::Unit is missing, so I get this :
LoadError: cannot load such file -- test/unit/assertions
/Users/jlecour/.gem/ruby/2.2.0/gems/activesupport-4.1.6/lib/active_support/dependencies.rb:247:in `require'
/Users/jlecour/.gem/ruby/2.2.0/gems/activesupport-4.1.6/lib/active_support/dependencies.rb:247:in `block in require'
/Users/jlecour/.gem/ruby/2.2.0/gems/activesupport-4.1.6/lib/active_support/dependencies.rb:232:in `load_dependency'
/Users/jlecour/.gem/ruby/2.2.0/gems/activesupport-4.1.6/lib/active_support/dependencies.rb:247:in `require'
/Users/jlecour/.gem/ruby/2.2.0/bundler/gems/activewarehouse-etl-98a9d7a87ccb/lib/etl/control/control.rb:5:in `<class:Context>'

I don't really understand how Test::Unit is used here and how it coud be replaced by Minitest (present in Ruby 2.2.0).

In fact, in Ruby 2.2.0, both Test::Unit and Minitest have been removed from the standard library (cf. https://www.ruby-lang.org/en/news/2014/12/25/ruby-2-2-0-released/).
Adding gem 'test-unit' in the application's Gemfile is enough to make it work again.
I guess adding a proper dependency in the gem spec would be better.

@jlecour good catch! I've not tested aw-etl on "modern" configurations recently. I added you to the member teams so you can push the change directly if you will. Thanks!

We've been using a fork of AW/ETL for some time, but without formal tests. I'm not yet confident to push our changes back in the main repository.
Here is the diff from where we are to the master we're based on : https://github.com/Autrement/activewarehouse-etl/compare/master...rails4
If you want to take a look, we could try to properly validate and merge this in.
Thanks

Well at that point I'm not maintaining aw-etl anymore, just casually responding to requests. I don't have the bandwidth to review / validate / merge (but honestly I'd definitely trust you to do a good job here if you want to go ahead).

test-unit-minitest gem includes the removed libraries


bundix: 2.2.0

Motivation for this change
Major improvements for stability and speed.
Things done

[x] Tested using sandboxing
(nix.useChroot on NixOS,
or option build-use-chroot in nix.conf
on non-NixOS)
Built on platform(s)

[x] NixOS
[x] OS X
[ ] Linux

[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Fits CONTRIBUTING.md.

I propose we revert this until https://github.com/manveru/bundix/pull/7 is resolved


Adds Kafka image and TRANSPORT_TYPE to select it with

This adds an image for Kafka/Zookeeper, which can be used to test kafka
integration of zipkin with instrumentation including Finagle, Brave and
HTrace. This exports a new variable to designate the TRANSPORT_TYPE,
with choices of scribe and kafka.
At the moment, there's no Finagle Kafka integraion. I tested this with
an experimental Brave span transport, used in the zipkin-java project.
The image bundles together Kafka and Zookeeper using Runit as the entry-
point. This is for simplicitly and to keep the requirements as small as
possible to run tests (ex. we are already several containers). The
approach used is similar to baseimage-docker.
See https://github.com/openzipkin/brave/issues/102
See https://github.com/openzipkin/zipkin/issues/682
See http://phusion.github.io/baseimage-docker

relevant snippet from my patched docker-compose file in the zipkin-java project
kafka:
  image: openzipkin/zipkin-kafka:1.17.2
  ports:
    - 2181:2181
    - 9092:9092
collector:
  image: openzipkin/zipkin-collector:1.17.2
  environment:
    - TRANSPORT_TYPE=kafka
    - STORAGE_TYPE=mysql
  expose:
    - 9410
  ports:
    - 9410:9410
    - 9900:9900
  links:
    - kafka
    - mysql:storage
query:
  build: zipkin-java-server
  environment:
    - TRANSPORT_TYPE=kafka
    - STORAGE_TYPE=mysql
  expose:
    - 9411
  ports:
    - 9411:9411
  links:
    - kafka
    - mysql:storage

Looks great. docker-zipkin is shaping up to be a nice runnable blueprint collection of possible zipkin setups. :heart: :+1:

published and verified the public docker images.


Evaluate our use of lib/intercom.js, lib/eventemitter.js

The intercom.js project has been dead for a while.  We've got a small version of it in lib/intercom.js, which we use to create filesystem watches across windows.
I suspect there are more modern ways to do this, which we can either build or use via a dep.  Let's see if we can update this.

I worked on this a bit tonight, trying to remove https://github.com/filerjs/filer/blob/master/lib/eventemitter.js and instead use a bundled/node.js EventEmitter via Parcel.  I got stuck because it requires surgery to intercom.js.  Probably doing both as part of the same bug makes sense.

https://github.com/pubkey/broadcast-channel might be a better solution here.

BroadcastChannels are available in Chrome and Firefox, but not in Edge or Safari (yet), see https://caniuse.com/#feat=broadcastchannel.
We'd have to polyfill for a while to make this work.  I think that's probably OK, since the polyfill does exactly what intercom is doing now: use localStorage events.


Replacing Uninformative Variables from GPfit

I've been working with GPfit recently, and it returns constraints that look like:
gpkit.PosynomialInequality(w_fit**3.5 >= 1.38e+41*u_fit_(0,)**-0.15*u_fit_(1,)**-9.5 + 1.62e-07*u_fit_(0,)**0.0064*u_fit_(1,)**-0.16)

It's possible that this exists, but it would be really nice if there were a property of a constraint, such as cosntraint.replace() that could allow me to do something like:
from gpkit import Variable

x = Variable("x", "-")
y = Variable("y", "-")
z = Variable("z", "-")

const = x >= y

const.replace(x,z)
print const

and have it return:
z >= y

IE, the variable x has simply been replaced in the constraint by the variable z.  Something of this sort would be nice so I can replace u_1 with C_L, u_2 with Re, w with C_D etc...  And since GPfit returns vector variables, that would be great too in addition to the scalar example I've got here.

Does subinplace({"w": "x"}) work here?
On Jan 5, 2017 3:07 PM, "cjk12"<EMAIL_ADDRESS>wrote:

I've been working with GPfit recently, and it returns constraints that
look like:
gpkit.PosynomialInequality(w_fit3.5 >= 1.38e+41*u_fit_(0,)-0.15u_fit_(1,)**-9.5 + 1.62e-07u_fit_(0,)0.0064*u_fit_(1,)-0.16)
It's possible that this exists, but it would be really nice if there were
a property of a constraint, such as cosntraint.replace() that could allow
me to do something like:
from gpkit import Variable
x = Variable("x", "-")
y = Variable("y", "-")
z = Variable("z", "-")
const = x >= y
const.replace(x,z)print const
and have it return:
z >= y
IE, the variable x has simply been replaced in the constraint by the
variable z. Something of this sort would be nice so I can replace u_1
with C_L, u_2 with Re, w with C_D etc... And since GPfit returns vector
variables, that would be great too in addition to the scalar example I've
got here.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/hoburg/gpkit/issues/1028, or mute the thread
https://github.com/notifications/unsubscribe-auth/ABagGAqqCk33h2sWgq8lW3mJ4levUDZKks5rPXfFgaJpZM4LcOxp
.

This has some strange behavior:
import unittest
from numpy import log, exp, log10, vstack
from gpfit.fit import fit
from numpy.random import random_sample

Vdd = random_sample(1000,) + 1
Vth = 0.2*random_sample(1000,) + 0.2
P = Vdd**2 + 30*Vdd*exp(-(Vth-0.06*Vdd)/0.039)
u = vstack((Vdd,Vth))
x = log(u)
y = log(P)
K = 4

cstrt, rmsErr, a, b, c = fit(x,y,K,"ISMA")

from gpkit import Variable, Model
Re = Variable("Re", "-", "Reynold's number")
cstrt.subinplace({"w_fit":"Re"})
print cstrt

0.0128*Re_fit**-0.66*u_fit_(0,)**2.6*u_fit_(1,)**-2.6 + 0.308*Re_fit**-0.31*u_fit_(0,)**0.64*u_fit_(1,)**-0.82 + 0.891*Re_fit**-0.25*u_fit_(0,)**0.5*u_fit_(1,)**-0.093 + 1.01*Re_fit**-0.076*u_fit_(0,)**0.15*u_fit_(1,)**0.012 <= 1

Strings values will just rename: try cstrt.subinplace({"w_fit": Re}), if
that doesn't work we should fix that, but for now cstrt.subinplace({"w_fit":
Re.key}) should work.

This appears to work:
import unittest
from numpy import log, exp, log10, vstack
from gpfit.fit import fit
from numpy.random import random_sample

Vdd = random_sample(1000,) + 1
Vth = 0.2*random_sample(1000,) + 0.2
P = Vdd**2 + 30*Vdd*exp(-(Vth-0.06*Vdd)/0.039)
u = vstack((Vdd,Vth))
x = log(u)
y = log(P)
K = 4

cstrt, rmsErr, a, b, c = fit(x,y,K,"ISMA")

from gpkit import Variable, Model
Re = Variable("Re", "-", "Reynold's number")
cstrt.subinplace({"w_fit":Re})
cstrt

Thanks @bqpd

Actually, is this any different for vector variables?  For example, if I have u with two elements, how do I replace the first one with variable x and the second element with variable y?

.sub({"u": [x, y]}) seems to work just fine.
Note that if you have an older version of gpfit, u is not a vector variable.

Awesome, thanks.  Just saved me a lot of time.
On Fri, Jan 6, 2017 at 8:55 PM bqpd<EMAIL_ADDRESS>wrote:

Closed #1028 https://github.com/hoburg/gpkit/issues/1028.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/hoburg/gpkit/issues/1028#event-914820955, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AIO1rcDRQ9rPa_DFLzxhhAwzMYqSYACOks5rPvCVgaJpZM4LcOxp
.

Great!
(note that new gpfit only returns cstrt, rmsErr from fit)

Yea I hacked that.  Thanks for the heads up.
On Fri, Jan 6, 2017 at 9:11 PM bqpd<EMAIL_ADDRESS>wrote:

Great!
(note that new gpfit only returns cstrt, rmsErr from fit)
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/hoburg/gpkit/issues/1028#issuecomment-271055532, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AIO1rTMiTK2Lgr02OE9JNCDDwqfAT3Onks5rPvRbgaJpZM4LcOxp
.


Limit rows printed to avoid console locking up during print

:j for large journals may print thousands of lines, taking a long time. Add a limit (say, 1000 rows), and a flag to optionally turn this limit off.

All done.


add favicon

Adds a favicon so it's easier to visually find the right tab quickly.
Favicon takes the default roamresearch.com favicon and adds a `TOOLS' label over it.

Wrong MIME Type. I've created a new PR https://github.com/ryanguill/roam-tools/pull/10


[DOC]Disabling hybridization steps added

Steps to Disable hybridization added!
As the document previously did not mention the steps to disable hybridization i.e using net.hybridize(active=False), that has been added in the commit.

Thanks for the guidance @szha

Hello @szha, I wanted to know if this PR is merged? If not then are there any other changes that need to be done?


[p4] Adding table with both exacts and optionals in table key.

PR Description :
[p4] Adding table with both exacts and optionals in table key.
Build Results :
bibhuprasad@eaba4ee0062c:/sonic/src/sonic-p4rt/sonic-pins$ bazel build $BAZEL_BUILD_OPTS p4_pdpi/...
INFO: Analyzed 66 targets (0 packages loaded, 0 targets configured).
INFO: Found 66 targets...
INFO: Elapsed time: 25.761s, Critical Path: 8.00s
INFO: 39 processes: 7 internal, 32 linux-sandbox.
INFO: Build completed successfully, 39 total actions
bibhuprasad@eaba4ee0062c:/sonic/src/sonic-p4rt/sonic-pins$
bibhuprasad@eaba4ee0062c:/sonic/src/sonic-p4rt/sonic-pins$ bazel build $BAZEL_BUILD_OPTS p4rt_app/...
INFO: Analyzed 100 targets (0 packages loaded, 0 targets configured).
INFO: Found 100 targets...
INFO: Elapsed time: 0.859s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
bibhuprasad@eaba4ee0062c:/sonic/src/sonic-p4rt/sonic-pins$
UT Results :
bibhuprasad@eaba4ee0062c:/sonic/src/sonic-p4rt/sonic-pins$ bazel build $BAZEL_BUILD_OPTS p4rt_app/...
INFO: Analyzed 100 targets (0 packages loaded, 0 targets configured).
INFO: Found 100 targets...
INFO: Elapsed time: 0.859s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
bibhuprasad@eaba4ee0062c:/sonic/src/sonic-p4rt/sonic-pins$
bibhuprasad@eaba4ee0062c:/sonic/src/sonic-p4rt/sonic-pins$
bibhuprasad@eaba4ee0062c:/sonic/src/sonic-p4rt/sonic-pins$
bibhuprasad@eaba4ee0062c:/sonic/src/sonic-p4rt/sonic-pins$
bibhuprasad@eaba4ee0062c:/sonic/src/sonic-p4rt/sonic-pins$
bibhuprasad@eaba4ee0062c:/sonic/src/sonic-p4rt/sonic-pins$
bibhuprasad@eaba4ee0062c:/sonic/src/sonic-p4rt/sonic-pins$ bazel test $BAZEL_BUILD_OPTS p4_pdpi/...
INFO: Analyzed 66 targets (0 packages loaded, 0 targets configured).
INFO: Found 39 targets and 27 test targets...
INFO: Elapsed time: 1.587s, Critical Path: 0.45s
INFO: 15 processes: 25 linux-sandbox.
INFO: Build completed successfully, 15 total actions
//p4_pdpi/netaddr:ipv4_address_and_network_address_test         (cached) PASSED in 0.2s
//p4_pdpi/netaddr:ipv6_address_test                             (cached) PASSED in 0.2s
//p4_pdpi/netaddr:mac_address_test                              (cached) PASSED in 0.2s
//p4_pdpi/string_encodings:bit_string_test                      (cached) PASSED in 0.2s
//p4_pdpi/string_encodings:byte_string_test                     (cached) PASSED in 0.3s
//p4_pdpi/string_encodings:decimal_string_test                  (cached) PASSED in 0.0s
//p4_pdpi/string_encodings:decimal_string_test_runner           (cached) PASSED in 0.0s
//p4_pdpi/string_encodings:hex_string_test                      (cached) PASSED in 0.1s
//p4_pdpi/string_encodings:hex_string_test_runner               (cached) PASSED in 0.0s
//p4_pdpi/string_encodings:readable_byte_string_test            (cached) PASSED in 0.2s
//p4_pdpi/testing:mock_p4_runtime_server_test                   (cached) PASSED in 0.3s
//p4_pdpi/utils:annotation_parser_test                          (cached) PASSED in 0.2s
//p4_pdpi/utils:ir_test                                         (cached) PASSED in 0.3s
//p4_pdpi:ir_tools_test                                                  PASSED in 0.4s
//p4_pdpi/testing:helper_function_test                                   PASSED in 0.3s
//p4_pdpi/testing:info_test                                              PASSED in 0.1s
//p4_pdpi/testing:info_test_runner                                       PASSED in 0.1s
//p4_pdpi/testing:main_pd_test                                           PASSED in 0.0s
//p4_pdpi/testing:packet_io_test                                         PASSED in 0.1s
//p4_pdpi/testing:packet_io_test_runner                                  PASSED in 0.0s
//p4_pdpi/testing:rpc_test                                               PASSED in 0.1s
//p4_pdpi/testing:rpc_test_runner                                        PASSED in 0.1s
//p4_pdpi/testing:sequencing_test                                        PASSED in 0.1s
//p4_pdpi/testing:sequencing_test_runner                                 PASSED in 0.1s
//p4_pdpi/testing:table_entry_gunit_test                                 PASSED in 0.4s
//p4_pdpi/testing:table_entry_test                                       PASSED in 0.1s
//p4_pdpi/testing:table_entry_test_runner                                PASSED in 0.1s
Executed 14 out of 27 tests: 27 tests pass.
INFO: Build completed successfully, 15 total actions
bibhuprasad@eaba4ee0062c:/sonic/src/sonic-p4rt/sonic-pins$ bazel test $BAZEL_BUILD_OPTS p4rt_app/...
INFO: Analyzed 100 targets (0 packages loaded, 0 targets configured).
INFO: Found 65 targets and 35 test targets...
INFO: Elapsed time: 0.389s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
//p4rt_app/event_monitoring:app_state_db_port_table_event_test  (cached) PASSED in 0.6s
//p4rt_app/event_monitoring:config_db_node_cfg_table_event_test (cached) PASSED in 0.8s
//p4rt_app/event_monitoring:config_db_port_table_event_test     (cached) PASSED in 0.6s
//p4rt_app/event_monitoring:state_verification_events_test      (cached) PASSED in 0.6s
//p4rt_app/p4runtime:ir_translation_test                        (cached) PASSED in 0.5s
//p4rt_app/p4runtime:p4info_verification_schema_test            (cached) PASSED in 0.4s
//p4rt_app/p4runtime:p4info_verification_test                   (cached) PASSED in 0.5s
//p4rt_app/p4runtime:packetio_helpers_test                      (cached) PASSED in 0.5s
//p4rt_app/sonic:app_db_acl_def_table_manager_test              (cached) PASSED in 0.4s
//p4rt_app/sonic:app_db_manager_test                            (cached) PASSED in 0.4s
//p4rt_app/sonic:app_db_to_pdpi_ir_translator_test              (cached) PASSED in 0.4s
//p4rt_app/sonic:packetio_impl_test                             (cached) PASSED in 0.3s
//p4rt_app/sonic:packetio_port_test                             (cached) PASSED in 0.3s
//p4rt_app/sonic:response_handler_test                          (cached) PASSED in 0.4s
//p4rt_app/sonic:state_verification_test                        (cached) PASSED in 0.2s
//p4rt_app/sonic:vrf_entry_translation_test                     (cached) PASSED in 0.4s
//p4rt_app/sonic/adapters:fake_sonic_db_table_test              (cached) PASSED in 0.1s
//p4rt_app/tests:acl_table_test                                 (cached) PASSED in 0.9s
//p4rt_app/tests:action_set_test                                (cached) PASSED in 0.7s
//p4rt_app/tests:api_access_test                                (cached) PASSED in 0.7s
//p4rt_app/tests:arbitration_test                               (cached) PASSED in 0.8s
//p4rt_app/tests:fixed_l3_tables_test                           (cached) PASSED in 2.2s
//p4rt_app/tests:forwarding_pipeline_config_test                (cached) PASSED in 3.2s
//p4rt_app/tests:grpc_behavior_test                             (cached) PASSED in 5.1s
//p4rt_app/tests:p4_constraints_test                            (cached) PASSED in 0.1s
//p4rt_app/tests:p4_constraints_test_runner                     (cached) PASSED in 0.2s
//p4rt_app/tests:p4_programs_test                               (cached) PASSED in 1.0s
//p4rt_app/tests:packetio_test                                  (cached) PASSED in 3.3s
//p4rt_app/tests:port_name_and_id_test                          (cached) PASSED in 0.9s
//p4rt_app/tests:response_path_test                             (cached) PASSED in 1.7s
//p4rt_app/tests:role_test                                      (cached) PASSED in 0.7s
//p4rt_app/tests:state_verification_test                        (cached) PASSED in 1.0s
//p4rt_app/tests/lib:app_db_entry_builder_test                  (cached) PASSED in 0.0s
//p4rt_app/utils:event_data_tracker_test                        (cached) PASSED in 0.0s
//p4rt_app/utils:table_utility_test                             (cached) PASSED in 0.4s
Executed 0 out of 35 tests: 35 tests pass.
INFO: Build completed successfully, 1 total action
bibhuprasad@eaba4ee0062c:/sonic/src/sonic-p4rt/sonic-pins$

Co-authored-by: Srikishen Pondicherry Shanmugam<EMAIL_ADDRESS>

@kishanps , can you pl. review


Support automerge for Github PullRequest

Support automerge for Github PullRequest
Fix #488
This PR introduces an new parameter in Github PullRequest spec named "automerge".
When set to true, updatecli enables automerge to pullrequest that it creates.
Test
To test this pull request, you can run the following commands:
cp <to_package_directory>
go test

Additional Information
Tradeoff
At this stage, I missing the ability to define the merge method.
Potential improvement

I did some quick testing and it works but
The github API doesn't return a clear message when auto-merge doesn't meet requirements.
If auto-merge is not enabled at the repository level or if the PR is not in a blocked state due to a branch restriction, for instance, the API just returns "Pull request is not in the correct state to enable auto-merge"
Also, I am now wondering if we should also merge the PR even if the Github auto-merge feature is not available.

I am still missing the ability to set the merge type like rebase/squash/merge

I am still missing the ability to set the merge type like rebase/squash/merge

I think changing merge_method to squash or rebase in addition  to auto-merge should do the trick: https://docs.github.com/en/rest/reference/pulls#merge-a-pull-request

I did some quick testing and it works but The github API doesn't return a clear message when auto-merge doesn't meet requirements. If auto-merge is not enabled at the repository level or if the PR is not in a blocked state due to a branch restriction, for instance, the API just returns "Pull request is not in the correct state to enable auto-merge"

For this check we can retrieve the allow_squash_merge value of the repository before enabling auto-merge: https://docs.github.com/en/rest/reference/repos#get-a-repository

allow_auto_merge: Either true to allow auto-merge on pull requests, or false to disallow auto-merge. Default: false

Also, I am now wondering if we should also merge the PR even if the Github auto-merge feature is not available.

I'm not sure I understand what do you mean here, could you expand a little bit on this?

Also, I am now wondering if we should also merge the PR even if the Github auto-merge feature is not available.

Let's say a updatecli configuration contains automerge: true
Scenario 1: The repository allows auto-merge pullrequest as defined at the repository level. So we enable the flag "automerge" which is the goal of the current pullrequest
Scenario 2: The repository disallows automerge, so updatecli just merges the pullrequest without using the automerge feature. The usecase that I see is if a project use release-drafter to generate changelog based on pull request but doesn't care about reviewing pull requests generated by updatecli.

The repository disallows automerge, so updatecli just merges the pullrequest without using the automerge feature.

I wouldn't do that and bypass the repository security setting.
I would instead fail or warn in the output.

+1 with @lemeurherve . You are trying to solve multiple problems at once. The purpose of the PR here is to enable the "automerge" feature of GitHub pullrequests: let's keep that scope.

Scenario 2: The repository disallows automerge, so updatecli just merges the pullrequest without using the automerge feature.

=> In this case, I expect updatecli to have the PR opened BUT to have failed with an error message that tells the end user that automerge is not allowed on this repository.

I am still missing the ability to set the merge type like rebase/squash/merge

Do you mind proposing different YAML examples to show the different cases:

No automerge (current case)
Enable automerge (and let GitHub handle the default behavior)
Enable automerge with squash
Enable automerge with rebase
Enable automerge with merge

Are there other cases?

I am still missing the ability to set the merge type like rebase/squash/merge
Do you mind proposing different YAML examples to show the different cases

I wouldn't even bother, as it's in itself a new feature, independent of the automerge, the possibility to set the merge_method

I'm only asking to see examples "as a end user" :)

Agreed, for those with this new mergeMethod parameter on a separate PR though?

+1 with @lemeurherve . You are trying to solve multiple problems at once. The purpose of the PR here is to enable the "automerge" feature of GitHub pullrequests: let's keep that scope.

It depends who you see it hence my question :p
If you put "automerge: true" in an updatecli configuration.
Are you expecting to merge automatically the pullrequest or to enable "automerge" feature on the pullrequest. The latter case expect you to understand what the automerge flag from a GitHub point of vue.

Do you mind proposing different YAML examples to show the different cases:
While I don't think it's mandatory in this PR, I think it would be useful to identify how the configuration will evolve

I was thinking something like
pullrequests:
  helm-charts:
    kind: "github"
    scmID: "helm-charts"
    targets:
      - "helm-charts"
    spec:
      automerge: true
      mergemethod: squash # available options are sqash, merge,rebase
      labels:
        - "dependencies"
        - "epinio"

Agreed, for those with this new mergeMethod parameter on a separate PR though?

I agree but I would be interested to identify if it affects the "automerge" wording

Anyway the merge method will not be implemented in this PR because GitHub API v4 documentation warn that it's evolving
Preview notice
mergeStateStatus is available under the Merge info preview. During the preview period, the API may change without notice.

So I am mainly interested to identify blockers

Are you expecting to merge automatically the pullrequest or to enable "automerge" feature on the pullrequest.

No more than Github offer as native feature, the activation of the "auto-merge" github feature, no less no more.

+1 with @lemeurherve . You are trying to solve multiple problems at once. The purpose of the PR here is to enable the "automerge" feature of GitHub pullrequests: let's keep that scope.

It depends who you see it hence my question :p If you put "automerge: true" in an updatecli configuration. Are you expecting to merge automatically the pullrequest or to enable "automerge" feature on the pullrequest. The latter case expect you to understand what the automerge flag from a GitHub point of vue.

Do you mind proposing different YAML examples to show the different cases:
While I don't think it's mandatory in this PR, I think it would be useful to identify how the configuration will evolve

I was thinking something like
pullrequests:
  helm-charts:
    kind: "github"
    scmID: "helm-charts"
    targets:
      - "helm-charts"
    spec:
      automerge: true
      mergemethod: squash # available options are squash, merge,rebase
      labels:
        - "dependencies"
        - "epinio"

I expect the "automerge" of the "GitHub Pull Request" object in updatecli to map to the "automerge" feature of... GitHub Pull Requests.
Because in the alternativ scnearion (e.g. updatecli merges immediately after opening the PR) seems unrealistic or really really edgy and I'm not sur to see why updatecli should do it:

PR have branch protection quite often: that would be against the protections (and really hard to document
I don't really see the value of creating a PR to be merged compared to commit-pushing directly. I understand that it would mean release-drafter  cannot be used but there are so much tools to generate changelogs based on git commit messages that a PR in such cases looks really a hack
What about the PR checks? How can you manage updatecli to wait for the check to complete: this would be an "asynchronous" task while updatecli is built as a synchronous system from its core

I agree but I would be interested to identify if it affects the "automerge" wording

It's two distinct things IMO, you can set merge/squash/rebase independently of auto-merge. The combined github UI isn't mandatory

For the merge-method feature I would state its value in the output: "merge-method: merge"
And for this other feature I would just state "auto-merge: enabled"

Anyway the merge method will not be implemented in this PR because GitHub API v4 documentation warn that it's evolving

Not sure a lot would change, and we don't know when, should be trivial to adapt when needed, I would go for it anyway.
WDYT?

Not sure a lot would change, and we don't know when, should be trivial to adapt when needed, I would go for it anyway.

I could go with it I just have to find the correct API call

Because in the alternativ scnearion (e.g. updatecli merges immediately after opening the PR) seems unrealistic or really really edgy and I'm not sur to see why updatecli should do it:
* PR have branch protection quite often: that would be against the protections (and really hard to document

* I don't really see the value of creating a PR to be merged compared to commit-pushing directly. I understand that it would mean release-drafter  cannot be used but there are so much tools to generate changelogs based on git commit messages that a PR in such cases looks really a hack

* What about the PR checks? How can you manage updatecli to wait for the check to complete: this would be an "asynchronous" task while updatecli is built as a synchronous system from its core

I am fine relying on the GH automerge feature. it's more secure by default because you have to explicitly configure branch protection and less code for me now.

For the automerge key, I would go with automerge: true as it's boolean with default state to false

Thanks a lot @olblak for the brainstorming and the code ❤
Thanks a lot @lemeurherve for the ideas, feedbacks, proposal (and the issue for the automerge type ;) )

Thanks, everybody for the reviews. This PR is ready to be shipped as is, I am preparing a PR to update the document.
I propose to target 0.19 release. @lemeurherve @dduportal any objections?

No check of allow_auto_merge on the repo prior to the mutation ?

No check of allow_auto_merge on the repo prior to the mutation ?

subsequent PR that would fix the issue you created (or did I misundertood)?

The issue I've created concerns the merge_method, no relation to PR automerge nor repo allow_auto_merge

cc @olblak ^^

The issue I've created concerns the merge_method, no relation to PR automerge nor repo allow_auto_merge
For me this last check has to be done in order to validate or warn the user trying to set automerge on a PR

This requires significantly more because we need to get repository information to know  if "automerge" is allowed at the repository level and verify that branch protection exists on the targeted branch.
Imho it's a UX issue that should be tracked on a specific issue.
What do you think @lemeurherve

Don't we already have access to the repository via scms since this feature concerns only github?
Is it really significant more work and not just a verification call on the repo specified in values?

It could be added later as it's only silently failing, not that big of a deal, but it would be better to have this included by default.

I just want to point that if a part of a feature require significant work, then it means that we should restrain to merge it even if we are interested. I approved this pr so i am culprit also but we need to be careful in the future


Question: reproducible results

Hi,
Is there a way to fix a seed in order to get reproducible results?
Many thanks,
Ivan

Hi @ivan-marroquin, the results would be reproducible if using only one thread during the construction.

Hi @yurymalkov thanks for the clarification. Do you think that it is possible to ensure reproducible results when using several CPUs? It will be a great addition to a great package!
Thanks for all,
Ivan

@ivan-marroquin it is possible, but it is complicated. It also prevents incremental one-vector updates :you will likely need to process everything in batches. There is also no obvious benefit to this. If indexing parameters M and efConstruction are sufficiently large, usually what we see are very small variations in performance.

Hi @searchivarius , I will try using multiple batches. Since my input file has N >>> D (N: observations, D: features), I have to use M = 5 and efConstruction= 10 to deal with this large amount of data. Is it possible that I will get relative large variations?

Hi @ivan-marroquin it is not impossible. But it is not impossible that the method will not work well at all. I have seen some data sets where this happened. HNSW is relatively robust, but there are no guarantees. It is best to build the index and to test.
BTW, for large amounts of data, you can easily trade-off some efficiency for performance if you index in chunks. Say, index a chunk not larger than 10M records. Then, combine results. If you have say 1B records, it can be an order of magnitude in terms of indexing time, so you would be able to trade it off for increased accuracy (i.e., by setting larger  M and efConstruction).


Installs out-of-date python-craigslist (1.0.0 not 1.0.3)

Requirements.txt pulls in v1.0.0, which no longer works with craigslist due to changes made on their end. Running python-craigslist 1.0.3 fixes the problem.
This program will return 0 results and write nothing to the listings database until python-craigslist v1.0.3 is used.

Can you update the docker image to reflect this code change as well?
Thanks.


Develop

Coverage increased (+0.8%) to 82.883% when pulling d5247849a104a36a0a328aebcc011844f5e006b6 on develop into 7fea1338bc88e1a83b79a7e52157676e9b1a7e75 on qa.


Specify deprecated theme option

See https://github.com/pydata/pydata-sphinx-theme/issues/1492

That’s correct, I initially added that fix in a backport PR.


Http retries

Tearing my hair out over this one. Sometimes Chrome just seems not to completely start up, which makes my tests fail with a ResponseTimeout exception. It seems to have got more frequent with a recent Chrome update - almost every test run has at least one session which fails to start up correctly.
This is a sticking-plaster approach - just retry the request if it times out, and only give up if it fails 10 times in a row. PR opened for discussion in case you know of a better way to do this, rather than because I think it's a serious long-term fix.
Cheers,
David

Anything noteworthy from the Selenium server logs?

I'm going to mess around with some stuff to see if I can identify the cause. My initial guess is that the lower bound dependency for http-client needs to be bumped, based on issue #69 and the relevant fix at https://github.com/snoyberg/http-client/issues/116. While that issue is only reported on Windows, it appears to be similar and there may be other networking problems that will be resolved by enforcing newer versions of http-client. If you could identify which version of http-client you're using that would greatly help my search.
If tweaking the http-client dependency doesn't work, then the next best place to examine would be Selenium commits and issues.

Sure, here's some more details.
It's a hspec testsuite that uses hspec-webdriver to interface to this library, which then talks (via http-client) to a WebDriver server (we have tried 2.43.1, 2.45.0 and 2.46.0). This then talks to a ChromeDriver (we have tried 2.12, 2.15 and 2.16) which then talks to Chrome (currently 44.0.2403.155 but various other versions in the recent past too). All of the above are on the same machine.
The way we've got it set up at the moment involves using a different Chrome session per functional area, which is now in the region of 20. It is increasingly rare that all 20 of these manage to start successfully, causing the test failures we're seeing. There doesn't seem to be a pattern to the failures - it's different each run, and sometimes we've seen two or three fail. The failure is always a ResponseTimeout HttpException and always at session creation. Of course, the number of functional areas has also been increasing so the apparent creeping flakiness of the test suite could well be to do with that.
We've taken a tcpdump trace of a full run. Murphy's law applied, of course: the first such run didn't blooming well fail. When we did capture a failure, the effect we saw is a whole sequence of HTTP requests that receive no response: the test suite process POSTs to WebDriver to create a new session, which causes a POST to ChromeDriver which then does a GET on the debugging port of the Chrome instance it's just started. Eventually http-client gives up, although the other requests remained stuck until past the end of the run. Some of them were even merrily exchanging TCP keepalives.
I guessed that the upstream requests are blocked on the downstream one, so started looking at what that Chrome instance is playing at. It is indeed running, and a by-hand GET to the debugging port does indeed hang. It's running in xvfb so we can't see it directly, but a screenshot of the xvfb desktop was blank.
A bit of faffing around with strace and lsof sort-of hints that it's waiting for one of its child processes to do something, but the child process seems to be stuck in a loop calling epoll and immediately receiving ETIMEDOUT. I did have a half-hearted look at the Chrome source code but frankly lost the will to continue at around that point so gave up and stuck this retry loop in. The tests have been rock solid since then.
In short, I couldn't find any evidence that the Haskell libs were doing anything wrong - it looks like something weird going on at the other end of the pipeline, in Chrome itself.
A configurable retry count would certainly be nice. If you would like me to attempt a more refined PR along those lines then I can do so. Would you add the retry-count parameter to the Capabilities type?

Thanks for the detailed report. I might pass this along to the Selenium dev team as it sounds like something they might be interested in, or might have more information on.

In short, I couldn't find any evidence that the Haskell libs were doing anything wrong - it looks like something weird going on at the other end of the pipeline, in Chrome itself.

This is often the case in my experience. The implementation of the Haskell library is very straightforward so very few logical issues arise. I suppose much of that stability can be attributed to the nature of Haskell code. Almost all of the maintenance involves interfacing between these much larger dependencies which do the real heavy lifting, and keeping our API up to date with changes in those dependencies.

The way we've got it set up at the moment involves using a different Chrome session per functional area, which is now in the region of 20. It is increasingly rare that all 20 of these manage to start successfully, causing the test failures we're seeing.

So it sounds possible that ChromeDriver is breaking under the stress.  Perhaps there's a deadlock or race condition happening somewhere along the pipeline. I don't know for certain the thread-safety of our various dependencies and target servers. For example, I don't know what happens when you send multiple concurrent requests with http-client using different Managers, which would be the default arrangement if you make a runSession call for each individual thread.
If concurrency doesn't appear to be an issue, perhaps tweaking the test suite by adding delays or rate limiting would prevent the hang-up.

A configurable retry count would certainly be nice. If you would like me to attempt a more refined PR along those lines then I can do so. Would you add the retry-count parameter to the Capabilities type?

So far Capabilities is only used to implement capabilities as defined within the WebDriver spec, with hs-webdriver-specific configuration being specified within WDConfig and then translated into the actual session state stored/access by `WDSession'. This is mostly an artifact of the simplicity of the interface I initially created, being more-or-less a direct implementation of the spec without too much conceptual abstraction. It has the benefit of keeping the implementation very easy to maintain, since there's parity between the location of data within the configuration structs and how they're implemented. While not perfect, In the short term this is the preferred approach for implementing new configuration options.
As more client-side features have been added, the weakness of that approach has become apparent; it's not at all intuitive, I think, for a user of the library to specify session configuration in multiple locations. The solution I have in mind is to hide Capabilities from the simplest use cases, and instead use only WDConfig as a front-facing configuration interface. The WDConfig struct is then used to build Capabilities at session creation. A more advanced user could then access Capabilities to tweak settings, but it would not be mandatory for setting basic configuration such as browser choice. While not directly related to your problem, any feedback or suggestions on this proposed API would be helpful.
I would recommend keeping the number of retries at 1 by default, since I feel this is a feature that you would only want to use after you've discovered a problem. Retrying HTTP requests by default might hide undiscovered network issues if someone isn't looking carefully enough.

The tests are run sequentially, but there's no pause between the end of one session and the start of the next so it's possible it's something to do with one Chrome not having fully shut down before the next one starts up. They're all started with a different --user-data-dir argument so shouldn't be interacting, but maybe they are. Could be a race condition in the Chrome-to-Chrome interaction, failing to handle the exiting instance going away at just the wrong moment, perhaps?
I prefer retrying to pausing between sessions because if we paused we'd have to decide for how long (which may vary depending on other load on the test machine etc. etc). The retrying approach typically works first time, without a pause, and if it doesn't then the HTTP timeout is in effect a large pause.
I've pushed a commit that exposes my magical 10 as a parameter in WDConfig as you suggest, with the plumbing via WDSession. It preserves the existing behaviour by default by asking for zero retries (i.e. a single try).

Thanks!
Just spotted I left a hPutStrLn stderr in this PR, Oops. Suggest either removing it or else adding a webdriver: prefix so it's clear to others where that message is coming from.


Update raylib v4 5

Here is an rlgl module, a complete? 4.5 function list, and a ported rlgl example.  This code has been has been working with a game I have been porting to V, which uses a number of rlgl calls.  Only tested on Linux.

LGTM! Thanks so much for helping out this project, it's so nice to see :)


Remove unwrap() and fix clippy warnings

Description

This PR makes changes to the entire codebase based on running cargo clippy. Changes have been made where deemed appropriate. It also removes the use of unwrap() in favour of expect (with the exception of in mock/ test code)

Context & Background

The ticket to remove unwrap was brought up in TRN-211 from Red4Sec which is what this PR aims to address

Related Issues

https://futureverse.atlassian.net/browse/TRN-230?atlOrigin=eyJpIjoiNjkzZWM3YjU1ZmRjNDIzNGI3MWY0NmM5ZDVjMDEwMjQiLCJwIjoiaiJ9

Release Notes
Key Changes
Type of Change

[x] Runtime Changes
[ ] Client Changes

CI fix required

As part of this PR the CI should also be updated to enforce cargo clippy compliancy for the codebase.


Using the aider open large git repository

Issue
Here I use aider to open the Linux kernel repository, which is about 6GB size. The following error is displayed:
If it is because the git repository is too large, can compile_commands.json be used to reduce the index of the git repository source code?
$ aider
Traceback (most recent call last):
  File "/home/test/.local/bin/aider", line 8, in <module>
    sys.exit(main())
  File "/home/test/.local/lib/python3.10/site-packages/aider/main.py", line 414, in main
    coder.show_announcements()
  File "/home/test/.local/lib/python3.10/site-packages/aider/coders/base_coder.py", line 356, in show_announcements
    for line in self.get_announcements():
  File "/home/test/.local/lib/python3.10/site-packages/aider/coders/base_coder.py", line 159, in get_announcements
    num_files = len(self.repo.get_tracked_files())
  File "/home/test/.local/lib/python3.10/site-packages/aider/repo.py", line 213, in get_tracked_files
    for blob in commit.tree.traverse():
  File "/home/test/.local/lib/python3.10/site-packages/git/objects/util.py", line 575, in _traverse
    addToStack(stack, item, branch_first, nd)
  File "/home/test/.local/lib/python3.10/site-packages/git/objects/util.py", line 535, in addToStack
    lst = self._get_intermediate_items(item)
  File "/home/test/.local/lib/python3.10/site-packages/git/objects/tree.py", line 207, in _get_intermediate_items
    return tuple(index_object._iter_convert_to_object(index_object._cache))
  File "/home/test/.local/lib/python3.10/site-packages/gitdb/util.py", line 253, in __getattr__
    self._set_cache_(attr)
  File "/home/test/.local/lib/python3.10/site-packages/git/objects/tree.py", line 214, in _set_cache_
    self._cache: List[TreeCacheTup] = tree_entries_from_data(ostream.read())
  File "/home/test/.local/lib/python3.10/site-packages/gitdb/base.py", line 138, in read
    return self[3].read(size)
  File "/home/test/.local/lib/python3.10/site-packages/gitdb/stream.py", line 487, in read
    bl = self._size - self._br      # bytes left
  File "/home/test/.local/lib/python3.10/site-packages/gitdb/util.py", line 253, in __getattr__
    self._set_cache_(attr)
  File "/home/test/.local/lib/python3.10/site-packages/gitdb/stream.py", line 461, in _set_cache_brute_
    apply_delta_data(bbuf, src_size, ddata, len(ddata), tbuf.write)
  File "/home/test/.local/lib/python3.10/site-packages/gitdb/fun.py", line 673, in apply_delta_data
    raise ValueError("unexpected delta opcode 0")
ValueError: unexpected delta opcode 0

Version and model info
Aider v0.39.0

Thanks for trying aider and filing this issue.
This may be due to the index version of that repo. See #211 for more info.

Thanks for your reply.
My git version is 2.34.1, it is an unknown option --show-index-version, but the aider --no-git command is normally working.
After adding a file to the aider, let analysis a function, but another error appears below.
It looks like this file has too much content, can the file content be sent in multiple parts?
$ aider --no-git
Aider v0.39.0
Models: gpt-4o with diff edit format, weak model gpt-3.5-turbo
Git repo: none
Repo-map: disabled
Use /help to see in-chat commands, run with --help to see cmd line args
────────────────────────────────────────────────────────
> /add mm/vmscan.c

Added /home/test/workplaces/upstream/linux/mm/vmscan.c to the chat
────────────────────────────────────────────────────────
mm/vmscan.c
> analysis shrink_folio_list function
litellm.APIConnectionError: Error code: 429 - {'error': {'message': 'Request too large for gpt-4o in organization org-xxx on tokens per min (TPM): Limit 30000, Requested 54736. The input or output tokens must be red
uced in order to run successfully. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}
Traceback (most recent call last):
  File "/home/test/.local/lib/python3.10/site-packages/litellm/llms/openai.py", line 799, in completion
    raise e
  File "/home/test/.local/lib/python3.10/site-packages/litellm/llms/openai.py", line 718, in completion
    return self.streaming(
  File "/home/test/.local/lib/python3.10/site-packages/litellm/llms/openai.py", line 896, in streaming
    response = openai_client.chat.completions.create(**data, timeout=timeout)
  File "/home/test/.local/lib/python3.10/site-packages/openai/_utils/_utils.py", line 277, in wrapper
    return func(*args, **kwargs)
  File "/home/test/.local/lib/python3.10/site-packages/openai/resources/chat/completions.py", line 606, in create
    return self._post(
  File "/home/test/.local/lib/python3.10/site-packages/openai/_base_client.py", line 1240, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "/home/test/.local/lib/python3.10/site-packages/openai/_base_client.py", line 921, in request
    return self._request(
  File "/home/test/.local/lib/python3.10/site-packages/openai/_base_client.py", line 1005, in _request
    return self._retry_request(
  File "/home/test/.local/lib/python3.10/site-packages/openai/_base_client.py", line 1053, in _retry_request
    return self._request(
  File "/home/test/.local/lib/python3.10/site-packages/openai/_base_client.py", line 1005, in _request
    return self._retry_request(
  File "/home/test/.local/lib/python3.10/site-packages/openai/_base_client.py", line 1053, in _retry_request
    return self._request(
  File "/home/test/.local/lib/python3.10/site-packages/openai/_base_client.py", line 1020, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.RateLimitError: Error code: 429 - {'error': {'message': 'Request too large for gpt-4o in organization org-xxx on tokens per min (TPM): Limit 30000, Requested 54736. The input or output tokens must be reduced
in order to run successfully. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}

It looks like your OpenAI API key doesn't have enough quote to work with large files yet.

It looks like your OpenAI API key doesn't have enough quote to work with large files yet.

So how should we deal with large code repositories? Sending only part of the source code will affect the context. Now it seems that only Gemini 1.5pro can handle about 200m tokens, which is the upper limit.

This is a problem with your specific OpenAI API key. OpenAI usually limits new API users to a very low quota. As you use the API more and pay them more money they usually give you full access.

This is a problem with your specific OpenAI API key. OpenAI usually limits new API users to a very low quota. As you use the API more and pay them more money they usually give you full access.

Can the file content be sent in multiple parts? If yes, we can avoid this limit.

Unfortunately no. You need to get your OpenAI API key working correctly.

Unfortunately no. You need to get your OpenAI API key working correctly.

Hi, paul.
But I feel that this is not a problem with OpenAI tokens. Even if you are an enterprise user's token, it is limited by LLM long context memory. OpenAI currently only supports up to 128k input. However, this should not be enough for user tokens in large code repositories.

openai.RateLimitError: Error code: 429 - {'error': {'message': 'Request too large for gpt-4o in organization org-xxx on tokens per min (TPM): Limit 30000, Requested 54736. The input or output tokens must be reduced
in order to run successfully. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}

I was responding about the error above that was reported earlier in the issue. If you follow the url in the error message it will provide more detail about OpenAI rate limits.
Their rate limits have nothing to do with the models context window. Here's some info about how aider manages files and other repo context:
https://aider.chat/docs/usage.html#adding-files

I'm going to close this issue for now, but feel free to add a comment here and I will re-open or file a new issue any time.


Prevent interactive on provisioning

When installing Ruby on provisioning it would hang completely
I think it was because a prompt is blocking installation

DEBIAN_FRONTEND=noninteractive should prevent interactive during provisioning
Note that this happens regardless of using the -y flag on apt-get

PS: This might have been related to this Ubuntu bug:
https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1832919

I have just provisioned a new VM without using this environment variable. The base box ubuntu/disco64 got updated to version v20191217.0.0, though, I do not know if that made a difference.
As you noted, theapt-get command includes the -y flag which according to the documentation implies non-interactive mode:
-y, --yes, --assume-yes
  Automatic yes to prompts. Assume "yes" as answer to all prompts and run
  non-interactively. If an undesirable situation, such as changing a held package
  or removing an essential package, occurs then apt-get will abort.

And this has worked fine for years.
Could you please update the base box and see if provisioning completes for you too?

@fxn yeah, this was really strange. I've had this env working fine before.
I think it was the update to VirtualBox 6.x that caused the problems.
I had some other trouble with disco64, so I downgraded it to bionic, and that's when I got this issue. All this started when migrating to VirtualBox 6.x
I think this is really some problem upstream.
And the common denominator seems to always be libssl:

https://bugs.launchpad.net/ubuntu/+source/ansible/+bug/1833013 (note that they use -y)
https://unix.stackexchange.com/questions/146283/how-to-prevent-prompt-that-ask-to-restart-services-when-installing-libpq-dev

I can re-test everything to see if I can replicate the issue.
To be clear: this is more of a workaround before things are fixed wherever they are broken. Setting DEBIAN_FRONTEND is, as I understood, not recommended.

Ok, I just retried everything, and to be sure I purged the image and re-downloaded it.
└─▪vagrant box remove ubuntu/disco64
Removing box 'ubuntu/disco64' (v20191217.0.0) with provider 'virtualbox'...

Seems to have worked... Possibly my problem was in the cached image?
Or the problem got fixed? I don't know :)

Thanks for reporting it anyway!


Identity Provider OutBound Provisioning Role existence is not checked.

When I was trying to add an Identity Provider OutBound Provisioning Role under the Role configuration section of an Identity Provider, it allows to add a role which is not existing. Before adding the Identity Provider OutBound Provisioning Role it is required to validate if such role is already existing. If the role is not available a proper error message should be generated for the user.

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!


Refactor and improve caching in getTypeOfSymbol callees

Always cache calls to getTypeOfSymbol, even in the error case.
JS expando types are now cached on the original symbol as well as the cloned symbol. Previously they were only cached on the cloned symbol.
Large callees of getTypeOfSymbol (variable/param/property, func/class/enum/module, and accessors) now handle only caching and delegate to -Worker functions whose return values are cached
unconditionally (unlike previously). This makes the code in the -Worker functions easier to write and reason about.
Previously, successfully obtaining a type from a js special property declaration would forget to pop the circularity detection stack and check its value.

Note: Turning on 'Ignore whitespace' is practically a requirement for reading this review.


Text Overflow Bug

Bevy version
0.13
Relevant system information
cargo 1.77.0-nightly (1ae631085 2024-01-17)
AdapterInfo { name: "Apple M1 Pro", vendor: 0, device: 0, device_type: IntegratedGpu, driver: "", driver_info: "", backend: Metal }
What you did
Added a TextBundle with a max_width of 80% of the parent, Overflow::Clip on both axis.
Item holder (Parent) (Corresponds to any slot that's blue in the screenshot)
    let item_holder = commands.spawn(NodeBundle {
        style: Style {
            position_type: PositionType::Absolute,
            grid_column,
            grid_row,
            margin: UiRect::all(Val::Auto),
            width: Val::Percent(100.0),
            height: Val::Percent(100.0),
            border: GRID_INNER_OCCUPIED_BORDER,
            display: Display::Flex,
            justify_content: JustifyContent::Center,
            align_items: AlignItems::Center,
            ..Default::default()
        },
        border_color: GRID_INNER_OCCUPIED_BORDER_COLOR,
        background_color: item.item.item.get_inventory_background_color(),
        z_index: ZIndex::Global(1),
        focus_policy: FocusPolicy::Block,
        ..Default::default()
    }).id();

Text inside parent (This is where the problem is)
commands.spawn(TextBundle {
        text: Text::from_section(item.item.item.get_item_properties().name, TextStyle {
            font_size: 13.0,
            color: Color::WHITE,
            ..Default::default()
        }).with_justify(JustifyText::Right),
        style: Style {
            position_type: PositionType::Absolute,
            right: Val::Px(1.0),
            top: Val::Px(1.0),
            overflow: Overflow { x: OverflowAxis::Clip, y: OverflowAxis::Clip },
            max_width: Val::Percent(80.0),
            max_height: Val::Px(13.0),
            ..Default::default()
        },
        z_index: ZIndex::Local(2),
        ..Default::default()
    })
    .set_parent(item_holder);

What went wrong
As this Screenshot shows, almost all the items have names that would take up more than the allowed width. But weirdly enough, the ones that have the first word take up the max_width or more, overflow. The ones that would take up >= max_width but with separate words in between them don't overflow. They clip, as expected.

Just so you have an Idea of how long the names are, here it is without a max_height property:

Additional information
Just wanted to know if this is considered a bug, (I'm pretty sure it is). Or if it's expected behaviour.
This isn't really the worst bug, I could just limit the characters and remove any length of characters past some formula calculating available width.
Thanks to the bevy devs for your work!

I wonder if this is related to #6879.
You might need to wrap the text in a separate nodebundle to do the clipping.

It seems like there's something very interesting happening with PositionType::Absolute and clipping.


Replace \begin{} and \end{} with [ and ]

Supports "\begin{}" and "\end{}" and replaces them with "[" and "]"
Fixes #3

done


Dev tools security concern.

I have a slight concern about having devtools always available with any Vue build type.
This is because if the application has a security exploit, the dev tools make it increasingly easy to exploit this vulnerability. Such as poorly handled authentication. I know this is purely down to bad development, but the dev tools should only be available when hooks are available, this will prevent exploitation.
Could there be an additional option to make vue-devtools always available. To me it makes more logical sense to be defensive for production builds.

That's a valid point - to achieve this we need to add a config option to Vue core:
Vue.config.devtools = process.env.NODE_ENV !== 'production'

We can also make it default to false in production builds. Because this is a change in Vue core, tracking it here instead: https://github.com/vuejs/vue/issues/2453

Nice catch @blake-newman!


Deprecated paths in tests

Responding to a couple of warnings

These fixes are also in #2162 -- any chance I could tempt you to review that PR instead? :)


[feature request] in-expression

I think Python-like in-expressions would be a nice addition to allow for checking whether an object is in a table. It could be done like this:
mylist = { "foo", "bar", "biz", "baz" }

if "biz" in mylist
    print("Has biz")

has_bar = "bar" in mylist

print("has_bar = #{has_bar}")

which compiles to:
local mylist = {
    "foo",
    "bar",
    "biz",
    "baz"
}
if (function()
    local result
    result = false
    for k, v in pairs(mylist) do
        if v == "biz" then
            result = true
            break
        end
    end
    return result
end)() then
    print("Has biz")
end
local has_bar
do
    local result
    result = false
    for k, v in pairs(mylist) do
        if v == "bar" then
            result = true
            break
        end
    end
    has_bar = result
end
return print("has_bar = " .. tostring(has_bar))

Well... 'in' in python would make sense because python have 'dict' and 'list'.
print(1 in [1,2,3])
print("a" in {"a": 1})

But in lua/yue, 'in' can be very ambiguous. It is unclear to check key or value in table? is this table is 'list like' or 'dict like'? Should loop with 'pairs' or 'ipairs'?
Choice implements any option would lead to not satisfy for other cases!
And make thing more complicated, Yue already has support for 'in' constant table:
if "biz" in { "foo", "bar", "biz", "baz" }
  print 'found'

Thanks and regards!

Well... there are never has any edge cases like this in python, this example is simple don't exist because nobody code like this:
mydict = {
    "foo": "bar"
}

class UndefinedField:
    pass

if mydict.get("biz", UndefinedField) is UndefinedField:
    # ...

but:
if "biz" in mydict:
  pass

It work in python because with dict 'in' check the key in keys, with list 'in' find value in values! That is common sense.
But lua is the language without common sense! 😓
lua can not compare to python [Beautiful, Explicit, Explicit] because in lua all we get is table. And Yue cannot escape it!
If yue has 'in'. What would it do?

find value in a table that act like list with 'ipairs' => make sense, but what about list with gaps?
find value in a table that act like dict with 'pairs' => make no sense, nobody that code like that!

I would like to have 'in' dynamic in Yue too. I have try with macro. But there I see some downside that I don't like and simple giveup on that idea:

Create and run anonymous, the big no no when I run lua in game.
Because I work a lot with python, sometime I make a mistake when work with table dict like and create the silent error that took a ton of time to debug!
It simple to make a function for that like all the lua coder code!

Yue have a lot of rich features, but add more features that not well consider can create more burden to maintain language for the author! I already happy with author focus maintain the language for now!
Thanks and regards!

compiles to... this mess
local has
do
    local _val_0 = "foo"
    return {
        [1] = "bar",
        x = "foo"
    } == _val_0
end

Confirmed it is a bug and made a fix. This code should compile to:
local has
do
    local _val_0 = "foo"
    has = {
        [1] = "bar",
        x = "foo"
    } == _val_0
end

The expressions in the braces of in { } are just used for comparison.

mylist = { "foo", "bar", "biz", "baz" }

if "biz" in mylist
    print("Has biz")

has_bar = "bar" in mylist

print("has_bar = #{has_bar}")

How about generating this code to:
local mylist = {
  "foo",
  "bar",
  "biz",
  "baz"
}
if (#mylist > 0 and (function()
  for i = 1, #mylist do
    if mylist[i] == "biz" then
      return true
    end
  end
  return false
end)() or mylist["biz"] ~= nil) then
  print("Has biz")
end
local has_bar
do
  local result = false
  for i = 1, #mylist do
    if mylist[i] == "biz" then
      result = true
      break
    end
  end
  if not result then
    result = mylist["biz"] ~= nil
  end
  has_bar = result
end
return print("has_bar = " .. tostring(has_bar))

If a table has only an array part, the in expression will match its elements by values. And if a table has only a hash part, the in expression will skip the array part then match hash part by key and tell if the paired value exists.
If a table has both an array part and a hash part, the in expression will first try matching the array part, if nothing matches, continue to do the hash part matching.

has = "foo" in { [1]: "bar", x: "foo" }

How about change { } => [ ] make it more clear that: it not a 'lua table', but a 'yue constant list'!
has = "foo" in [ [1]: "bar", x: "foo" ]

The brackets in in-expression are already used for representing closed intervals.
if num in [1, 10]
  print "number is in 1 to 10"
list = {1, 2, 3} -- is currently a valid list literal

Maybe we can throw an error, when the content in the in-expression is in a form of complex table literal instead of just a simple list.

Had a second thought that the in-expression should not be used for table hash part existence checking. Because tab[key]? is already sufficient for this purpose. Adding similar functions to in-expression will lead to generating lots of redundant codes when you are just for hash part checking.  Tried adding the list table checking to Yuescript by commit 85a2da5cbbbbfb27f3b65ac67d083dd2443d223d.


appended National Science Foundation wherever required

Only one mentioning of National Center for Atmospheric Research which I could find (README.md).
There is no mentioning of "National Center for Atmospheric Research" in copyright headers in source code and no mentioning in the docs folder as well
In docs/source/conf.py l17 and l18:
copyright = f'2024-{datetime.datetime.now().year}, NCAR/UCAR'
author = 'NCAR/UCAR'

Do changes need to be made in these lines as well? No other mentioning in acronyms either.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 79.90%. Comparing base (8bd10e1) to head (9077685).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #283   +/-   ##
=======================================
  Coverage   79.90%   79.90%
=======================================
  Files          12       12
  Lines         821      821
=======================================
  Hits          656      656
  Misses        165      165

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests


Fixes extra intra-word-spaces problem with 4.0

#Fixes https://github.com/tesseract-ocr/tesseract/issues/988
preserve_interword_spaces 1

Yes,  langdata and langdata_lstm also need changes. Those will effect any new training being done.
The existing traineddata files have to be updated with the new config files.
The various _vert.traineddata may also need update, I haven't looked into it.
PS. Writing from my phone, don't have access to the repo to check details right now.


Why do I need an "id" to write new data?

When I post new data to my API, an "id" is required for some reason.  I'm using JAGQL and this MongoStore.  According to the JAGQL docs, it says that an id will be auto-generated unless you put generateId:false in the configuration for the server.
What am I missing?  Thank you!

This is actually not an issue with this library.  See here for complete answer:
https://stackoverflow.com/questions/55911148/jagql-why-do-i-need-an-id-for-a-post-call


[lit-html, reactive-element] Revert change to pause update cycle while disconnected

Reverts the change in Lit 2 to pause ReactiveElement's update cycle while the element is disconnected.
This change was originally added to Lit 2 to ensure that once AsyncDirectives are disconnected, they do not receive further updates until they are reconnected.
However, the change in Lit 2 to pause updates while the element was disconnected was found during internal testing to be subtly breaking in potentially dangerous ways, which could lead to memory leaks if the update cycle was relied on to clean up external event handlers/subscriptions.
Instead, the onus will shift to AsyncDirective authors to check the this.isConnected flag during update and avoid doing any potentially leaky work while disconnected, since they may never be reconnected in that state.
Note that calling setValue from an asyncDirective while disconnected will now synchronously commit the change (since it seems arbitrary now to not do that; that was only in to maintain the same guarantee we're removing above (that disconnected directives don't receive updates).
A follow-on PR (#2044) re-works the disconnection handling for first-party AsyncDirectives (until, asyncAppend, asyncReplace) to allow them to be GC'ed before their awaited promises resolve.

LGTM, though I still prefer the name RootPart. RootChildPart just sounds a bit awkward and contradictory to me.

@justinfagnani Ok I think I changed my mind. I liked RootChildPart from the point of view of it being important for users to understand that it had all the API of a ChildPart plus the RootPart API (setConnected). But on further reflection, there's actually very-little-to-no useful ChildPart API that a user would need to know about/use on the value returned from render. So I think I'm good with that just being RootPart.