Feat: Remove gruvbox-material and everforest themes

Requested feature
Hello, I'm the author of gruvbox-material and everforest, I've added support for lualine recently, with support for various configuration options (e.g. &background, g:gruvbox_material_background, g:gruvbox_material_statusline_style, etc.).
The theme files are located in my repo, which I think might be conflict with the theme files in yours, since they have the same runtime path.
Could you remove the theme files in your repo? Thanks!

Glad to know you added support for lualine :)
No they won't conflict. Themes are loaded with custom loader . it prefers theme files from external sources.
So when your colorschemes are installed lualine will load theme from your repo and when colorschemes aren't installed theme will be loaded from this repo.

That's fine, so it's unnecessary to remove the theme files in this repo.
I'm going to close this issue.


Repackaging a class leads to wrong Coder behavior

Create a class SomeClas in package SomeWrongPackage
After creating, change the package to SomeCorrectPackage
Accept the refactoring even if it's says the category does not exist.
You get this:

Even though the definition seems correct:


Submit ENA run and experiment XMLs

After we converted the HCA experiment to an ENA experiment and run XML we need to send it to ENA to be able to store it in that archive.
To be able to send the ENA experiment and run XML or any other ENA object(s), we also need an accompanying submission XML in a separate file. We need to name this file to submission.xml.
Here is the documentation from ENA how to do it:
https://ena-docs.readthedocs.io/en/latest/submit/reads/programmatic.html

Prabhat will be looking into this today. But it is dependant on at least a ticket that is in Code Review.

Unit tests are done but testing to the actual archives is in progress.

PR https://github.com/ebi-ait/ingest-archiver/pull/79


Create uqat

http://uqat.ca/
students have emails ending by :  uqat.ca
http://programmes.uqat.ca/programmes/4613.html
thank you

Sorry for the delay - i've added the domain manually and therefore closing the request

Thank you very much, I appreciate it a lot.
Have a good day
Nicole Gingras

De : Elena Ganenkova<EMAIL_ADDRESS>Envoyé : 27 mars 2015 08:57
À : JetBrains/swot
Cc : Gingras, Nicole
Objet : Re: [swot] Create uqat (#818)
Closed #818https://github.com/JetBrains/swot/pull/818.
—
Reply to this email directly or view it on GitHubhttps://github.com/JetBrains/swot/pull/818#event-266853104.


Create a Telugu translation of the basic web server guide

Based on the English version of the guide (https://github.com/devadvance/terminalcheatsheet/blob/staging/guides/basic-web-server-macos-linux.md), create a Telugu translation in https://github.com/devadvance/terminalcheatsheet/blob/staging/i18n/te/guides/basic-web-server-macos-linux.md.
It is OK to use the English images/GIFs. Look for TODO in the file for suggested places for translations.
Please comment on this issue before starting work so people are informed that you are working on it.

So, we have to translate the page file in First link and store it in second ?
I can do this if this is unassigned yet

@manishkrjha yes that is the objective. Assigning this issue to you.


Zero ra, decs in the reduced M31 data

This is just a bookmark about the issue with
RA,DECs  being zero for a large number of sources
in the the fastspecfit processing of the special reduction M31 data.
n [45]: T2 = atpy.Table().read("fastspec-m31.fits",2)

In [46]: T2
Out[46]:
<Table length=3200>
   TARGETID       RA     DEC   TILEID ... FLUX_IVAR_Z FLUX_IVAR_W1 FLUX_IVAR_W2
    int64      float64 float64 int64  ...   float32     float32      float32
-------------- ------- ------- ------ ... ----------- ------------ ------------
37466597753522     0.0     0.0  80715 ...         0.0          0.0          0.0
37466601947510     0.0     0.0  80715 ...         0.0          0.0          0.0
37466601947567     0.0     0.0  80715 ...         0.0          0.0          0.0

There's an assert in the code now to ensure that the coordinates are never zero (although I've lost track of the exact commit).
For the record, the M31 project has moved over to here--
https://github.com/moustakas/fastspecfit-projects/blob/main/m31/fastspecfit-m31


ruby_install_gem, gem installed but not working

`: -->
Issue Type

Bug Report / Support Request

Your Environment

vagrant 1.9.1
virtualbox 5.1.10

Your OS

Windows (8)

Summary
First, I want thank you Geerlingguy for this project ! I'm a beginner on linux vm and I was  before working with wamp for my drupal theming projects. It's really help me to introduce linux environment.
So I've a little issue with ruby_install_gems. When I add to my config.yml:
ruby_install_gems:
  - compass
  - sass
  - less

After vagrant up, gems are ok in the gem list with versions. But I can't use them, for exemple when I enter 'compass -v' the console say there is no program compass installed. same for a compass watch command.
as you can see below :
vagrant@drupalvmGEMBUG:~$ gem list

*** LOCAL GEMS ***

bigdecimal (1.2.8)
bundler (1.13.7)
chunky_png (1.3.8)
commonjs (0.2.7)
compass (1.0.3)
compass-core (1.0.3)
compass-import-once (1.0.5)
did_you_mean (1.0.0)
ffi (1.9.14)
io-console (0.4.5)
json (1.8.3)
less (2.6.0)
minitest (5.8.4)
multi_json (1.12.1)
net-telnet (0.1.1)
power_assert (0.2.7)
psych (2.0.17)
rake (10.5.0)
rb-fsevent (0.9.8)
rb-inotify (0.9.7)
rdoc (4.2.1)
sass (3.4.23)
test-unit (3.1.7)
vagrant@drupalvmGEMBUG:~$ compass -v
The program 'compass' is currently not installed. You can install it by typing:
sudo apt install ruby-compass

Other gems seems to work, for exemple $ bundler -v return the version.
So to make my gems working after a vagrant up I have to install them in a .sh file (linked in config.yml) with :
yes | sudo apt install ruby-compass
yes | sudo apt install ruby-sass

Did I miss something to make it work simply with ruby_install_gem ?
Thanks in advance

This is a bug https://github.com/geerlingguy/ansible-role-ruby/issues/33

The fix is in geerlingguy.ruby 2.4.2.

Nice ! thank you


New Crossover Patch Bug

https://github.com/Marqasa/overwatch-crossover-patch/assets/64818186/85aeb1c9-0481-4ac7-bb1d-e7fea4ef39ef
Limited knowledge of how this all works, though with the most recent update of Crossover, it appears that you have to switch from ESync to MSync, but there's random black flashing on the screen, especially when there are a lot of characters close together in battle.
By chance, is there a way to resolve this? Or a setting I need to disable/enable?
macOS Sonoma 14.1.2
M1 MacBook Pro
Crossover v23.7

i have the same issue. Did you solve the problem?

I haven't been able to replicate this issue. Have you tried a fresh install of CrossOver + patch?

Same issue here, latest patch and crossover v23.7 macOS Sonoma 14.1.2

@thinksamwp I've had to rebuild it 4 times to finally get it to work. Just DVXC (whatever it's called) cache and high resolution mode is enabled. M/E Sync can't be enabled otherwise it breaks everything.

just repatch overwatch with this it fixed it for me


Windows npm.bat pathname bug prevents build with unclear errors (with verified bug fix )

Development Environment:
OS: Windows 7 Professional SP1
Visual Studio 15 Community \ C++ Tools \ SDK
VS Code: 1.9.1 master release
Node: 6.9.1
NPM:  3.10.8
Description Of Bug:
npm.bat will fail on Windows if the vscode code tree/repository is in a subdirectory
with any directories with spaces in the path.  The error does not fatally
stop the script making it easy to miss the "cannot open" output from
the npm script.  The error also does not show up in the npm_debug.txt
file after many modules have been processed.
The bug is in line five, in the findstr command -
5 for /f "tokens=2 delims=:, " %%a in ('findstr /R /C:""electronVersion":.*" %~dp0..\package.json') do set npm_config_target=%%~a
The file string:   %~dp0..\package.json
Will to evaluate to multiple files/folders for any path with spaces and folder names.
Thus %electronVersion does not get set but no fatal error stops the script.
Since the command output gets overrun from the npm output it's easy to miss the warning.
The actual error building electron comes much later and gives no clue as to the cause
(empty electronVersion value)
Steps to reproduce:
(working example) CheckOut code to:  C:\Dev\vscode
(failure example) CheckOut code to:  C:\VSC Dev\vscode
(Initial output to terminal NOT npm_debug_out)
FINDSTR: Cannot open C:\VSC
FINDSTR: Cannot open Dev\vscode\scripts..\package.json
...subsequent npm output until electron failure
The Fix: Simply enclose the full file path string in double quotes
5 for /f "tokens=2 delims=:, " %%a in ('findstr /R /C:""electronVersion":.*" "%~dp0..\package.json"') do set npm_config_target=%%~a
I also added an error level output so code.bat could detect an error.
Additionally command output with an explicit fatal error.
Can I submit these fixes?  I am embarrassed, but it took me three days
to catch this as all other similar issues I read on github pointed
to a toolchain issue. There our problems with the C++ tools install
that can lead you down a wrong path. I would like to add some notes
to the contribution wiki page.
Looking forward to contributing especially for accessibility
as a disabled\ALS user using voice programming - I really love
the vscode editor !
Cheers Christopher

Hi Christopher,
Thanks a lot for your fix! I just pushed it 👍


[FLINK-10627][E2E tests] Test s3 output for streaming file sink

What is the purpose of the change
This PR extends streaming file sink end-to-end test with the case of s3 output.
Brief change log

Add end-to-end test java utility module with s3 console tool.
Add common-s3.sh with s3 bash functions, move there previous s3 functions from common.sh.
modify s3 shaded dependency tests to use common-s3.sh
Detect end of tests by number of lines in output files.
Use OUT_TYPE var to distinguish testing local and s3 output in streaming file sink end-to-end test.
Add s3 case to nightly tests.

Verifying this change
build flink and run:
cd flink-end-to-end-tests
FLINK_DIR=../build-target
ARTIFACTS_AWS_BUCKET=
ARTIFACTS_AWS_ACCESS_KEY=
ARTIFACTS_AWS_SECRET_KEY=
./run-single-test.sh test-scripts/test_streaming_file_sink.sh <local|s3>
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

thanks @kl0u, I have rebased and addressed the issue


gulp-jest running the same test file for each other test file

I have a gulp watch task watching for file changes in the test file directory. Test file updates are detected correctly. Then a test task is run that pipes the changes file through to gulp-jest, but gulp-jest always runs the same test file again an again - once for each test file in the test directory.
const gjest  = require('gulp-jest').default;
const debug = require('gulp-debug');

module.exports = function(gulp, config) {
    return function() {
        return gulp
            .src(config.srcPath + 'scripts/tests/*.test.js')
            .pipe(debug())
            .pipe(gjest({
                'config': {
                    'transformIgnorePatterns': [
                        '<rootDir>/bin/',
                        '<rootDir>/capistrano/',
                        '<rootDir>/config/',
                        '<rootDir>/coverage/',
                        '<rootDir>/src/',
                        '<rootDir>/web/',
                        '<rootDir>/node_modules/',
                    ],
                    'automock': false,
                    'notify': true,
                },
            }));
    };
};

The irritating output after I change one test file is this (note the debug output showing the file names of all the test files in my test file directory):
[16:39:40] Starting 'eslint'...
[16:39:40] File /home/roman/Development/project-ongr/app/Resources/scripts/tests/templates-helper.test.js was changed, keep watching ...
[16:39:42] Finished 'eslint' after 1.69 s
[16:39:42] Starting 'test-scripts'...
[16:39:42] gulp-debug: app/Resources/scripts/tests/price-formatting.test.js
[16:39:42] gulp-debug: app/Resources/scripts/tests/templates-helper.test.js
 PASS  app/Resources/scripts/tests/price-formatting.test.js
  priceFormatting
    ✓ formatPrice returns a correctly formatted price (4ms)
    ✓ formatPriceValue returns a correctly formatted price value (1ms)
    ✓ formatPricesIn fills price containers with formatted prices using their unformatted-price data attributes (16ms)
    ✓ exposePriceFormatting exposes the price formatting functions to the global project object (1ms)

Test Suites: 1 passed, 1 total
Tests:       4 passed, 4 total
Snapshots:   0 total
Time:        0.547s, estimated 1s
Ran all test suites.
 PASS  app/Resources/scripts/tests/price-formatting.test.js
  priceFormatting
    ✓ formatPrice returns a correctly formatted price (1ms)
    ✓ formatPriceValue returns a correctly formatted price value (1ms)
    ✓ formatPricesIn fills price containers with formatted prices using their unformatted-price data attributes (2ms)
    ✓ exposePriceFormatting exposes the price formatting functions to the global project object (1ms)

Test Suites: 1 passed, 1 total
Tests:       4 passed, 4 total
Snapshots:   0 total
Time:        0.088s, estimated 1s
Ran all test suites.
[16:39:42] gulp-debug: app/Resources/scripts/tests/utils.test.js
[16:39:42] gulp-debug: 3 items
 PASS  app/Resources/scripts/tests/price-formatting.test.js
  priceFormatting
    ✓ formatPrice returns a correctly formatted price (2ms)
    ✓ formatPriceValue returns a correctly formatted price value (1ms)
    ✓ formatPricesIn fills price containers with formatted prices using their unformatted-price data attributes (2ms)
    ✓ exposePriceFormatting exposes the price formatting functions to the global project object (1ms)

Test Suites: 1 passed, 1 total
Tests:       4 passed, 4 total
Snapshots:   0 total
Time:        0.087s, estimated 1s
Ran all test suites.
[16:39:42] Finished 'test-scripts' after 807 ms

This looks like a gulp-jest related error to me.

Sorry, I now figured out that this only happens when I only save the test file without any real code change. If I change the test code, everything runs as expected. My bad!


Move pClk frequency setting to the clocks class.

It was configured in the wrong place (the CCI-P interface class).

DCP tests pass: http://sw-pert.altera.com/pert/pert.php?test_run_id=2237248


How to reproduce the repainting results in a paper

Thanks for your great work!
I tried to reproduce the results in the bottom right of fig3 in the paper based on the “imagenet_256.yml” and “256*256_duffusion_uncond.pt” you provided, but my reconstructed image was poor.
May I ask if I didn't notice anything that caused this poor result?

The visualization results in fig3 are generated by "hq_demo", you can find instructions on the main page.

But I notice  “hq_demo/evaluation.sh” only have the super-resolution inference, is there inpainting implement?

Yes, you can see the Colab for more demonstrations.

Thanks for your reply.
But I only find face inpainting based on celebahq in the Colab, so how can I realize the Inpainting on DIV2K.

Just change the --deg, for example, python main.py --config confs/inet256.yml --path_y data/datasets/gts/inet256/323.png --class 323 --deg "inpainting" -i butterfly_sr.
Maybe you need to delete and conf.name=='face256' in gaussian_diffusion.py, line 601.

Okey I see, thank you so much for you patient answer

我按照你说的去做了，但是我发现这是有分类引导的，如果我尝试复现论文中Fig3右下角的水母的图片，在 classifier_scale=0时，会生成一些桔子的图片，这是怎么一回事。所以如何复现paper里fig3中一摸一样mask设置的结果
I did as you said, but I found that this is classifier guided.
the command is "python main.py --config confs/inet256.yml --path_y data/datasets/gts/inet256/paper_fig3_jellyfish.jpg --deg "inpainting" -i imagnet"
And I try to reproduce the image of jellyfish picture in the lower right of Fig3 in your paper, with the classifier_scale=0, some images of oranges will be generated, I don't know why.
So how to reproduce the result of exactly the same mask setting in fig3 in paper.

You need to set the correct class label to guide the denoiser toward that class content. For example, jellyfish is 107.

https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a you can find the class here


Support aten.min/max with retuning values and indices

Both aten.min.dim and aten.max.dim return not only the min/max value of the given dim but also the index of the value.
Currently in ttnn we have three related functions:

ttnn.min
ttnn.max
ttnn.argmax

For aten.max.dim, while we can use ttnn.max to get the max values and ttnn.argmax to get the corresponding indices, it will run the same reduction process twice. I think we should create an option for ttnn.argmax to return (max values, indices)
For aten.min.dim, we at least need ttnn.argmin to get the indices

The conversion is OK imo.  The efficiency of the hardware should outweigh the cost of a second pass.  (Imagine doing this with GPU.)
Max pooling is another story.  If we don't compute argmax on the fly, there is no way to recover the indices.


Task queued, but corresponding feature is disabled

The scheduled task mollie.subscriptions.renewal_reminder is queued, but the subscriptions feature is disabled.
I had the issue that the task was running but never finished. Had to manually trigger the task. All other tasks were skipped for half a month, until I recognized that.

I updated to version 4.10.1 and it seems the mentioned memory usage was causing the issue.
But the subscription task is still scheduled. Isn't it better to schedule the task only if the corresponding feature is enabled?

@carstenjaksch i have to check if there is a way to remove tasks in shopware, but currently if the feature is disabled, this task does nothing right now

@carstenjaksch https://developer.shopware.com/docs/guides/plugins/plugins/plugin-fundamentals/add-scheduled-task.html  it seems like i can only register scheduled tasks but i cannot remove them after wards, they are installed during plugin update


[DevToolbar] Investigate if we need to include frame-ancestors header

Should we also include this header in the /iframe response if we have validated that the requested domain is valid?
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/frame-ancestors
Some thoughts:

We would still want to validate first, because we send a different response depending on if the origin is allowed or not
We would never want to use the Content-Security-Policy: frame-ancestors <space separated list of sources>; form because that could leak user data and expose origins that are not being requested right now

For prod frame-ancestors includes 'self', so we're good there.
However we will need to look into: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/script-src to run the inline scripts in the response templates.

The comment above is wrong. 'self' refers to the server, sentry.io, not the browser. frame-ancestors takes precedence over X-Frame-Options, so we do need this. Opening a ticket to track CSP-related tasks soon.

See https://github.com/getsentry/sentry/issues/78237. Closing this one now investigation's over


Styleguide global assets, like js and scss outside of components?

First, thanks for making this! It seems like it could be really helpful if I can get if figured out.
Loading the styleguide to view a component shows the css and javascript that goes along with that component, but what about other javascript and css, not contained within the component folder?
For example, a component that relies on styles not kept within its own stylesheet. Like, a card that uses .o-heading objects. I have a css file for headings, but that doesn't belong with the card component. This means it doesn't get loaded on the styleguide page, though. It would only pull it in if I @import all my scss at the top of every component? I'm sure I'm missing something there.
Also, say the card uses another, separate js file like some custom functions to interact with an API. These functions are used elsewhere, not just with this component so they don't belong in the component folder. How do you get that included on the styleguide page, as well?

Hi @jonathanrbowman, thanks for your words!
Regarding the external CSS in the stylesheet, there's a configuration option for including that: https://github.com/devnacho/mountain_view#global-stylesheets
Hope that helps, and do not hesitate to ask if any further questions.
Regarding JS, there's currently no option for that. But seems like it could be a good idea to add one. Happy to accept a PR if you wanna take on it!


Removed duplicates

@meve can you please perform step 2 of https://github.com/elasticsearch/logstash/blob/master/CONTRIBUTING.md#contribution-steps

@suyograo done.

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run; then  say 'jenkins, test it'.


eventing-contrib image URL broken: Can not pull images, image URL 404

Expected Behavior
docker pull knative.dev/eventing-contrib/github/cmd/controller:latest

pull image successfully
Actual Behavior
sudo docker pull knative.dev/eventing-contrib/github/cmd/controller
Using default tag: latest
Error response from daemon: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE html>\n\n  \n    <meta charse
t=\"utf-8\">\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no\">\n\n    <title>Page Not Found</title>\n    <link href='https://fonts.googleapis.com/css?family=Roboto:400,700&subset=latin,latin-ext' rel='stylesheet' type='text/css'>\n    \n  \n  \n
\n
\n
\n
Page Not Found
\n
\n
\n
Looks like you've followed a broken link or entered a URL that doesn't exist on this site.
\n
\n            \n              <svg xmlns=\"http://www.w3.org/2000/svg\" width=\"16\" height=\"16\" viewBox=\"0 0 16 16\">\n                <path fill=\"#00ad9f\" d=\"M11.9998836,4.09370803 L8.55809517,7.43294953 C8.23531459,7.74611298 8.23531459,8.25388736 8.55809517,8.56693769 L12,11.9062921 L9.84187871,14 L4.24208544,8.56693751 C3.91930485,8.25388719 3.91930485,7.74611281 4.24208544,7.43294936 L9.84199531,2 L11.9998836,4.09370803 Z\"/>\n              </svg>\n              Back to our site\n             \n
\n
\n
\n
\n    \n  \n\n\n"

I have tried multiple URL like:
knative.dev/eventing-contrib/kafka/channel/cmd/channel_controller
knative.dev/eventing-contrib/prometheus/cmd/controller

all these images are same error.
However, docker pull like:
us.gcr.io/probable-summer-223122/cmd-03315b715ae8f3e08e3a9378df706fbb@sha256:2656f39a7fcb6afd9fc79e7a4e215d14d651dc674f38020d1d18c6f04b220700
gcr.io/knative-releases/github.com/knative/eventing-sources/cmd/event_display

all these images can be pulled successfully.
Are these eventing-contrib image URL broken?
Steps to Reproduce the Problem

docker pull eventing-contrib images

Additional Info

I'm guessing the URL was found here:
https://github.com/knative/eventing-contrib/blob/dcf7dbfe0006e843183a2a0a9f0c6d98019ee8f2/github/config/500-controller.yaml#L34
All of the YAMLs in this repo require the use of ko to deploy, as described here. ko will build the image and replace the URL with a working one.
If you do not want to build the images yourself, use one of the releases, such as https://github.com/knative/eventing-contrib/releases/tag/v0.9.0, which is intended to be applied with kubectl. Concretely github.yaml uses the  gcr.io/knative-releases/knative.dev/eventing-contrib/github/cmd/receive_adapter@sha256:30e4fca7e293490b637087b57961f1487823a47725a91c9903323e8fe3ed3833 image.


Add additional RDS TLS communications checks

Context
Add additional RDS transport level encrypted checks for support RDS versions
Description
Added additional checks for MySQL, MariaDB, Aurora PostgreSQL, and Aurora MySQL DB instances.
Corrected Microsoft SQL Server checks as well to include all the engine versions available.
License
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Closed in favor of https://github.com/prowler-cloud/prowler/pull/3964


Update Portainer to use the Helm chart

Description
Updated Portainer to use the offical helm chart to install.
Motivation and Context

[ ] I have raised an issue to propose this change, which has been given a label of design/approved by a maintainer (required)
[X] Issue #587

How Has This Been Tested?
Tested arkade install portainer with different namespaces, with and without persistence and with changing the Service type.
Are you a GitHub Sponsor (Yes/No?)

[ ] Yes
[X] No

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Documentation

[X] I have updated the list of tools in README.md if (required) with ./arkade get -o markdown
[X] I have updated the list of apps in README.md if (required) with ./arkade install --help

Checklist:

[X] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[X] I've read the CONTRIBUTION guide
[X] I have signed-off my commits with git commit -s
[ ] I have added tests to cover my changes.
[X] All new and existing tests passed.

[X] I have tested this on arm, or have added code to prevent deployment

Thanks for this @Shikachuu
I reckon your next app or PR is going to be merged much more quickly, now that you have been through a few iterations.


fix(chatty-errors): Compact error msg for long list of k8s resources

Description
This should prevent really chatty error messages like here: https://redhat-internal.slack.com/archives/C02HX541YQN/p1707422841968359
Checklist

[x] Investigated and inspected CI test results

N/A

[ ] Unit test and regression tests added
[ ] Evaluated and added CHANGELOG entry if required
[ ] Determined and documented upgrade steps
[ ] Documented user facing changes (create PR based on openshift/openshift-docs and merge into rhacs-docs)

If any of these don't apply, please comment below.
Testing Performed
Here I tell how I validated my change

Manually on GKE using the following steps:

Install Cluster from this branch
Set Sensor image manually to 4.2
Run the Sensor Upgrader
Navigate to URL/main/clusters to check the upgrade status
Notice Sensor Upgrade error as in the picture below:

Reminder for reviewers
In addition to reviewing code here, reviewers must also review testing and request further testing in case the
performed one does not seem sufficient. As a reviewer, you must not approve the change until you understand the
performed testing and you are satisfied with it.

@mclasmeier, @Maddosaurus,  I made a screenshot from the new version of this error and pasted it into the PR description. PTAL
I invited @theencee to give this a look as well. I will merge that soon if I don't hear any requests for further changes.


Pairtree Derivative Paths

Closes #66

Given an image stored at tmp/08/61/2n/57/q-intermediate_file.jp2, there should be a way to get the string 08%2F61%2F2n%2F57%2Fq-intermediate_file.jp2, i.e., the URL-encoded ID that the image server expects. Am I missing it? This seems like the time to add it.
Loris lets you get away with not having it encoded because this, but we should use the correct syntax in our HTML view and manifests.

@jpstroop Looking at the IIIF Presentation spec I'm not seeing where to put that. Could you modify the following manifest to show the way it should be updated?
{
   "@context":"http://iiif.io/api/presentation/2/context.json",
   "@id":"http://<IP_ADDRESS>:3000/concern/scanned_books/08612n57q/manifest",
   "@type":"sc:Manifest",
   "label":"Test",
   "description":"Test",
   "viewingHint":"individuals",
   "viewingDirection":"left-to-right",
   "sequences":[
      {
         "@id":"http://<IP_ADDRESS>:3000/concern/scanned_books/08612n57q/manifest/sequence/normal",
         "@type":"sc:Sequence",
         "canvases":[
            {
               "@id":"http://<IP_ADDRESS>:3000/concern/scanned_books/08612n57q/manifest/canvas/bz60cw30w",
               "@type":"sc:Canvas",
               "label":"color.tif",
               "images":[
                  {
                     "@id":"http://<IP_ADDRESS>:3000/concern/scanned_books/08612n57q/manifest/annotation/bz60cw30w-image",
                     "@type":"oa:Annotation",
                     "motivation":"sc:painting",
                     "resource":{
                        "@id":"http://<IP_ADDRESS>:5004/bz/60/cw/30/w-intermediate_file.jp2/full/!200,200/0/default.jpg",
                        "@type":"dcterms:Image",
                        "format":"image/jpeg",
                        "width":200,
                        "height":287,
                        "service":{
                           "@context":"http://iiif.io/api/image/2/context.json",
                           "@id":"http://<IP_ADDRESS>:5004/bz/60/cw/30/w-intermediate_file.jp2",
                           "profile":"http://iiif.io/api/image/2/level2.json"
                        }
                     },
                     "on":"http://<IP_ADDRESS>:3000/concern/scanned_books/08612n57q/manifest/canvas/bz60cw30w"
                  }
               ],
               "width":200,
               "height":287
            },
            {
               "@id":"http://<IP_ADDRESS>:3000/concern/scanned_books/08612n57q/manifest/canvas/zs25x852h",
               "@type":"sc:Canvas",
               "label":"gray.tif",
               "images":[
                  {
                     "@id":"http://<IP_ADDRESS>:3000/concern/scanned_books/08612n57q/manifest/annotation/zs25x852h-image",
                     "@type":"oa:Annotation",
                     "motivation":"sc:painting",
                     "resource":{
                        "@id":"http://<IP_ADDRESS>:5004/zs/25/x8/52/h-intermediate_file.jp2/full/!200,200/0/default.jpg",
                        "@type":"dcterms:Image",
                        "format":"image/jpeg",
                        "width":200,
                        "height":256,
                        "service":{
                           "@context":"http://iiif.io/api/image/2/context.json",
                           "@id":"http://<IP_ADDRESS>:5004/zs/25/x8/52/h-intermediate_file.jp2",
                           "profile":"http://iiif.io/api/image/2/level2.json"
                        }
                     },
                     "on":"http://<IP_ADDRESS>:3000/concern/scanned_books/08612n57q/manifest/canvas/zs25x852h"
                  }
               ],
               "width":200,
               "height":256
            }
         ]
      }
   ]
}

Yup, sorry, I could have been clearer.
In the Prezi API under 6.4:

If a IIIF Image API service is available for the image, then a link to the service’s base URI should be included. The base URI is the URI up to the identifier, but not including the trailing slash character or any of the subsequent parameters.

So, in the service blocks above (this is the first one):
"service": {
    "@context":"http://iiif.io/api/image/2/context.json",
    "@id":"http://<IP_ADDRESS>:5004/bz/60/cw/30/w-intermediate_file.jp2", // here
    "profile":"http://iiif.io/api/image/2/level2.json"
}

that @id should have a value that is the Base URI for an IIIF image service. Assuming that the identifier portion of the URI in @id above is bz/60/cw/30/w-intermediate_file.jp2, that part needs to be escaped to bz%2F60%2Fcw%2F30%2Fw-intermediate_file.jp2. So the whole URI would be:
http://<IP_ADDRESS>:5004/bz%2F60%2Fcw%2F30%2Fw-intermediate_file.jp2",

Yar?
See also:

Image API Section 3: http://iiif.io/api/image/2.0/#identifier
Image API Section 9: http://iiif.io/api/image/2.0/#uri-encoding-and-decoding

@jpstroop Is this better?
{
   "@context":"http://iiif.io/api/presentation/2/context.json",
   "@id":"http://<IP_ADDRESS>:3000/concern/scanned_books/08612n57q/manifest",
   "@type":"sc:Manifest",
   "label":"Test",
   "description":"Test",
   "viewingHint":"individuals",
   "viewingDirection":"left-to-right",
   "sequences":[
      {
         "@id":"http://<IP_ADDRESS>:3000/concern/scanned_books/08612n57q/manifest/sequence/normal",
         "@type":"sc:Sequence",
         "canvases":[
            {
               "@id":"http://<IP_ADDRESS>:3000/concern/scanned_books/08612n57q/manifest/canvas/bz60cw30w",
               "@type":"sc:Canvas",
               "label":"color.tif",
               "images":[
                  {
                     "@id":"http://<IP_ADDRESS>:3000/concern/scanned_books/08612n57q/manifest/annotation/bz60cw30w-image",
                     "@type":"oa:Annotation",
                     "motivation":"sc:painting",
                     "resource":{
                        "@id":"http://<IP_ADDRESS>:5004/bz%2F60%2Fcw%2F30%2Fw-intermediate_file.jp2/full/!200,200/0/default.jpg",
                        "@type":"dcterms:Image",
                        "format":"image/jpeg",
                        "width":200,
                        "height":287,
                        "service":{
                           "@context":"http://iiif.io/api/image/2/context.json",
                           "@id":"http://<IP_ADDRESS>:5004/bz%2F60%2Fcw%2F30%2Fw-intermediate_file.jp2",
                           "profile":"http://iiif.io/api/image/2/level2.json"
                        }
                     },
                     "on":"http://<IP_ADDRESS>:3000/concern/scanned_books/08612n57q/manifest/canvas/bz60cw30w"
                  }
               ],
               "width":200,
               "height":287
            },
            {
               "@id":"http://<IP_ADDRESS>:3000/concern/scanned_books/08612n57q/manifest/canvas/zs25x852h",
               "@type":"sc:Canvas",
               "label":"gray.tif",
               "images":[
                  {
                     "@id":"http://<IP_ADDRESS>:3000/concern/scanned_books/08612n57q/manifest/annotation/zs25x852h-image",
                     "@type":"oa:Annotation",
                     "motivation":"sc:painting",
                     "resource":{
                        "@id":"http://<IP_ADDRESS>:5004/zs%2F25%2Fx8%2F52%2Fh-intermediate_file.jp2/full/!200,200/0/default.jpg",
                        "@type":"dcterms:Image",
                        "format":"image/jpeg",
                        "width":200,
                        "height":256,
                        "service":{
                           "@context":"http://iiif.io/api/image/2/context.json",
                           "@id":"http://<IP_ADDRESS>:5004/zs%2F25%2Fx8%2F52%2Fh-intermediate_file.jp2",
                           "profile":"http://iiif.io/api/image/2/level2.json"
                        }
                     },
                     "on":"http://<IP_ADDRESS>:3000/concern/scanned_books/08612n57q/manifest/canvas/zs25x852h"
                  }
               ],
               "width":200,
               "height":256
            }
         ]
      }
   ]
}

That's correct. Were there changes in the code to make this happen? Not seeing them if so.

@jpstroop Wanted confirmation before I pushed up.

@jpstroop Should be ready for review.

:+1:


Incompatibility between Tensorflow and Theano

I've written a custom Layer for Keras: It combines a 2D convolution with a GRU and calculates the gates for the GRU with a convolution.
The new layer seems to work nicely. The code still needs to be cleaned up, but it is rather straightforward, the code is here:
http://pastebin.com/P7ZeiVjz
Here's how this would be used in a model:
inputs = Input((1, 40, 40, 40))

# now reshape to a sequence
reshaped = Reshape((40, 1, 40, 40))(inputs)

# now create a convnet which is fed the time-distributed data
# this model will perform a segmentation on a 2D slice of data.

conv_inputs = Input((1, 40, 40))
base_filters_size = 64
conv1 = Convolution2D(64, 3, 3, activation='relu', border_mode='same')(conv_inputs)
#conv1 = Convolution2D(64, 3, 3, activation='relu', border_mode='same')(conv1)
conv1 = Convolution2D(32, 3, 3, activation='relu', border_mode='same')(conv1)
#conv1 = Convolution2D(32, 3, 3, activation='relu', border_mode='same')(conv1)
conv1 = Convolution2D(3, 3, 3, activation='relu', border_mode='same')(conv1)
convmodel = Model(input=conv_inputs, output=conv1)

convmodel.summary()

#apply the segmentation to each layer
time_dist=TimeDistributed(convmodel)(reshaped)
temp_model=Model(input=inputs, output=time_dist)
temp_model.summary()

# now a model with the two GRUs which move in different directions

up=CGRU(go_backwards=False, return_sequences=True, name="up", input_shape=[40,3,40,40])(time_dist)
down=CGRU(go_backwards=True, return_sequences=True, name="down", input_shape=[40,3,40,40])(time_dist)

merged_updown=merge([up,down], mode="concat", concat_axis=2)
reshaped=Reshape((40,2,40,40))(merged_updown)

mergeconv_inputs=Input((2,40,40))
mergeconv=Convolution2D(1,3,3, activation="sigmoid", border_mode="same")(mergeconv_inputs)
merge_model=Model(input=mergeconv_inputs, output=mergeconv)

final_time_dist=TimeDistributed(merge_model)(reshaped)

output=Reshape((1,40,40,40))(final_time_dist)

model=Model(input=inputs, output=output)
print(model.summary())

This code runs perfectly fine on Theano.
The output of the final model.summary() is
Layer (type)                     Output Shape          Param #     Connected to
====================================================================================================
input_4 (InputLayer)             (None, 1, 40, 40, 40) 0
____________________________________________________________________________________________________
reshape_4 (Reshape)              (None, 40, 1, 40, 40) 0           input_4[0][0]
____________________________________________________________________________________________________
timedistributed_3 (TimeDistribute(None, 40, 3, 40, 40) 19971       reshape_4[0][0]
____________________________________________________________________________________________________
up (CGRU)                        (None, 40, 1, 40, 40) 0           timedistributed_3[0][0]
____________________________________________________________________________________________________
down (CGRU)                      (None, 40, 1, 40, 40) 0           timedistributed_3[0][0]
____________________________________________________________________________________________________
merge_2 (Merge)                  (None, 40, 2, 40, 40) 0           up[0][0]
                                                                   down[0][0]
____________________________________________________________________________________________________
reshape_5 (Reshape)              (None, 40, 2, 40, 40) 0           merge_2[0][0]
____________________________________________________________________________________________________
timedistributed_4 (TimeDistribute(None, 40, 1, 40, 40) 19          reshape_5[0][0]
____________________________________________________________________________________________________
reshape_6 (Reshape)              (None, 1, 40, 40, 40) 0           timedistributed_4[0][0]
====================================================================================================
Total params: 19990

But with Tensorflow as backend, somehow the shape inference is messed up:
There is this error message:

ValueError: Shapes (?, ?, 40, 40) and (40, ?, 40) are not compatible

This is really strange. It's not just a matter of reshaping, the number of dimensions is wrong, too.
I think I'm missing some important configuration.
Could you point me in the right direction ?
Here is the whole error message:

ValueError                                Traceback (most recent call last)
 in ()
28 # now a model with the two GRUs which move in different directions
29
---> 30 up=CGRU(go_backwards=False, return_sequences=True, name="up", input_shape=[40,3,40,40])(time_dist)
31 down=CGRU(go_backwards=True, return_sequences=True, name="down", input_shape=[40,3,40,40])(time_dist)
32
/usr/local/lib/python2.7/dist-packages/keras/engine/topology.pyc in call(self, x, mask)
513         if inbound_layers:
514             # this will call layer.build() if necessary
--> 515             self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
516             input_added = True
517
/usr/local/lib/python2.7/dist-packages/keras/engine/topology.pyc in add_inbound_node(self, inbound_layers, node_indices, tensor_indices)
571         # creating the node automatically updates self.inbound_nodes
572         # as well as outbound_nodes on inbound layers.
--> 573         Node.create_node(self, inbound_layers, node_indices, tensor_indices)
574
575     def get_output_shape_for(self, input_shape):
/usr/local/lib/python2.7/dist-packages/keras/engine/topology.pyc in create_node(cls, outbound_layer, inbound_layers, node_indices, tensor_indices)
148
149         if len(input_tensors) == 1:
--> 150             output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
151             output_masks = to_list(outbound_layer.compute_mask(input_tensors[0], input_masks[0]))
152             # TODO: try to auto-infer shape if exception is raised by get_output_shape_for
/usr/local/lib/python2.7/dist-packages/keras/layers/recurrent.pyc in call(self, x, mask)
211                                              constants=constants,
212                                              unroll=self.unroll,
--> 213                                              input_length=input_shape[1])
214         if self.stateful:
215             self.updates = []
/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.pyc in rnn(step_function, inputs, initial_states, go_backwards, mask, constants, unroll, input_length)
1193             parallel_iterations=32,
1194             swap_memory=True,
-> 1195             sequence_length=None)
1196
1197         if nb_states > 1:
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/rnn.pyc in _dynamic_rnn_loop(cell, inputs, initial_state, parallel_iterations, swap_memory, sequence_length, dtype)
1023     shape = _state_size_with_prefix(
1024         output_size, prefix=[const_time_steps, const_batch_size])
-> 1025     output.set_shape(shape)
1026
1027   final_outputs = nest.pack_sequence_as(
/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.pyc in set_shape(self, shape)
406         this tensor.
407     """
--> 408     self._shape = self._shape.merge_with(shape)
409
410   @property
/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_shape.pyc in merge_with(self, other)
568       except ValueError:
569         raise ValueError("Shapes %s and %s are not compatible" %
--> 570                          (self, other))
571
572   def concatenate(self, other):
ValueError: Shapes (?, ?, 40, 40) and (40, ?, 40) are not compatible

You might want to check your "image_dim_ordering" in ~/.keras/keras.json. For theano,  it is set to "th". When switching to tensorflow if you don't change it to "tf" and vice-versa, this kind of error occurs. So simply editing the file will fix your issue.
Hope this helps!

Thank you for the tip.
I had indeed hardcoded the theano image layout into the layer, but I assumed that this wouldn't matter, as long as "th" was selected in the keras.json.
I've now fixed the code and added support for both "th" and "tf" layout.
The model was also adapted: Now I've got two models, one for "th" and one for "tf".
With theano as backend, both image layouts run fine. The network is able to learn from data, the new layer seems to work.
But when I switch to tensorflow, it invariably crashes. The layout doesn't seem to have an effect.
Here is my new code:
import numpy as np
from keras import backend as K
from keras import initializations, activations
from keras.engine import InputSpec
from keras.layers.recurrent import Recurrent

# copied from the keras github source. removed lots of unnecessary (for me) code

# assuming a 2D Convolution was run by hand before this layer.
# please note that this has no variables of its own.
# TODO: incorporate the 2D Convolution into this layer

class CGRU(Recurrent):
    def __init__(self,
                 init='glorot_uniform', inner_init='orthogonal',
                 activation='tanh', inner_activation='hard_sigmoid', **kwargs):

        self.init = initializations.get(init)
        self.inner_init = initializations.get(inner_init)
        self.activation = activations.get(activation)
        self.inner_activation = activations.get(inner_activation)

        self.dim_ordering = K.image_dim_ordering()
        assert self.dim_ordering in {'tf', 'th'}, '`dim_ordering` must be in {"tf", "th"}.'

        # removing the regularizers and the dropout

        super(CGRU, self).__init__(**kwargs)

        # this seems necessary in order to accept 5 input dimensions
        # (samples, timesteps, features, x, y)
        self.input_spec = [InputSpec(ndim=5)]

    def build(self, input_shape):
        self.input_spec = [InputSpec(shape=input_shape)]

        if self.dim_ordering == "th":
            nb_features = input_shape[2]
            x_dim = input_shape[3]
            y_dim = input_shape[4]

            # the input_dim was broken and didn't cause any problems. I'm commenting it out to see
            # whether it is really necessary
            # self.input_dim = (3,x_dim, y_dim)

            # moved here from the constructor. Curiously it seems like batch_size has been removed from here.
            # I also don't understand why I don't need to specify the time_sequence if there is the flag
            # output_sequence is set to True
            # TODO: debug where this is used
            self.output_dim = [1, x_dim, y_dim]

        elif self.dim_ordering == "tf":
            nb_features = input_shape[4]
            x_dim = input_shape[2]
            y_dim = input_shape[3]

            # see above
            # self.input_dim = (x_dim, y_dim, 3)

            # see above
            self.output_dim = [x_dim, y_dim, 1]

        assert (nb_features == 3, "the CGRU expects 3 input features")

        if self.stateful:
            self.reset_states()
        else:
            # initial states: all-zero tensor of shape (output_dim)
            self.states = [None]

    def reset_states(self):

        assert self.stateful, 'Layer must be stateful.'
        input_shape = self.input_spec[0].shape
        if not input_shape[0]:
            raise Exception('If a RNN is stateful, a complete ' +
                            'input_shape must be provided (including batch size).')

        if self.dim_ordering == "th":
            nb_features = input_shape[2]
            x_dim = input_shape[3]
            y_dim = input_shape[4]

        elif self.dim_ordering == "tf":
            nb_features = input_shape[4]
            x_dim = input_shape[2]
            y_dim = input_shape[3]

        if hasattr(self, 'states'):
            K.set_value(self.states[0],
                        np.zeros((input_shape[0], x_dim, y_dim)))
        else:
            self.states = [K.zeros((input_shape[0], x_dim, y_dim))]

    def preprocess_input(self, x):
        # TODO: if at some point, this layer does calculations, revisit and optimise this code
        return x

    def step(self, x, states):
        h_tm1 = states[0]  # previous memory

        # usually here would be matrix multiplications for the different gates
        # I assume that a convolution before this layer has prepared 3 features per coordinate
        # I'll simply use those 3 features
        # in fact: this layer doesn't have any trainable weights

        # TODO: is an if else problematic wrt performance.
        # TODO: since I'm currently trying to get the layer to work with tensorflow, I'll now hardcode for tf
        if self.dim_ordering == "th":
            z = self.inner_activation(x[:, 0, :, :])
            r = self.inner_activation(x[:, 1, :, :])
            hh = self.activation(x[:, 2, :, :])
        else:  # self.dim_ordering == "th":
            z = self.inner_activation(x[:, :, :, 0])
            r = self.inner_activation(x[:, :, :, 1])
            hh = self.activation(x[:, :, :, 2])

        h = z * h_tm1 + (1 - z) * hh
        return h, [h]

    def get_initial_states(self, x):
        initial_state = K.zeros_like(x)  # (samples, timesteps, input_dim)
        # input_dim = (3, x_dim, y_dim) for th
        # input_dim = (x_dim, y_dim, 3) for tf
        # please note: the number of features is expected to be 3, one feature for each gate

        if self.dim_ordering == "th":
            initial_state = K.sum(initial_state, axis=(1, 2))  # (samples, x_dim, y_dim)
        else:  # "tf"
            initial_state = K.sum(initial_state, axis=(1, -1))  # (samples, x_dim, y_dim)

        return [initial_state]

    def get_output_shape_for(self, input_shape):
        if self.return_sequences:
            return tuple([input_shape[0], input_shape[1]] + self.output_dim)
        else:
            return tuple([input_shape[0]] + self.output_dim)

    def get_config(self):
        config = {'output_dim': self.output_dim,
                  'init': self.init.__name__,
                  'inner_init': self.inner_init.__name__,
                  'activation': self.activation.__name__,
                  'inner_activation': self.inner_activation.__name__}

        # removed the various regularizers and dropouts.
        # surely this isn't needed if not present ?
        base_config = super(CGRU, self).get_config()
        return dict(list(base_config.items()) + list(config.items()))

Sorry to flood you with so much code, I've tried to comment and format it as well as possible.
Here's the setup, this is how I use the code:
if K.image_dim_ordering()=="tf":
# this is the actual input, fed to the network
inputs = Input((40, 40, 40, 1))
# now reshape to a sequence
#reshaped = Reshape((40, 1, 40, 40))(inputs)

# now create a model like uNet which is fed the time-distributed data
# this model will perform a segmentation on a 2D slice of data.

conv_inputs = Input((40, 40, 1))
base_filters_size = 16
conv1 = Convolution2D(16, 3, 3, activation='relu', border_mode='same')(conv_inputs)
#conv1 = Convolution2D(64, 3, 3, activation='relu', border_mode='same')(conv1)
conv1 = Convolution2D(16, 3, 3, activation='relu', border_mode='same')(conv1)
#conv1 = Convolution2D(32, 3, 3, activation='relu', border_mode='same')(conv1)
conv1 = Convolution2D(3, 3, 3, activation='relu', border_mode='same')(conv1)
convmodel = Model(input=conv_inputs, output=conv1)

convmodel.summary()

#apply the segmentation to each layer
time_dist=TimeDistributed(convmodel)(inputs)

# now a model with the two GRUs which move in different directions
time_dist=Reshape((40,40,40, 3))(time_dist)

# The next line will crash
up=CGRU(go_backwards=False, return_sequences=True, name="up", input_shape=[40,40,40,3])(time_dist)
down=CGRU(go_backwards=True, return_sequences=True, name="down", input_shape=[40,40,40,3])(time_dist)

# merge the two outputs
merged_updown=merge([up,down], mode="concat", concat_axis=2)
reshaped=Reshape((40,40,40,2))(merged_updown)

# and apply a final convolution to map from 2 features back to 1
mergeconv_inputs=Input((40,40,2))
mergeconv=Convolution2D(1,3,3, activation="sigmoid", border_mode="same")(mergeconv_inputs)
merge_model=Model(input=mergeconv_inputs, output=mergeconv)

final_time_dist=TimeDistributed(merge_model)(reshaped)

output=Reshape((40,40,40,1))(final_time_dist)

model=Model(input=inputs, output=output)
print(model.summary())
model.compile(optimizer=Adam(lr=1e-4), loss=dice_coef_loss, metrics=[dice_coef])

The very same code, with the "tf" image layout, runs fine under theano and crashes with tensorflow.
The error message is:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-8-295ff1bf414f> in <module>()
     25     # now a model with the two GRUs which move in different directions
     26     time_dist=Reshape((40,40,40, 3))(time_dist)
---> 27     up=CGRU(go_backwards=False, return_sequences=True, name="up", input_shape=[40,40,40,3])(time_dist)
     28     down=CGRU(go_backwards=True, return_sequences=True, name="down", input_shape=[40,40,40,3])(time_dist)
     29

/usr/local/lib/python2.7/dist-packages/keras/engine/topology.pyc in __call__(self, x, mask)
    513         if inbound_layers:
    514             # this will call layer.build() if necessary
--> 515             self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
    516             input_added = True
    517

/usr/local/lib/python2.7/dist-packages/keras/engine/topology.pyc in add_inbound_node(self, inbound_layers, node_indices, tensor_indices)
    571         # creating the node automatically updates self.inbound_nodes
    572         # as well as outbound_nodes on inbound layers.
--> 573         Node.create_node(self, inbound_layers, node_indices, tensor_indices)
    574
    575     def get_output_shape_for(self, input_shape):

/usr/local/lib/python2.7/dist-packages/keras/engine/topology.pyc in create_node(cls, outbound_layer, inbound_layers, node_indices, tensor_indices)
    148
    149         if len(input_tensors) == 1:
--> 150             output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
    151             output_masks = to_list(outbound_layer.compute_mask(input_tensors[0], input_masks[0]))
    152             # TODO: try to auto-infer shape if exception is raised by get_output_shape_for

/usr/local/lib/python2.7/dist-packages/keras/layers/recurrent.pyc in call(self, x, mask)
    211                                              constants=constants,
    212                                              unroll=self.unroll,
--> 213                                              input_length=input_shape[1])
    214         if self.stateful:
    215             self.updates = []

/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.pyc in rnn(step_function, inputs, initial_states, go_backwards, mask, constants, unroll, input_length)
   1193             parallel_iterations=32,
   1194             swap_memory=True,
-> 1195             sequence_length=None)
   1196
   1197         if nb_states > 1:

/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/rnn.pyc in _dynamic_rnn_loop(cell, inputs, initial_state, parallel_iterations, swap_memory, sequence_length, dtype)
   1023     shape = _state_size_with_prefix(
   1024         output_size, prefix=[const_time_steps, const_batch_size])
-> 1025     output.set_shape(shape)
   1026
   1027   final_outputs = nest.pack_sequence_as(

/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.pyc in set_shape(self, shape)
    406         this tensor.
    407     """
--> 408     self._shape = self._shape.merge_with(shape)
    409
    410   @property

/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_shape.pyc in merge_with(self, other)
    568       except ValueError:
    569         raise ValueError("Shapes %s and %s are not compatible" %
--> 570                          (self, other))
    571
    572   def concatenate(self, other):

ValueError: Shapes (?, ?, 40, 40) and (40, ?, 40) are not compatible


[tabs] Various issues

Hi,
First of all thanks for providing a tabs plugin for vitepress!
I tried it out in an effort to migrate an existing documentation from VuePress 1 to Vitepress/trying to find a replacement for vuepress-plugin-tabs-less. I ran into a couple of issues. Let me know if you would prefer separate issues for them, but to me it looks like the plugin is currently pretty broken, so not sure which of these make sense to be fixed independently.
Things I noticed:

Using spaces in the title of the tabs is currently not supported, which is IMHO something that should be supported
Using more advanced Markdown features such as a :::tip section within a tab also causes parser errors
Using only the example from the documentation leads to the following error at runtime (causing a blank page)

useTabs.ts:39 Uncaught (in promise) Error: [vitepress-plugin-tabs] TabsSharedState should be injected
    at useTabs (useTabs.ts:39:11)
    at setup (PluginTabs.vue:12:30)
    at callWithErrorHandling (chunk-WICYTZ2N.js?v=af802301:1565:18)
    at setupStatefulComponent (chunk-WICYTZ2N.js?v=af802301:8631:25)
    at setupComponent (chunk-WICYTZ2N.js?v=af802301:8592:36)
    at mountComponent (chunk-WICYTZ2N.js?v=af802301:6997:7)
    at processComponent (chunk-WICYTZ2N.js?v=af802301:6963:9)
    at patch (chunk-WICYTZ2N.js?v=af802301:6436:11)
    at mountChildren (chunk-WICYTZ2N.js?v=af802301:6680:7)
    at mountElement (chunk-WICYTZ2N.js?v=af802301:6587:7)

Reproduction: https://stackblitz.com/edit/github-zbjrsg-ytw6ky?file=index.md

Hi

Using spaces in the title of the tabs is currently not supported, which is IMHO something that should be supported

I simply didn't implement that as I created these plugins as an experiment. Contributions are welcome.

Using more advanced Markdown features such as a :::tip section within a tab also causes parser errors

Tabs are already a container thus I didn't allow them to exist inside the tab.

Using only the example from the documentation leads to the following error at runtime (causing a blank page)

It seems vite.optimizeDeps.exclude is required.
export default defineConfig({
  vite: {
    optimizeDeps: {
      exclude: ['vitepress-plugin-tabs']
    }
  }
})

https://stackblitz.com/edit/github-zbjrsg-cg6vvk?file=.vitepress%2Fconfig.ts,.vitepress%2Ftheme%2Findex.ts
I guess it would fixed if this Symbol is changed into a string.
https://github.com/sapphi-red/vitepress-plugins/blob/7f4c8fe683650f6e8b82addc92c187fe07801fd4/packages/vitepress-plugin-tabs/src/client/useTabs.ts#L6

Fixed with #24


[Graphics] Template games don't render

Release Type: Github
Version: Latest as of writing
Platform(s): Most likely all of them
Describe the bug
The game doesn't render, a warning related to camera slot is reported to the console.
To Reproduce
Steps to reproduce the behavior:

Build template
Play template
Observe the console warning and black screen

Expected behavior
Renders as it did before.
Additional context
This issue is most likely related to #805 could you take a look @azeno ?

Argh sorry will have a look.

Thanks azeno !
This issue should be fixed through #826 , feel free to reopen if it isn't


how to solve core of  SSL_CTX_new?

The call stack like this:
#0  0x00007f3354a48fd7 in raise () from /lib64/libc.so.6
#1  0x00007f3354a4a3aa in abort () from /lib64/libc.so.6
#2  0x00007f3354a88164 in __libc_message () from /lib64/libc.so.6
#3  0x00007f3354a8d5d6 in malloc_printerr () from /lib64/libc.so.6
#4  0x00007f3354a8d972 in malloc_consolidate () from /lib64/libc.so.6
#5  0x00007f3354a8f3a5 in _int_malloc () from /lib64/libc.so.6
#6  0x00007f3354a91460 in malloc () from /lib64/libc.so.6
#7  0x00007f3356dcf268 in CRYPTO_malloc () from /lib64/libcrypto.so.1.0.0
#8  0x00007f3356b35245 in ssl_create_cipher_list () from /lib64/libssl.so.1.0.0
#9  0x00007f3356b2d657 in SSL_CTX_new () from /lib64/libssl.so.1.0.0
#10 0x000000000051f8df in Client::InitCtx (this=0x256a2e0) at Client.cpp:155

The openssl version is OpenSSL 1.0.2p-fips

The callstack shows that an error message is printed to stderr in frame 3 (malloc_printerr). It’s most likely some memory corruption detected by the C runtime which is causing the abort. Check the console output for the error message if available. If not try to figure it out by inspecting frame 3 in the debugger.

How to inspect frame 3 in the debugger,  i haven't source code.

You might be able to run this with address sanitizer or under valgrind.

If i call SSL_CTX_new in thread A, and call SSL_CTX_free in thread B, will it cause the above problem?

If i call SSL_CTX_new in thread A, and call SSL_CTX_free in thread B, will it cause the above problem?

You cannot share a single SSL_CTX object between multiple threads at the same time. It is allowed to call SSL_CTX functions on different threads as long as each thread is using a different SSL_CTX object.
Since SSL_CTX_new() is creating a new SSL_CTX there is never any possibility that the SSL_CTX being created has been shared with other threads yet, so this is always safe.

Until SSL_CTX_new() returns, how does a different thread get the pointer to the context to free it?
I'm sure it's possible to do this, I don't think it would be easy or accidental.

OpenSSL does not use a per-thread malloc heap, so to answer your original question it should be fine.

Do you mean the memory allocated by thread A, the memory free must in same thread not another thread?

Do you mean the memory allocated by thread A, the memory free must in same thread not another thread?

No, I mean the exact opposite. Memory allocated in thread A can be freed by any thread, including A B C and so on.
If that doesn't work, it's not OpenSSL's fault, but the platform it is running on is doing something strange.

This would depend on your system's C library: OpenSSL uses the system's malloc/free by default.

This is not like Windows, etc., having a malloc heap for a shared library. In Posix, memory allocation is global no matter which thread allocates, uses, or frees it.

My system is
LSB Version:	n/a
Distributor ID:	SUSE
Description:	SUSE Linux Enterprise Server 12 SP5
Release:	12.5
Codename:	n/a
Linux linux-ra5s 4.12.14-120-default #1 SMP Thu Nov 7 16:39:09 UTC 2019 (fd9dc36) x86_64 x86_64 x86_64 GNU/Linux

Like I said, it should work.

@okayabc my understanding is that your call stack shows only the location where the memory corruption has been detected, which in many cases can be logically unrelated to the location where the corruption was caused. The malloc does some internal consolidation which aborts deliberately because it detects some problem.
You are looking at the wrong location, the error happened earlier (maybe even in another thread). Look out for malloc_printerr() lines in malloc_consolidate().

@mspncp Currently i can't get the output of malloc_printerr.

You might get some information if you run your program with the address sanitizer or under valgrind, as @paulidale suggested. We can't help you with the details, you need to google it, or ask a friend or colleague for help.

ok, thanks all!


How to write a media-query custom rule?

For example, to match this className:
<div className="mb_20_24-500_1280" />

These need to be generated by rules:
@media(min-width: 500px) {
  margin-bottom: 20px;
}
@media((min-width: 500px) and (max-width: 1280px)) {
  margin-bottom: calc(20px + (24 - 20) / (1280 - 500) * (100vw - 500px));
}
@media(min-width: 1280px) {
  margin-bottom: 24px;
}

So the margin-bottom of this element will change smoothly from 20px to 24px when window width is between 500 and 1280.
I realize one of the advantages unocss has over tailwindcss is that I can write my own parser rule, but it's not able to add media-query rule now, right?

@jiangyijie27 now you can do multiple rules and media queries in v0.3.0 https://github.com/antfu/unocss#full-controlled-rules


Parser error when using "when" in diverts

When "when" is used in a divert like this:
-> some block {when some_condition}
the parser gives an error. When "when" is omitted, everything works.
Here is a script which demonstrates the problem.
{set foo = 1}
-> main

== main
foo equals 1 {foo == 1} -- this works
-> test one {foo == 1} <- -- this works without "when"
foo equals 2 {when foo == 2} -- this also works
-> test two {when foo == 2} -- parser error on Clyde CLI 0.1.0

this is the end of the test
-> END

== test one
value of foo is one
{set foo = 2}
<-

== test two
value of foo is two
<-

which gives the error: Error: Unexpected token "when" on line 8 column 93. Expected }

After some more testing, I found more problems with this particular pattern of -> block {condition}.
here is another test script:
{set foo = true}

test 1
{foo} -- this works
    -> bar

test 1b
{foo} -> bar -- this also works

test 2
-> bar {foo} -- this comment on the same line somehow prevents the parser error

test 3 -- the following line will cause a parser error
-> bar {foo}

test 4
-> bar {foo} <- -- this variation works, but is not equivalent to the other lines

test is completed
-> END

== bar
this is a test
<-

which gives the error: ERROR: Unexpected token "text" on line 16 column 1. Expected INDENT

Thanks for finding this issue @lapspider45  and giving examples.
The problem here is that I believe I didn't add support to conditions after diverts. You are able to {foo} -> bar, but not -> bar {foo}. I'll take a look into this.
About test 4 -> bar {foo} <-. That's a tricky situation.
The thing is, logic blocks are intended to be placed before or after a line. In that case, it is in the middle.  The compiler wouldn't know if you meant
-> bar {foo}
<-

or
-> bar
{foo} <-

As at the moment the compiler only supports condition before the divert, the result is equivalent to the second one.
But this actually makes me think of a second issue, -> bar <- {foo}. After I fix it, the condition will only apply to the divert to parent, which may be misleading. I'll have to put some thought on this one.
I'll let you know when the fix is ready :+1:
Cheers

The fix is already available in the latest NPM libraries and it just went live to Godot Asset Library https://godotengine.org/asset-library/asset/870
Thanks again :partying_face:


Change Account Callback

...to not set fee currency if there is native balance in the account.

fixes issues in https://github.com/galacticcouncil/Basilisk-node/pull/514

Note that session benchmarks will need to run again to update weights.


adding playerIndex to message

Make the ReplayParser parse the player of the message correctly.

What are you wanting the player index for?
If 'MessageSender' is not pointing to the correct player, we should fix that directly

I'm developing a tool that can save the parsing result to a local db. Users can search replays with some conditions( like chat message ).
I saved the chat messages to a seperate database table. In order to know which player the message is belong to, it's convenice to have a player index field in Message , since the replay storing the players using an array.
I can also use the MessageSender property, but in that case I have to write a for-loop and compare the player name + BattleTag to locate the player.
Maybe there's an easy way, I'm glad to know.


delete duplicate GAME-RULES.MD

#993
Relevant #993

Resolves #993
Delete Duplicate GAME-RULES.MD

Please check the following

[x] Do the tests still pass? (see Run the Tests)
[x] Is the code formatted properly? (see Linting (Formatting))
For New Features:

[x] Have tests been added to cover any new features or fixes?
[x] Has the documentation been updated accordingly?

Please describe additional details for testing this change

Thank You So Much for Approving the Pull Request Respectively.

Sorry a duplicate came in while this was closed/reopened. Thanks for your contribution, this was indeed the right change. Feel free to grab any other issue


wast lexer size assert not compatible with randomized struct layouts

https://github.com/bytecodealliance/wasm-tools/blob/d68b3828f0ebbc46ce0b4fff9c4d5d881487114a/crates/wast/src/lexer.rs#L64-L66
breaks when compiling with layout randomization
If the assert is necessary for correctness then repr(C) should be used to get a deterministic layout.
If it's just an optimization then it's probably better to just check it in tests/CI.
Found this while trying to build the rust compiler with randomized layouts. https://github.com/rust-lang/rust/pull/101339

Thanks for the report, and sorry for the breakage! Is this something that you'd prefer to see urgently fixed? If so I can work on getting a patch release out for rust-lang/rust to update to. (it's a bit nonstandard to do patch releases in this repo unfortunately but I can make it work)

Not urgent, I can probably work around it for now by excluding a bunch of things from randomization.
I see that wast is updated monthly or so, that should be fine.

Looks like that wasn't sufficient, there are another compile time size assert:
https://github.com/bytecodealliance/wasm-tools/blob/537ca81024e47b349551a95ab6ecba229f3795ba/crates/wast/src/core/expr.rs#L1214

There are more
https://github.com/bytecodealliance/wasm-tools/blob/537ca81024e47b349551a95ab6ecba229f3795ba/crates/wasmparser/src/validator/types.rs#L109-L111
generally all such asserts should be moved to tests, otherwise the crates break under layout randomization.

Oops, thanks for pointing that out! I've submitted https://github.com/bytecodealliance/wasm-tools/pull/1894 to relax some more assertions.
To confirm though, in the second case you mentioned, that assert is technically ok to leave as a const-assert right? It's an assertion about a #[repr(C)] structure which -Zrandomize-layouts doesn't affect, right?

Oh yeah, I didn't look at the struct declaration. That one is fine.


upgrade upgrade hudi version from 0.13.0 to 0.13.1

Proposed changes
Upgrade hudi version from 0.13.0 to 0.13.1, and keep the hudi version of jni scanner the same as that of FE.
This may fix the bug of the table schema is not same as parquet schema.

run buildall


Make separate API for encryption with detached payload

The convention used for signing and MAC is to have one API for use without a detached payload and separate API for one with a detached payload.  The two APIs are similar, but not the same.  The same should be done for encryption and decryption.

Fixed by #159


Make color and shape of error markers configurable

From @Franziskus1988 on March 3, 2016 14:6
Sometimes it's difficult to find/see the green marked linting errors. Especially when a file contains a lot of comments (normally green).
Would it be possible to make the color and maybe the line thickness configurable?
Copied from original issue: Microsoft/vscode-tslint#31

Rendering diagnostics is a general vscode feature.
Moving to vscode

Is there any workaround for this in the meantime? With dark themes it is very easy to miss the green-on-black, rendering tslint far less useful than it should be.

@bmayen
Yes, there is :)

Install Custom CSS and JS Loader extension, then use the following style as the custom CSS:
.monaco-editor.vs-dark .greensquiggly,
.monaco-editor.vs .greensquiggly {
  /* yellow squiggly */
  background: url("data:image/svg+xml;charset=utf-8,<svg xmlns='http://www.w3.org/2000/svg' height='3' width='6'><g fill='#FF0'><path d='M5.5 0l-3 3H1.1l3-3z'/><path d='M4 0l2 2V.6L5.4 0zM0 2l1 1h1.4L0 .6z'/></g></svg>") repeat-x 0 100%;

  /* rectangle */
  /* background: none;
     border: 1px solid yellow; */

  /* underline */
  /* background: none;
     border-bottom: 1px solid yellow; */
}

Please let me know whether it worked :]

Thank you! Will let you know if I have any problems.

@lambda4fun: I've installed this extension, created the custom css file, updated settings to point to that file at "vscode_custom_css.imports": ["file:///C:/Users/me/.vscode/custom.css"], and started VSCode as administrator and enabled the the extension. It doesn't inject the css though. I open developer tools and there is a VSCODE-CUSTOM-CSS comment block, but it's empty. Searching for the css defined in the custom.css file also returns 0 results.

@bmayen
Maybe, there's one extra '/' in the path. Try changing the file:/// to file://

@bmayen
Try executing 'Update Custom CSS and JS' from Command Palette.

That did it. Thank you, good sir!

https://github.com/Microsoft/vscode/issues/9819


How to add hands pose?

Hi, great work!
when I set no_hand to False, I meet error about model param dimension. Could you run demo with hands, and fix bug?

Hi, thanks for your interest in Animate-X.
no_hand is a parameter we left over during debugging, and it has no actual significance in the current code.

Hi,
The results on the hands are not good. I find input_dim in DwposeEncoder is 36, but If you add the features of the hands and face, the dimension is 128, it dose not work. Will you release the checkpoint with 128?

Your understanding about the input_dim in DwposeEncoder is right.
But I am sorry that we currently have no plan to release the checkpoint with 128.
If we have enough time later, we'd like to try your suggestion~


Preserve mem format

Codecov Report

Merging #255 (74aa5ed) into master (852439b) will decrease coverage by 0.05%.
The diff coverage is 90.00%.

@@            Coverage Diff             @@
##           master     #255      +/-   ##
==========================================
- Coverage   84.48%   84.42%   -0.06%
==========================================
  Files          25       25
  Lines        1573     1567       -6
  Branches      357      357
==========================================
- Hits         1329     1323       -6
  Misses        140      140
  Partials      104      104

Impacted Files
Coverage Δ

torch_optimizer/adabound.py
81.69% <66.66%> (ø)

torch_optimizer/qhadam.py
83.58% <100.00%> (ø)

torch_optimizer/qhm.py
76.74% <100.00%> (ø)

torch_optimizer/sgdp.py
76.71% <100.00%> (-0.92%)
:arrow_down:

torch_optimizer/swats.py
73.17% <100.00%> (ø)

torch_optimizer/adamp.py
89.87% <0.00%> (-0.38%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 852439b...74aa5ed. Read the comment docs.

Codecov Report

Merging #255 (74aa5ed) into master (852439b) will decrease coverage by 0.05%.
The diff coverage is 90.00%.

@@            Coverage Diff             @@
##           master     #255      +/-   ##
==========================================
- Coverage   84.48%   84.42%   -0.06%
==========================================
  Files          25       25
  Lines        1573     1567       -6
  Branches      357      357
==========================================
- Hits         1329     1323       -6
  Misses        140      140
  Partials      104      104

Impacted Files
Coverage Δ

torch_optimizer/adabound.py
81.69% <66.66%> (ø)

torch_optimizer/qhadam.py
83.58% <100.00%> (ø)

torch_optimizer/qhm.py
76.74% <100.00%> (ø)

torch_optimizer/sgdp.py
76.71% <100.00%> (-0.92%)
:arrow_down:

torch_optimizer/swats.py
73.17% <100.00%> (ø)

torch_optimizer/adamp.py
89.87% <0.00%> (-0.38%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 852439b...74aa5ed. Read the comment docs.


Title cannot be inferred from Markdown with Documenter

In Documenter.jl the title of a page can be inferred from the first header. However, when converting a notebook due to the HTML structure the inferred title is just -. Would there be a way to solve that? Here is a screenshot of what I mean:

It does work, see for example TuringGLM.jl. Make sure that you pass pairs to pages like
pages = ["Home" => "index.md", "Tutorials" => ["Max" => "max.md", "Min" => "min.md"]
makedocs(...; pages, ...)

Does that solve your issue?

Not really :sweat_smile:
What I meant was that if I have a folder full of notebooks and do not necessarily have names for each of them so writing pairs is not really an option.
Using documenter, if I passed a file my_example.md :
# Title1

...content

as a String and not Pair{String,String} it would automatically give it the title Title1.

That's unfortunate. Still, I think an important part of software development is to say no. In this case, I'm gonna say no because I don't see a way in which this can be elegantly handled at the PlutoStaticHTML side. In the worst-case, the PlutoStaticHTML logic would differ from the Documenter logic which causes things to behave unexpectedly. Taking in Documenter as a dependency also seems too heavy-handed because many use-cases of this package don't use the Documenter output. Also, I would like to avoid Requires.
What I do like about passing pairs of "title => filename" is that it is very explicit. For example, what we did is to define our notebook titles as, for example, ["Foo", "Bar and baz"] and generate the filenames from that as ["foo", "bar-and-baz"].
I'm gonna close for now. Feel free to change my mind and/or reopen.

Thanks for the answer! I also thought this could actually be solved on the Documenter side! I fully understand that this is not a priority for you!


Better description/organization of developer install

Currently, there are a few issues with the directions about how developers can install prompt2model:

We have no description in contributing.md about how we can install the toolkit locally
pyprojects.toml doesn't have requirements for developer tools such as pre-commit and pytest. We can add these with pyproject.optional-dependencies: https://setuptools.pypa.io/en/latest/userguide/dependency_management.html

Once done, we should be able to run pip install ".[dev]" to install the requirements and the development dependencies, and add these directions to contributing.md

I'll take this one up too @neubig @viswavi


Merged PR 1712 introduces a new issue

Please see my comment here: https://github.com/bluefireteam/audioplayers/pull/1712#issuecomment-1964860737
If the audioplayer is stopped and seek is called, the process will hang for 30 seconds (defined here: https://github.com/bluefireteam/audioplayers/blob/81e5ea542578f27c558f9a049996ecd8cb95c002/packages/audioplayers/lib/src/audioplayer.dart#L291) and raise a TimeoutException when the future fails to complete.
Prior to this PR, the hang would not occur.
Note: I wasn't exactly sure where to file this since it involves an unreleased version of AudioPlayers, with a PR that was approved and merged.  If there's a better way to log this issue, let me know, thanks!

@japostoles can you fill the complete issue template, plz? This gives us more detailed information about what might cause the issue (e.g. what OS and Flutter version you're using). Thank you :)


container sanity_check() error when babs-init

Hi @djarecka !
Regarding the container error you mentioned in PR #19, I guess it's because I've added a method called sanity_check() for Container class in PR #19. It performs some sanity checks when babs-init clones the container DataLad dataset. It might not be robust enough.
The error regarding analysis/containers/.datalad/environments/not_really_an_image.simg/image might come from this line, but I'd love to know more details of the error! Could you please say a bit more regarding: e.g., how you prepared your container datalad dataset, the command you used when calling BABS (including the --container_name you specified), the error messages, etc?
Moving this sanity checks earlier would be nice, but I'm just concerned it might not be able to cover cases that the container datalad dataset comes from a remote link (e.g., OSF, github, etc), so it will be hard for BABS to perform sanity checks before it clones to local BABS project.
Thank you so much!
Chenying

Hi @djarecka I think we can close this issue then after debugging it on zoom just now! I'm closing it now but feel free to let me know if you have further issues regarding this!


Cilium envoy crd

Note: Generic fixes in this series are to be reviewed in their own PRs. This PR will be rebased as these PRs are merged:

[x] https://github.com/cilium/cilium/pull/17591
[x] https://github.com/cilium/cilium/pull/17592
[x] https://github.com/cilium/cilium/pull/17593
[x] https://github.com/cilium/cilium/pull/17595
[x] https://github.com/cilium/cilium/pull/17596
[x] https://github.com/cilium/cilium/pull/17720
[x] https://github.com/cilium/cilium/pull/17718
[ ] https://github.com/cilium/cilium/pull/17758
[ ] https://github.com/cilium/cilium/pull/17637

Add alpha-level support for arbitrary configurations for the Cilium Envoy host proxy. The new CiliumEnvoyConfig CRD supports listeners, routes, clusters, and "cluster endpoint sets" (ClusterLoadAssignment) xDS resource types. In addition a listener configured in a CRD can be bound to k8s services, causing load balancing of the services to be postponed to the Envoy listener. The listener must be configured to route to the service backends via a clusters named as "namespace/service". Cilium agent will automatically push a simple EDS resources with the service backends matching those names. The CRD can use any other load balancing strategy via configuring a Cluster and ClusterLoadAssignment resources using some other name.
Cilium agent will automatically fill in the listener address and xDS config source information, if missing.
Limitations of this release include:

Listeners on pod ingress will not work yet (not tested)
For egress, normal bpf datapath policy enforcement is performed after the L7 LB proxy by tail calling from bpf_host to source bpf_lxc. This has the effect where the Envoy proxy will return a 503 reply if L3 to a backend is dropped.
There is no K8s level validation on the Envoy resources in the new CRD. Any errors will only be visible in Cilium agent log error/warning messages.

Datapath implementation adds a new flag to the main LB map value that causes LB to be bypassed in the socket layer, and the traffic to be redirected to the Envoy listener port on the TC layer. Redirection is done using the Linux transparent proxy mechanism so the Envoy listener receives the traffic with the original destination address (e.g., the ClusterIP of the k8s service).
The new datapath flag is visible in cilium bpf lb list command output as l7-load-balancer.
A new proxy magic mark value is used for upstream connections from the L7 LB proxy. This mark includes the source endpoint ID so that the datapath can tail call to the source pod's bpf egress path for policy enforcement.
A new CT table value flag from_l7lb is added to allow reply packets to be forwarded back to the L7 LB proxy.
CEC CRD can specify backendServices causing backends of those services to be synced with Envoy. This is useful for configuring an k8s-level Ingress service that load balances to backends of other services.
A CEC CRD listener will be reachable on a NodePort if services contains a service with a node port. This is useful for configuring an k8s-level Ingress service, e.g., of type LoadBalancer.
Note: Currently service redirected to L7 LB must have backends for Cilium Agent's service cache to wire the service to eBPF datapath. L7 LB continues to work even after such backends are removed. Best practice for now for K8s-level ingress service is to define a dummy backend in some IP address and never use it as a destination in the Listener route configuration. Future versions will lift this requirement.
Note: CEC CRD service access from a local node host namespace requires Linux kernel 5.12 or newer, and may not work properly in Kind.
Note: Service backends see traffic from external sources via Node Port as coming from the local or remote node (node/remote-node). This is not ideal for policy enforcement.
Examples of the new CRDs are available at https://github.com/cilium/cilium-cli/pull/550 (see this PR description for demo instructions):

Simple prometheus listener for the host port 9090: connectivity/manifests/envoy-prometheus-metrics-listener.yaml
Prometheus listener using multiple xDS resources: connectivity/manifests/envoy-prometheus-metrics-listener-multiresource.yaml
L7 LB CRD for the cilium-test/echo-same-node and cilium-test/echo-other-node services deployed by cilium connectivity test: connectivity/manifests/envoy-test.yaml

HTTP paths starting with /foo is always rewritten to /
Both services are 50/50 load balanced to the backends of both services

K8s Ingress service my-ingress:80 and corresponding CEC CRD building on top of the previous example: connectivity/manifests/envoy-ingress-test.yaml

Initial support for CiliumEnvoyConfig CRD with L7 load balancing is added.

test-me-please

Just from reading commit messages it seems like quite a few commits could be split into a separate PR that can be merged ahead of this one, to it looks that would at least half of the commit that are currently included here.

This is still very much a work-in-progress. I'll clean and split this up in a later stage.

At first glance, I guess the initial review pass should maybe look over the new CRD and check that we're happy with the way those configurations would work? Maybe @cilium/sig-k8s members can help with that?

Normal bpf datapath policy enforcement is bypassed. However, the cilium.network (Network) and cilium.l7policy(HTTP) filters can be configured to perform policy enforcement on the CRD listeners. The required Cilium listener filter is automatically injected to the CRD listener configuration.

From a functional perspective, I don't think it matters whether BPF or Envoy performs the network filtering. The danger is that someone configures the envoy setup and silently breaks policy. Should we just provide default examples that always have cilium.network configuration to ensure that it's there, and/or warn users in the documentation that failure to configure that properly will cause policy enforcement to break? Or auto-inject this if there's any Envoy configuration? Or should we add warnings / status checks that report a red flag if this new functionality is used without configuring cilium.network?
One other related comment here, Cilium always applies service translation first and then applies network policy based on the pod<->pod traffic (ie not pod->service). Is that constraint retained here?
Seems like this should work with BPF tproxy without any further changes, though the only way to confirm this would be to enable the combination of the flags for bpf tproxy + this new functionality and test it. Currently there is only one test in test/k8sT/Policies.go that does this, grep for bpf.tproxy for more details.
Other reviewers may benefit from some combination of reviewing this commit-by-commit and/or using git diff  HEAD~13..HEAD -- ':(exclude)vendor/' to narrow down the changed files (down to order ~2-3K LoC instead of the 65K+ when considering vendor changes).

One other related comment here, Cilium always applies service translation first and then applies network policy based on the pod<->pod traffic (ie not pod->service). Is that constraint retained here?

I forgot that this part of the PR was half baked in the haste of getting the PR out :-)
Actually, no. The whole point of postponing the load balancing decision to L7 is to not do the service translation before the L7 proxy. To retain the current model we'll need to do one of these:

Add ClusterIP/NodePort/HostPort level policies to the bpf datapath so that we can do policy enforcement before the L7 proxy

Not sure if it will be possible to form a coherent security ID for these (esp. Node/HostPort), ClusterIP could potentially be considered to have the intersection of the security labels of the backends.
Maybe HostPort/NodePort access to policies already requires policy allow on the host/node rather than the backend, as the service translation happens after the (egress) policy enforcement in this case, at least when kube-proxy is used. Not sure if KPR changes this dynamic, though?

Do the policy enforcement after the L7 proxy. We could add a new socket mark pattern so that the datapath would know to perform policy enforcement, reusing the existing host firewall code paths.

This would keep the existing model and not complicate userspace policy management at all.
Additional overhead of host firewall and another proxy traversal (for L3-dependent egress L7 policies) would be needed.

The above applies to egress. Ingress is not functional yet anyway, but similar considerations will be there as well depending on whether it is about pod ingress (service translation done at the source host) or cluster ingress (service translation needs to be done, somewhat like with node port).

test-me-please

test-me-please

test-me-please

One other related comment here, Cilium always applies service translation first and then applies network policy based on the pod<->pod traffic (ie not pod->service). Is that constraint retained here?

Actually, no. The whole point of postponing the load balancing decision to L7 is to not do the service translation before the L7 proxy. To retain the current model we'll need to do one of these:

Add ClusterIP/NodePort/HostPort level policies to the bpf datapath so that we can do policy enforcement before the L7 proxy

Not sure if it will be possible to form a coherent security ID for these (esp. Node/HostPort), ClusterIP could potentially be considered to have the intersection of the security labels of the backends.
Maybe HostPort/NodePort access to policies already requires policy allow on the host/node rather than the backend, as the service translation happens after the (egress) policy enforcement in this case, at least when kube-proxy is used. Not sure if KPR changes this dynamic, though?

Hmm, curious - so is this effectively, we'll pretend to do service translation at BPF layer in order to understand what the identity of the backend is, then apply that policy, then forward to the L7 proxy in order to make the real L7 forwarding decision? Sounds a bit weaker from a security model since it assumes that all backends have the same identity and that the BPF translation will match the L7 service translation. If there's any reason that these assumptions do not hold, then the policy could be improperly applied. :thinking:

Do the policy enforcement after the L7 proxy. We could add a new socket mark pattern so that the datapath would know to perform policy enforcement, reusing the existing host firewall code paths.

This would keep the existing model and not complicate userspace policy management at all.
Additional overhead of host firewall and another proxy traversal (for L3-dependent egress L7 policies) would be needed.

Yeah would need to be a different pattern and include a way to identify the original source in the mark as well (either endpoint ID or security ID) so that we can still apply the policy based on the pod that initiated the connections rather than applying the host's policy to this traffic. Nice property of retaining the single enforcement point, admittedly at the cost of making L7 policy enforcement more complicated/expensive. I think that's fine as a first cut though, we can optimize the latter case after we get a bit more experience with this implementation.

The above applies to egress. Ingress is not functional yet anyway, but similar considerations will be there as well depending on whether it is about pod ingress (service translation done at the source host) or cluster ingress (service translation needs to be done, somewhat like with node port).
Some additional considerations:

Node/Host port is already NOT translated to backend before egress policy enforcement

I was under the impression that for NodePort there will still typically be an underlying BPF service in which case even when kube-proxy is present, Cilium will also implement the service for traffic egressing a pod. But yes for hostport/portmap chaining case without BPF hostport, I think you're right. That traffic will go to backend before egress policy enforcement. Bit of a gap in the model at the moment. Also yes if traffic comes in from outside the node then it would go to kube-proxy first, but that means that the packets should be translated before they reach Cilium's BPF enforcement layer as well. So that side should be consistent at least.

Backends may have different security labels, so a current policy may allow/deny depending on which backend is selected
Some services may have non-pod backends, which may not have any labels, but for which a CIDR policy may apply, for example

Yeah, this has the potentially to be a bit weird (eg two requests to the same service could have different behaviour), but that's the model we defined for a good reason. At the end of the day, if policy is defined on the translated addresses then you know that the policy must allow traffic from that source to that real destination. Regardless of how the pod reaches out to the service or how the service has been implemented, the network policy can be separately configured to enforce these constraints. Organizationally this means also that whoever has the RBAC access for network policy will define the restrictions regardless of how the app team configures the services. If you have a threat model where someone is able to gain access to an app team's cluster roles, then they would not be able to skip the network policy just by reconfiguring services; they'd have to escalate to a more privileged cluster role.

test-me-please

/test

/test

/test

/test

/test

/test

/test

/test

/test

rebased to get #17488 to potentially help with helm issues.

/test

/test

/test

/test

test-me-please

/test

/test

k8s-1.21 fails on nodeport service having no backends:
./bugtool-cilium-dxdzp/cmd/iptables-legacy-save--c.md:[14:840] -A KUBE-EXTERNAL-SERVICES -p tcp -m comment --comment "default/test-nodeport-local-k8s2:http has no endpoints" -m addrtype --dst-type LOCAL -m tcp --dport 31317 -j REJECT --reject-with icmp-port-unreachable

I wonder if the Envoy proxies absolutely must run in the host network namespace? I do also think that at some stage the proxies will need to be managed outside of Cilium pod, but presently, it'd make sense to at least outline the reasoning for why proxies are not running in their own network namespaces. Supposedly, use of host network is just a historic assumption, but that still would be good to confirm. One would think that host port management is not very trivial sort of business, especially as ports can be a scarce resource. On the other hand, having the Envoy proxies exposed to the host network by default also seems like a security concern. Other than that, not much has come to my attention while reading the code. I can see that a few reviewers had looked at the changes line by line already.

The decision to run the proxy in the host network namespace makes transparent redirection easier from the bpf datapath running in the host network namespace. One detail is that bpf datapath, iptables rules, and proxies need to be able to pass socket/skb marks between each other, which would not be possible if passing through a veth pair between network namespaces.
While transparent proxying might not be a requirement for k8s Ingress implementation, it is needed for L7 redirection for policy enforcement, and we want to reuse the same Envoy instances for both purposes.

Pushed with one new experimental commit on top.

/test

/test

All upgrade tests are now failing due to increasing cilium calls map size. This will be fixed by #17744, will rebase after that is merged.

/test

Rebased to master

/test

/test

/test

All Jenkins pipelines fail on upgrade/dowgrade tests, which is expected, as this PR adds new tail calls to the calls map and #17744 has not been merged yet.

/test

EKS tests hit the maximum number of VPCs, failing because of it.

All CI pipelines expectedly fail on the upgrade/downgrade test. K8s 1.22 / kernel 4.9 is hitting #16680, which needs manual testing to figure out if the datapath is broken by this PR for service hairpin access in this case.

This was merged in an updated PR #18894, closing.


Adding Google Tag Manager script

Paste this code as high in the  of the page as possible:

Additionally, paste this code immediately after the opening  tag:

<iframe src=https://www.googletagmanager.com/ns.html?id=GTM-WJDGM92
height="0" width="0" style="display:none;visibility:hidden">

Hello @kpierduta google tag manager was added to the Website please have a look and verify it. thanks