jigen failure

RuleException:
IndexError in file /home/ubuntu/georg/DomainLab/domainlab/exp_protocol/benchmark.smk, line 59:
index 9 is out of bounds for axis 0 with size 9
  File "/home/ubuntu//DomainLab/domainlab/exp_protocol/benchmark.smk", line 59, in __rule_run_experiment
  File "/home/ubuntu/DomainLab/domainlab/exp_protocol/run_experiment.py", line 87, in run_experiment
  File "/home/ubuntu/DomainLab/domainlab/compos/exp/exp_main.py", line 48, in execute
  File "/home/ubuntu/DomainLab/domainlab/algos/trainers/train_visitor.py", line 43, in tr_epoch
  File "/home/ubuntu/DomainLab/domainlab/algos/trainers/train_basic.py", line 26, in tr_epoch
  File "/home/ubuntu/mambaforge/envs/gdl/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 681, in __next__
  File "/home/ubuntu/mambaforge/envs/gdl/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 721, in _next_da
ta
  File "/home/ubuntu/mambaforge/envs/gdl/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
  File "/home/ubuntu/mambaforge/envs/gdl/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listco
mp>
  File "/home/ubuntu/mambaforge/envs/gdl/lib/python3.8/site-packages/torch/utils/data/dataset.py", line 235, in __getitem__
  File "/home/ubuntu/DomainLab/domainlab/dsets/utils_wrapdset_patches.py", line 108, in __getitem__
  File "/home/ubuntu/DomainLab/domainlab/dsets/utils_wrapdset_patches.py", line 108, in <listcomp>
  File "/home/ubuntu/mambaforge/envs/gdl/lib/python3.8/concurrent/futures/thread.py", line 57, in run
Shutting down, this might take some time.                                                                                Exiting because a job execution failed. Look above for error message

reproducible with
tpath: examples/tasks/task_vlcs.py  # dataset

test_domains:
  - caltech

output_dir: zoutput/benchmarks/demo_benchmark

num_param_samples: 2
sampling_seed: 1  # Optional

epos: 3
bs: 2
startseed: 1
endseed: 2  # currently included

Task4:  # name
  aname: jigen

  hyperparameters:
    nperm:
      distribution: categorical
      values:
        - 30
        - 31
        - 100
      datatype: int

    pperm:
      min: 0.1
      max: 0.5
      distribution: uniform

    grid_len:
      min: 2
      max: 5
      step: 1
      distribution: uniform

    early_stop:
      distribution: categorical
      datatype: int
      values:
        - 1
        - 5
        - 10

    gamma_reg:
      min: 0.01
      max: 10
      distribution: loguniform

command:
snakemake --cores 1 -s "domainlab/exp_protocol/benchmark.smk" --configfile "examples/yaml/demo_benchmark.yaml"

@Car-la-F @schoersch , i am back to work an hour ago, I have added some tests to jigen, and try to reproduce now:
using
git checkout carla_benchmark
snakemake --cores 1 -s "domainlab/exp_protocol/benchmark.smk" --configfile "examples/yaml/demo_benchmark.yaml"

What I got:  will take sometime to delve deep into it
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job                   count    min threads    max threads
------------------  -------  -------------  -------------
agg_results               1              1              1
all                       1              1              1
parameter_sampling        1              1              1
run_experiment            5              1              1
total                     8              1              1

Select jobs to execute...

[Tue Feb 21 15:41:15 2023]
rule parameter_sampling:
    input: /home/sunxd/domainlab_master/examples/yaml/demo_benchmark.yaml
    output: zoutput/benchmarks/demo_benchmark/hyperparameters.csv
    jobid: 3
    reason: Missing output files: zoutput/benchmarks/demo_benchmark/hyperparameters.csv
    resources: tmpdir=/tmp

[Tue Feb 21 15:41:15 2023]
Finished job 3.
1 of 8 steps (12%) done
Select jobs to execute...

[Tue Feb 21 15:41:15 2023]
rule run_experiment:
    input: zoutput/benchmarks/demo_benchmark/hyperparameters.csv
    output: zoutput/benchmarks/demo_benchmark/rule_results/1.csv
    jobid: 4
    reason: Missing output files: zoutput/benchmarks/demo_benchmark/rule_results/1.csv; Input files updated by another job: zoutput/benchmarks/demo_benchmark/hyperparameters.csv
    wildcards: index=1
    resources: tmpdir=/tmp

[Tue Feb 21 15:42:25 2023]
Finished job 4.
2 of 8 steps (25%) done
Select jobs to execute...

[Tue Feb 21 15:42:25 2023]
rule run_experiment:
    input: zoutput/benchmarks/demo_benchmark/hyperparameters.csv
    output: zoutput/benchmarks/demo_benchmark/rule_results/2.csv
    jobid: 5
    reason: Missing output files: zoutput/benchmarks/demo_benchmark/rule_results/2.csv; Input files updated by another job: zoutput/benchmarks/demo_benchmark/hyperparameters.csv
    wildcards: index=2
    resources: tmpdir=/tmp

[Tue Feb 21 15:44:21 2023]
Finished job 5.
3 of 8 steps (38%) done
Select jobs to execute...

[Tue Feb 21 15:44:21 2023]
rule run_experiment:
    input: zoutput/benchmarks/demo_benchmark/hyperparameters.csv
    output: zoutput/benchmarks/demo_benchmark/rule_results/3.csv
    jobid: 6
    reason: Missing output files: zoutput/benchmarks/demo_benchmark/rule_results/3.csv; Input files updated by another job: zoutput/benchmarks/demo_benchmark/hyperparameters.csv
    wildcards: index=3
    resources: tmpdir=/tmp

Removing output files of failed job run_experiment since they might be corrupted:
zoutput/benchmarks/demo_benchmark/rule_results/3.csv
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2023-02-21T154115.105733.snakemake.log

@smilesun
I think jigen is no part of this demo. I created the jigen_failure, which reproduces the error above via
git checkout jigen_failure
snakemake --cores 1 -s "domainlab/exp_protocol/benchmark.smk" --configfile examples/yaml/jigen_test.yaml

thanks, i merged master into Jigen to say if there will be a difference
https://github.com/marrlab/DomainLab/pull/130 , will check

FileNotFoundError: [Errno 2] No such file or directory: 'examples/yaml/jigen_test.yaml'

I get a different error:
RuleException:
NotImplementedError in file /home/carla/Dokumente/Uni/Master/Computational_Methods_for_Single-Cell_Biology/DomainLab/domainlab/exp_protocol/benchmark.smk, line 59:
Namespace(config_file=None, lr=0.0001, gamma_reg=0.08072307884499755, es=10, seed=1, nocu=False, gen=False, keep_model=False, epos=3, epo_te=1, debug=False, dmem=False, no_dump=True, out='zoutput', dpath='zdpath', tpath='examples/tasks/task_vlcs.py', npath=None, npath_dom=None, npath_argna2val=None, nname_argna2val=None, nname=None, nname_dom=None, apath=None, exptag='exptag', aggtag='aggtag', msel=None, aname='jigen', acon=None, task=None, bs=2, split=0, te_d='caltech', tr_d=None, san_check=False, san_num=8, zd_dim=64, zx_dim=0, zy_dim=64, topic_dim=3, topic_h_dim=8, img_h_dim=8, nname_topic_distrib_img2topic=None, npath_topic_distrib_img2topic=None, nname_encoder_sandwich_layer_img2h4zd=None, npath_encoder_sandwich_layer_img2h4zd=None, gamma_y=None, gamma_d=None, beta_t=1.0, beta_d=1.0, beta_x=1.0, beta_y=1.0, warmup=100, tau=0.05, epos_per_match_update=5, epochs_ctr=None, epochs_erm=None, penalty_ws=0.1, nperm=31, pperm=0.49887392437554745, grid_len=5, dial_steps_perturb=3, dial_noise_scale=0.001, dial_lr=0.003, dial_epsilon=0.031, result_file='zoutput/benchmarks/demo_benchmark/rule_results/0.csv', params={'nperm': 31, 'pperm': 0.49887392437554745, 'grid_len': 5, 'early_stop': 5, 'gamma_reg': 0.08072307884499755}, benchmark_task_name='Task4', param_index=0) does not exist
  File "/home/carla/Dokumente/Uni/Master/Computational_Methods_for_Single-Cell_Biology/DomainLab/domainlab/exp_protocol/benchmark.smk", line 59, in __rule_run_experiment
  File "/home/carla/Dokumente/Uni/Master/Computational_Methods_for_Single-Cell_Biology/DomainLab/domainlab/exp_protocol/run_experiment.py", line 85, in run_experiment
  File "/home/carla/Dokumente/Uni/Master/Computational_Methods_for_Single-Cell_Biology/DomainLab/domainlab/compos/exp/exp_main.py", line 33, in __init__
  File "/home/carla/Dokumente/Uni/Master/Computational_Methods_for_Single-Cell_Biology/DomainLab/domainlab/algos/builder_jigen1.py", line 33, in init_business
  File "/home/carla/Dokumente/Uni/Master/Computational_Methods_for_Single-Cell_Biology/DomainLab/domainlab/compos/zoo_nn.py", line 42, in __call__
  File "/home/carla/Dokumente/Uni/Master/Computational_Methods_for_Single-Cell_Biology/DomainLab/domainlab/compos/pcr/p_chain_handler.py", line 84, in handle
  File "/home/carla/Dokumente/Uni/Master/Computational_Methods_for_Single-Cell_Biology/DomainLab/domainlab/compos/pcr/p_chain_handler.py", line 84, in handle
  File "/home/carla/Dokumente/Uni/Master/Computational_Methods_for_Single-Cell_Biology/DomainLab/domainlab/compos/pcr/p_chain_handler.py", line 84, in handle
  File "/home/carla/Dokumente/Uni/Master/Computational_Methods_for_Single-Cell_Biology/DomainLab/domainlab/compos/pcr/p_chain_handler.py", line 88, in handle
  File "/home/carla/miniconda3/envs/DomainLab/lib/python3.9/concurrent/futures/thread.py", line 58, in run

reproducible with:
tpath: examples/tasks/task_vlcs.py  # dataset

test_domains:
  - caltech

output_dir: zoutput/benchmarks/demo_benchmark

num_param_samples: 2
sampling_seed: 1  # Optional

epos: 3
bs: 2
startseed: 1
endseed: 2  # currently included

Task4:  # name
  aname: jigen

  hyperparameters:
    nperm:
      distribution: categorical
      values:
        - 30
        - 31
        - 100
      datatype: int

    pperm:
      min: 0.1
      max: 0.5
      distribution: uniform

    grid_len:
      min: 2
      max: 5
      step: 1
      distribution: uniform

    early_stop:
      distribution: categorical
      datatype: int
      values:
        - 1
        - 5
        - 10

    gamma_reg:
      min: 0.01
      max: 10
      distribution: loguniform

command: snakemake --cores 1 -s "domainlab/exp_protocol/benchmark.smk" --configfile "examples/yaml/demo_benchmark.yaml"

I can reproduce this one

@Car-la-F @schoersch
by adding one line:
nname: alexnet
to example yaml file I could get the following error
index 9 is out of bounds for axis 0 with size 9
  File "/home/sunxd/domainlab_master/domainlab/exp_protocol/benchmark.smk", line 59, in __rule_run_experiment
  File "/home/sunxd/domainlab_master/domainlab/exp_protocol/run_experiment.py", line 87, in run_experiment
  File "/home/sunxd/domainlab_master/domainlab/compos/exp/exp_main.py", line 48, in execute
  File "/home/sunxd/domainlab_master/domainlab/algos/trainers/train_visitor.py", line 43, in tr_epoch
  File "/home/sunxd/domainlab_master/domainlab/algos/trainers/train_basic.py", line 26, in tr_epoch
  File "/home/sunxd/anaconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 681, in __next__
  File "/home/sunxd/anaconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 721, in _next_data
  File "/home/sunxd/anaconda3/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
  File "/home/sunxd/anaconda3/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
  File "/home/sunxd/anaconda3/lib/python3.9/site-packages/torch/utils/data/dataset.py", line 235, in __getitem__
  File "/home/sunxd/domainlab_master/domainlab/dsets/utils_wrapdset_patches.py", line 108, in __getitem__
  File "/home/sunxd/domainlab_master/domainlab/dsets/utils_wrapdset_patches.py", line 108, in <listcomp>
  File "/home/sunxd/anaconda3/lib/python3.9/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Shutting down, this might take some time.

@Car-la-F @schoersch , thanks for finding the bug! I am sorry for my carelessness, we can not support other grid-len other than 3*3 currently!
So we have to remove
    grid_len:
      min: 2
      max: 5
      step: 1
      distribution: uniform


Fix CSS for Italian "The secret is to begin"

In English, the design looks like this:
https://crimethinc.com/2016/09/28/feature-the-secret-is-to-begin-getting-started-further-resources-frequently-asked-questions

In the Italian translation, it looks like this:
https://crimethinc.com/2016/09/28/il-segreto-sta-nelliniziare-iniziare-altre-risorse-faq

TODO

[ ] Reposition the links to not overlap with header text, on big screen
[ ] Resize header text to fit, on mobile

Since this is an article specific CSS change, it'll get added to this article's css field in the database. So, the deliverable for this issue is a block of CSS as a comment that can be copied / pasted into the database.
Feel free to save a copy of the HTML page and experiment in a <style> tag.

this issue is not specific for this page rather it is a common problem. if a word in the title is too long then this problem might reoccur. for example in german texts lengthy words are pretty common. consider this as a general fix.
	.header {
		display: grid;
		grid-gap: 10px;
		grid-template-columns: repeat(2,1fr);
	}
	.h1 {
	        overflow-wrap: break-word;
	        word-break: break-word;
	        hyphens: auto;
	        justify-self: end;
	        width: 50%;
	        margin: 0px;
	        font-size: 7vw;
	}
	h4{
	        margin: 0px;
	        align-self: end;
	}

@media (min-device-width:200px) and (max-width:599px) {
	.header {
		display: grid;
		grid-template-columns: 1fr;
		grid-gap: 10px;
	}
	h1 {
	    order: 2;
	    margin: 0;
	    word-wrap: break-word;
	    overflow-wrap: break-word;
	    hyphens: auto;
	    padding-left: 10%;
	    padding-right: 10%;
	    overflow: hidden;
	    width: auto;
	}
	h4{
	    order: 1;
	    margin: 0;
	    padding-left: 10%;
	}
}


Adjustments so everything fits nicely

This PR includes the following changes:

Misspell of OpenAPI on the main header
Update the GitHub Actions code sample
Use the same colors for active buttons
Align code samples on the page boxes
Prevent scrolling on laptop sized screens
Fix mobile ordering of RatingExamples component

Before:

Now:

Great! 👏
Just something that came to my mind:
In another step, to save some more space, we could remove the header on top that serves no real purpose. We still could keep the logo and put it to the left or so

I tried my best to keep it cause I agree it serves no real purpose lol


Allow filled candle-stick diagrams

Implements fill for candle-stick diagrams.
Also uses this functionality in the provided stocks example.
In contrast to #226, this implementation reuses the gain/loss style attribute and does not add an extra field.
The change is thus much simpler.
Fixes #281

Given https://github.com/38/plotters/issues/333, I don't think this is going anywhere. If there should ever be a fork still interested in this, feel free to contact me.

Hi!
Given that you only closed this three days ago, I think that https://github.com/38/plotters/issues/341 may be of interest to you.


Cookie match for multipart requests

During the implementation of the integration tests notices the problem with registering disposable handler for multipart requests.
Initially, I thought to use the NewCookieMatcher to match the multipart response, but ofp.MultipartRequest header does not implement CookieJar interface, while the body of multipart request does (e.g. AggregateStatsRequest).
Need a solution how to register disposable handler for multipart responses.

Probably, the appropriate way to implement multipart requets is to defined the standalone structure for each type, which will implement the CookieJar interface.

In fact, the multipart request does not accept cookie (body could contain cookie but only to match interested entries), so the matching the response by cookies is useless.


Spit out an error if we have a WEBIRC block with no hosts specified

What it says on the tin. If we aren't doing this already then we should do it, to help peeps work out what's going on when the block doesn't work.

Hm. To help with WEBIRC setups, maybe we could also spit out a warn/info line if a WEBIRC block is specified but the associated IP doesn't have an exemption? That might be trying to be a bit too helpful though.

As of #989, the default config accepts PROXY from localhost so there's no issue with accepting WEBIRC from localhost as well.
However, WEBIRC is just straight up deprecated now that we have native websockets. Maybe that's not sufficiently clear to people?

Totally agree on the localhost. I wouldn't really consider WEBIRC deprecated, it's gonna take a long time for people to switch over to using them natively. And the WEBIRC command's still gonna be useful to software that keeps proxying connections rather than connecting directly to the IRC server from the browser.


[Suggestion] Option to only generate .osdb

Since Collection Manager can be used to download missing beatmaps, it would be nice if there was an option to only generate the .osdb file and use Collection Manager to download the beatmaps so that the already existing maps don't have to be downloaded again.

Hello! Thanks for the suggestion.
You can actually close the terminal once the .osdb file was generated (Since it was generated first before the download starts).
But yeah, it can be convenient for some people, if that option exists. I will consider it, thanks again.

Hello vpertys, thank you for your suggestion. After careful consideration, I have decided to reject it as I want to keep the focus of the project on providing an easy way to download every beatmap set from the website. Adding more options could make the project more complex and detract from its main purpose. However, I will be implementing a 3-second delay before the download begins to allow users to close the terminal once the .osdb file is generated.


Updating documentation and updating the version of the colors library.

Updating/fixing a bunch of documentation, updating the version of the colors library being pulled in, and adding a bunch of method PHP Docs to make intellisense in an IDE.

this looks like a really good quality PR!!
:+1: great work


Bug when comparing size of len(msg.Pack) and msg.Len()

Following https://github.com/miekg/dns/issues/657 I updated DNS lib to 1.0.5 into Consul but ran into another issue and I am still able to reproduce case where len(msg.Pack()) is not equal to msg.Len()
https://github.com/pierresouchay/consul/commit/f85db7e7b78eac095fd3c0ec40ce643a6513268c
It crashes with the following output:
2018/04/02 15:25:33 [DEBUG] err:= %!s(<nil>), dns: DIFF between resp.Len() := 65481 and len(buf) := 65465 for ;; opcode: QUERY, status: NOERROR, id: 42437
;; flags: qr aa rd; QUERY: 1, ANSWER: 647, AUTHORITY: 0, ADDITIONAL: 647

;; QUESTION SECTION:
;redis.service.consul.	IN	 SRV

;; ANSWER SECTION:
redis.service.consul.	0	IN	SRV	1 1 8000 host-redis-1-146.test.acme.com.node.dc1.consul.
redis.service.consul.	0	IN	SRV	1 1 8000 host-redis-2-31.test.acme.com.node.dc1.consul.

[643 SRV similar Entries Skipped...]

redis.service.consul.	0	IN	SRV	1 1 8000 host-redis-3-100.test.acme.com.node.dc1.consul.
redis.service.consul.	0	IN	SRV	1 1 8000 host-redis-3-56.test.acme.com.node.dc1.consul.

;; ADDITIONAL SECTION:
host-redis-1-146.test.acme.com.node.dc1.consul.	0	IN	CNAME	fx.168.1.146.
host-redis-2-31.test.acme.com.node.dc1.consul.	0	IN	CNAME	fx.168.2.31.
host-redis-1-226.test.acme.com.node.dc1.consul.	0	IN	CNAME	fx.168.1.226.
host-redis-1-59.test.acme.com.node.dc1.consul.	0	IN	CNAME	fx.168.1.59.

[639 CNAME Entries Skipped...]

host-redis-3-108.test.acme.com.node.dc1.consul.	0	IN	CNAME	fx.168.3.108.
host-redis-3-85.test.acme.com.node.dc1.consul.	0	IN	CNAME	fx.168.3.85.
host-redis-3-100.test.acme.com.node.dc1.consul.	0	IN	CNAME	fx.168.3.100.
host-redis-3-56.test.acme.com.node.dc1.consul.	0	IN	CNAME	fx.168.3.56.
    2018/04/02 15:25:33 [DEBUG] http: Request PUT /v1/catalog/register (443.311µs) from=<IP_ADDRESS>:51799
    2018/04/02 15:25:33 [DEBUG] dns: request for name redis.service.consul. type SRV class IN (took 69.958983ms) from client <IP_ADDRESS>:51798 (tcp)
panic: Bug in DNS lib

goroutine 2140 [running]:
github.com/hashicorp/consul/agent.(*DNSServer).trimTCPResponse(0xc4204266c0, 0xc42016e120, 0xc42016e1b0, 0x0)
	/Users/p.souchay/go/src/github.com/hashicorp/consul/agent/dns.go:761 +0x638
github.com/hashicorp/consul/agent.(*DNSServer).trimDNSResponse(0xc4204266c0, 0x20e2a9f, 0x3, 0xc42016e120, 0xc42016e1b0, 0x289)
	/Users/p.souchay/go/src/github.com/hashicorp/consul/agent/dns.go:827 +0xc0
github.com/hashicorp/consul/agent.(*DNSServer).serviceLookup(0xc4204266c0, 0x20e2a9f, 0x3, 0xc4205058ec, 0x3, 0xc420af2100, 0x5, 0x0, 0x0, 0xc42016e120, ...)
	/Users/p.souchay/go/src/github.com/hashicorp/consul/agent/dns.go:911 +0x54f
github.com/hashicorp/consul/agent.(*DNSServer).dispatch(0xc4204266c0, 0x20e2a9f, 0x3, 0xc42016e120, 0xc42016e1b0)
	/Users/p.souchay/go/src/github.com/hashicorp/consul/agent/dns.go:423 +0xd9a
github.com/hashicorp/consul/agent.(*DNSServer).handleQuery(0xc4204266c0, 0x2a295c0, 0xc420cfc780, 0xc42016e120)
	/Users/p.souchay/go/src/github.com/hashicorp/consul/agent/dns.go:273 +0x4d4
github.com/hashicorp/consul/agent.(*DNSServer).(github.com/hashicorp/consul/agent.handleQuery)-fm(0x2a295c0, 0xc420cfc780, 0xc42016e120)
	/Users/p.souchay/go/src/github.com/hashicorp/consul/agent/dns.go:117 +0x48
github.com/hashicorp/consul/vendor/github.com/miekg/dns.HandlerFunc.ServeDNS(0xc420182100, 0x2a295c0, 0xc420cfc780, 0xc42016e120)
	/Users/p.souchay/go/src/github.com/hashicorp/consul/vendor/github.com/miekg/dns/server.go:84 +0x44
github.com/hashicorp/consul/vendor/github.com/miekg/dns.(*ServeMux).ServeDNS(0xc4201820c0, 0x2a295c0, 0xc420cfc780, 0xc42016e120)
	/Users/p.souchay/go/src/github.com/hashicorp/consul/vendor/github.com/miekg/dns/server.go:210 +0x65
github.com/hashicorp/consul/vendor/github.com/miekg/dns.(*Server).serve(0xc4203be000, 0x2a1dfe0, 0xc420708030, 0x2a15ea0, 0xc4201820c0, 0xc42083e1c0, 0x3d, 0x3d, 0x0, 0x0, ...)
	/Users/p.souchay/go/src/github.com/hashicorp/consul/vendor/github.com/miekg/dns/server.go:567 +0x2c3
github.com/hashicorp/consul/vendor/github.com/miekg/dns.(*Server).serveTCP.func1(0xc420182200, 0x2a29740, 0xc42022e0e0, 0x77359400, 0xc4203be000, 0xc420182220)
	/Users/p.souchay/go/src/github.com/hashicorp/consul/vendor/github.com/miekg/dns/server.go:481 +0x141
created by github.com/hashicorp/consul/vendor/github.com/miekg/dns.(*Server).serveTCP
	/Users/p.souchay/go/src/github.com/hashicorp/consul/vendor/github.com/miekg/dns/server.go:475 +0x2e3

I am gonna try to provide a test case

Here is a failing Test Case:
func TestMsgCompressLengthLargeRecordsAllValues(t *testing.T) {
	msg := new(Msg)
	msg.Compress = true
	msg.SetQuestion("redis.service.consul.", TypeSRV)
	for i := 0; i < 900; i++ {
		target := fmt.Sprintf("host-redis-%d-%d.test.acme.com.node.dc1.consul.", i/256, i%256)
		msg.Answer = append(msg.Answer, &SRV{Hdr: RR_Header{Name: "redis.service.consul.", Class: 1, Rrtype: TypeSRV, Ttl: 0x3c}, Port: 0x4c57, Target: target})
		msg.Extra = append(msg.Extra, &CNAME{Hdr: RR_Header{Name: target, Class: 1, Rrtype: TypeCNAME, Ttl: 0x3c}, Target: fmt.Sprintf("fx.168.%d.%d.", i/256, i%256)})
		predicted := msg.Len()
		buf, err := msg.Pack()
		if err != nil {
			t.Error(err)
		}
		if predicted != len(buf) {
			t.Fatalf("predicted compressed length is wrong for %d records: predicted %s (len=%d) %d, actual %d", i, msg.Question[0].Name, len(msg.Answer), predicted, len(buf))
		}
	}
}

It fails with the following:
--- FAIL: TestMsgCompressLengthLargeRecordsAllValues (0.10s)
	/Users/p.souchay/go/src/github.com/miekg/dns/length_test.go:170: predicted compressed length is wrong for 249 records: predicted redis.service.consul. (len=250) 22835, actual 22818
FAIL
FAIL	github.com/miekg/dns	0.115s
Error: Tests failed.

[ Quoting<EMAIL_ADDRESS>in "[miekg/dns] Bug when comparing size..." ]

Following https://github.com/miekg/dns/issues/635 I updated DNS lib to 1.0.5 into Consul but ran into another issue and I am still able to reproduce case where len(msg.Pack()) is not equal to msg.Len()

There could, maybe, be an one-off in Len(), but I don't have the time to deepdive
that atm.

@miekg I tried to find the issue... it seems it is related to the compression of data of Extra... I was not exactly able to find what is the issue, but this is definitly related to tests such as len < maxCompressionOffset...
I'll try to find some time to dig a bit more

[ Quoting<EMAIL_ADDRESS>in "Re: [miekg/dns] Bug when comparing ..." ]

@miekg I tried to find the issue... it seems it is related to the compression of data of Extra... I was not exactly able to find what is the issue, but this is definitly related to tests such as len < maxCompressionOffset...
I'll try to find some time to dig a bit more

Thanks.
Yes, the len < maxCompressionOffset should maybe be <=, but I haven't given
this a lot of thought.

It took me a while to figure out the issue (a few hours), it was too simple :-)
Since len was added before trying to add label to compression and since it is legal to have a start offset before 14 bits, we just had to more the increase of len at the end of loop :-)

@miekg  Ok, the patch I made seems to work fine now.
I added lots of tests cases, including TestMsgCompressLengthLargeRecordsWithPaddingPermutation which test all permutations with padding around 14bits, so it should work well in all cases now


[4186] Admin funding view

Context
https://trello.com/c/9euPhMgK/4186-admin-funding-view
Changes proposed in this pull request

Adds two new controllers SystemAdmin::Funding::PaymentSchedulesController and SystemAdmin::Funding::LeadSchoolsController that serve data for four new pages on following URLs:

/providers/id/funding/payment-schedule
/lead-schools/id/funding/payment-schedule
/providers/id/funding/trainee-summary
/lead-schools/id/funding/trainee-summary

Reuses the view objects and existing functionality for the views

Creates new view object Funding::NavigationView to deal with the logic for rending the correct tab links, depending on whether you're an admin / viewing a lead school / viewing a provider

Adds a links to funding from provider and lead school show pages

Adds a new table to providers show page to show some information about the provider you're looking at

Guidance to review

Sign in a system admin
Go to providers -> select a provider
Click 'View funding' link
See that the correct payment schedule is displayed
Click 'Trainee summary'
See that the correct trainee summary is displayed

Repeat the above steps for a lead school (choose one that's linked to another persona)

Important business

[ ] Does this PR introduce any PII fields that need to be overwritten or deleted in db/scripts/sanitise.sql?
[ ] Does this PR change the database schema? If so, have you updated the config/analytics.yml file and considered whether you need to send 'import_entity' events?

NB: Please notify the #twd_data_insights team and ask for a review if new fields are being added to analytics.yml

Not a massive deal - but I wonder if we should hide 'Funding' in the nav when the current user is system admin?

Not a massive issue but the back links from the payment schedule and summary pages go back to the home page rather than the provider/school page.

Not a massive issue but the back links from the payment schedule and summary pages go back to the home page rather than the provider/school page. We could card that up as a bug and get this in if it's fiddly.

Hettie and I chatted about this and we're gonna card it up - it's not difficult but we want to check what should happen with the back links for normal users too (currently it just goes back to root_path from where ever they were)


[AIRFLOW-861] make pickle_info endpoint be login_required

Dear Airflow Maintainers,
Please accept this PR that addresses the following issues:

https://issues.apache.org/jira/browse/AIRFLOW-861

Testing Done:

Unittests pass

landscape errors are unrelated

Codecov Report

Merging #2077 into master will increase coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #2077      +/-   ##
==========================================
+ Coverage   66.94%   66.96%   +0.01%
==========================================
  Files         140      140
  Lines       10635    10640       +5
==========================================
+ Hits         7120     7125       +5
  Misses       3515     3515

Impacted Files
Coverage Δ

airflow/www/views.py
69.73% <100%> (+0.02%)
:white_check_mark:

airflow/models.py
86.13% <ø> (-0.03%)
:x:

airflow/jobs.py
72.59% <ø> (+0.1%)
:white_check_mark:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b0ae70d...5872fc1. Read the comment docs.

LGTM


Rewrite the tests to avoid precision issue in different node versions

Stable still fails, see https://github.com/node-nock/nock/issues/922


how to install the module mc?

You can proceed by following the README.md.
Omit mc.

Sorry, but it is too slow if not using the memorycache.

It is better if you can provide the memory cached method, or I have to solve this with other repo

I don't use mc because it's fast on my training servers.


CDPSDX-839 Allow a role to be deleted when deleting a host or service

An optional way to delete roles was added to the kerberos management
delete host and delete service principal APIs. An attempt is made to
ensure the role is not in use, although the verification has a race
condition, the check will handle most common use cases and it is
better than nothing.
This was tested manually using a local deployment of cloudbreak.
Closes #CDPSDX-839

I got the comment on the last PR about refactoring KerberosMgmtV1Service due to the size of the class. I will get that next and I created CB-3420 for it.

int test is going to be fixed with https://github.com/hortonworks/cloudbreak/pull/6164, please rerun when it's merged


Problem if multiple worker and using q.on('progress', ...)

While having Multiple worker, if i use q.on('progress',..) the other worker also, responds. Why?
My code:
Process.js
const Queue = require('rethinkdb-job-queue')
const qOptions = {
    name: 'Mathematics' // The queue and table name
}
const cxnOptions = {
    db: 'JobQueue', // The name of the database in RethinkDB
}

const q = new Queue(cxnOptions, qOptions)

function getJobs() {
    q.process((job, next) => {

        try {
            let result = job.numerator / job.denominator
            console.log('result', result);
            // Do something with your result
            return next(null, result)
        } catch (err) {
            console.error(err)
            return next(err)
        }
    })
    q.on('completed', async(queueId, jobId, isRepeating) => {
        // console.log('Processing Queue: ' + queueId)

            console.log('Job completed: ' + jobId)

    })
    q.on('progress', async(queueId, jobId, percent) => {

        console.log('Job progress: ' + percent)

    })
}
getJobs()

Create.js
const Queue = require('rethinkdb-job-queue')
const qOptions = {
    name: 'Mathematics' // The queue and table name
}
const cxnOptions = {
    db: 'JobQueue', // The name of the database in RethinkDB
}

const q = new Queue(cxnOptions, qOptions)

const job = q.createJob({
    numerator: 123,
    denominator: 456
})

return q.addJob(job).catch((err) => {
    console.error(err)
})

Screeshot of output.

Hi @gkumar9
As you can see here: https://github.com/grantcarthew/node-rethinkdb-job-queue/blob/ae3111cdd7596c552e05d60a32daadd4088af63f/src/job-completed.js#L13
When a job is complete its progress gets set to 100%.
If you are only interested in progress reports for local jobs, just filter by the job ids in your worker.

@grantcarthew
Thats fine, but the problem is as below :
I have 3 workers (worker1, worker2, worker3) running simultaneously waiting for the job to be added inside the queue.
The job is added through create.js
All workers (worker1, worker2, worker3) (named as process.js inside screenshot above) having identical code and containing q.on('progress') and q.on('completed')
What my understanding is, as the job is added, only one of the worker (i.e., worker1) should start running job and that same worker should only respond to progress and completed events.
But as you can see in the screenshot, the other 2 workers (worker2, worker3) are giving the response of progress. This is misleading.
Can you please explain this scenario.
Thanks

This is all documented @gkumar9
Start here: https://github.com/grantcarthew/node-rethinkdb-job-queue/wiki/Queue-Events
More here: https://github.com/grantcarthew/node-rethinkdb-job-queue/wiki/Queue-PubSub#subscriber-queue-object
Finish here: https://github.com/grantcarthew/node-rethinkdb-job-queue/wiki/Queue-Options#queue-changefeed-option
To summarize the above:

Change feed enabled will cause the Queue object to receive events for the whole queue.
Change feed disabled will cause the Queue object to not respond to jobs added to the queue by other publishers. It will only work with jobs added to the queue by its self (and jobs found after adding a job).
The subscribers do not poll the database, the only way they receive updates is by the change feed.
The Master Queue is an exception to the above rule.

You have two options:

Unlike your above event handler code which is ignoring the jobId, use it to filter events to locally processed jobs.
Construct your own change feed for the subscriber that filters for the jobId.

The second option is not difficult however I would suggest the top option to be a better one.
Something like this (pay attention to the jobIdCache variable):

const Queue = require('rethinkdb-job-queue')
const qOptions = {
    name: 'Mathematics' // The queue and table name
}
const cxnOptions = {
    db: 'JobQueue', // The name of the database in RethinkDB
}
const jobIdCache = new WeakSet()

const q = new Queue(cxnOptions, qOptions)

function getJobs() {
    q.process((job, next) => {
        jobIdCache.add(job.id)
        try {
            let result = job.numerator / job.denominator
            console.log('result', result);
            // Do something with your result
            return next(null, result)
        } catch (err) {
            jobIdCache.delete(job.id)
            console.error(err)
            return next(err)
        }
    })
    q.on('completed', async(queueId, jobId, isRepeating) => {
        // console.log('Processing Queue: ' + queueId)
        if (jobIdCache.has(jobId)) {
            console.log('Job completed: ' + jobId)
            jobIdCache.delete(jobId)
        }
    })
    q.on('progress', async(queueId, jobId, percent) => {

        console.log('Job progress: ' + percent)

    })
}
getJobs()

This is a silly example of a process handler because it is not asynchronous so the job is complete as soon as you call next in the handler. It does the job for a description though.

Closing this @gkumar9 as is it documented and you are not responding.


Class constructor Store cannot be invoked without 'new'

I got this error when try to use your package:
/opt/node/app/web/server/util/session-store/index.js:31
    var _this = _possibleConstructorReturn(this, (RedisStore.__proto__ || Object.getPrototypeOf(RedisStore)).call(this));
                                                                                ^

TypeError: Class constructor Store cannot be invoked without 'new'
    at new RedisStore (/opt/node/app/web/server/util/session-store/index.js:6:17)
    at Object.<anonymous> (/opt/node/app/web/server/index.js:60:10)
    at Module._compile (module.js:571:32)
    at loader (/opt/node/app/web/node_modules/babel-register/lib/node.js:144:5)
    at Object.require.extensions.(anonymous function) [as .js] (/opt/node/app/web/node_modules/babel-register/lib/node.js:154:7)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/opt/node/app/web/bin/server.js:1:1)
    at Module._compile (module.js:571:32)
    at loader (/opt/node/app/web/node_modules/babel-register/lib/node.js:144:5)
    at Object.require.extensions.(anonymous function) [as .js] (/opt/node/app/web/node_modules/babel-register/lib/node.js:154:7)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)
    at Function.Module.runMain (module.js:605:10)
    at Object.<anonymous> (/opt/node/app/web/node_modules/babel-cli/lib/_babel-node.js:154:22)
    at Module._compile (module.js:571:32)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:488:32)

Store.js
import Redis from 'ioredis'
import { Store } from 'koa-session2'

export default class RedisStore extends Store {

  constructor() {
    super()
    this.redis = new Redis()
  }

  async get(sid) {
    const data = await this.redis.get(`SESSION:${sid}`)
    return JSON.parse(data)
  }

  async set(session, { sid =  this.getID(24), maxAge = 1000000 } = {}) {
    try {
      // Use redis set EX to automatically drop expired sessions
      await this.redis.set(`SESSION:${sid}`, JSON.stringify(session), 'EX', maxAge / 1000)
    } catch (e) {}

    return sid
  }

  async destroy(sid) {
    return await this.redis.del(`SESSION:${sid}`)
  }
}

app:
import RedisStore from './util/session-store'

app.use(session({
  key: 'MY_KEY',
  httpOnly: false,
  store: new RedisStore()
}))

i meet same error, how can solve it??

@uerbin I think it's babel issue, but I don't know how should I deal with that because I need babel-cli to run my app

@raminious  did you use the babel version koa-session2@babel?

@raminious  i use babel-register ^6.24.0
.babelrc
{
"presets": ["latest"],
"plugins": ["transform-object-rest-spread", "transform-class-constructor-call"]
}
at first , plugins is  ["transform-object-rest-spread"], "transform-class-constructor-call" is added later, but not work

"devDependencies": {
"babel-loader": "^6.4.0",
"babel-plugin-transform-class-constructor-call": "^6.22.0",
"babel-plugin-transform-object-rest-spread": "^6.23.0",
"babel-preset-latest": "^6.24.0",
"babel-register": "^6.24.0",
"css-loader": "^0.27.3",
"extract-text-webpack-plugin": "^2.1.0",
"file-loader": "^0.10.1",
"koa-webpack": "^0.3.1",
"node-sass": "^4.5.0",
"nodemon": "^1.11.0",
"sass-loader": "^6.0.3",
"style-loader": "^0.13.2",
"url-loader": "^0.5.8",
"webpack": "^2.2.1"
}

maybe i know it
because the module write in new syntax, and not compile it to es5 version and publish it, so when use it online, it can't run, must with babel-register.
use the option can solve it, but cause new problem.
require('babel-register')({
	ignore: false
});

@uerbin you can run npm install --save koa-session2@babel to use the babel version.
Here is the doc.

Thanks @Secbone @uerbin
I will close this issue

@raminious  thanks.

@Secbone hello, did you merge the latest commits to babel branch?

@ilaipi Koa2 is required node 7.6.0+, so it doesn't need babel anymore.

@Secbone my node is 8.14.0 still not work.

@ilaipi did you use the master version?  could you show me some details?

@Secbone maybe it's because I use babel-node, because I need import&export, so I think the babel branch is still needed?

@ilaipi no, you can use the latest version directly, and use import export  with babel.

@Secbone my babel plugin don't compile node_modules dir only my src dir.

@Secbone have u ever tried this? I just upgraded my babel-node to 7.x, but it still don't work.
// babel-node src/app.js
TypeError: Class constructor Store cannot be invoked without 'new'

@ilaipi could you try to disable 'babel-plugin-transform-classes to run it?
like this:
presets: [
    ["env", { exclude: ["babel-plugin-transform-es2015-classes"] }]
  ]

And I think maybe you don't need babel-node anymore, you can use the latest version of node (11.4 now).

@Secbone do u use import & export ?  if don't use babel-node, how to use them?
@babel/plugin-transform-es2015-classes became @babel/plugin-transform-classes.

// but not work.
The plugins/built-ins '@babel/plugin-transform-classes' passed to the 'exclude' option are not
// the same as babel-plugin-transform-es2015-classes

@ilaipi Okay, I will test it later, maybe tonight I will publish a new version of babel branch. 😄

@Secbone thank you!

@ilaipi Okay, I will test it later, maybe tonight I will publish a new version of babel branch. 😄

@Secbone Hello, I have switch my session manager lib to koa-session-minimal and koa-redis. It seems koa-session2 don't compatible with koa-passport.


DOGECOIN IS ALIVE: A SOLUTION FOR THE WORLD

DOGECOIN IS ALIVE
ISSUES:

Crypto Mining Energy Consumption
Air Pollution (CO2 EMISSION)
Land and water pollution caused by human actions such as unrecycling and upgrading products globally

SOLUTION:
With the increasing carbon emission of the technological innovation that brings adversities to us humans I would propose a solution and a plan for the circulation of products globally. The rise of Cryptocurrencies needs more energy more than the past decades. There is a market for Cryptocurrencies and something is missing which is a market to spend this currencies. The solution is to have an entity that would exchange used products (not all) that would be sold(exchange) in an online commerce platform this would help lessen the CO2 emission through recycling also the profit that we'll be making will finance other entity that would focus on research in making green energy.
A RELATIONSHIP REPRESENTATION OF THE PROPOSED SOLUTION
An unnamed currency that is equalled to real money will be distributed to circulate in the platform.The pointing system will be based on the quality of product, demand and other details for evaluation.
Situation no. 1
(Person 1) A person that has camera without money and he wants to buy a watch (camera  worth-4,000 points)
(The entity) The entity would decide on the following options:
Option no. 1 - The entity would buy the camera of person 1 (converted it into points) to be sold to person 2
Option no. 2- Make a biding for the product based on points of other users acquisition.
(Person 2) A person looking for camera but only has a cellphone and ipad (worth 5,000 points)
(Person 3) Has a watch posted in the platform only wants to have a points in the system to be spent in the platform in the future( acquired-3,000 points in the system)
(In the situation the entity acquired a cellphone, ipad and fees that will serve as the entity revenues)
The entity will be collecting fees with every transaction done.
Situation no. 2
The company acquired the cellphone which is equivalent to 2,500 points
(Person 4) He wants to have cellphone but don't have money only had a mp3 player(worth-500 points), shoes(worth-600 points), sunglasses (worth- 400 points), xbox 360(worth-1000 points), headphones (rejected due to off qualifications)
The entity would convert all the items into points thus Person 1 could have a cellphone with the assets he had turned into points.
(This will make an infinite cash flow)The entity will have more items to acquire and to be sold using points or in the long run using(DOGECOIN!)
In addition and accordance to Cryptocurrency the entity will only accept Dogecoin later on as the entity established a name and reputation globally, with the right management, marketing and funding this could be the new revolutionized market.
I believe in innovation that we could make as well as making opportunities and discoveries without sacrificing our planet. I am Spencer R. Augusto from the Philippines a teen worrying about the future of our world, if we don't act today there will be no tomorrow, give me a chance to make our world a better place.
If you are interested for an elaborated presentation please contact me at<PHONE_NUMBER>4 or in email<EMAIL_ADDRESS>please help me change the world, I need your help.
DOGECOIN is not just a meme it is alive!

What’s this have to do with doge? I don’t see how this would happen, if you are interested in doing this I would look into making a crypto that YOU built around this point. Dogecoin was built for another reason, if you believe in changing the world so much do it. Because to get all this dogecoin someone would have to mine it causing an emissions issue all in its own.
Get Outlook for iOShttps://aka.ms/o0ukef

From: spencer-coder1 @.>
Sent: Tuesday, May 25, 2021 7:03:21 AM
To: dogecoin/dogecoin @.>
Cc: Subscribed @.***>
Subject: [dogecoin/dogecoin] DOGECOIN IS ALIVE: A SOLUTION FOR THE WORLD (#2096)
DOGECOIN IS ALIVE
ISSUES:

Crypto Mining Energy Consumption
Air Pollution (CO2 EMISSION)
Land and water pollution caused by human actions such as unrecycling and upgrading products globally

SOLUTION:
With the increasing carbon emission of the technological innovation that brings adversities to us humans I would propose a solution and a plan for the circulation of products globally. The rise of Cryptocurrencies needs more energy more than the past decades. There is a market for Cryptocurrencies and something is missing which is a market to spend this currencies. The solution is to have an entity that would exchange used products (not all) that would be sold(exchange) in an online commerce platform this would help lessen the CO2 emission through recycling also the profit that we'll be making will finance other entity that would focus on research in making green energy.
A RELATIONSHIP REPRESENTATION OF THE PROPOSED SOLUTION
An unnamed currency that is equalled to real money will be distributed to circulate in the platform.The pointing system will be based on the quality of product, demand and other details for evaluation.
Situation no. 1
(Person 1) A person that has camera without money and he wants to buy a watch (camera worth-4,000 points)
(The entity) The entity would decide on the following options:
Option no. 1 - The entity would buy the camera of person 1 (converted it into points) to be sold to person 2
Option no. 2- Make a biding for the product based on points of other users acquisition.
(Person 2) A person looking for camera but only has a cellphone and ipad (worth 5,000 points)
(Person 3) Has a watch posted in the platform only wants to have a points in the system to be spent in the platform in the future( acquired-3,000 points in the system)
(In the situation the entity acquired a cellphone, ipad and fees that will serve as the entity revenues)
The entity will be collecting fees with every transaction done.
Situation no. 2
The company acquired the cellphone which is equivalent to 2,500 points
(Person 4) He wants to have cellphone but don't have money only had a mp3 player(worth-500 points), shoes(worth-600 points), sunglasses (worth- 400 points), xbox 360(worth-1000 points), headphones (rejected due to off qualifications)
The entity would convert all the items into points thus Person 1 could have a cellphone with the assets he had turned into points.
(This will make an infinite cash flow)The entity will have more items to acquire and to be sold using points or in the long run using(DOGECOIN!)
In addition and accordance to Cryptocurrency the entity will only accept Dogecoin later on as the entity established a name and reputation globally, with the right management, marketing and funding this could be the new revolutionized market.
I believe in innovation that we could make as well as making opportunities and discoveries without sacrificing our planet. I am Spencer R. Augusto from the Philippines a teen worrying about the future of our world, if we don't act today there will be no tomorrow, give me a chance to make our world a better place.
If you are interested for an elaborated presentation please contact me at<PHONE_NUMBER>4 or in email<EMAIL_ADDRESS>please help me change the world, I need your help.
DOGECOIN is not just a meme it is alive!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/dogecoin/dogecoin/issues/2096, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APVV5RUDLL4ANGYI35ULG5TTPOGYTANCNFSM45PGTNCA.

I'm confused, did you just suggest adding a currency-like thing to a currency?

Hi! Very interesting! I think this has great potential to be turned into a white paper. Let us know on reddit how you progress!
https://www.reddit.com/r/dogecoindev/

No Ross I was suggesting you create your own, handshake is a crypto use to decentralize the internet, helium is used to decentralize ISP’s. So it would benefit you more making a crypto then piggy back on another. Because what you are asking to do would cause dogecoin to be mined more or at a higher rate which wouldn’t be good for emissions or the environment. So making a PoS or PoC would be better for the environment then you can base its core functions on what you are trying to perform. Just my two cents, doing this would either hurt the value of dogecoin or B.) turn into the bitcoin where states ban mining because of emissions. (Don’t think doge would hit the value of bitcoin just the mining part)
Get Outlook for iOShttps://aka.ms/o0ukef

From: Patrick Lodder @.>
Sent: Tuesday, May 25, 2021 10:53:17 AM
To: dogecoin/dogecoin @.>
Cc: raggingsoldier @.>; Comment @.>
Subject: Re: [dogecoin/dogecoin] DOGECOIN IS ALIVE: A SOLUTION FOR THE WORLD (#2096)
Hi! Very interesting! I think this has great potential to be turned into a white paper. Let us know on reddit how you progress!
https://www.reddit.com/r/dogecoindev/
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/dogecoin/dogecoin/issues/2096#issuecomment-847989597, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APVV5RWLFH6DWBG7ZXCUERLTPPBW3ANCNFSM45PGTNCA.


[Bug]: Improve K8s List Calls

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
We need a way to limit / paginate as needed List calls that we make to k8s. It causes a spike in processing when we make these calls against a potential large size return. We need to avoid bogging down things when we are just requesting.
Perf noticed this with multiple users and the nb-events endpoint to get K8sEvent objects. We don't limit the return.
There are other list calls we should look at too -- we should definitely not keep it unbound when fetching list endpoints from k8s.
Expected Behavior
Let us start with endpoints we can find. Investigate it from a high level... ...List... calls from the K8s library is a good spot to start.
Main goal is to see if we can trim the events from the backend. We need to make those smaller and to the point of this spawn. Maybe we can find a uid or resourceVersion in the metadata of the K8sEvent objects that can help us with the current gen of an object so we don't get unrelated events to the old notebook instances.
Steps To Reproduce
Spawner Page - Create Notebook.
Let us start with the nb-events endpoint and see if we can get other ones as well.
Workaround (if any)
No response
OpenShift Infrastructure Version
No response
Openshift Version
No response
What browsers are you seeing the problem on?
No response
Open Data Hub Version
2.3.0

Relevant log output
No response

We resolved this for the nb-events end point. We should look at API calls when we know they are issues to the performance.


Badge in Text SegmentedOption

Would it be nice to put notification badges on the SegmentControls, has anyone been able to implement them?

So no possibility of implementation? So it was closed?


Cleanup: Consider unifying sync and async code paths

Using the pattern where we use async methods all the way down, but pass a flag that controls whether the actual I/O operation is performed asynchronously. See Npgsql for example.

@smitpatel not needed?

I don't remember what was basic idea was before. But this is what current mechanism in place.

Query Compilation is agnostic to sync/async except

Generating final QueryingEnumerable which would be used to enumerate database result
Reducing cardinality of result on client side when final result is not enumerable

Shaper is also sync agnostic. We only use sync/async to read row from database side in particular mode. And of course the cancellation token for async enumeration.

If there is anything else we can do and should do then we can update description accordingly. Nothing specific comes to my mind. Up to @andriysvyryd and @roji

We still have quite a few places in EF (outside query compilation) where we have code duplication for sync/async, I think there's value in doing this - unless there are objections.
One theoretical issue is that this introduces a slight CPU perf hit on the sync path because methods are now async, i.e. have a state machine, even if they always complete synchronously. That's likely to be completely negligible though.


ERR: Unexpected Status code 403 while joining token

It seems like it obtains a fingerprint but the joining token fails..
I tried with a lot of tokens by now..

403 is forbidden, your token might be locked or incorrect.


Update README.md

Better visibility of component prop.
When I read new doc first time, I miss component prop on line 42. Think that render prop is enough for route construction.

I mean, this is actually the code style I use on all of my projects though. The docs don't even mention the render callback until the end.

Fine, we'll be consistent about newlines.


chore(core): update package icons version to 1.x

https://pipelinesghubeus22.actions.githubusercontent.com/HeUToew9cTjZafWvRwfFsAbmUuLakH3Eax7Th2vD9nOGVpsEKf/_apis/pipelines/1/runs/182/signedlogcontent/13?urlExpires=2023-10-13T12%3A14%3A59.7710858Z&urlSigningMethod=HMACV1&urlSignature=vi%2FR3Z%2BBJmaroWdHENpvrcwituONyZaTe1JyMCO8Fkk%3D
Runner 2 is failing


Collection of Improvements for next major release of Version 2.0

Fix #2
Add plots of *.csv results
Add summary plot to project page
Move frontend-code into Java for performance improvements

improvement 3 is moved to later version.


Population density and land value determine transport preferences?

Could transport preference (and car/cycle ownership) be determined by where the cim lives?
The two metrics I'm thinking of are Population Density and Land Value – I think those have more effect on transport choices than wealth alone. Ideally the metrics would be based on home district, with fallback to building the cim lives in if it's not in a district.
The two metrics would be graded as High, Medium or Low. And based on the combination of grades, transport preference modifiers would be determined (via predefined lookup table). Those modifiers would then be passed in to the pathfinder.
A sample table can be found on google docs.
The premise is that with so many transport options, being able to more readily reject options based on transport preferences would create much more distinct transport patterns (eg. high density residents more likely to use PT than low density, or posh people avoiding busses) and potentially also reduce CPU load due to faster culling and/or choosing of available paths.
In addition, users could specify their own transport preference profiles (just like people can create their own profiles for Ambient Sounds Tuner, or their own District Themes, etc). People might create and share profiles for specific cities, countries, etc. For example, Nordic countries are more likely to use public transport and own bicycles.
I have no idea how the game currently determines which cims have a car or bike, or what sort of transport cims prefer, so no idea if there is already something like this or if its even feasible to implement. But just thought I'd put the idea up in case it holds any interest.

Agreed on the car ownership aspect, but for pathfinder related stuff it would be difficult (and likely performance degrading) to facilitate multiple mods changing that.

Related tickets: #167 , #166 , #106


Investigate and improve current E2E tests related to header/proof system

Describe the Issue
We currently have an option to check tx proof in DepositEth and DepositBtc functions.
However these are not complete tests of the header/proof system, it only check with the Prove query
We should:

Add more complete tests as isolated tests for testing adding trackers with proofs
Make the block header voting more optimized for localnet. Right now the header check has been put in the deposit methods used at the beginning of the test because the client can't catch up with headers on Bitcoin

Closing since not considered for the time being. Keeping the EPIC open for tracking.


NameError: name 'max_accuracy' is not defined

I have tried to run the eval_linear.py after training dino on a custom dataset. I get the folowing error:
`
Traceback (most recent call last):
File "/home/ubuntu/dino/eval_linear.py", line 250, in
eval_linear(args)
File "/home/ubuntu/dino/eval_linear.py", line 142, in eval_linear
"Top-1 test accuracy: {acc:.1f}".format(acc=max_accuracy))
NameError: name 'max_accuracy' is not defined
Killing subprocess 44659
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/envs/pytorch/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/ubuntu/anaconda3/envs/pytorch/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/ubuntu/anaconda3/envs/pytorch/lib/python3.9/site-packages/torch/distributed/launch.py", line 340, in
main()
File "/home/ubuntu/anaconda3/envs/pytorch/lib/python3.9/site-packages/torch/distributed/launch.py", line 326, in main
sigkill_handler(signal.SIGTERM, None)  # not coming back
File "/home/ubuntu/anaconda3/envs/pytorch/lib/python3.9/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/ubuntu/anaconda3/envs/pytorch/bin/python', '-u', 'eval_linear.py', '--local_rank=0', '--data_path', 'images', '--pretrained_weights', 'runs/checkpoint.pth']' returned non-zero exit status 1.
`
After skimming through the lines above it, I believe it should be best_acc rather than max_accuracy ?
Note: I have changed the dataset class (pytorch dataset) so as to work on my custom dataset but I believe the error still stands as I tried finding max_accuracy in the entire file and found only one occurrence (i.e., not defined earlier).

True.
https://github.com/facebookresearch/dino/blob/8aa93fdc90eae4b183c4e3c005174a9f634ecfbf/eval_linear.py#L114-L115
vs.
https://github.com/facebookresearch/dino/blob/8aa93fdc90eae4b183c4e3c005174a9f634ecfbf/eval_linear.py#L98-L100

Thanks, I have merged the PR.


HistoryVault.com is broken

Checklist

[x] I'm reporting a broken site support
[x] I've verified that I'm running youtube-dl version 2020.07.28
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar issues including closed ones

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-F', '-u', 'PRIVATE', '-p', 'PRIVATE', '-v', 'https://watch.historyvault.com/shows/hunting-hitler/season-1/episode-1/full-episode']

[debug] Encodings: locale cp932, fs utf-8, out utf-8, pref cp932
[debug] youtube-dl version 2020.07.28
[debug] Python version 3.8.3 (CPython) - Windows-7-6.1.7601-SP1
[debug] exe versions: ffmpeg git-2019-10-23-1f327f5, ffprobe git-2019-10-23-1f327f5
[debug] Proxy map: {}
[generic] full-episode: Requesting header
WARNING: Falling back on generic information extractor.
[generic] full-episode: Downloading webpage
[generic] full-episode: Extracting information
ERROR: Unsupported URL: https://watch.historyvault.com/shows/hunting-hitler/season-1/episode-1/full-episode
Traceback (most recent call last):
  File "c:\users\zzzz\appdata\local\programs\python\python38-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 797, in extract_info
    ie_result = ie.extract(url)
  File "c:\users\zzzz\appdata\local\programs\python\python38-32\lib\site-packages\youtube_dl\extractor\common.py", line 530, in extract
    ie_result = self._real_extract(url)
  File "c:\users\zzzz\appdata\local\programs\python\python38-32\lib\site-packages\youtube_dl\extractor\generic.py", line 3382, in _real_extract
    raise UnsupportedError(url)
youtube_dl.utils.UnsupportedError: Unsupported URL: https://watch.historyvault.com/shows/hunting-hitler/season-1/episode-1/full-episode

Description

Tried downloading an episode of Hunting Hitler on History Vault but it doesn't even detect the website anymore.

@remitamine May I ask why this is considered a duplicate? The only other issue existing for this site is #18460 and that was asking for the site to be implemented. That was almost two years ago.
I have good faith this is not a duplicate issue.

It's a duplicate of https://github.com/ytdl-org/youtube-dl/issues/23363

@llacb47 Thank you for giving an issue number.


Update bicep from build templates

Here are some optimizations and cleanup from project we created for build.

Sorted params added descriptions
Added support for ACA Addons with the ServiceType and ServiceBinds properties
Allow user to choose to not include dashboard in app insights
Allow user to create secrets in bulk

Repoman Generation Results
Repoman pushed changes to remotes for the following projects:
Project: todo-csharp-cosmos-sql
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-csharp-cosmos-sql -b pr/2369

View Changes | Compare Changes

Project: todo-csharp-sql-swa-func
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-csharp-sql-swa-func -b pr/2369

View Changes | Compare Changes

Project: todo-csharp-sql
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-csharp-sql -b pr/2369

View Changes | Compare Changes

Project: todo-java-mongo-aca
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-java-mongo-aca -b pr/2369

View Changes | Compare Changes

Project: todo-java-mongo
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-java-mongo -b pr/2369

View Changes | Compare Changes

Project: todo-nodejs-mongo-aca
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-nodejs-mongo-aca -b pr/2369

View Changes | Compare Changes

Project: todo-nodejs-mongo-aks
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-nodejs-mongo-aks -b pr/2369

View Changes | Compare Changes

Project: todo-nodejs-mongo-swa-func
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-nodejs-mongo-swa-func -b pr/2369

View Changes | Compare Changes

Project: todo-nodejs-mongo
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-nodejs-mongo -b pr/2369

View Changes | Compare Changes

Project: todo-python-mongo-aca
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-python-mongo-aca -b pr/2369

View Changes | Compare Changes

Project: todo-python-mongo-swa-func
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-python-mongo-swa-func -b pr/2369

View Changes | Compare Changes

Project: todo-python-mongo
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-python-mongo -b pr/2369

View Changes | Compare Changes

Repoman Generation Results
Repoman pushed changes to remotes for the following projects:
Project: todo-csharp-cosmos-sql
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-csharp-cosmos-sql -b pr/2369

View Changes | Compare Changes

Project: todo-csharp-sql-swa-func
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-csharp-sql-swa-func -b pr/2369

View Changes | Compare Changes

Project: todo-csharp-sql
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-csharp-sql -b pr/2369

View Changes | Compare Changes

Project: todo-java-mongo-aca
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-java-mongo-aca -b pr/2369

View Changes | Compare Changes

Project: todo-java-mongo
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-java-mongo -b pr/2369

View Changes | Compare Changes

Project: todo-nodejs-mongo-aca
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-nodejs-mongo-aca -b pr/2369

View Changes | Compare Changes

Project: todo-nodejs-mongo-aks
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-nodejs-mongo-aks -b pr/2369

View Changes | Compare Changes

Project: todo-nodejs-mongo-swa-func
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-nodejs-mongo-swa-func -b pr/2369

View Changes | Compare Changes

Project: todo-nodejs-mongo
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-nodejs-mongo -b pr/2369

View Changes | Compare Changes

Project: todo-python-mongo-aca
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-python-mongo-aca -b pr/2369

View Changes | Compare Changes

Project: todo-python-mongo-swa-func
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-python-mongo-swa-func -b pr/2369

View Changes | Compare Changes

Project: todo-python-mongo
Remote: azure-samples-staging
Branch: pr/2369
You can initialize this project with:
azd init -t Azure-Samples/todo-python-mongo -b pr/2369

View Changes | Compare Changes

Waiting on a test pass before we merge.


Annotation processors must be explicitly declared now

I'm trying to migrate from 1.5.6 to community version 2.0.1
Dependencies in the build.gradle:
    implementation "com.github.moxy-community:moxy:$moxyVersion"
    implementation "com.github.moxy-community:moxy-androidx:$moxyVersion"
    implementation "com.github.moxy-community:moxy-ktx:$moxyVersion"
    kapt "com.github.moxy-community:moxy-compiler:$moxyVersion"

And i got an error:
> Configure project :app
WARNING: API 'variant.getMergeResources()' is obsolete and has been replaced with 'variant.getMergeResourcesProvider()'.
It will be removed at the end of 2019.
For more information, see https://d.android.com/r/tools/task-configuration-avoidance.
To determine what is calling variant.getMergeResources(), use -Pandroid.debug.obsoleteApi=true on the command line to display more information.

> Task :app:preBuild UP-TO-DATE
> Task :app:preStagingDebugBuild UP-TO-DATE
> Task :app:compileStagingDebugAidl NO-SOURCE
> Task :app:checkStagingDebugManifest UP-TO-DATE
> Task :app:generateStagingDebugBuildConfig UP-TO-DATE
> Task :app:compileStagingDebugRenderscript NO-SOURCE
> Task :app:mainApkListPersistenceStagingDebug UP-TO-DATE
> Task :app:mergeStagingDebugShaders UP-TO-DATE
> Task :app:compileStagingDebugShaders UP-TO-DATE
> Task :app:generateStagingDebugAssets UP-TO-DATE
> Task :app:mergeStagingDebugAssets UP-TO-DATE
> Task :app:processStagingDebugGoogleServices UP-TO-DATE
> Task :app:createStagingDebugCompatibleScreenManifests UP-TO-DATE
> Task :app:generateStagingDebugResValues
> Task :app:processStagingDebugManifest UP-TO-DATE
> Task :app:fabricGenerateResourcesStagingDebug
> Task :app:generateStagingDebugResources

> Transform artifact moxy.jar (com.github.moxy-community.moxy:moxy:2.0.2) with DexingNoClasspathTransform
AGPBI: {"kind":"error","text":"Invoke-customs are only supported starting with Android O (--min-api 26)","sources":[{}],"tool":"D8"}

> Task :app:mergeStagingDebugResources

> Transform artifact javapoet.jar (com.squareup:javapoet:1.11.1) with DexingNoClasspathTransform
AGPBI: {"kind":"error","text":"Invoke-customs are only supported starting with Android O (--min-api 26)","sources":[{}],"tool":"D8"}

> Task :app:processStagingDebugResources

> Transform artifact guava.jar (com.google.guava:guava:23.0) with DexingNoClasspathTransform
AGPBI: {"kind":"error","text":"Default interface methods are only supported starting with Android N (--min-api 24): boolean com.google.common.base.Predicate.test(java.lang.Object)","sources":[{}],"tool":"D8"}

> Task :app:kaptGenerateStubsStagingDebugKotlin

> Task :app:kaptStagingDebugKotlin
w: [kapt] Incremental annotation processing requested, but support is disabled because the following processors are not incremental: com.squareup.moshi.kotlin.codegen.JsonClassCodegenProcessor (NON_INCREMENTAL), javaslang.match.PatternsProcessor (NON_INCREMENTAL).

> Task :app:compileStagingDebugKotlin
w: /home/dda/StudioProjects/hypermarket-android-app/app/src/main/kotlin/ru/ng/base/moxy/BasePresenter.kt: (15, 47): Parameter 'parameters' is never used
w: /home/dda/StudioProjects/hypermarket-android-app/app/src/main/kotlin/ru/ng/util/Logging.kt: (154, 21): Name shadowed: thr
w: /home/dda/StudioProjects/hypermarket-android-app/app/src/main/kotlin/ru/ng/util/Logging.kt: (171, 21): Name shadowed: thr
w: /home/dda/StudioProjects/hypermarket-android-app/app/src/main/kotlin/ru/ng/util/Logging.kt: (188, 21): Name shadowed: thr
w: /home/dda/StudioProjects/hypermarket-android-app/app/src/main/kotlin/ru/ng/util/Logging.kt: (205, 21): Name shadowed: thr
w: /home/dda/StudioProjects/hypermarket-android-app/app/src/main/kotlin/ru/ng/util/Logging.kt: (222, 21): Name shadowed: thr

> Task :app:javaPreCompileStagingDebug FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:javaPreCompileStagingDebug'.
> A failure occurred while executing com.android.build.gradle.internal.tasks.Workers$ActionFacade
   > Annotation processors must be explicitly declared now.  The following dependencies on the compile classpath are found to contain annotation processor.  Please add them to the annotationProcessor configuration.
       - moxy-compiler-2.0.2.jar (com.github.moxy-community.moxy:moxy-compiler:2.0.2)
     Alternatively, set android.defaultConfig.javaCompileOptions.annotationProcessorOptions.includeCompileClasspath = true to continue with previous behavior.  Note that this option is deprecated and will be removed in the future.
     See https://developer.android.com/r/tools/annotation-processor-error-message.html for more details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.4.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 33s
17 actionable tasks: 8 executed, 9 up-to-date

Seems like moxy-compiler dependency somehow got into compile classpath, while it should only be used in annotation processing. I tried but couldn't reproduce this problem.
We would appreciate if you provide minimal sample to reproduce it, or show your build.gradle and result of execution of ./gradlew app:dependencies.

Sorry for long response
build.gradle.app.txt
build.gradle.project.txt
dependencies.txt

or show your build.gradle and result of execution of ./gradlew app:dependencies.

Sorry for long response
build.gradle.app.txt
build.gradle.project.txt
dependencies.txt

I investigated your files, checked all the poms in every repository - everything is clear.
For your project to compile moxy-compiler dependency shouldn't be listed in compile classpath dependencies. If you take a look at your dependencies file, under any compileClasspath configuration, you can see it lists all moxy artifacts, not just 3 you requested in your build.gradle. This makes me think problem lies somewhere within project or dependency configuration. Maybe there's something in libs\ folder, or broken gradle cache.

I found the problem.
allprojects {
    repositories {
        google()
        mavenCentral()
        maven { url "https://jitpack.io" } <-- this is cause
        jcenter()
    }
}

All works if raise up jcenter() before maven { url "https://jitpack.io" } :
allprojects {
    repositories {
        google()
        mavenCentral()
        jcenter()
        maven { url "https://jitpack.io" }
    }
}


✅ Two-Stack Enqueue and Dequeue (Lazy Transfer)

✅ Fixes: #1498
Summary:
This PR provides, A complete C implementation of a queue using two stacks with detailed documentation explaining the approach and complexity. Also a flowchart showing the operation logic

The two-stack approach with lazy transfer provides efficient queue operations:
Enqueue: O(1) time complexity
Dequeue: O(1) amortized time complexity
Space complexity: O(n)

The lazy transfer strategy means we only move elements from the input stack to the output stack when necessary, which is more efficient than transferring on every operation.

@pankaj-bind can you review this PR, and assign this issue to me with proper labels

@pankaj-bind @can you review this PR, and add proper labels to it


Create the Preliminary SRD/SDD for Nucleus

Initial requirements defined: https://github.com/NASA-PDS/nucleus/blob/main/docs/requirements/REQUIREMENTS.md


FAQs for Kubearmor

FAQs we can answer:
General Queries

[x] What deployments (GKE, EKS, Tanzu, OpenShift) are supported by KubeArmor? How can I check whether my deployment will be supported?
[x] How is KubeArmor different from PodSecurityPolicy/PodSecurityContext?
[x] What are the different approaches to runtime security? What is kubearmor's approach?
[x] What is visibility that I hear of in KubeArmor and how to get visibility information?

How to get process events in the context of a specific pods?

[x] How is Kubearmor different from Admission Controllers (kyverno, OPA)? Are the use-cases different? What are the risks that kubearmor can additionally cover on top of admission controller?
[x] What are Kubearmor's policy actions? How are they compared to other runtime engines?
[x] What is the difference between KubeArmorHostPolicy and KubeArmorPolicy?
[x] Where can I find examples of realistic policies for real workloads?

It can be found in policy-templates repo.

[x] What are the Policy Actions supported by KubeArmor? (What happens if Block policy is used and enforcement is not supported on the platform?)
[ ] How to enable KubeArmorHostPolicy in k8s env?

Trouble shooting & Diagnostics

[ ] I am running into problems, where can I ask for help and what should I do before asking for help? #499
[ ] KubeArmor policy enforcement works on multiubuntu example but not on my pods. What could be the reasons?
[ ] I am trying the example sleep process deny policy and it is not working.
[ ] Why some of the examples do not work with alpine?
[ ] I am deploying an individual pod, but the policies in the context are not getting enforced.
[ ] I see Action: Audit (Block) but the action is not getting blocked.
[ ] I get this error, module kheaders not found in modules.dep Unable to find kernel headers. What does it mean?

Sample log from kubectl logs -n kube-system kubearmor-xxxxxxx
2022-06-13 20:12:31.893899	INFO	Initialized Containerd Handler
2022-06-13 20:12:31.897327	INFO	Build Time: 2022-06-13 19:59:36.007842487 +0000 UTC
2022-06-13 20:12:31.897402	INFO	Arguments [cluster:default coverageTest:false defaultCapabilitiesPosture:block defaultFilePosture:block defaultNetworkPosture:block enableKubeArmorHostPolicy:false enableKubeArmorPolicy:true enableKubeArmorVm:false gRPC:32767 host:ip-10-0-1-252 hostDefaultCapabilitiesPosture:block hostDefaultFilePosture:block hostDefaultNetworkPosture:block hostVisibility:default k8s:true logPath:/tmp/kubearmor.log seLinuxProfileDir:/tmp/kubearmor.selinux visibility:process,file,network,capabilities]
2022-06-13 20:12:31.897478	INFO	Configuration [{Cluster:default Host:ip-10-0-1-252 GRPC:32767 LogPath:/tmp/kubearmor.log SELinuxProfileDir:/tmp/kubearmor.selinux Visibility:process,file,network,capabilities HostVisibility:default Policy:true HostPolicy:true KVMAgent:false K8sEnv:true DefaultFilePosture:block DefaultNetworkPosture:block DefaultCapabilitiesPosture:block HostDefaultFilePosture:block HostDefaultNetworkPosture:block HostDefaultCapabilitiesPosture:block CoverageTest:false}]
2022-06-13 20:12:31.897497	INFO	Final Configuration [{Cluster:default Host:ip-10-0-1-252 GRPC:32767 LogPath:/tmp/kubearmor.log SELinuxProfileDir:/tmp/kubearmor.selinux Visibility:process,file,network,capabilities HostVisibility:none Policy:true HostPolicy:true KVMAgent:false K8sEnv:true DefaultFilePosture:block DefaultNetworkPosture:block DefaultCapabilitiesPosture:block HostDefaultFilePosture:block HostDefaultNetworkPosture:block HostDefaultCapabilitiesPosture:block CoverageTest:false}]
2022-06-13 20:12:31.897725	INFO	Initialized Kubernetes client
2022-06-13 20:12:31.897745	INFO	Started to monitor node events
2022-06-13 20:12:32.898044	INFO	Node Name: ip-10-0-1-252
2022-06-13 20:12:32.898093	INFO	Node IP: <IP_ADDRESS>
2022-06-13 20:12:32.898134	INFO	Node Annotations: map[kubearmor-policy:audited kubearmor-visibility:none node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true]
2022-06-13 20:12:32.898144	INFO	OS Image: Amazon Linux 2
2022-06-13 20:12:32.898152	INFO	Kernel Version: 5.4.156-83.273.amzn2.x86_64
2022-06-13 20:12:32.898164	INFO	Kubelet Version: v1.21.2-13+d2965f0db10712
2022-06-13 20:12:32.898170	INFO	Container Runtime: docker://19.3.13
2022-06-13 20:12:32.898512	INFO	Initialized KubeArmor Logger
2022-06-13 20:12:32.899236	INFO	checking if kernel headers path (/media/root/usr/src/linux-headers-5.4.156-83.273.amzn2.x86_64) exists
2022-06-13 20:12:32.899334	INFO	Initializing an eBPF program
modprobe: module kheaders not found in modules.dep
Unable to find kernel headers. Try rebuilding kernel with CONFIG_IKHEADERS=m (module) or installing the kernel development package for your running kernel version.
chdir(/lib/modules/5.4.156-83.273.amzn2.x86_64/build): No such file or directory
2022-06-13 20:12:32.903332	ERROR	Failed to initialize BPF (bpf module is nil)
github.com/kubearmor/KubeArmor/KubeArmor/log.Errf
	/usr/src/KubeArmor/KubeArmor/log/logger.go:102
github.com/kubearmor/KubeArmor/KubeArmor/core.(*KubeArmorDaemon).InitSystemMonitor
	/usr/src/KubeArmor/KubeArmor/core/kubeArmor.go:215
github.com/kubearmor/KubeArmor/KubeArmor/core.KubeArmor
	/usr/src/KubeArmor/KubeArmor/core/kubeArmor.go:407
main.main
	/usr/src/KubeArmor/KubeArmor/main.go:44
runtime.main
	/usr/local/go/src/runtime/proc.go:255
2022-06-13 20:12:32.903382	ERROR	Failed to initialize KubeArmor Monitor
github.com/kubearmor/KubeArmor/KubeArmor/log.Err
	/usr/src/KubeArmor/KubeArmor/log/logger.go:97
github.com/kubearmor/KubeArmor/KubeArmor/feeder.(*Feeder).Err
	/usr/src/KubeArmor/KubeArmor/feeder/feeder.go:460
github.com/kubearmor/KubeArmor/KubeArmor/core.KubeArmor

Further FAQs:

[ ] I am using KAta containers/KubeVirt/Firecracker ... How can KubeArmor help?
[ ] Deny ICMP is not working with Ubuntu/AppArmor
[ ] Is there a difference between the level of security offered by KubeArmor towards k8s vs VMs/Bare-metal environment?

How to enable KubeArmorHostPolicy in k8s cluster?
By default the host policies and visibility is disabled for k8s hosts.
If you use following command, kubectl logs -n kube-system <KUBEARMOR-POD> | grep "Started to protect"
you will see, 2023-08-21 12:58:34.641665      INFO    Started to protect containers.
This indicates that only container/pod protection is enabled.
If you have hostpolicy enabled you should see something like this, 2023-08-22 18:07:43.335232      INFO    Started to protect a host and containers
One can enable the host policy by patching the daemonset (kubectl edit daemonsets.apps -n kube-system kubearmor):
...
  template:
    metadata:
      annotations:
        container.apparmor.security.beta.kubernetes.io/kubearmor: unconfined
      creationTimestamp: null
      labels:
        kubearmor-app: kubearmor
    spec:
      containers:
      - args:
        - -gRPC=32767
        - -enableKubeArmorHostPolicy  <----- ADD THIS LINE
        env:
        - name: KUBEARMOR_NODENAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
...


generalize connect to basic_connect

Codecov Report
Attention: Patch coverage is 0% with 127 lines in your changes are missing coverage. Please review.

Project coverage is 0.00%. Comparing base (532458c) to head (39f9ebb).

Files
Patch %
Lines

include/aquarius/connect/basic_connect.hpp
0.00%
85 Missing :warning:

include/aquarius/connect/ssl.hpp
0.00%
23 Missing :warning:

include/aquarius/connect/no_ssl.hpp
0.00%
15 Missing :warning:

test/context.h
0.00%
2 Missing :warning:

test/connect.h
0.00%
1 Missing :warning:

test/proto_regist.h
0.00%
1 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@          Coverage Diff           @@
##           master     #61   +/-   ##
======================================
  Coverage    0.00%   0.00%
======================================
  Files          57      58    +1
  Lines        1616    1592   -24
  Branches      462     455    -7
======================================
+ Misses       1616    1592   -24

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


DBZ-804 Better schema comparison in tests; preparing byte conversion

Preparations in core for https://issues.jboss.org/browse/DBZ-804

@gunnarmorling Applied


Raspberry pi 3 ffmepg hw acceleration doesn't work

All work with my ezviz cam but not with hardware acceleration (black screen on tv)

@Pa0l0ne, can you try increase the GPU memory?
sudo raspi-config

7 Advanced Options ->  A3 Memory Split -> 256


readme updation

Hi,
I hope this message finds you well. I’m eager to contribute to the project and would love to take on the task of enhancing the README. I have already successfully completed one README task, where I made it more interactive and visually appealing by adding GIFs to demonstrate key features.
Please let me know if you can assign this task to me. I’m looking forward to your feedback and am excited to contribute further!
Thank you!
Best regards,
Eshal :)

We're not accepting readme contributions.


[en]fix terms in Kairship

application console, application workbench -> Workbench
configuration item -> configmap
container group -> Pod
expansion and contraction -> autoscaling
镜像仓库 image repository, image repositories -> container registry
image space -> registry space
multi-cloud -> multicloud
stateful workload -> statefulset
stateless workload -> deployment
strategy -> policy
the load instance -> the workload instance
time window -> period
timed task, scheduled task -> CronJob
working cluster, work cluster -> worker cluster
⋮ -> ┇

Related to #4956


Plucking Flowers

💥 Proposal
Jay's neighbour has a garden in which there are N flowers and each flower has a beauty associated with it i.e. ith flower has B[i] beauty. Jay wants to pluck exactly K flowers from his neighbour's garden such that the sum of beauties of all flowers he plucked is maximum but to not get caught he can't pluck more than one flower in a row, more formally he can't pluck two flowers if they are adjacent.
Your task is to find maximum sum of beauty of all flowers Jay can get.
Have you read the Contributing Guidelines on Pull Requests?
Yes

/assign

@HarshCasper  @bhav09  @gat786  can I work on this issue ? I have one PR pending in this project .


Add Backwards Compatibility to Upstart

Not a complete solution, by any means, as I've only added a custom template for centos-6 nodes, but it would be easy enough to add templates for other distros that use older versions of Upstart.

This should (eventually) be refactored into a more general solution, but let's not have perfect be the enemy of good. :+1:


Will update 1.21

Matters to be added to TODO - 将要加入TODO的事项
update gca to 1.21
Describe what this TODO will do - 描述这个TODO要做的事情
please update gca to 1.21

I think, as I have explained in the issue template, this is an internal issue template for a project, please do not use this template! Unless you are an internal member of the project, you should not use this issue template, which is ordering me! And you shouldn't order me to do anything, even if it is in my plan!

Matters to be added to TODO - 将要加入TODO的事项
将 GCA 更新到 1.21
Describe what this TODO will do - 描述这个TODO要做的事情
请将 GCA 更新到 1.21

你可以在行动中获取build构建


The windows mitmdump connection is slow when you download the file

Steps to reproduce the problem:
1.Start mitmdump on Windows
2.Listen for localhost: 8888
3.Click the download button on the web page
Any other comments? What have you tried so far?
At run time, memory is always high and there is no reduction in closing the browser.
System information

Mitmproxy: 4.0.3 binary
Python:    3.6.5
OpenSSL:   OpenSSL 1.1.0h  27 Mar 2018
Platform:  Windows-2008ServerR2-6.1.7601-SP1

It looks like you want to stream large response bodies?
https://docs.mitmproxy.org/stable/overview-features/#streaming

Thank you for your answer. I'll try it.


REST interface for analytics drilldown responds with 500 error code.

Not possible to fetch analytics drilldown data from api-umbrella instance:
GET https://<api-umbrella-domain>/api-umbrella/v1/analytics/drilldown request responds with 500 error code.

For example GET https://<api-umbrella-domain>/api-umbrella/v1/admins  works just fine.
X-Admin-Auth-Token & X-Api-Key headers were provided and are correct.
(Requests tested via POSTMAN)

I think this can happen if you don't pass in one of the required fields, or if the prefix parameter isn't in the right format. But we obviously need to do some better parameter validation on this API, so it provides better feedback rather than throwing a 500 error.
In the meantime, here's an example of a valid initial request request starting at the root:
/api-umbrella/v1/analytics/drilldown.json?interval=day&start_at=2016-07-04&end_at=2016-08-02&prefix=0%2F

Note prefix is 0/ (it must contain a slash). Once you get the root response, you can use the descendent_prefix field in the response to generate further prefix parameters to drill down into the results.

Thanks @GUI! It works fine when fields are passed.

Also, how is it possible to filter by API backend frontend_prefix value using URL above? I've tried a lot of different ways passing elasticsearch query as a search parameter but it did not work. Could you please provide a sample URL which does that?

@frenchbread: You'd filter by the request_path field. You can filter using the search URL parameter if you want to pass in a raw Lucene-style query, or by the query URL parameter if you want to use the query-builder JSON format the default admin tool uses. Here are a couple examples that filter for API traffic beginning with /foo/ (if you url-decode these, the examples will probably make more sense):
Using search (note that with the Lucene syntax forward slashes need to be escaped with backslashes):
/api-umbrella/v1/analytics/drilldown.json?interval=day&start_at=2016-07-04&end_at=2016-08-02&prefix=0%2F&search=request_path%3A%5C%2Ffoo%5C%2F*

Using query with a JSON format:
/api-umbrella/v1/analytics/drilldown.json?interval=day&start_at=2016-07-04&end_at=2016-08-02&prefix=0%2F&query=%7B%22condition%22%3A%22AND%22%2C%22rules%22%3A%5B%7B%22id%22%3A%22request_path%22%2C%22field%22%3A%22request_path%22%2C%22type%22%3A%22string%22%2C%22input%22%3A%22text%22%2C%22operator%22%3A%22begins_with%22%2C%22value%22%3A%22%2Ffoo%2F%22%7D%5D%7D

Hope that helps, but feel free to ask if you still have any other questions.

Hey, @GUI. I'd like to ask about query URL parameter. Try to use your example and I've got 500 error. Although, without this parameter everything is okay.
Could you please tell the right structure of query body?

@marla-singer: The query parameter takes URL-encoded JSON, with the structure coming from jQuery QueryBuilder's rules.
Although, the example query above still works for me, so if it's throwing a 500 error for you, I'm not entirely sure what's going on. You might try seeing what's in the /var/log/api-umbrella/web-puma/current log. If you can share any errors from there, then hopefully we can get to the bottom of the 500 error.

Thanks for your answer! I tested the example again and it worked as expected. There was my fault