Progress percentage not accurate

System OS
Windows
Python Version
3.9 (CPython)
Install Source
pip / PyPi
Install version / commit hash
v4.0.1
Expected Behavior vs Actual Behavior
Expected
Progress percentage and eta to be accurate.
Actual
If songs are skipped they don't seem to be added to the progress percentage. It just starts from 0 then counts up from there.

Also what is up with the search results for songs? On a new phone on a new sim I get the actual music I expect from the artists channels but all the links from spotdl seem to be from random 20-500 view videos with modified audio. I'm not complaining, this is truly great software, out of the couple thousand i have downloaded I would say every 1000 it misses around 40 and gets the wrong link (audio is different to original) wrong on around 10 of them. Would it help if it prioritized channels with "- Topic" in the name or that have the tag "Official Artist Channel", tho I don't think yt_dlp supports that. Anyway, love you guys and the work you do, btw that web UI is looking nice 👌
Steps to reproduce - Ensure to include actual links!
I waited until 1.5k ish originally but I'm sure any number works, just the more, the more obvious.
spotdl download https://open.spotify.com/playlist/7amAY9qfwE7Lp7nMoQ5U0j
Also got rate limited at 1.4k, waiting 10 mins allowed another 300 before I got limited again. Nothing on you guys, its just spotify, but is there any cooldown function that can be set?
Running the program after getting rate limited or getting a decent chuck will now start the percentage back at 0%
spotdl download https://open.spotify.com/playlist/7amAY9qfwE7Lp7nMoQ5U0j
Traceback
Comment is too long (maximum is 65536 characters)
https://pastebin.com/raw/X4jV8UkZ

Other details
Got rate limited at the end, hence the invalid client and 404. If I try again after a crash I get the old 429 error 😂

We use YouTube Music to get the search results, so the results with "- Topic" in the channel names are always prioritized. If you could send some links to wrong songs I would appreciate that.
If you don't want to be constantly rate limited you can lower the number of threwdw. Downloading will take longer but should be more stable.
You can except fixed progress bar/percentage in next release (no ETA yet)

fixed on dev branch

We use YouTube Music to get the search results, so the results with "- Topic" in the channel names are always prioritized. If you could send some links to wrong songs I would appreciate that.

Oh ok nice. Sorry about that then. I'll keep an eye out for any wrong songs that I find 👌

If you don't want to be constantly rate limited you can lower the number of threwdw. Downloading will take longer but should be more stable.

Yeah ok, thanks for that. Prob should have done that before wasting your time with that 😅

You can except fixed progress bar/percentage in next release (no ETA yet)

Sounds good man

Found another lil bug with the progress bar.
Mainly visual, but the weird characters of the song seem to give the progress bar some issues. So far I've only had it happen with this song.
Reproduce with this:
spotdl download https://open.spotify.com/track/58o6AY4WfMsGlVrpfsbtGs
Win 10 - Python 3.9 - Spotdl v4.0.1

ffmpeg_error_2022-10-30-21-30-31.txt


Customize router in CRC

General information

OS: Linux
Hypervisor: KVM
Did you run crc setup before starting it (Yes/No)? yes

CRC version
crc version: 1.2.0+c2e3c0f
OpenShift version: 4.2.8 (embedded in binary)

CRC status
CRC VM:          Running
OpenShift:       Running (v4.2.8)
Disk Usage:      11.14GB of 32.2GB (Inside the CRC VM)
Cache Usage:     11.01GB
Cache Directory: /home/riso/.crc/cache

CRC config
- skip-check-crc-dnsmasq-file           : true
- skip-check-network-manager-config     : true
- skip-check-network-manager-installed  : true
- skip-check-network-manager-running    : true

Host Operating System
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"

Hello, I am quite new in OpenShift and not sure if this is the proper place where to ask questions.
I installed CRC on Ubuntu after few workarounds it works fine. I am surprise, if there is any way how I could customize router in CRC, or add 2nd. router. I can not find any documentation about this topic for OpenShift 4, just for openshift 3 (https://docs.openshift.com/container-platform/3.11/install_config/router/customized_haproxy_router.html). Also I would like to try if I can use nginx as a router with my custom module. Thanks for help

This is not related to crc, please ask it either https://github.com/openshift/origin/issues or may be send a mail to openshift-user ML. There is lot of changes happen from 3.x to 4.x . Closing this issue from CRC side.


pxz: undefined reference to `lzma_code'

I'd think this means it's either including the wrong header or linking the wrong library, but I can't see any reason that might be.
https://gist.github.com/bukzor/ca7d5a5710c94f3822d8

The error is in the link command line. The linked libraries -l must come after the source files. -llzma pxz.c -o pxz should be pxz.c -o pxz -llzma.
/usr/bin/gcc-4.8 -Wall -Wshadow -Wcast-align -Wunreachable-code -Winline -Wextra -Wmissing-noreturn  -Os -w -pipe -march=core2 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DPXZ_BUILD_DATE=\"`date +%Y%m%d`\" -DPXZ_VERSION=\"4.999.9beta\" -O2 -fopenmp -L/home/buck/prefices/brew/lib -Wl,-rpath,/home/buck/prefices/brew/lib -llzma pxz.c -o pxz

Please report this bug to the upstream author of the software.

Shouldn't we find just the same bug on osx?
Is this different between clang/gcc?

On OS X the linker allows the -l command line options to precede the source files. On Linux it does not.


CIR w/o text gives better results on my custom dataset

Hi @Chiaraplizz,
I am running the CIR and CIR w/o text on my custom dataset. I followed the instructions provided and used the following configs for the experiments. However, I am seeing CIR w/o text is giving better results than CIR. Can you please look into the config files and let me know if they are correct? I do not have scenario_idx in my dataset.
CIR w/o Text:
use_ffcv:
  desc: set to True to use ffcv compressed datasets
  value: True
seed:
  desc: set the seed
  value: 0
dataset_splits:
  value: [ "train", "test", "validation"]
dataset_csvs:
  value: ["India_train_argo1m_format.csv", "India_test_argo1m_format.csv", "India_val_argo1m_format.csv"]
dataset_ffcvs:
  value: ["India_train_argo1m_format.beton", "India_test_argo1m_format.beton", "India_val_argo1m_format.beton"]
ffcv_pre_shuffle:
  desc: shuffle dataset before encoding
  value: True
ffcv_path:
  desc: ffcv base directory
  value: ./data/ego-exo4d
ffcv_order:
  desc: Order used for training. Evaluation is always sequential. RANDOM, QUASI_RANDOM or SEQUENTIAL.
  value: QUASI_RANDOM
ffcv_os_cache:
  desc: Set to true if whole dataset fits in memory (and use RANDOM).
  value: False
ffcv_batches_ahead:
  value: 3

csv_path:
  desc: Action annotation csv
  value: ./data_csv/ego-exo4d
csv_shuffle:
  desc: shuffle the dataloader for csv files
  value: True
csv_len:
  desc: only load n samples (useful for debugging). Set to -1 to ignore and use the full dataset.
  value: -1
base_path:
  desc: path where to save runs
  value: ./argo-1m-experiments

use_text:
  desc: whether to use text features or not
  value: False
CIR_text_dim:
  desc: dimensionality of the features for CIR_text
  value: 512
text_model:
  desc: type of text model used
  value: 'clip-ViT-B-32'
text_dim:
  desc: embedding output dimension of text model used
  value: 512
gen_attn:
  desc: type of attention for recnstruction
  value: 'other'
CIR_text_attn:
  desc: type of attention for building the CIR_text support set, choose between "fixed" and "learned"
  value: 'learned'
CIR_attn:
  desc:
  value: 'fixed'

feat_path:
  desc: Path to flattened features
  value: /workspace/data/ego-exo4d-DA-features

sample_mode:
  desc: How to sample features for a given action. E.g. start takes the first feature.
  value: customized
n_action_subsample:
  desc: Sequence length of loaded features inside the segment narration.
  value: 1
n_before_after_context:
  desc: Sequence length of loaded features before/after.
  value: 0
resume:
  value: False
last_epoch:
  value: 0
resume_date:
  value: None
save_model:
  value: True
labels:
  desc: Labels returned by the dataloader. Should include action label first, followed by domain labels.
  value: [ 'action', 'scenario_idx', 'location_idx' ]
store_dir:
  desc: Directory where to store the model
  value:  ./argo-1m-experiments
model:
  desc: The model to use for extracting video embeddings
  value: 'MLP'
model_prediction:
  desc: The model to use for the final prediction
  value: 'Classifier.logits'
model_types:
  desc: The model to use for training, as defined in models.py
  value: ['MLP', 'CIR', 'Classifier']
model_names:
  desc: Names corresponding to the model_types
  value: ['MLP', 'CIR', 'Classifier']
model_lrs:
  desc: Learning rate for the models
  value: [ 0.0002, 0.0002, 0.0002, 0.0002, 0.0002 ]
step:
  desc: Whether to do a lr step for the models
  value: [True, True, True, True, True]
model_inputs:
  desc: The model inputs
  value: [{ "x": "data.rgb_feat" },  { "x": "MLP.representations", 'target': 'data.target' },{ "x": "MLP.representations","x_mixed":"CIR.feat" },{ "video_features": "MLP.representations", "target": "data.target" }  ]
model_use_train:
  desc: Models used during training
  value: [ True, True, True]
model_use_eval:
  desc: Models used for evaluation
  value: [True, False, True]

bottleneck_dim:
  desc: bottleneck dimension for the learnable cross-attention
  value: 128
mlp_hidden_dims:
  desc: Dimensions of hidden layers in mlp. Needs at least one dimension.
  value: [ 4096, 512 ]
mlp_dropout:
  desc: dropout prob for use in mlp hidden layers
  value: 0.5
prompt_dim:
  desc: dim of the reduced feature vector
  value: 64

loss_types:
  desc: Loss function to use from losses.py
  value: [ "CE",  "CE" ]
loss_names:
  desc: Loss function names corresponding the the loss_types
  value: [ "CE", "CIR"]
loss_inputs:
  desc: Loss function inputs to feed to the loss
  value: [ { 'output': 'Classifier.logits', 'target': 'data.target'},{ 'output': 'Classifier.logits_mixed', 'target': 'data.target' }, { 'output': 'CIR_text.representations', 'target': 'data.target' }, ]

loss_weights:
  desc: Weightings applied to losses when summing them
  value: [ 1.0, 0.5, 1.0, 1.0 ]
LA_tau:
  value: 1.0
weight_decay:
  value: 0.0

boda_upsilon:
  desc: power of the BODA calibrated distance
  value: 1.0
boda_dist:
  desc: distance function between samples
  value: mean
boda_balance_domain:
  desc: Set to true for Eq. 3 and Eq. 4
  value: True
boda_calibrate_distance:
  desc: Set to true for Eq. 4
  value: True
boda_dists:
  desc: choose from mean or mahalanobis. Mahalanobis will probably NAN, as the representation dimension > n samples for some class/domain sets.
  value: mean
boda_level:
  desc: Level of feat we want to apply BODA to.
  value: -1

update_stat_fraction:
  desc: fraction of training dataset used to update stats each epoch
  value: 0.1
update_stat_alpha:
  desc: momentum used in running update of distribution statistics. refers to weight of previous epoch.
  value: 0.5
stat_update_start_epoch:
  desc: epoch to start tracking distribution stats
  value: 100
dist_stats:
  desc: distribution statistics for bode loss. Choose between ["c_d_means", "c_d_vars", "c_d_covs"]
  value: [ "c_d_means" ]

batch_size:
  value: 128

optimizer:
  value: Adam

sch_milestones:
  value: [ 30,40 ]

sch_gamma:
  value: 0.1

epochs:
  value: 50

n_gpu:
  value: 1

feat_dim:
  value: 2304

n_classes:
  value: 60

n_scenarios:
  value: 10

n_locations:
  value: 13

n_workers:
  desc: Number of dataloader workers.
  value: 1

CIR w/ Text:
use_ffcv:
  desc: set to True to use ffcv compressed datasets
  value: True
seed:
  desc: set the seed
  value: 0
dataset_splits:
  value: [ "train", "test", "validation"]
dataset_csvs:
  value: ["India_train_argo1m_format.csv", "India_test_argo1m_format.csv", "India_val_argo1m_format.csv"]
dataset_ffcvs:
  value: ["India_train_argo1m_format.beton", "India_test_argo1m_format.beton", "India_val_argo1m_format.beton"]
ffcv_pre_shuffle:
  desc: shuffle dataset before encoding
  value: True
ffcv_path:
  desc: ffcv base directory
  value: ./data/ego-exo4d
ffcv_order:
  desc: Order used for training. Evaluation is always sequential. RANDOM, QUASI_RANDOM or SEQUENTIAL.
  value: QUASI_RANDOM
ffcv_os_cache:
  desc: Set to true if whole dataset fits in memory (and use RANDOM).
  value: False
ffcv_batches_ahead:
  value: 3

csv_path:
  desc: Action annotation csv
  value: ./data_csv/ego-exo4d
csv_shuffle:
  desc: shuffle the dataloader for csv files
  value: True
csv_len:
  desc: only load n samples (useful for debugging). Set to -1 to ignore and use the full dataset.
  value: -1
base_path:
  desc: path where to save runs
  value: ./argo-1m-experiments

use_text:
  desc: whether to use text features or not
  value: True
CIR_text_dim:
  desc: dimensionality of the features for CIR_text
  value: 512
text_model:
  desc: type of text model used
  value: 'clip-ViT-B-32'
text_dim:
  desc: embedding output dimension of text model used
  value: 512
gen_attn:
  desc: type of attention for recnstruction
  value: 'other'
CIR_text_attn:
  desc: type of attention for building the CIR_text support set, choose between "fixed" and "learned"
  value: 'learned'
CIR_attn:
  desc:
  value: 'fixed'

feat_path:
  desc: Path to flattened features
  value: /workspace/data/ego-exo4d-DA-features

sample_mode:
  desc: How to sample features for a given action. E.g. start takes the first feature.
  value: customized
n_action_subsample:
  desc: Sequence length of loaded features inside the segment narration.
  value: 1
n_before_after_context:
  desc: Sequence length of loaded features before/after.
  value: 0
resume:
  value: False
last_epoch:
  value: 0
resume_date:
  value: None
save_model:
  value: True
labels:
  desc: Labels returned by the dataloader. Should include action label first, followed by domain labels.
  value: [ 'action', 'scenario_idx', 'location_idx' ]
store_dir:
  desc: Directory where to store the model
  value:  ./argo-1m-experiments
model:
  desc: The model to use for extracting video embeddings
  value: 'MLP'
model_prediction:
  desc: The model to use for the final prediction
  value: 'Classifier.logits'
model_types:
  desc: The model to use for training, as defined in models.py
  value: [ 'TextModel', 'MLP', 'CIR', 'Classifier', 'CIR_text']
model_names:
  desc: Names corresponding to the model_types
  value: [ 'TextModel', 'MLP',  'CIR', 'Classifier', 'CIR_text']
model_lrs:
  desc: Learning rate for the models
  value: [ 0.0002, 0.0002, 0.0002, 0.0002, 0.0002 ]
step:
  desc: Whether to do a lr step for the models
  value: [True, True, True, True, True]
model_inputs:
  desc: The model inputs
  value: [ { "text": "data.text" }, { "x": "data.rgb_feat" },  { "x": "MLP.representations", 'target': 'data.target' },{ "x": "MLP.representations","x_mixed":"CIR.feat" },{ "video_features": "MLP.representations", "text_features": "TextModel.text_feat", "target": "data.target" }  ]
model_use_train:
  desc: Models used during training
  value: [ True, True, True, True, True ]
model_use_eval:
  desc: Models used for evaluation
  value: [ False, True, False, True, False ]

bottleneck_dim:
  desc: bottleneck dimension for the learnable cross-attention
  value: 128
mlp_hidden_dims:
  desc: Dimensions of hidden layers in mlp. Needs at least one dimension.
  value: [ 4096, 512 ]
mlp_dropout:
  desc: dropout prob for use in mlp hidden layers
  value: 0.5
prompt_dim:
  desc: dim of the reduced feature vector
  value: 64

loss_types:
  desc: Loss function to use from losses.py
  value: [ "CE",  "CE",  "CIR_text_loss" ]
loss_names:
  desc: Loss function names corresponding the the loss_types
  value: [ "CE", "CIR", "CIR_text_loss"]
loss_inputs:
  desc: Loss function inputs to feed to the loss
  value: [ { 'output': 'Classifier.logits', 'target': 'data.target'},{ 'output': 'Classifier.logits_mixed', 'target': 'data.target' }, { 'output': 'CIR_text.representations', 'target': 'data.target' }, ]

loss_weights:
  desc: Weightings applied to losses when summing them
  value: [ 1.0, 0.5, 1.0, 1.0 ]
LA_tau:
  value: 1.0
weight_decay:
  value: 0.0

boda_upsilon:
  desc: power of the BODA calibrated distance
  value: 1.0
boda_dist:
  desc: distance function between samples
  value: mean
boda_balance_domain:
  desc: Set to true for Eq. 3 and Eq. 4
  value: True
boda_calibrate_distance:
  desc: Set to true for Eq. 4
  value: True
boda_dists:
  desc: choose from mean or mahalanobis. Mahalanobis will probably NAN, as the representation dimension > n samples for some class/domain sets.
  value: mean
boda_level:
  desc: Level of feat we want to apply BODA to.
  value: -1

update_stat_fraction:
  desc: fraction of training dataset used to update stats each epoch
  value: 0.1
update_stat_alpha:
  desc: momentum used in running update of distribution statistics. refers to weight of previous epoch.
  value: 0.5
stat_update_start_epoch:
  desc: epoch to start tracking distribution stats
  value: 100
dist_stats:
  desc: distribution statistics for bode loss. Choose between ["c_d_means", "c_d_vars", "c_d_covs"]
  value: [ "c_d_means" ]

batch_size:
  value: 128

optimizer:
  value: Adam

sch_milestones:
  value: [ 30,40 ]

sch_gamma:
  value: 0.1

epochs:
  value: 50

n_gpu:
  value: 1

feat_dim:
  value: 2304

n_classes:
  value: 60

n_scenarios:
  value: 10

n_locations:
  value: 13

n_workers:
  desc: Number of dataloader workers.
  value: 1

Thanks in advance!

Hello,
The scripts look correct!
Note that in Tab.2 of the main paper we show that in some splits CIR w/o text performs slightly better (e.g., Knitting in India), so that might be your case.
Best,
Chiara

Alright, thanks!

Hi @Chiaraplizz, I am not able to run the DoPrompt baseline. Is it possible to share the config file for the same? TIA!

Hi @Chiaraplizz, I am not able to run the DoPrompt baseline. Is it possible to share the config file for the same? TIA!

Sorry for the late reply, I was offline last week.
You can use this config file:
use_ffcv:
  desc: set to True to use ffcv compressed datasets
  value: True
seed:
  desc: set the seed
  value: 0
dataset_splits:
  value: [ "train", "test", "validation"]
dataset_csvs:
  value: ["Ar_ITA_train.csv", "Ar_ITA_test.csv", "Ar_ITA_test.csv"]
dataset_ffcvs:
  value: ["Ar_ITA_train.beton", "Ar_ITA_test.beton", "Ar_ITA_test.beton"]
ffcv_pre_shuffle:
  desc: shuffle dataset before encoding
  value: True
ffcv_path:
  desc: ffcv base directory
  value: /scratch/chiarap/
ffcv_order:
  desc: Order used for training. Evaluation is always sequential. RANDOM, QUASI_RANDOM or SEQUENTIAL.
  value: QUASI_RANDOM
ffcv_os_cache:
  desc: Set to true if whole dataset fits in memory (and use RANDOM).
  value: False
ffcv_batches_ahead:
  value: 3

csv_path:
  desc: Action annotation csv
  value:  /scratch/chiarap/
csv_shuffle:
  desc: shuffle the dataloader for csv files
  value: True
csv_len:
  desc: only load n samples (useful for debugging). Set to -1 to ignore and use the full dataset.
  value: -1
base_path:
  desc: path where to save runs
  value: /scratch/chiarap/

use_text:
  desc: whether to use text features or not
  value: False
CIR_text_dim:
  desc: dimensionality of the features for CIR_text
  value: 512
text_model:
  desc: type of text model used
  value: 'clip-ViT-B-32'
text_dim:
  desc: embedding output dimension of text model used
  value: 512
gen_attn:
  desc: type of attention for recnstruction
  value: 'other'
CIR_text_attn:
  desc: type of attention for building the CIR_text support set, choose between "fixed" and "learned"
  value: 'learned'
CIR_attn:
  desc:
  value: 'fixed'

feat_path:
  desc: Path to flattened features
  value: /path/to/feat/

sample_mode:
  desc: How to sample features for a given action. E.g. start takes the first feature.
  value: customized
n_action_subsample:
  desc: Sequence length of loaded features inside the segment narration.
  value: 3
n_before_after_context:
  desc: Sequence length of loaded features before/after.
  value: 0
resume:
  value: False
last_epoch:
  value: 0
resume_date:
  value: None
save_model:
  value: False
labels:
  desc: Labels returned by the dataloader. Should include action label first, followed by domain labels.
  value: [ 'label_idx', 'scenario_idx', 'location_idx' ]
store_dir:
  desc: Directory where to store the model
  value:  /path/to/save_dir/
model:
  desc: The model to use for extracting video embeddings
  value: 'MLP'
model_prediction:
  desc: The model to use. Names as in model.py
  value: 'CE_DoPrompt.out_prompt'
model_types:
  desc: The model to use. Names as in model.py
  value: [  'DoPrompt', 'DoPrompt_Classifier']
model_names:
  desc: The model to use. Names as in model.py
  value: [ 'DoPrompt', 'CE_DoPrompt']
model_lrs:
  desc: Learning rate for the models
  value: [0.000001, 0.000001]
step:
  desc: Whether to do a lr step for the models
  value: [ True, True, True]
model_inputs:
  desc: The model inputs
  value: [{ "x": "data.rgb_feat", 'target': 'data.target'}, {"x": "DoPrompt.feat_cls"}]
model_use_train:
  desc: Models used during training
  value: [ True, True, True]
model_use_eval:
  desc: Models used for evaluation
  value: [ True, True, False]

bottleneck_dim:
  desc: bottleneck dimension for the learnable cross-attention
  value: 128
mlp_hidden_dims:
  desc: Dimensions of hidden layers in mlp. Needs at least one dimension.
  value: [ 4096, 512 ]
mlp_dropout:
  desc: dropout prob for use in mlp hidden layers
  value: 0.5
prompt_dim:
  desc: dim of the reduced feature vector
  value: 4

loss_names:
  desc: Loss function to use from losses.py
  value: [ "CE_DoPrompt", "CE_DoPrompt_prompt", "CE_DoPrompt_adapt_scenario", "CE_DoPrompt_adapt_location"]
loss_types:
  desc: Loss function to use from losses.py
  value: [ "CE", "CE", 'CE_DoPrompt_adapt_S', 'CE_DoPrompt_adapt_L']
loss_inputs:
  desc: Loss function inputs to feed to the loss
  value: [ {'output': 'CE_DoPrompt.out_cls', 'target': 'data.target'}, { 'output': 'CE_DoPrompt.out_prompt', 'target': 'data.target' }, { 'output': 'DoPrompt.prompt_weights_scenario', 'target': 'data.target'}, { 'output': 'DoPrompt.prompt_weights_location', 'target': 'data.target'}]

loss_weights:
  desc: Weightings applied to losses when summing them
  value: [1.0, 1.0, 1.0, 1.0]
LA_tau:
  value: 1.0
weight_decay:
  value: 0.0

boda_upsilon:
  desc: power of the BODA calibrated distance
  value: 1.0
boda_dist:
  desc: distance function between samples
  value: mean
boda_balance_domain:
  desc: Set to true for Eq. 3 and Eq. 4
  value: True
boda_calibrate_distance:
  desc: Set to true for Eq. 4
  value: True
boda_dists:
  desc: choose from mean or mahalanobis. Mahalanobis will probably NAN, as the representation dimension > n samples for some class/domain sets.
  value: mean
boda_level:
  desc: Level of feat we want to apply BODA to.
  value: -1

update_stat_fraction:
  desc: fraction of training dataset used to update stats each epoch
  value: 0.1
update_stat_alpha:
  desc: momentum used in running update of distribution statistics. refers to weight of previous epoch.
  value: 0.5
stat_update_start_epoch:
  desc: epoch to start tracking distribution stats
  value: 100
dist_stats:
  desc: distribution statistics for bode loss. Choose between ["c_d_means", "c_d_vars", "c_d_covs"]
  value: [ "c_d_means" ]

batch_size:
  value: 128

optimizer:
  value: Adam

sch_milestones:
  value: [ 30,40 ]

sch_gamma:
  value: 0.1

epochs:
  value: 50

n_gpu:
  value: 1

feat_dim:
  value: 2304

n_classes:
  value: 60

n_scenarios:
  value: 11

n_locations:
  value: 13

n_workers:
  desc: Number of dataloader workers.
  value: 1


[TASK] Bump the version of SLES, RHEL, Ubuntu in the integration test pipeline

What's the task? Please describe.
Use the latest version of the SLES, Ubuntu and RHEL in the integration test pipeline
https://ci.longhorn.io/job/public/job/master/job/rhel/job/amd64/job/longhorn-tests-rhel-amd64/
Describe the items of the task (DoD, definition of done) you'd like.

[x] Use the latest ami for the instances running the Longhorn tests.
[ ] Run the test and verify the result

SLES jobs:

https://ci.longhorn.io/job/public/job/master/job/sles/job/amd64/job/longhorn-tests-sles-amd64/14/console
https://ci.longhorn.io/job/public/job/master/job/sles/job/amd64/job/longhorn-upgrade-tests-sles-amd64/7/console

RHEL jobs:

https://ci.longhorn.io/job/public/job/master/job/rhel/job/amd64/job/longhorn-tests-rhel-amd64/29/console
https://ci.longhorn.io/job/public/job/master/job/rhel/job/amd64/job/longhorn-upgrade-tests-rhel-amd64/19/console

RHEL 8.4.0: Terraform step will take longer time than expected, will check the root cause.

We've reverted to use RHEL 8.3 for now.

@meldafrawi Please close this if it's done.


Adjust condition to display delete button for tasks

Details

Fixed Issues

$ https://github.com/Expensify/App/issues/49408
PROPOSAL: https://github.com/Expensify/App/issues/49408#issuecomment-2361112776
Tests

QA Steps

[x] Verify that no errors appear in the JS console

Offline tests

QA Steps

Log in as user A
Open chat with user B
Create a task and assign it to user C
As user B open the task details
Verify that user B unable to complete a task
Click on header and delete task
Ensure that user B unable to delete task
As user A complete the task
Ensure that user B unable to incomplete task

[x] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native

https://github.com/user-attachments/assets/a1d6a6fc-0059-4610-a3cf-f5c3e342be17

Android: mWeb Chrome

https://github.com/user-attachments/assets/52fee8c2-a115-4080-8549-3e7bb120dfa8

iOS: Native
https://github.com/user-attachments/assets/6ce98d99-e527-442b-be9c-eeba0596eaeb

iOS: mWeb Safari

https://github.com/user-attachments/assets/10946bf2-6fe7-479f-8d67-86367d2f1186

MacOS: Chrome / Safari

https://github.com/user-attachments/assets/5847527e-44da-4c1e-93ed-e7933dc46149

MacOS: Desktop

https://github.com/user-attachments/assets/c5236d8a-773c-4fd8-aa08-6a44b1f6b4ff

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.


Add remaining DCT keys and key vault vars

Problem statement
DCT keys for signing images need to be added to a key vault in each environment, as it currently only resides on staging.
Once signing keys are available, add environment variables of the key vault names to the "deploy-backend" local action.
What you need to know
Blocked by #1647
Acceptance criteria

[ ] CDC approved signing keys added to key vault in each env
[ ] Env vars updated in "deploy-backend" local action to use they keys

We need to get a public AND private key for the production environment. Chris needs to do this.

Let's see if we can move forward with this without direct Azure access.


Package request: o3de

Project description
Open 3D Engine (O3DE) is an Apache 2.0-licensed multi-platform 3D engine that enables developers and content creators to build AAA games, cinema-quality 3D worlds, and high-fidelity simulations without any fees or commercial obligations.
Metadata

homepage URL: https://o3de.org/
source URL: https://github.com/o3de/o3de
license: mit, apache-2.0, unfree
platforms: unix

I can try working on a derivation, but I'm not sure if I will be able to build this locally.

Any news on this?

@QushyQushy there's actually a bunch of issues with building o3de, and most of it has to do with how the bash scripts fail, because of the shebang in Python and Shell script respectively. I don't have a lot of experience with using patches in Nix expressions. This project tries to download it's own Python binaries, which isn't reproducible, I think.  It would be better if someone with proper Nix proficiency looks into this. However, I can give this a last try.

@QushyQushy there's actually a bunch of issues with building o3de, and most of it has to do with how the bash scripts fail, because of the shebang in Python and Shell script respectively. I don't have a lot of experience with using patches in Nix expressions. This project tries to download it's own Python binaries, which isn't reproducible, I think.  It would be better if someone with proper Nix proficiency looks into this. However, I can give this a last try.
Here's the build instructions for Linux, if you want to try them for yourself.

Thank you so much! I have zero familiarity with the Nix build system unfortunately. I thought it would be as simple as downloading the .deb file and extracting it in the proper places but I have no idea in reality.

Can we reopen this? I don't want to make a duplicate issue

@polypoyo I've re-opened this.

I've tried to build o3de using nix + nixpkgs and here some thoughts about it. Even though build process pretty good documented o3de has one big issue regarding to building process: O3DE Packages. As well as other big c++/cmake projects this project invented its own cross-platform dependency management tooling. In general terms CMake scripts (engine itself, gems and project files) contains O3DE Packages declarations (here is the example from engine scripts: https://github.com/o3de/o3de/blob/c753ddf5c25c8cc7aa44845afd035e3996962e81/cmake/3rdParty/Platform/Linux/BuiltInPackages_linux_x86_64.cmake#L20-L48) with tar hashes. CMake functions tries to download, check hash, unpack and check unpacked file hashes of these O3DE Packages into $LY_3RDPARTY_PATH folder. CMake targets then can refer to these O3DE packages as 3rdParty::ZLIB, for example.
This approach obviously conflicts with nix + nixpkgs because:

nix build environment with enabled sandboxing will not allow build script to access network.
Even if we will pre-download all these dependencies binaries from these packages will not run in nixos due to incorrect interpreter path and incorrect RPATH.
Even if we will pre-download all required dependencies and run patchelf on them for engine code what to do with O3DE packages declared in gems and projects?

I see several solutions to fix this issue, one worse than other:

Pre-download and pre-patch all required dependencies for engine itself, require special environment for packages with all required dependencies pre-downloaded and pre-patched. Kinda true nix-way but it is pretty hard-to-use for package users.
Pre-download binaries for engine and hack somehow cmake functions in https://github.com/o3de/o3de/blob/development/cmake/3rdParty.cmake and https://github.com/o3de/o3de/blob/development/cmake/3rdPartyPackages.cmake so they will automatically patch binaries after downloading.

Maybe somebody here has more experience with dealing cmake-based package managers?
NB: it is just build-related issues, I think there can be bunch of other questions regarding to the usability: engine registration, packaging project for release, etc. I think this questions should be discussed separately.


Updating ozone docs

Changes:

changed customData format from string to array

Just a fix to the docs, @bretg .

ok cool - merged.


New Soft ban is banning a tone of users

The new soft ban now bans you so you don't see pokestops, many users say it is because of the new "car passenger" alert, because it isn't dismissed and the bot still takes pokestops, it bans you,
I got banned 2 times+ ban IP
I'm really pissed of rn,
was level 24 , got banned , then started an other account and was level 8 and got banned....
even after setting my walk speed to 4 with a delay of 8s between actions

is it a softball or permaban

Ok i got soft banned about 6 hours ago and heres what happened:
-At first I couldn't catch pokemon or spin pokestops
-After 2 hours I could catch pokemon but not spin pokestops
-After the full 6 hours I could play normally
looks like they're increasing the softban severity in a addition to issuing permabans to hardcore botters. Bots are no longer safe to use until a workaround is found to bypass this. The bot's own softban bypass no longer works,

It looks like they are banning on IP.
Try to use 4g tethered instead and provide us a result.
For me, it is passed !!!

I'm not ban IP, but i've got 2 accounts with "Failed to get game data from server"

Same here Tomygames :( played one account to level 22 (legit!) and then I used the bot on it and now its gone. :(

@MrAppleMania permaban hammer =/

Same here "Failed to get game data from server"
is this a ban?  or just server failure?

@myworkpool It's permaban

@tnsbm   Thanks,   :(

@tnsbm I got permabanned, thing is, I don't really care. The whole point of me using this bot was that it was a proof of concept. Now that we know that it works, we find a better way to hide the fact we're spoofing and botting.


Gear button crashes app when selecting non owned hotspot

I was trying to check hotspots in my area and accidentally hit it and the app closed itself to error page showed up to catch this occuring. It was occuring at the same time that I wasn't able to view un owned hotspots in my area in another issue I posted.

same as the other issue - app version and device, maybe screenshot of the error?
I'm still unable to reproduce.

No error got thrown when it crashed. It just went back to my home screen on android device.

I just tested this on Android, but on 3.2.2 since that's our next release and it'll be released later this week. I'm unable to reproduce. If it's an android, what android device is it?

checking in to see if this is happening with the 3.2.2 release @techman05 and if it is, please let me know what android device and OS version.
Thanks.

I haven't had that happen recently yet on recent but I did find out  that I can only use find a hotspot mode 3 times before the app crashes now. My rak miner wasn't turnes on enough  and I hit it enough times the app gave up.   This is on android 11 on samsung s20+


BUGFIX: Update args struct generation to add yml suffix instead of json

When runnning go-swagger3 only with --generate-yaml and no --output flag the file written is oas.json but the content is yaml. Added check to ensure a yaml file is generated instead
Fixes #13
Signed-off-by: Alvaro Frias Garay<EMAIL_ADDRESS>

Thanks @qequ


unexpected character " U+1B parsing .xmq

Strange bytes at the beginning of an .xmq prevent parsing it back by xmq.
xmq --version
xmq: 2.8.0

xmq simple.xml > simple.xmq

cat simple.xmq
root

file simple.xmq
simple.xmq: ASCII text, with escape sequences

hexdump simple.xmq
0000000 5b1b 3b30 3833 353b 313b 3636 726d 6f6f
0000010 0a74
0000012

xmq simple.xmq to-xml
simple.xmq:1:1: error: unexpected character " U+1B
root
 ^

Binary from https://github.com/libxmq/xmq/releases/download/2.8.0/xmq-macosx-release.gz .
Running OSX 14.4.1.

... forgot to mention that the source simple.xml is of course nothing strange:
cat simple.xml
<?xml version="1.0" encoding="UTF-8"?>
<root/>

hexdump simple.xml
0000000 3f3c 6d78 206c 6576 7372 6f69 3d6e 3122
0000010 302e 2022 6e65 6f63 6964 676e 223d 5455
0000020 2d46 2238 3e3f 3c0a 6f72 746f 3e2f 000a
000002f

Hi! Here's the debug:
xmq --debug simple.xml > simple.xmq
[xmq] detected terminal xterm-256color
(xmq) COLORFGBG means dark
[xmq] option --debug
(xmq) added to-xmq command
(xmq) added print command
[xmq] performing load
(xmq) cmd-load simple.xml
[XMQ] file size 47
[XMQ] read 47 bytes total 0
[XMQ] read total 47 bytes fsize 47 bytes
[XMQ] content detected as xml since <?xml found
[XMQ] fixup comments after readin
[XMQ] fixup comments |root element
[XMQ] trim element
(xmq) cmd-load 47 bytes from simple.xml
[xmq] performing to-xmq
(xmq) cmd-to xmq render terminal
[xmq] performing print
(xmq) cmd-print output
(xmq) cmd-unload document
[XMQ] freeing source name
[XMQ] freeing xml doc
[XMQ] freeing xmq doc

od -t x1 simple.xmq
0000000    1b  5b  30  3b  33  38  3b  35  3b  31  36  36  6d  72  6f  6f
0000020    74  0a
0000022

Thanks, it was the COLORFGBG that forced color even when writing to a file! Bug. This is now fixed. Release 2.8.1
I also changed the ordering of background color lookup, if you are running in an xterm which can report
the actual background color, then it uses that information before using COLORFGBG.
You can force with XMQ_THEME=darkbg XMQ_THEME=lightbg XMQ_THEME=mono
Please test!

Thanks, Fredrik!
2.8.1 pulled, built, and tested. Works fine!
As a side note, I was seriously impressed how gracefully xmq handles namespaces:
cat xmlnshell.xml
<?xml version="1.0" encoding="UTF-8"?>
<root>
  <hello xmlns="http://a" xmlns:x="http://x" x:attn="attv">
    <world xmlns="http://b" xmlns:x="http://y" x:attn="attv"/>
  </hello>
</root>

xmq xmlnshell.xml > xmlnshell.xmq

cat xmlnshell.xmq
root {
    hello(x:attn  = attv
          xmlns   = http://a
          xmlns:x = http://x)
    {
        world(x:attn  = attv
              xmlns   = http://b
              xmlns:x = http://y)
    }
}

xmq xmlnshell.xmq to-xml
<?xml version="1.0" encoding="utf8"?>
<root><hello xmlns:x="http://x" xmlns="http://a" x:attn="attv"><world xmlns="http://b" xmlns:x="http://y" x:attn="attv"/></hello></root>


AudioManager remains "connected" to the voice channel after being voice kicked

General Troubleshooting

[x] I have checked for similar issues.
[x] I have updated to the latest JDA version.
[x] I have checked the branches or the maintainers' PRs for upcoming bug fixes.

Bug Report
When the bot is kicked from a voice channel, the audio manager remains in a "connected" state.
Expected Behavior
The audio manager should handle the voice kick and report that it's been disconnected from the voice channel.
Code Example or Reproduction Steps
https://github.com/nikammerlaan/jda-voice-bug/blob/master/src/main/java/Main.java
This occurs in the latest versions of both JDA 3 and JDA 4, though the code may vary slightly.
Exception or Error
N/A

I get this using the exact same code on 4.BETA.0_30:
19:09:40: Executing task 'Main.main()'...

> Task :compileJava
> Task :processResources NO-SOURCE
> Task :classes

> Task :Main.main()
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
SLF4J: Failed to load class "org.slf4j.impl.StaticMDCBinder".
SLF4J: Defaulting to no-operation MDCAdapter implementation.
SLF4J: See http://www.slf4j.org/codes.html#no_static_mdc_binder for further details.
[main] INFO JDA - Login Successful!
[JDA MainWS-WriteThread] INFO WebSocketClient - Connected to WebSocket
[JDA MainWS-ReadThread] INFO JDA - Finished Loading!
Before Kick
===========
Voice State Channel:   VC:Helpers Music(357195653006360576)
Audio Manager Channel: VC:Helpers Music(357195653006360576)
Audio Manager Status:  CONNECTED

After Kick
===========
Voice State Channel:   null
Audio Manager Channel: null
Audio Manager Status:  NOT_CONNECTED

Voice kick handling was specifically addressed in version 4.ALPHA.0_87. I believe v3 never received a change for this so the bot will just reconnect which at this point is intended behavior there as it represents the legacy handling.

And I get this using the exact code I provided.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
SLF4J: Failed to load class "org.slf4j.impl.StaticMDCBinder".
SLF4J: Defaulting to no-operation MDCAdapter implementation.
SLF4J: See http://www.slf4j.org/codes.html#no_static_mdc_binder for further details.
[main] INFO JDA - Login Successful!
[JDA MainWS-ReadThread] INFO WebSocketClient - Connected to WebSocket
[JDA MainWS-ReadThread] INFO JDA - Finished Loading!
Before Kick
===========
Voice State Channel:   VC:💡 Testing(355943910750683137)
Audio Manager Channel: VC:💡 Testing(355943910750683137)
Audio Manager Status:  CONNECTED

After Kick
===========
Voice State Channel:   null
Audio Manager Channel: VC:💡 Testing(355943910750683137)
Audio Manager Status:  CONNECTED

I've experienced this on different PC's, OS's, networks so I'm not sure what could be up.

Can you provide debug logs?

https://hasteb.in/yizovoma.m
Unfortunately, doesn't look like there's anything too helpful in there.

No this is helpful. Discord didn't disconnect the websocket so the connection remains open. Kick is supposed to close the socket with code 4014 which tells JDA not to reconnect. Since it was never closed by discord the connection is not closed either. Either the close frame was blocked by your setup or this is a discord bug.

Which voice region did you use when you tested the code? I wonder if I can reproduce your result if I use that region.

I used region london but you can just test every region

Huh, how strange. It does seem really inconsistent. I'll have to bring this up with the discord devs. I guess I can close this issue then 😄


Naming imported chunks causes all chunks to download

When using the new chunk naming feature introduced in Webpack 2.4.0 it causes all imports to run immediately instead of when the system import is lazily loaded.
Used to lazy load account-settings module within a react project with react-router:
 getComponent: (state: Object, cb: Function) => (
   System.import(/* webpackChunkName: "account-settings" */ 'src/modules/account/settings')
   ..
 )

The comment successful names the chunk but it causes it to download the chunk immediately. By removing the webpackChunkName comment it will allow this module to lazy load again.

I am noticing that there is a difference between System.import and import here.
System.import correctly names all the chunks but causes the lazy loaded chunks to fetch immediately.
import correctly names the first chunk that is loaded and does NOT cause lazy chunks to fetch immediately but when the chunks are fetched they are not named.

This featured was added in #4573

Did not notice this, I will try to fix it if I can.

I had a similar problem but with react-router v4. The default/index route was always loaded even though I navigated to a sibling route. I fixed it by having no index route (kind of a pain I know). Not sure how relevant it is.

@huy-nguyen Could you please provide a reproduceable repo or gist to help us figure out it? Thanks :)

The default/index route was always loaded even though I navigated to a sibling route.

@huy-nguyen Did you set the exact property to your index route ? If you didn't set it, this is a feature in react-router v4 AFAIK.
And I've make a v4 Demo today. It looks ok.
You can try https://mc-zone.github.io/Demo_react_router_lazy_load_with_webpack_import_named_chunk/v4/index.html#/bar
and the source code is here.


Store raw document bytes

This refactors the corpus models / jobs so that we store the raw bytes for the documents. Before, we were just storing the extracted plain text for documents. Eg, when we ingested WARCs from the scraper, text extraction was done immediately, and the raw bytes were thrown out.
There are a variety of reasons that we might want the original docs, I've realized. For example, when extracting dates for the syllabi - PDFs often include metadata in the file header about when the document was created or modified. Or, in the (more distant) future - when extracting text assignments, we could conceivably hook on to formatting features in the text to improve accuracy, things like whether or not words in a candidate title are italicized. Which could theoretically be recovered from the raw document, but which get lost in the plain text.

Documents no longer have a text field? I'm pretty sure this was in use elsewhere in the codebase (here at least: https://github.com/opensyllabus/osp-pipeline/blob/master/osp_pipeline/institutions/matcher.py#L224)

That's right, thanks, I forgot that the institution matching is already using the text to find emails.
I forgot to say in the PR - text certainly isn't disappearing, I'm just moving it to a separate dataframe. So, in the same way that, eg, the classifier job will write out the classifications as a separate schema of (document id, classification), I'm going to add an extract_text job that writes the text and inferred mime type into a adjacent table. Something like:
class DocumentText(metaclass=ModelMeta):

    key = 'document_texts'

    schema = T.StructType([
        T.StructField('document_id', T.StringType(), nullable=False),
        T.StructField('mime_type', T.StringType()),
        T.StructField('text', T.StringType()),
    ])

So, that way the raw document bytes are stored in Document, and we can join on DocumentText to get the text.
I was going to do that as a separate PR, but I'll do ahead and do this here, and update the institution matching. I'll also see if I can write an integration test to check the interaction between the two modules.

Hey @wearpants,
Ok, I just pushed the changes that make this work with the institution matching, along with a little bit of framework-y code for structuring the joins.
To recap - instead of storing document metadata + text on in a single table, we now just store the raw document bytes in Document:
class Document(metaclass=ModelMeta):
    """Unified data model (UDM) for the various raw corpus sources.
    """

    s3_key = 'documents'

    schema = T.StructType([
        T.StructField('id', T.StringType(), nullable=False),
        T.StructField('corpus', T.StringType(), nullable=False),
        T.StructField('corpus_id', T.StringType(), nullable=False),
        T.StructField('url', T.StringType()),
        T.StructField('source_url', T.StringType()),
        T.StructField('retrieved', T.TimestampType()),
        T.StructField('data', T.BinaryType()),
    ])

And, the mime type and text are extracted to a separate table that links on the document ids:
class DocumentText(metaclass=ModelMeta):

    s3_key = 'document_texts'

    schema = T.StructType([
        T.StructField('document_id', T.StringType(), nullable=False),
        T.StructField('mime_type', T.StringType()),
        T.StructField('text', T.StringType()),
    ])

This pattern having an "entity" table, annotated by "metadata" tables, will come up lots - eg, when we write out results for deduping / classification.
So, with institutions - for the matching, we use the document id, URL, source URL, and plain text, which are now broken across two tables, so we need a join. In the tests, it's also nice to be able to easily mock one of the rows that would get produced by the join. I also wanted some way to explicitly represent the structure of the joins, similar to how to organize the models. To do this, I added a JoinRow construct, which is very similar to the Model class we use. Basically, a join row just defines a set of columns that will be present in the joined row, and, in a build_df classmethod, implements the join itself and selects out the needed columns. For institutions:
from osp_pipeline.core.joins import JoinRowMeta

class MatchDocument(metaclass=JoinRowMeta):

    fields = ['id', 'url', 'source_url', 'text']

    @classmethod
    def build_df(cls, docs, texts):
        """Join document URLs + extracted text.

        Args:
            docs (DataFrame of Document)
            texts (DataFrame of DocumentText)
        """
        return (docs
                .join(texts, docs.id == texts.document_id)
                .select(cls.fields))

And then, in the job itself, we just load up the model DFs, pass them to .build_rdd() classmethod on the join row, which executes the join and wraps each row as a MatchDocument:
docs = Document.read(spark)
texts = DocumentText.read(spark)

match_docs = MatchDocument.build_rdd(docs, texts)

The other nice thing about this is that it makes it easy to write domain logic for the joined rows - eg, we could just start adding row-level methods to MatchDocument, just like a regular model.
All tests are passing on this, let me know what you think.

Hmm... so there's nothing really specific to joining about JoinRow / JoinRowMeta - it's a way of representing any computed dataframe... I was kind of expecting it to be declarative, instead of needing to implement build_df yourself:
from osp_pipeline.core.joins import JoinRowMeta

class MatchDocument(metaclass=JoinRowMeta):
    fields = ['id', 'url', 'source_url', 'text']
    models = (Document, DocumentText)
    on = ('id', 'document_id')

And having the base class implement the join (using same logic as you did).
Not entirely sure what the present class gets us above and beyond standalone implementations of build_df()? I think I'm missing something...

Yeah, you're right, this is really more convention than code at this point. The immediate impetus was just to get the institution code working again, specifically the TextDomain class, which wraps around a Document row, and uses the old text fields. Since text got broken off into a separate table, though, it doesn't work to pass a raw Document anymore. This can be fixed in the job itself just by doing the join and mapping on the joined RDD, but there was also the question of how to handle this in the tests, where it's necessary (or at least much much easier) to just spin up testing Document rows and pass them into the code under test, like you do here:
https://github.com/opensyllabus/osp-pipeline/blob/master/tests/osp_pipeline/institutions/test_matcher.py#L32
So MatchDocument really just provides an easy way to mock one of the joined rows, like here:
https://github.com/opensyllabus/osp-pipeline/blob/store-bytes/tests/osp_pipeline/institutions/test_matcher.py#L32
And gives (very) loose structure to the code that produces the real join in the jobs.
But yeah, I really like this idea:
class MatchDocument(metaclass=JoinRowMeta):
    fields = ['id', 'url', 'source_url', 'text']
    models = (Document, DocumentText)
    on = ('id', 'document_id')

Only thought that pops to mind - will we ever need to more complex joins? Eg, 3+ tables, etc? I'll play with this later today and update here.
Thanks.

Only thought that pops to mind - will we ever need to more complex joins?
Eg, 3+ tables, etc?

Right, at some point this generalizes to functions taking and returning
dataframes...
On Jun 26, 2017 2:07 PM, "David McClure"<EMAIL_ADDRESS>wrote:

Yeah, you're right, this is really more convention than code at this
point. The immediate impetus was just to get the institution code working
again, specifically the TextDomain class, which wraps around a Document
row, and uses the old text fields. Since text got broken off into a
separate table, though, it doesn't work to pass a raw Document anymore.
This can be fixed in the job itself just by doing the join and mapping on
the joined RDD, but there was also the question of how to handle this in
the tests, where it's necessary (or at least much much easier) to just spin
up testing Document rows and pass them into the code under test, like you
do here:
https://github.com/opensyllabus/osp-pipeline/blob/master/tests/osp_
pipeline/institutions/test_matcher.py#L32
So MatchDocument really just provides an easy way to mock one of the
joined rows, like here:
https://github.com/opensyllabus/osp-pipeline/blob/store-bytes/tests/osp_
pipeline/institutions/test_matcher.py#L32
And gives (very) loose structure to the code that produces the real join
in the jobs.
But yeah, I really like this idea:
class MatchDocument(metaclass=JoinRowMeta):
fields = ['id', 'url', 'source_url', 'text']
models = (Document, DocumentText)
on = ('id', 'document_id')
Only thought that pops to mind - will we ever need to more complex joins?
Eg, 3+ tables, etc? I'll play with this later today and update here.
Thanks.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/opensyllabus/osp-pipeline/pull/42#issuecomment-311136983,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAe2AH2Zd1KKwyY1f__622frmyIWOdXbks5sH_NSgaJpZM4NzeUR
.

@wearpants, want to go ahead and merge this as-is, for now, and loop back later to the question of how structured the API should be for the joins?


entsql.IndexWhere queries for partial indexes are not escaped, resulting in wrong go code

[x] The issue is present in the latest release.
[x] I have searched the issues of this repository and believe that this is not a duplicate.

Current Behavior 😯
index.Fields("a").
    Annotations(
        entsql.IndexWhere(`"b" AND "c" > 0`),
    )

results in wrong go code. It generates this in schema.go:
				Annotation: &entsql.IndexAnnotation{
					Where: ""b" AND "c" > 0",
				},

which is not a valid go, which leads to weeping and gnashing of teeth.
Expected Behavior 🤔
				Annotation: &entsql.IndexAnnotation{
					Where: "\"b\" AND \"c\" > 0",
				},

Or, maybe better
				Annotation: &entsql.IndexAnnotation{
					Where: `"b" AND "c" > 0`,
				},

The bug might be on this line
https://github.com/ent/ent/blob/master/entc/gen/template/migrate/schema.tmpl#L183

Thanks for reporting this, @karelbilek 🙏.
The issue was resolved with https://github.com/ent/ent/pull/3578.


Request/Recommendation: Not prioritized icon

Alexander - this is a great idea and I love the icons. Any chance you'd be able to come up with something for not yet prioritized items? Perhaps a circle or something neutral?

Hi @ThePaulMichaels Thank you :) What about simple question mark symbol? It might be self explanatory.

Question mark, or a circle with a question mark may work and be fairly universal? Maybe both options?

Looks like @sbrouillard may have added a notset.png in their fork. Same concept.


Node: doc for clipArea and setClipArea doesn't match

Nodes.js:
277    'clipArea', // {Bounds2|Shape|null} - Makes things outside of a shape invisible, see setClipArea() for more documentation

But setClipArea apparently doesn't take a Bounds2.
    /**
     * Sets a clipped shape where only content in our local coordinate frame that is inside the clip area will be shown
     * (anything outside is fully transparent).
     * @public
     *
     * @param {Shape|null} shape
     */
    setClipArea: function( shape ) {
      assert && assert( shape === null || shape instanceof Shape, 'clipArea needs to be a kite.Shape, or null' );

Assigning to @jonathanolson, since I'm not sure whether the doc or implementation is incorrect.

Thanks for the catch. It does only take Shape/null, fixed in the first case.


phpPackages.protobuf: 3.11.1 -> 3.11.2

Motivation for this change
https://github.com/protocolbuffers/protobuf/releases/tag/v3.11.2 fixes the build with PHP 7.4.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Result of nix-review 1

6 package were built:

php72Packages-unit.protobuf
php72Packages.protobuf
php73Packages-unit.protobuf
php73Packages.protobuf
php74Packages-unit.protobuf
php74Packages.protobuf

Notify maintainers
cc @

@GrahamcOfBorg build php72Packages.protobuf php73Packages.protobuf php74Packages.protobuf


Update JavaDoc comments

#169 and #75

@navyasric Thanks for reviewing. Yes I plan on changing the redirect links to point to the microsoft docs (other than the client-credentials link) when we update the code snippets.


Where is the application?

Where is the application?

master branch has no applications and comes as bare desktop environment. Check out xterm branch (comes with xterm, correspondingly) or minimal (firefox, eye of gnome, gnome-terminal, so on.)


add qDevice and qNetd configuration

This PR adds the configuration of qDevice and qNetd daemons to the cluster_setup_ha playbook. The qDevice and qNetd daemons are used to provide a quorum without the need of a third corosync node. This is useful when the cluster has only two or an even number of nodes.

qDevice / qNetd are too complex.


Immutable memtables

When a memtable is full, convert it to an immutable memtable and keep it in memory until it can be flushed to disk.

See:
func (tree *LsmTree) walJob()

Would need to store a list of immutable memtables in the LSM tree instance as well as the "primary" memtable. Or is that just the first one in the list?

The advantage of this strategy is that we don't have to hold a lock while the immutable memtable is flushed to disk. Basically the sequence of operations is:

Create immutable memtable (requires lock)
Fire/wakeup an async job to flush this memtable to disk (create new SST file)
After flush, lock the tree, move in the new SST file, remove the immutable memtable

One complication is that right now we roll the WAL once the memtable is flushed. We won't be able to do that with this new approach. One solution is to keep 2 WAL's at once. Stop writing to old WAL when memtable becomes immutable, and create new WAL. Then when we remove the immutable memtable we can safely delete the old WAL as that data has now been committed to disk in the new SST file.


Compass

Has anyone found a way to get http://compass-style.org/ to work with this package? No matter what I try it doesn't seem to work. I see some posts in here from 2016 and prior, but nothing definite.

You may try to use https://github.com/Igosuki/compass-mixins instead of http://compass-style.org/
npm install compass-mixins
Then change normal compass import with this:
@import '/absolute/path/to/node_modules/compass-mixins/lib/_compass';


We still have public links to old repo

We still have links to old repo here: https://sveltesociety.dev/help/components/

I did this PR https://github.com/svelte-society/sveltesociety-2021/pull/26 to solve it. 😀


Jupyter NoteBook Issue Urgent

Missing viewType for 'file:///c%3A/Users/Victus/Desktop/Python/a.ipynb'.
View Jupyter log for further details.

Code and Mardown not shown in my VS code for Data Science also Says missing View Type please tell solution as soon as possible

@Ayushporwal07 does it happen all the time when you try to open the ipynb file? Does it work if you close the editor and then reload the window?

@Ayushporwal07 does it happen all the time when you try to open the ipynb file? Does it work if you close the editor and then reload the window?

No bro I reset my PC now problem solved


DNA small molecule tutorial

If we run it under the tutorial account it will not be deleted.

+Once your run has completed you will be presented with a result page showing the cluster statistics and some graphical representation of the data (and if registered, you will also be notified by email). Such an example output page can be found [here](https://wenmr.science.uu.nl/haddock2.4/run/1667417750/117344-netropsin_DNA https://wenmr.science.uu.nl/haddock2.4/run/1667417750/117344-netropsin_DNA) in case you don't want to wait for the results of your docking run.
Do you want to keep this forever? This link needs to be fixed in the database

Would be simpler to run it again under the tutorial account then everything is consistent (and also properly defined in the database)

Took the liberty of fix the html and some of the links that were missing the _blank tag

and you should also add information on what are the requirements to run the generate_conformers.py (how to install, which version, etc)
» python generate_conformers.py
Traceback (most recent call last):
  File "generate_conformers.py", line 9, in <module>
    from rdkit import Chem
ModuleNotFoundError: No module named 'rdkit

We probably have that info in the ligand shape tutorial

and you should also add information on what are the requirements to run the generate_conformers.py (how to install, which version, etc)
» python generate_conformers.py
Traceback (most recent call last):
File "generate_conformers.py", line 9, in
from rdkit import Chem
ModuleNotFoundError: No module named 'rdkit

We do:
https://www.bonvinlab.org/education/HADDOCK24/shape-small-molecule/#requirements-and-setup https://www.bonvinlab.org/education/HADDOCK24/shape-small-molecule/#requirements-and-setup
There is a requirement file for conda installation

No we don't have more info in the shape ligand tutorial, I will add it to both

We do: https://www.bonvinlab.org/education/HADDOCK24/shape-small-molecule/#requirements-and-setup https://www.bonvinlab.org/education/HADDOCK24/shape-small-molecule/#requirements-and-setup There is a requirement file for conda installation
…
No we don't have more info in the shape ligand tutorial, I will add it to both

I missed it, sorry !

I added:

New results with the permanent link
The requirements
Updated the index.md file with the last results


Added method call saveAssignments to \yii\rbac\PhpManager::removeItem

…tem`

It happened because we do not have a proper test.
@rezident1307 could you add a test to PhpManagerTest.php:143?

Ok. I will add a new commit later

Look a new pool request #10109


Players spawning at 0,0 below ground inside a hill

This apparently is still happening.
In Forest and Genesis Plaza. Could reproduce with a /goto -5,82

Lets wait until the chest smart item issue is resolved before validating this one

I haven't seen this happen for a long time. Is it still a problem?


Move conditioning

Starting to work on the general condition of movement checks:

Board edge detection,
Enemy piece square occupancy (target),
Friendly piece square occupancy (block).

This is a rather complex condition already, I see if it semantically splitable between the classes. This means that objects will end up playing ball with this condition, but that's the best I got so far.

Not sure you guys saw this. This is the latest work, so I will try to commit the corrections you proposed here.

I think I took care of most of the issue in #15. The branch was rebranded, because from this point on I started working on a new feature, before #15 ever got the change to be merged.

The next step is capturing and blocking logic, probably provideth by Board.


Allow preprocessing part of year

Some species, such as Chimney Swift are well modeled by ebirdst for part of the year but not for the whole year, for both species that I've looked at with this pattern they disappear from ebirdst into the Amazon.
Add an argument to preprocess_species() to allow just running for part of the year seems like a good solution.  It should cross reference against the metadata and allow running a subset even with skip_quality_checks = FALSE as long as all the timesteps included in the model are fully modeled.
Possible argument names:
from, to
date_range
This would allow building and using Chimney Swift models and fix some of the (secondary) problems related to #32.

And create a stand alone function to subset an existing, fitted model to part of the year.


Persistent mode with config watching

Hello. I'd like propose idea about persistent mode whenever something like gulp will watch that composer config has changed, and composer will re-update vendor components. It's useful for development stage.

Workaround example with shell script and composer hooks.
composer.json
{
  "scripts":
  {
    "post-install-cmd":
    [
      "/scripts/composer-watch-update.sh _COMPOSER_JSON_CONFIG_PATH_"
    ]
  },
  "config":
  {
    "process-timeout": 0,
  },

composer-watch-update.sh
#!/bin/bash

FILE="$1"
CMD="$2"
DIR=$(dirname "${FILE}")

LAST=`ls -l "$FILE"`
while true; do
  sleep 1
  NEW=`ls -l "$FILE"`
  if [ "$NEW" != "$LAST" ]; then
    cd $DIR
    composer update
    LAST="$NEW"
  fi
done

docker-compose.yml
application:
  image: busybox
  volumes:
    - ./scripts/:/scripts/:ro
    - ./tmp/development/payload/:/app/
    - ./config/development/composer.json:/app/composer.json:ro
  command: "true"
# platform dependency manager
payload:
  image: composer/composer
  mem_limit: 256m
  memswap_limit: 512m
  command: 'install'
  volumes_from:
    - application

Oh, very interesting. Does Composer natively have something along these lines?

Oh, Hello. :)
No, there is not native feature like that.


Is this module still work now? I got an error, outdated?

Yes it seems to be true.
You can use another libs like pytion

:(


Docker private registry make too many AWS S3 LIST Requests

Hi Docker team,
We have set up our own Docker private registry (v2) with AWS S3 as the backend storage.
We have observed that there are many S3 LIST requests made to the S3 bucket which probably means Docker registry is keeping scanning objects in the bucket for some purpose.
Can anyone help point out why it is doing this? and any optimization we can do to reduce the number of calls (and save cost)?

Please follow these instructions to help us diagnose your issue

create a new issue, with a succinct title that describes your issue:

bad title: "It doesn't work with my docker"
good title: "Private registry push fail: 400 error with E_INVALID_DIGEST"

copy the output of:

docker version
docker info
docker exec <registry-container> registry -version

copy the command line you used to launch your Registry
restart your docker daemon in debug mode (add -D to the daemon launch arguments)
reproduce your problem and get your docker daemon logs showing the error
if relevant, copy your registry logs that show the error
provide any relevant detail about your specific Registry configuration (e.g., storage backend used)
indicate if you are using an enterprise proxy, Nginx, or anything else between you and your Registry

Hi @TimeInvestor,
we're facing a similar issue where we see a huge amount of LIST requests reported by S3. Did you end up finding the cause of this?

The root cause for our issue is that we accidentally have local Docker registry set up in our AMI (AWS Image Instance). Hence, each time we boot up a machine using that AMI, the machine tries to initialise local Docker registry and cause S3 LIST request.

I have this issue too. I'm using simple docker registry for very small installation and it does around 10k list requests per day according to my provider statistics. That's with 500 get requests and 55 put requests and 250 MB egress. It's not a lot but this list request spending is like 99.9% of my S3 expenses.

I had similar issue. It turned out that /debug/health health check checked S3 health every 10 seconds and it caused this excessive traffic. I set REGISTRY_HEALTH_STORAGEDRIVER_INTERVAL=15m and it seems to solve this issue.

+1


how to create custom circular progress button on image ?

Hi, i need to create custom circular progress button on image (like whatsapp) ?? please tell me

Thanks @chinnuios ! :smile:
This doesn't seem specific to the library. StackOverflow is a more appropriate place to ask this question. :blush:
I'm going to close this issue, but the community should feel free to continue the discussion here.
Other ways to get help:

Ask on StackOverflow with the jsqmessagesviewcontroller tag
See the FAQ
Search previous questions


Minor improvements of BilinearForm and MixedBilinearForm [najlkin:pr6]

Small improvements of the BilinearForm and MixedBilinearForm classes to provide a more complete and coherent set of features. These include:

boundary trace face integrators for MixedBilinearForm and boundary attribute markers system (comparable with BilinearForm)
Compute(Bdr)ElementMatrix() methods for the both classes and a description to the existing BilinearForm::ComputeElementMatrix()
Assemble(Bdr)ElementMatrix() for the MixedBilinearForm enabling manual assembly of the form similarly to BilinearForm

@tzanio , @v-dobrev this PR seems good to go. Adds minor but useful bi-forms features regarding BC for DG. Unifies member names between BilinearForm and MixedBilinearForm.

Can you please address the following regression test errors:
fem/bilinearform.cpp(1208): warning #823: reference is to variable "k" (declared at line 1148) -- under old for-init scoping rules it would have been variable "k" (declared at line 1183)
           for (k = 0; k < bbfi.Size(); k++)
                ^

fem/bilinearform.cpp(1210): warning #823: reference is to variable "k" (declared at line 1148) -- under old for-init scoping rules it would have been variable "k" (declared at line 1183)
              if (bbfi_marker[k] &&
                              ^

fem/bilinearform.cpp(1211): warning #823: reference is to variable "k" (declared at line 1148) -- under old for-init scoping rules it would have been variable "k" (declared at line 1183)
                  (*bbfi_marker[k])[bdr_attr-1] == 0) { continue; }
                                ^

fem/bilinearform.cpp(1213): warning #823: reference is to variable "k" (declared at line 1148) -- under old for-init scoping rules it would have been variable "k" (declared at line 1183)
              bbfi[k] -> AssembleElementMatrix2 (*trial_fes -> GetBE(i),
                   ^

Does it solve the problem? Unfortunately, I cannot replicate the error.

These warnings are produced by the intel c++ compiler. I think my last commit should fix them.


Laravel 10.x Compatibility

This is an automated pull request from Shift to update your package code and dependencies to be compatible with Laravel 10.x.
Before merging, you need to:

Checkout the l10-compatibility branch
Review all comments for additional changes
Thoroughly test your package

If you do find an issue, please report it by commenting on this PR to help improve future automation.

:warning: The latest version of Laravel has a dependency for phpunit/phpunit of ^9.5.10. If you would like to also upgrade your tests, you may run the PHPUnit 9 Shift for free.

:alembic: Using this package? If you would like to help test these changes or believe them to be compatible, you may update your project to reference this branch.
To do so, temporarily add Shift's fork to the repositories property of your composer.json:
{
    "repositories": [
        {
            "type": "vcs",
            "url": "https://github.com/laravel-shift/Adldap2-Laravel.git"
        }
    ]
}

Then update your dependency constraint to reference this branch:
{
    "require": {
        "adldap2/adldap2-laravel": "dev-l10-compatibility",
    }
}

Finally, run: composer update


[Coverity CID :203431]Memory - corruptions in /tests/bluetooth/uuid/src/main.c

Static code scan issues seen in File: /tests/bluetooth/uuid/src/main.c
Category: Memory - corruptions
Function: test_uuid_cmp
Component: Tests
CID: 203431
Please fix or provide comments to square it off in coverity in the link: https://scan9.coverity.com/reports.htm#v32951/p12996

@jhedberg

Marked as false positive in Coverity


added WKWebViewOnly to Readme

added WKWebViewOnly to Readme to config Readme explanation.
Motivation and Context
This is (afaik) needed in order to be able to upload new apps to the iOS App Store. I assume because without it both WebViews get delivered which then triggers Apples ITMS-90809: Deprecated API Usage Error on processing of the App
Testing
try upload app to AppStoreConnect via XCode without the added xml tag and without.
with the tag:  ITMS-90809: Deprecated API Usage Error
with the tag: passes

Thanks for your contribution!
I would actually add a note to this preference stating that WKWebViewOnly only works on cordova-ios versions >= 5.1.0

thank you! implemented the version limitation.
is there information to be referenced about the behaviour above 5.1.0 that could be included additionally?

you might also have to you might have to also

Think you might have doubled up some words here :)

is there information to be referenced about the behaviour above 5.1.0 that could be included additionally?

I don't believe so... this plugin is not usable on cordova-ios@6, but that's already documented from https://github.com/apache/cordova-plugin-wkwebview-engine/commit/486e5a74ec826b88d156dafe427aabf0827162ab
There isn't an official conclusion from Apache yet, but I'm pretty sure this plugin is going to become deprecated soon.

awkward. sorry for the extra work.
this is what i get for multitasking.

If I don't receive a second review by Monday I'll just go ahead and merge since this is a trivial change.

thanks! i commited those. indeed better wording!


Python Rewrite

TODO:

[x] Switch to postgres
[x] Allow using a config instead of hardcoded stuff  (Added with ENV vars)
[ ] Add authentication for owners
[x] Pagination for users and endpoints
[ ] Set up keycloak-based authentication

Locking this PR from other people to prevent off-topic.

My bad, it seems that deleting that branch kills off the PR

pls undo

Can't. Sorry fam.

Note that DataTypes.md is out of date but I cba to fix it


Request fails if host issues a self sign certificate

Hi,
Can we make it possible to allow self signed certificates from the host?
Also an option to start a https server instead of http could be good

OK I found out that setting process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0' solves this issue


terraform-docs tfvars hcl does not generate tfvars file

Describe the bug
Running terraform-docs does not produce auto generated tfvars files (but indicates success in output)
How can we reproduce it?
.pre-commit-config.yaml
repos:
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.82.0
    hooks:
      - id: terraform_docs
        name: Generating / updating terraform.tfvars.auto (terraform_docs)
        args:
          - tfvars hcl --output-file terraform.tfvars.auto .

 Create /etc/containers/nodocker to quiet msg.
    [INFO] Initializing environment for https://github.com/antonbabenko/pre-commit-terraform.
    Generating / updating terraform.tfvars.auto (terraform_docs).......................Passed

Directory structure
.
├── README.md
├── bootstrapping_node.puml
├── data.tf
├── locals.tf
├── main.tf
├── outputs.tf
├── precommit.sh   # can be ignored
├── providers.tf
├── templates         # can be ignored
│   ├── metadata.tpl
│   └── userdata.yaml.tpl
└── variables.tf

Additional information
Running terraform-docs binary directly on the same project generates the corresponding file containing all variables (incl. defaults)
terraform-docs tfvars hcl --output-file terraform.tfvars.auto .

Running terraform_docs in .pre-commit-config.yaml to generate markdown works like a charm
repos:
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.82.0
    hooks:
      - id: terraform_docs
        name: Generating / updating README.md (terraform_docs)
        args:
          - --hook-config=--create-file-if-not-exist=true
          - --hook-config=--path-to-file=README.md

Environment information

OS:  Windows 10 (WSL / Podman Desktop)
Docker image tag: ghcr.io/antonbabenko/pre-commit-terraform:v1.82.0 (official)

It is a subset of https://github.com/antonbabenko/pre-commit-terraform/issues/179

@MaxymVlasov : I am not 100% sure why this issue is closed.
Is it because pre-commit-terraform does not support the --output-file parameter?
All other prerequisistes for the linked ticket are already given (terraform-docs uses v0.16.0 and directly executing "terraform-docs tfvars hcl . --output-file test.tfvars" seems to work in the container).
In case this is the issue, may I kindly request to get rid of the official documentation stating the support for tfvars example file generation? This is the corresponding documentation that mentions exactly this command: https://github.com/antonbabenko/pre-commit-terraform/blob/master/README.md?plain=1#L502C14
Side Note / Question
Generation of tfvars files could be also achieved without using the --output-file parameter by piping the stdout to a file.
What I tried was this, but it seems like it is not working (no errors, but file is as well not created). Directly executing the command works fine.
  - id: terraform_docs
    name: Generating / updating terraform.tfvars.auto (terraform_docs)
    args:
      - tfvars hcl . > terraform.tfvars.auto

Any tips on how to get this working?

@MaxymVlasov : I am not 100% sure why this issue is closed.

This issue will be resolved during #179 resolution because that problem goes from unmaintainable logic located somewhere in https://github.com/antonbabenko/pre-commit-terraform/blob/master/hooks/terraform_docs.sh

Any tips on how to get this working?

Feel free to submit PR to make #179 live, that's the only possibility which I see here


Remove reference to 'tap' or 'click', replace with what will happen

e.g., "Call for help" Wording to come

Removed 'tap'


PHP 7.4 error in implode call

Environment details

OS: ALL
PHP version: 7.4
Package name and version: google/apiclient/src/Google/Service/Resource.php

290    if (count($queryVars)) {
291       $requestUrl .= '?' . implode($queryVars, '&');   // <<<<<<< here php 7.4 complains about parameter inversion
292     }

ErrorException
implode(): Passing glue string after array is deprecated. Swap the parameters

This was fixed in v2.2.4, released in April 2019. Upgrade to a more recent version of the client and you should be all set.


Update typing to be pyright-compliant (mostly)

When using this lib, pyright threw a lot of typing errors when using the methods. I forked the project and fixed the type errors, using the builtin typing lib.

Holy shit brother that's a lot that was wrong lmao
Not surprised because i've typed everything myself without really looking into any docs..
Thanks a ton for doing all this, i glanced over it and it isn't just typing but way more (raising errors, etc..)
Appreciate the work!

Yeah, sorry maybe should have mentioned some of that when I made the PR. The error raising stuff is not only good practice but better than returning None when there was a non-Optional typehint. Mainly that and using .get rather than slicing for accessing dictionaries in case some entries weren't present. Glad to help with this change, most other siege API projects I found have been abandoned and I was glad to find this one. Thanks for accepting the PR brother!

Yeah definitely. I approve all your changes, they make sense.
I wrote that stuff when Ranked 2.0 came out so i was more interested in making it work api-wise rather than making it work code-wise.
With the thought I'll fix it later (which never happened, as usual..)


Prevent focusedKey from scrolling into view on click/press

Closes #2285
✅ Pull Request Checklist:

[ ] Included link to corresponding React Spectrum GitHub Issue.
[ ] Added/updated unit tests and storybook for this change (for new code or code which already has tests).
[ ] Filled out test instructions.
[ ] Updated documentation (if it already exists for this component).
[ ] Looked at the Accessibility Practices for this feature - Aria Practices

📝 Test Instructions:

🧢 Your Project:
RSP

@LFDanLu I've incorporated this PR into #2767 with https://github.com/adobe/react-spectrum/pull/2767/commits/1fa895b57a4880f5458c6dcc91c2417e7c4eedc2


Add steve jobs animated character (new)

This pr is referenced from #269
Resolved merge conflict, rebased to make the pr more clean, and also fixed eslint formatting error.

Updated avatars package
Introduced isTextStreaming state in App.jsx
Altered steve job prompts to include emotion [tag]
Extracting emotion tag in Conversations/index.js
Fixes default character voice id -> change back to male id
*NEW Fix eslint formatting error

https://github.com/Shaunwei/RealChar/assets/99272387/160c7d41-f2e6-4167-b92a-3485fe783fa6

Also thanks for the eslint fixes! We need to run a release so I merge them first, please just merge main.


adding support for level custom fields

this simple example will show how to print the text in the current level's custom field:
    -- load custom data for selected level:
    customData = LDtk.get_customData( level_name )
    if customData.greeting then
        print(customData.greeting)
    else
        print("no custom greeting found for level", level_name)
    end

I change the function from LDtk.get_customData() to LDtk.get_custom_data()


WIP [backend] add label_type in dataset table to support segmentation

merged back to https://github.com/IndustryEssentials/ymir/pull/1230


Tic-tac-Toe-game

Description
This is a fully-fledged tic-tac-toe game made using java and Lottie animations.
Checklist:

[ ✔] My PR follows the style guidelines of this project
[ ✔] I have performed a self-review of my own code or materials
[ ✔] My branch is up-to-date with the Upstream development branch
[ ✔] I have added necessary images/screenshots of my changes

This is basic tic-tac-toe which is a beginner-friendly project. Please merge it. Thank you in advance!!


Statement seems to have no effect

Hey, @Mcamargo85,
Does next mean continue in this for-loop?
https://github.com/AdaptiveBProcess/Simod/blob/31acdb14223297fc798b1d4903e0960250141e0a/support_modules/writers/xes_writer.py#L101

Hi!
Yeah, it means continue Jaja, but it seems like it was an uncommented line about a possible transition that I didn't solve. In any case, is continue.
Regards,
Manuel
From: Ihar Suvorau @.>
Sent: Tuesday, July 27, 2021 2:15 PM
To: AdaptiveBProcess/Simod @.>
Cc: Manuel Alejandro Camargo Chavez @.>; Mention @.>
Subject: [AdaptiveBProcess/Simod] Statement seems to have no effect (#17)
Hey, @Mcamargo85https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMcamargo85&data=04|01|ma.camargo11%40uniandes.edu.co|dc92736202824d66c54308d950efba04|fabd047cff48492a8bbb8f98b9fb9cca|0|0|637629812809886686|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=2glYQQBvdlEeVqztFe%2B%2BWSIxjrQoe%2BCeFmaRetXMGuY%3D&reserved=0,
Does next mean continue in this for-loop?
https://github.com/AdaptiveBProcess/Simod/blob/31acdb14223297fc798b1d4903e0960250141e0a/support_modules/writers/xes_writer.py#L101https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAdaptiveBProcess%2FSimod%2Fblob%2F31acdb14223297fc798b1d4903e0960250141e0a%2Fsupport_modules%2Fwriters%2Fxes_writer.py%23L101&data=04|01|ma.camargo11%40uniandes.edu.co|dc92736202824d66c54308d950efba04|fabd047cff48492a8bbb8f98b9fb9cca|0|0|637629812809886686|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=cMLXbT9YAx9ntYvLzS0cGnisgJlPyJwUv6YRGxxzzN8%3D&reserved=0

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAdaptiveBProcess%2FSimod%2Fissues%2F17&data=04|01|ma.camargo11%40uniandes.edu.co|dc92736202824d66c54308d950efba04|fabd047cff48492a8bbb8f98b9fb9cca|0|0|637629812809896640|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=GRqE%2Ffx86zWj%2BJQ70z44CBII3rquG0VBakfLFMo4Spc%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJEHTCYBXSJCAGAQOY4KOJLTZ2IJ5ANCNFSM5BCALVIQ&data=04|01|ma.camargo11%40uniandes.edu.co|dc92736202824d66c54308d950efba04|fabd047cff48492a8bbb8f98b9fb9cca|0|0|637629812809896640|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=lHSKWM7XZDqjA5Wtf1jr52Dw6zdq0kwYB5awGOPXifo%3D&reserved=0.


Create and design database for supporting extension directory

The goal is to create and design a database used to support a postgreSQL extension directory. Conceptually-speaking, this would be a postgreSQL equivalent to npmjs.org (JavaScript), pypi.org (Python), and crates.io (Rust).
This would be a backend storage, to which source and built binary distributions would be published and indexed.

[ ] Where will the packages be physically stored?
[ ] How to best organize metadata (e.g., author(s), versions)
[ ] Defined steps to publish extensions

This is resolved by the Trunk Registry work :)


Fix MIDI channel off by 1 error

All the channels are 1 higher than what they are supposed to be (11 instead of 10, etc.)

done


Q: Does the operator perform state management?

The upstream kbs deployment recipes lack explicit state management and store resources like secrets, policies on a local container instance. There are APIs that allow altering of that pod-local state, which leads to unpredictable results. If the operator would provide facilities and defaults for explicit state (StatefulSet with Volume?) that would be good additional value, and we could point users to the operator as a default means for deploying KBS.

Currently the operator takes most of the config via configmaps. Would that suffice ?

I was more thinking about making the kbs a StatefulSet deployment and volume provisioning. But if the operator would provide options to mount secrets and policies via k8s read-only secrets/configmaps that would be also a good-enough solution IMO.

I was more thinking about making the kbs a StatefulSet deployment and volume provisioning. But if the operator would provide options to mount secrets and policies via k8s read-only secrets/configmaps that would be also a good-enough solution IMO.

We can aim to keep the core KBS as stateless and provide all config and secrets via K8s configMap and Secrets respectively. I think this will be easier to maintain, help with scaling of KBS/AS as needed and integrate with 3rd party vault storage.

We can aim to keep the core KBS as stateless and provide all config and secrets via K8s configMap and Secrets respectively. I think this will be easier to maintain, help with scaling of KBS/AS as needed and integrate with 3rd party vault storage.

in general that makes sense, I think @kartikjoshi21 is trialing out using configmaps for policies, maybe we can implement it in the scope of this repo.
it would make sense to have a config option "policies/resources.readonly" on kbs so that we disable the respective endpoints and give sensible error messages.
with regard to full statelessness: the kbs protocol is stateful, so a kbs pod has state in memory
solutions that come to mind:

sticky sessions (potentially litte/no changes to kbs required, but maybe elaborate k8s machinery, not sure)
implement a redis (or similar) backend for sessions (should be trivial, but we defer the problem of state to either a managed redis or a vanilla k8s stateful redis, the latter is probaby pretty mature, so that's still a win over everything we build ourselves)

We can aim to keep the core KBS as stateless and provide all config and secrets via K8s configMap and Secrets respectively. I think this will be easier to maintain, help with scaling of KBS/AS as needed and integrate with 3rd party vault storage.

in general that makes sense, I think @kartikjoshi21 is trialing out using configmaps for policies, maybe we can implement it in the scope of this repo.

Sounds good

it would make sense to have a config option "policies/resources.readonly" on kbs so that we disable the respective endpoints and give sensible error messages.

Can you please point me to the kbs endpoints that will require disabling

with regard to full statelessness: the kbs protocol is stateful, so a kbs pod has state in memory
solutions that come to mind:

sticky sessions (potentially litte/no changes to kbs required, but maybe elaborate k8s machinery, not sure)

I was thinking about sticky sessions/session affinity when having multiple kbs pod replicas running.  Example - https://kubernetes.io/docs/reference/networking/virtual-ips/#session-affinity
https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/

Can you please point me to the kbs endpoints that will require disabling

I think it's those endpoints: https://github.com/confidential-containers/trustee/blob/main/kbs/src/api/src/http/config.rs

I was thinking about sticky sessions/session affinity when having multiple kbs pod replicas running. Example - https://kubernetes.io/docs/reference/networking/virtual-ips/#session-affinity https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/

those would be the stateful RCAR protocol's endpoints: https://github.com/confidential-containers/trustee/blob/main/kbs/src/api/src/http/attest.rs
we might have to make to set some headers to allow sticky sessions, not sure.

@mkulke @kartikjoshi21  opa policy via configmaps is now supported with the operator - https://github.com/confidential-containers/trustee-operator/pull/33
I'm experimenting with session stickiness using ingress capabilities and will update soon. Here is what I have till now

Expose ingress route (enable session stickiness as part of the config)
Increase the replica count of the kbs deployment to 2
Open two console to look at the logs of the two kbs deployments
Use curl from two different pods and send request to the ingress

Verify from the logs how the requests are handled by kbs - Typically it'll be handled in a round-robin fashion (by default).

Add a cookie to the curl (header) request

Verify from the logs how the requests are handled -  the requests should only be served by a specific kbs deployment
I'll repeat this with some different configs just to be sure and based on that we can decide how to implement it on the client side.


Daily Data/Data Dashboard Missing Messages

Problem statement
WA, NY, and OK are missing messages since 1/15/2024.
What you need to know
Data dashboard shows last sent 1/15/2024. Daily data shows last sent 1/18/2024.
To do

Fix bug and resend messages not sent in data dashboard.

Associated resends numbers 13027, 13028, 13029 have been completed and follow-up was sent 1/23/24.


Traceback (most recent call last):   File "C:\Users\sabah\RDDoS_Tool\RDDoS_Tool\RDDoS_Tool.py", line 10, in      from   tqdm.auto import tqdm ModuleNotFoundError: No module named 'tqdm'

Traceback (most recent call last):
File "C:\Users\sabah\RDDoS_Tool\RDDoS_Tool\RDDoS_Tool.py", line 10, in
from   tqdm.auto import tqdm
ModuleNotFoundError: No module named 'tqdm'
FIX PLEASE

Bro, you just need install the module

mb mb

Yeah, I have that same problem, reinstalled, and installed tqdm, yet it still doesn't work.


Figure 3: Inclusion of merged objects and downloading projects

By the time we submit this paper, we hope to have made merged objects available on the portal. Along with that we will have a report that is included with the merged objects. The content of this report are TBD, but I assume we will want to show some of that content in a figure.
I'm filing this as a placeholder right now, because I think once we discuss that report we will have a better idea of what we might want to show here.

I'm updating this issue to reflect that this figure should be focused on downloading project data from the portal. This would include downloading data as a merged object. So here we would show:

Illustration of project download as individual files
Merged workflow overview
Illustration of project download ad merged file
Maybe UMAPs (we will decide this once we have more established merged objects)


Unable to complete wallet creation on low res smartphones

There is no "Next" button visible when installing IoPay on a Samsung J5 smartphone. It seems like a "Next" button is supposed to be showed somewhere while assigning a name to the very first wallet, but none is visible:

This the screen on a Samsung J5 - Italian language.

Let's track this in iopay-android repo


OS#19078643: Fix constant branch folding when equal sources can be Undefined

We constant fold branches with same sources. The outcome of this folding is different when the source has a valueType of Undefined.
Previously, we checked for this case with ValueType::IsUndefined. But this check does not include valueTypes like - Undefined | Boolean, in which case we cannot constant fold.
With this fix, we handle the case where we can have a definite valueType which could be Undefined.

@wyrichte Can you take a look at this as well ? This is similar to https://github.com/Microsoft/ChakraCore/pull/6086 that you reviewed.

non-Windows 10 checks had passed before, I'll merge this in.


REQ: Output option to have output file as MKV or MP4

Hi
Can you add MKV as an output option please
Reason being is that MKV files can be previewed during render so we could test a render while it's busy to see if we happy with options, and can then stop without the full render have to complete.
Will help save time for those of us with older GFX cards
Thank you

Thank you for fulfilling this request