Numpy arrays have no data after running prepare_data.py

Hello and thank you for this awesome program!
I am trying to take JSON files and run prepare_data.py to convert them to numpy arrays, but they always become empty files. My goal is to run this program and use it for training a model for the handwriting synthesis by Sean Vasquez.  I am not super familiar with numpy, but all of my dependencies are the same version as listed in requirements.txt. Any help on this is greatly appreciated! Thanks.

Hello! Glad you are making use of my project.
How many prompts did you complete to generate your data? This could just be
caused by too little data. If not, I'll need some way to reproduce the
issue to fix it, so if you can show off your handwriting JSON data too that
would be helpful.
Andrew Mattson
@.***
On Wed, Dec 27, 2023, 12:33 AM DrakeHooks @.***> wrote:

Hello and thank you for this awesome program!
I am trying to take JSON files and run prepare_data.py to convert them to
numpy arrays, but they always become empty files. My goal is to run this
program and use it for training a model for the handwriting synthesis by
Sean Vasquez. I am not super familiar with numpy, but all of my
dependencies are the same version as listed in requirements.txt. Any help
on this is greatly appreciated! Thanks.
files.png (view on web)
https://github.com/acmattson3/handwriting-data/assets/52050841/811ccbdc-8778-4741-ab85-2ba773e96430
—
Reply to this email directly, view it on GitHub
https://github.com/acmattson3/handwriting-data/issues/1, or unsubscribe
https://github.com/notifications/unsubscribe-auth/A22PHG7Q2JOE7WUY73VXXY3YLPTQHAVCNFSM6AAAAABBEEWQ5GVHI2DSMVQWIX3LMV43ASLTON2WKOZSGA2TOMJTHA2TGNY
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Hey! I got part of the issue above resolved by editing the paths. However, when i go to run my numpy arrays through rnn.py, I am getting this error for zero size array and that the train size is 0. Is this just a result of too little data? I only did about 10 JSON files before preparing the data. Thanks for all the help!
train size 0
val size 1
test size 1
new run with parameters:
{'attention_mixture_components': 10,
'batch_size': 32,
'batch_sizes': [32, 64, 64],
'beta1_decay': 0.9,
'beta1_decays': [0.9, 0.9, 0.9],
'checkpoint_dir': 'checkpoints',
'early_stopping_steps': 1500,
'enable_parameter_averaging': False,
'grad_clip': 10,
'keep_prob_scalar': 1.0,
'learning_rate': 0.0001,
'learning_rates': [0.0001, 5e-05, 2e-05],
'log_dir': 'logs',
'log_interval': 20,
'logging_level': 20,
'loss_averaging_window': 100,
'lstm_size': 400,
'min_steps_to_checkpoint': 2000,
'num_restarts': 2,
'num_training_steps': 100000,
'optimizer': 'rms',
'output_mixture_components': 20,
'output_units': 121,
'patiences': [1500, 1000, 500],
'prediction_dir': 'predictions',
'reader': <main.DataReader object at 0x00000157C34334E0>,
'regularization_constant': 0.0,
'restart_idx': 0,
'validation_batch_size': 32,
'warm_start_init_step': 0}
all parameters:
[('Variable:0', []),
('Variable_1:0', []),
('Variable_2:0', []),
('rnn/LSTMAttentionCell/lstm_cell/kernel:0', [476, 1600]),
('rnn/LSTMAttentionCell/lstm_cell/bias:0', [1600]),
('rnn/LSTMAttentionCell/attention/weights:0', [476, 30]),
('rnn/LSTMAttentionCell/attention/biases:0', [30]),
('rnn/LSTMAttentionCell/lstm_cell_1/kernel:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_1/bias:0', [1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/kernel:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/bias:0', [1600]),
('rnn/gmm/weights:0', [400, 121]),
('rnn/gmm/biases:0', [121]),
('rnn/LSTMAttentionCell/lstm_cell/kernel/RMSProp:0', [476, 1600]),
('rnn/LSTMAttentionCell/lstm_cell/kernel/RMSProp_1:0', [476, 1600]),
('rnn/LSTMAttentionCell/lstm_cell/bias/RMSProp:0', [1600]),
('rnn/LSTMAttentionCell/lstm_cell/bias/RMSProp_1:0', [1600]),
('rnn/LSTMAttentionCell/attention/weights/RMSProp:0', [476, 30]),
('rnn/LSTMAttentionCell/attention/weights/RMSProp_1:0', [476, 30]),
('rnn/LSTMAttentionCell/attention/biases/RMSProp:0', [30]),
('rnn/LSTMAttentionCell/attention/biases/RMSProp_1:0', [30]),
('rnn/LSTMAttentionCell/lstm_cell_1/kernel/RMSProp:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_1/kernel/RMSProp_1:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_1/bias/RMSProp:0', [1600]),
('rnn/LSTMAttentionCell/lstm_cell_1/bias/RMSProp_1:0', [1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/kernel/RMSProp:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/kernel/RMSProp_1:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/bias/RMSProp:0', [1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/bias/RMSProp_1:0', [1600]),
('rnn/gmm/weights/RMSProp:0', [400, 121]),
('rnn/gmm/weights/RMSProp_1:0', [400, 121]),
('rnn/gmm/biases/RMSProp:0', [121]),
('rnn/gmm/biases/RMSProp_1:0', [121])]
trainable parameters:
[('rnn/LSTMAttentionCell/lstm_cell/kernel:0', [476, 1600]),
('rnn/LSTMAttentionCell/lstm_cell/bias:0', [1600]),
('rnn/LSTMAttentionCell/attention/weights:0', [476, 30]),
('rnn/LSTMAttentionCell/attention/biases:0', [30]),
('rnn/LSTMAttentionCell/lstm_cell_1/kernel:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_1/bias:0', [1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/kernel:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/bias:0', [1600]),
('rnn/gmm/weights:0', [400, 121]),
('rnn/gmm/biases:0', [121])]
trainable parameter count:
3632431
2023-12-27 18:29:15.528539: I C:\tf_jenkins\workspace\rel-win\M\windows\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
built graph
Traceback (most recent call last):
File "c:/Users/fishh/Downloads/handwriting-synthesis-master/handwriting-synthesis-master/rnn.py", line 246, in
nn.fit()
File "c:\Users\fishh\Downloads\handwriting-synthesis-master\handwriting-synthesis-master\tf_base_model.py", line 134, in fit
train_batch_df = next(train_generator)
File "c:/Users/fishh/Downloads/handwriting-synthesis-master/handwriting-synthesis-master/rnn.py", line 65, in batch_generator
max_x_len = np.max(batch['x_len'])
File "<array_function internals>", line 6, in amax
File "C:\Users\fishh\miniconda3\envs\myenv\lib\site-packages\numpy\core\fromnumeric.py", line 2621, in amax
keepdims=keepdims, initial=initial, where=where)
File "C:\Users\fishh\miniconda3\envs\myenv\lib\site-packages\numpy\core\fromnumeric.py", line 90, in _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation maximum which has no identity

I used 70 prompts for my first attempt and that worked. 10 prompts doesn't
sound like nearly enough data. The original handwriting synthesis AI used
multiple hundreds of megabytes of data, and even 70 prompts is only a
couple megabytes.  In short you'll want a
lot more data.
If you want to download some of mine to test, I left instructions at the
bottom of the readme on how to download that data using mongoDB compass.
On Wed, Dec 27, 2023, 2:32 PM DrakeHooks @.***> wrote:

Hey! I got part of the issue above resolved by editing the paths. However,
when i go to run my numpy arrays through rnn.py, I am getting this error
for zero size array and that the train size is 0. Is this just a result of
too little data? I only did about 10 JSON files before preparing the data.
Thanks for all the help!
train size 0
val size 1
test size 1
new run with parameters:
{'attention_mixture_components': 10,
'batch_size': 32,
'batch_sizes': [32, 64, 64],
'beta1_decay': 0.9,
'beta1_decays': [0.9, 0.9, 0.9],
'checkpoint_dir': 'checkpoints',
'early_stopping_steps': 1500,
'enable_parameter_averaging': False,
'grad_clip': 10,
'keep_prob_scalar': 1.0,
'learning_rate': 0.0001,
'learning_rates': [0.0001, 5e-05, 2e-05],
'log_dir': 'logs',
'log_interval': 20,
'logging_level': 20,
'loss_averaging_window': 100,
'lstm_size': 400,
'min_steps_to_checkpoint': 2000,
'num_restarts': 2,
'num_training_steps': 100000,
'optimizer': 'rms',
'output_mixture_components': 20,
'output_units': 121,
'patiences': [1500, 1000, 500],
'prediction_dir': 'predictions',
'reader': <main.DataReader object at 0x00000157C34334E0>,
'regularization_constant': 0.0,
'restart_idx': 0,
'validation_batch_size': 32,
'warm_start_init_step': 0}
all parameters:
[('Variable:0', []),
('Variable_1:0', []),
('Variable_2:0', []),
('rnn/LSTMAttentionCell/lstm_cell/kernel:0', [476, 1600]),
('rnn/LSTMAttentionCell/lstm_cell/bias:0', [1600]),
('rnn/LSTMAttentionCell/attention/weights:0', [476, 30]),
('rnn/LSTMAttentionCell/attention/biases:0', [30]),
('rnn/LSTMAttentionCell/lstm_cell_1/kernel:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_1/bias:0', [1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/kernel:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/bias:0', [1600]),
('rnn/gmm/weights:0', [400, 121]),
('rnn/gmm/biases:0', [121]),
('rnn/LSTMAttentionCell/lstm_cell/kernel/RMSProp:0', [476, 1600]),
('rnn/LSTMAttentionCell/lstm_cell/kernel/RMSProp_1:0', [476, 1600]),
('rnn/LSTMAttentionCell/lstm_cell/bias/RMSProp:0', [1600]),
('rnn/LSTMAttentionCell/lstm_cell/bias/RMSProp_1:0', [1600]),
('rnn/LSTMAttentionCell/attention/weights/RMSProp:0', [476, 30]),
('rnn/LSTMAttentionCell/attention/weights/RMSProp_1:0', [476, 30]),
('rnn/LSTMAttentionCell/attention/biases/RMSProp:0', [30]),
('rnn/LSTMAttentionCell/attention/biases/RMSProp_1:0', [30]),
('rnn/LSTMAttentionCell/lstm_cell_1/kernel/RMSProp:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_1/kernel/RMSProp_1:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_1/bias/RMSProp:0', [1600]),
('rnn/LSTMAttentionCell/lstm_cell_1/bias/RMSProp_1:0', [1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/kernel/RMSProp:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/kernel/RMSProp_1:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/bias/RMSProp:0', [1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/bias/RMSProp_1:0', [1600]),
('rnn/gmm/weights/RMSProp:0', [400, 121]),
('rnn/gmm/weights/RMSProp_1:0', [400, 121]),
('rnn/gmm/biases/RMSProp:0', [121]),
('rnn/gmm/biases/RMSProp_1:0', [121])]
trainable parameters:
[('rnn/LSTMAttentionCell/lstm_cell/kernel:0', [476, 1600]),
('rnn/LSTMAttentionCell/lstm_cell/bias:0', [1600]),
('rnn/LSTMAttentionCell/attention/weights:0', [476, 30]),
('rnn/LSTMAttentionCell/attention/biases:0', [30]),
('rnn/LSTMAttentionCell/lstm_cell_1/kernel:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_1/bias:0', [1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/kernel:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/bias:0', [1600]),
('rnn/gmm/weights:0', [400, 121]),
('rnn/gmm/biases:0', [121])]
trainable parameter count:
3632431
2023-12-27 18:29:15.528539: I
C:\tf_jenkins\workspace\rel-win\M\windows\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:140]
Your CPU supports instructions that this TensorFlow binary was not compiled
to use: AVX2
built graph
Traceback (most recent call last):
File
"c:/Users/fishh/Downloads/handwriting-synthesis-master/handwriting-synthesis-master/rnn.py",
line 246, in
nn.fit()
File
"c:\Users\fishh\Downloads\handwriting-synthesis-master\handwriting-synthesis-master\tf_base_model.py",
line 134, in fit
train_batch_df = next(train_generator)
File
"c:/Users/fishh/Downloads/handwriting-synthesis-master/handwriting-synthesis-master/rnn.py",
line 65, in batch_generator
max_x_len = np.max(batch['x_len'])
File "<array_function internals>", line 6, in amax
File
"C:\Users\fishh\miniconda3\envs\myenv\lib\site-packages\numpy\core\fromnumeric.py",
line 2621, in amax
keepdims=keepdims, initial=initial, where=where)
File
"C:\Users\fishh\miniconda3\envs\myenv\lib\site-packages\numpy\core\fromnumeric.py",
line 90, in _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation maximum which has no
identity
—
Reply to this email directly, view it on GitHub
https://github.com/acmattson3/handwriting-data/issues/1#issuecomment-1870691352,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A22PHGYTG7T7DFIJOITPOYLYLSVZFAVCNFSM6AAAAABBEEWQ5GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZQGY4TCMZVGI
.
You are receiving this because you commented.Message ID:
@.***>

I used 70 prompts for my first attempt and that worked. 10 prompts doesn't sound like nearly enough data. The original handwriting synthesis AI used multiple hundreds of megabytes of data, and even 70 prompts is only a couple megabytes.  In short you'll want a lot more data. Make sure you modified the original AI code as described in the readme in the prompts directory, too, as one of those modifications fixes a "too little data" error. If you want to download some of mine to test, I left instructions at the bottom of the readme on how to download that data using mongoDB compass.
…
On Wed, Dec 27, 2023, 2:32 PM DrakeHooks @.**> wrote: Hey! I got part of the issue above resolved by editing the paths. However, when i go to run my numpy arrays through rnn.py, I am getting this error for zero size array and that the train size is 0. Is this just a result of too little data? I only did about 10 JSON files before preparing the data. Thanks for all the help! train size 0 val size 1 test size 1 new run with parameters: {'attention_mixture_components': 10, 'batch_size': 32, 'batch_sizes': [32, 64, 64], 'beta1_decay': 0.9, 'beta1_decays': [0.9, 0.9, 0.9], 'checkpoint_dir': 'checkpoints', 'early_stopping_steps': 1500, 'enable_parameter_averaging': False, 'grad_clip': 10, 'keep_prob_scalar': 1.0, 'learning_rate': 0.0001, 'learning_rates': [0.0001, 5e-05, 2e-05], 'log_dir': 'logs', 'log_interval': 20, 'logging_level': 20, 'loss_averaging_window': 100, 'lstm_size': 400, 'min_steps_to_checkpoint': 2000, 'num_restarts': 2, 'num_training_steps': 100000, 'optimizer': 'rms', 'output_mixture_components': 20, 'output_units': 121, 'patiences': [1500, 1000, 500], 'prediction_dir': 'predictions', 'reader': <main.DataReader object at 0x00000157C34334E0>, 'regularization_constant': 0.0, 'restart_idx': 0, 'validation_batch_size': 32, 'warm_start_init_step': 0} all parameters: [('Variable:0', []), ('Variable_1:0', []), ('Variable_2:0', []), ('rnn/LSTMAttentionCell/lstm_cell/kernel:0', [476, 1600]), ('rnn/LSTMAttentionCell/lstm_cell/bias:0', [1600]), ('rnn/LSTMAttentionCell/attention/weights:0', [476, 30]), ('rnn/LSTMAttentionCell/attention/biases:0', [30]), ('rnn/LSTMAttentionCell/lstm_cell_1/kernel:0', [876, 1600]), ('rnn/LSTMAttentionCell/lstm_cell_1/bias:0', [1600]), ('rnn/LSTMAttentionCell/lstm_cell_2/kernel:0', [876, 1600]), ('rnn/LSTMAttentionCell/lstm_cell_2/bias:0', [1600]), ('rnn/gmm/weights:0', [400, 121]), ('rnn/gmm/biases:0', [121]), ('rnn/LSTMAttentionCell/lstm_cell/kernel/RMSProp:0', [476, 1600]), ('rnn/LSTMAttentionCell/lstm_cell/kernel/RMSProp_1:0', [476, 1600]), ('rnn/LSTMAttentionCell/lstm_cell/bias/RMSProp:0', [1600]), ('rnn/LSTMAttentionCell/lstm_cell/bias/RMSProp_1:0', [1600]), ('rnn/LSTMAttentionCell/attention/weights/RMSProp:0', [476, 30]), ('rnn/LSTMAttentionCell/attention/weights/RMSProp_1:0', [476, 30]), ('rnn/LSTMAttentionCell/attention/biases/RMSProp:0', [30]), ('rnn/LSTMAttentionCell/attention/biases/RMSProp_1:0', [30]), ('rnn/LSTMAttentionCell/lstm_cell_1/kernel/RMSProp:0', [876, 1600]), ('rnn/LSTMAttentionCell/lstm_cell_1/kernel/RMSProp_1:0', [876, 1600]), ('rnn/LSTMAttentionCell/lstm_cell_1/bias/RMSProp:0', [1600]), ('rnn/LSTMAttentionCell/lstm_cell_1/bias/RMSProp_1:0', [1600]), ('rnn/LSTMAttentionCell/lstm_cell_2/kernel/RMSProp:0', [876, 1600]), ('rnn/LSTMAttentionCell/lstm_cell_2/kernel/RMSProp_1:0', [876, 1600]), ('rnn/LSTMAttentionCell/lstm_cell_2/bias/RMSProp:0', [1600]), ('rnn/LSTMAttentionCell/lstm_cell_2/bias/RMSProp_1:0', [1600]), ('rnn/gmm/weights/RMSProp:0', [400, 121]), ('rnn/gmm/weights/RMSProp_1:0', [400, 121]), ('rnn/gmm/biases/RMSProp:0', [121]), ('rnn/gmm/biases/RMSProp_1:0', [121])] trainable parameters: [('rnn/LSTMAttentionCell/lstm_cell/kernel:0', [476, 1600]), ('rnn/LSTMAttentionCell/lstm_cell/bias:0', [1600]), ('rnn/LSTMAttentionCell/attention/weights:0', [476, 30]), ('rnn/LSTMAttentionCell/attention/biases:0', [30]), ('rnn/LSTMAttentionCell/lstm_cell_1/kernel:0', [876, 1600]), ('rnn/LSTMAttentionCell/lstm_cell_1/bias:0', [1600]), ('rnn/LSTMAttentionCell/lstm_cell_2/kernel:0', [876, 1600]), ('rnn/LSTMAttentionCell/lstm_cell_2/bias:0', [1600]), ('rnn/gmm/weights:0', [400, 121]), ('rnn/gmm/biases:0', [121])] trainable parameter count: 3632431 2023-12-27 18:29:15.528539: I C:\tf_jenkins\workspace\rel-win\M\windows\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 built graph Traceback (most recent call last): File "c:/Users/fishh/Downloads/handwriting-synthesis-master/handwriting-synthesis-master/rnn.py", line 246, in nn.fit() File "c:\Users\fishh\Downloads\handwriting-synthesis-master\handwriting-synthesis-master\tf_base_model.py", line 134, in fit train_batch_df = next(train_generator) File "c:/Users/fishh/Downloads/handwriting-synthesis-master/handwriting-synthesis-master/rnn.py", line 65, in batch_generator max_x_len = np.max(batch['x_len']) File "<array_function internals>", line 6, in amax File "C:\Users\fishh\miniconda3\envs\myenv\lib\site-packages\numpy\core\fromnumeric.py", line 2621, in amax keepdims=keepdims, initial=initial, where=where) File "C:\Users\fishh\miniconda3\envs\myenv\lib\site-packages\numpy\core\fromnumeric.py", line 90, in _wrapreduction return ufunc.reduce(obj, axis, dtype, out, passkwargs) ValueError: zero-size array to reduction operation maximum which has no identity — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/A22PHGYTG7T7DFIJOITPOYLYLSVZFAVCNFSM6AAAAABBEEWQ5GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZQGY4TCMZVGI . You are receiving this because you commented.Message ID: @.>

I see. I just added more data and it looks like the train size is now 1.  I still get the StopIteration error with 20 prompts but hopefully that will stop when I add significantly more data. I have made sure to modify data_frame.py and tf_base_model.py and I will make an effort to download your data to test later tonight! Thanks again for all the help and clarification.

Hey! Just reporting back after trying to download the dataset on MongoDB Compass. MongoDB compass wants to download everything into one .JSON file so I was unable to get it downloaded locally. I noticed that your .JSON files have an ObjectId and mine only have one "id" and "writer_id" . I'm not sure if this is causing any issues for training but it is just an observation. I'm at 75 prompts completed and I still get StopIteration after running rnn.py . I will keep working on prompts to see if it still needs more data.

@acmattson3  Hey! Just reporting back after trying to download the dataset on MongoDB Compass. MongoDB compass wants to download everything into one .JSON file so I was unable to get it downloaded locally. I noticed that your .JSON files have an ObjectId and mine only have one "id" and "writer_id" . I'm not sure if this is causing any issues for training but it is just an observation. I'm at 75 prompts completed and I still get StopIteration after running rnn.py . I will keep working on prompts to see if it still needs more data.

The StopIteration problem is an issue in the handwriting synthesis AI. I may have made a change to the files that I didn't put in the readme, or you might have missed one of the changes I made. I will check if I missed anything.

@acmattson3 Hey! Just reporting back after trying to download the dataset on MongoDB Compass. MongoDB compass wants to download everything into one .JSON file so I was unable to get it downloaded locally. I noticed that your .JSON files have an ObjectId and mine only have one "id" and "writer_id" . I'm not sure if this is causing any issues for training but it is just an observation. I'm at 75 prompts completed and I still get StopIteration after running rnn.py . I will keep working on prompts to see if it still needs more data.

I found one more difference that is probably causing the issue. In my rnn.py file, in the if __name__=="__main__"": section, here is my code:
dr = DataReader(data_dir='data/processed/')

    nn = rnn(
        reader=dr,
        log_dir='logs',
        checkpoint_dir='checkpoints',
        prediction_dir='predictions',
        #learning_rates=[.0001, .00005, .00002],
        learning_rates=[.001, .0005, .0002],
        #batch_sizes=[32, 64, 64],
        batch_sizes=[8, 16, 16],
        #patiences=[1500, 1000, 500],
        patiences=[1000, 500, 250],
        beta1_decays=[.9, .9, .9],
        validation_batch_size=32,
        optimizer='rms',
        #num_training_steps=100000,
        num_training_steps=2500,
        warm_start_init_step=0,
        regularization_constant=0.0,
        keep_prob=1.0,
        enable_parameter_averaging=False,
        #min_steps_to_checkpoint=2000,
        min_steps_to_checkpoint=1000,
        log_interval=20,
        grad_clip=10,
        lstm_size=400,
        output_mixture_components=20,
        attention_mixture_components=10
    )
    nn.fit()

Those commented-out lines are probably the trick.

@acmattson3 Just made those changes and it looks like it is training. This is only 83 prompts completed of my own data and I think that is why it still does a StopIteration.
train size 10
val size 1
test size 11
new run with parameters:
{'attention_mixture_components': 10,
'batch_size': 8,
'batch_sizes': [8, 16, 16],
'beta1_decay': 0.9,
'beta1_decays': [0.9, 0.9, 0.9],
'checkpoint_dir': 'checkpoints',
'early_stopping_steps': 1000,
'enable_parameter_averaging': False,
'grad_clip': 10,
'keep_prob_scalar': 1.0,
'learning_rate': 0.001,
'learning_rates': [0.001, 0.0005, 0.0002],
'log_dir': 'logs',
'log_interval': 20,
'logging_level': 20,
'loss_averaging_window': 100,
'lstm_size': 400,
'min_steps_to_checkpoint': 1000,
'num_restarts': 2,
'num_training_steps': 2500,
'optimizer': 'rms',
'output_mixture_components': 20,
'output_units': 121,
'patiences': [1000, 500, 250],
'prediction_dir': 'predictions',
'reader': <main.DataReader object at 0x00000276021A34E0>,
'regularization_constant': 0.0,
'restart_idx': 0,
'validation_batch_size': 32,
'warm_start_init_step': 0}
all parameters:
[('Variable:0', []),
('Variable_1:0', []),
('Variable_2:0', []),
('rnn/LSTMAttentionCell/lstm_cell/kernel:0', [476, 1600]),
('rnn/LSTMAttentionCell/lstm_cell/bias:0', [1600]),
('rnn/LSTMAttentionCell/attention/weights:0', [476, 30]),
('rnn/LSTMAttentionCell/attention/biases:0', [30]),
('rnn/LSTMAttentionCell/lstm_cell_1/kernel:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_1/bias:0', [1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/kernel:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/bias:0', [1600]),
('rnn/gmm/weights:0', [400, 121]),
('rnn/gmm/biases:0', [121]),
('rnn/LSTMAttentionCell/lstm_cell/kernel/RMSProp:0', [476, 1600]),
('rnn/LSTMAttentionCell/lstm_cell/kernel/RMSProp_1:0', [476, 1600]),
('rnn/LSTMAttentionCell/lstm_cell/bias/RMSProp:0', [1600]),
('rnn/LSTMAttentionCell/lstm_cell/bias/RMSProp_1:0', [1600]),
('rnn/LSTMAttentionCell/attention/weights/RMSProp:0', [476, 30]),
('rnn/LSTMAttentionCell/attention/weights/RMSProp_1:0', [476, 30]),
('rnn/LSTMAttentionCell/attention/biases/RMSProp:0', [30]),
('rnn/LSTMAttentionCell/attention/biases/RMSProp_1:0', [30]),
('rnn/LSTMAttentionCell/lstm_cell_1/kernel/RMSProp:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_1/kernel/RMSProp_1:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_1/bias/RMSProp:0', [1600]),
('rnn/LSTMAttentionCell/lstm_cell_1/bias/RMSProp_1:0', [1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/kernel/RMSProp:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/kernel/RMSProp_1:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/bias/RMSProp:0', [1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/bias/RMSProp_1:0', [1600]),
('rnn/gmm/weights/RMSProp:0', [400, 121]),
('rnn/gmm/weights/RMSProp_1:0', [400, 121]),
('rnn/gmm/biases/RMSProp:0', [121]),
('rnn/gmm/biases/RMSProp_1:0', [121])]
trainable parameters:
[('rnn/LSTMAttentionCell/lstm_cell/kernel:0', [476, 1600]),
('rnn/LSTMAttentionCell/lstm_cell/bias:0', [1600]),
('rnn/LSTMAttentionCell/attention/weights:0', [476, 30]),
('rnn/LSTMAttentionCell/attention/biases:0', [30]),
('rnn/LSTMAttentionCell/lstm_cell_1/kernel:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_1/bias:0', [1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/kernel:0', [876, 1600]),
('rnn/LSTMAttentionCell/lstm_cell_2/bias:0', [1600]),
('rnn/gmm/weights:0', [400, 121]),
('rnn/gmm/biases:0', [121])]
trainable parameter count:
3632431
2023-12-28 22:26:24.866368: I C:\tf_jenkins\workspace\rel-win\M\windows\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
built graph
max_x: 1199
max_c: 44
max_x: 1199
max_c: 47
[[step        0]]     [[train 11.7543s]]     loss: 4.34547901       [[val 1.2226s]]     loss: 4.18537235
Traceback (most recent call last):
File "c:/Users/fishh/Downloads/handwriting-synthesis-master/handwriting-synthesis-master/rnn.py", line 251, in
nn.fit()
File "c:\Users\fishh\Downloads\handwriting-synthesis-master\handwriting-synthesis-master\tf_base_model.py", line 150, in fit
val_batch_df = next(val_generator)
StopIteration

@acmattson3 I was able to get a model trained by tweaking tf_base_model.py a bit. Do you have any instructions on how to use that model for the samples? The original repo only supports the .npy files found in the styles directory. Thanks again.


mini card completed

Looks good!
Comments for future work:

what if the cocktail has more than 3 ingredients?
could we use a cocktail shaker or a barman icon for the ReadyToGo cocktails


--out-dir doesn't seem to preserve dir structure

I've tried:

**/*
{*,**}
***
*
{*,**/*}
{*,**,**/*}
etc.

imagemin-cli seems to be using meow. meow seems to be using minimist. I can't figure out if minimist is using any glob lib.
Can't figure it out. Any help would be appreciated.
I'd like to be able to imagemin a folder of images to another folder while preserving dir structure.

It seems like this has been a recurring issue:

https://github.com/imagemin/imagemin/issues/87
https://github.com/imagemin/imagemin/issues/78

Further investigation looks like imagemin directly uses https://github.com/sindresorhus/globby
https://github.com/imagemin/imagemin/blob/master/index.js#L53 nodir: true looks suspicious.
Seems like there is some chatter going on about nodir in the globby repo: https://github.com/sindresorhus/globby/issues/33
Having a hard time reverse-engineering https://github.com/imagemin/imagemin/blob/master/index.js to see why nodir is being set to false, but I'll try to whip together a bandage CLI.

https://github.com/corysimmons/imagemin-cli-preserve-dir-structure seems to work.
API is: imagemin-dir src dist

Closing this for one of the other issues.


Rename ballerinna-launcher module as ballerina-tool

Purpose

This PR will rename ballerina-launcher module as ballerina-tool. Hence the imports are changed as accordingly.

Codecov Report

Merging #16876 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #16876   +/-   ##
=======================================
  Coverage   68.85%   68.85%
=======================================
  Files           5        5
  Lines         122      122
  Branches       21       21
=======================================
  Hits           84       84
  Misses         38       38

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a73d19a...05d35dd. Read the comment docs.


To do: add more metadata on obsolete terms in MONDO

Need to add more metadata on obsolete terms in MONDO that reference non-obsolete classes in external ontologies.

I think we are doing this now


Upgrade autoneg-controller to an upstream release

We used (and built) the existing fork/branch to get a bug fix,
but Google's upstream repo has since fixed the bug and published releases.

Is setup/scripts/fix_autoneg.sh related to the retry-register-with-backend problem? If so, should we also update/remove it (and its references in cloudbuild.yaml and README.md)?

@JanCVanB can you reconcile the changed in the manifest too?
https://github.com/GoogleCloudPlatform/gke-autoneg-controller/blob/master/deploy/autoneg.yaml

@JanCVanB can you reconcile the changed in the manifest too? https://github.com/GoogleCloudPlatform/gke-autoneg-controller/blob/master/deploy/autoneg.yaml

Is https://github.com/selkies-project/selkies/pull/48/commits/bfc2076d4e5be2abdb5907fb463e32869cfaf587 what you had in mind?

@JanCVanB can you reconcile the changed in the manifest too? https://github.com/GoogleCloudPlatform/gke-autoneg-controller/blob/master/deploy/autoneg.yaml

Is bfc2076 what you had in mind?

Not really, need to diff the upstream manifest with the current one.

@JanCVanB can you reconcile the changed in the manifest too? https://github.com/GoogleCloudPlatform/gke-autoneg-controller/blob/master/deploy/autoneg.yaml

Is bfc2076 what you had in mind?

Not really, need to diff the upstream manifest with the current one.

Ah, the entire file, not just the changed subsection - on it

@JanCVanB can you reconcile the changed in the manifest too? https://github.com/GoogleCloudPlatform/gke-autoneg-controller/blob/master/deploy/autoneg.yaml

@danisla how about this: https://github.com/selkies-project/selkies/pull/48/commits/18d013e55b69446171954ae82ac1fb22358c2db7

@danisla Thanks for those (now-overwritten, whoops) comments, they made perfect sense. How does https://github.com/selkies-project/selkies/pull/48/commits/d7a520d1796e014d4a2402f9dbeb605fc1f460df look?

Closing for staleness


Simplify travis (jruby-head runs 2.2 compatible by default)

JRuby fails due to an encoding error, see: jruby/jruby#2581

Coverage remained the same at 98.82% when pulling 7b1d978d6df12b7aa8af939f584dc9242ef38d68 on PragTob:simplify-travis into d07fd445ac3af1ae97c3f6c36b45313f69d48b68 on lotus:master.

Coverage remained the same at 98.82% when pulling 7b1d978d6df12b7aa8af939f584dc9242ef38d68 on PragTob:simplify-travis into d07fd445ac3af1ae97c3f6c36b45313f69d48b68 on lotus:master.

Coverage remained the same at 98.82% when pulling 7b1d978d6df12b7aa8af939f584dc9242ef38d68 on PragTob:simplify-travis into d07fd445ac3af1ae97c3f6c36b45313f69d48b68 on lotus:master.


review and remove instances of message

as per https://github.com/cloudflare/wrangler/pull/391 let's review insances of message and refactor where appropriate

this is not actually necsssary, whoops


Plugin Says Update is available

When I go into the Plugin it says there is an update available.  It says 0.5.3 is available when GitHub shows the latest at 0.5.0.  Not sure if something is out of sync, but I figured I'd post an issue for it if it helps.

Thanks for mentioning that, I forgot to create a tag and version on my last merges.
I have been working on automating things a bit so that in the future when I do changes to the plugin I do not have to publish it it manually. So far this is what I have done.
I will keep this issue opened as a reminder to automate the creation of a tag and a release on a merge.

Thanks for mentioning that, I forgot to create a tag and version on my last merges.
I have been working on automating things a bit so that in the future when I do changes to the plugin I do not have to publish it it manually. So far this is what I have done.
I will keep this issue opened as a reminder to automate the creation of a tag and a release on a merge.


[BUG - Experimental] Trying RE-UE4SS with a New Blank Project in 5.4.4

Branch or Release
Experimental
Game and Engine Version
Just a blank new empty game, in UE 5.4.4, packaged for windows
Describe the bug
UE4SS not loading
Mods directory
Just the default mods
To Reproduce
Create a new project in 5.4.4 and package it

Launch game is working
Attempt UHT Dump ? How could I do that? I don't see any logs so not sure if the UE4SS is loading

Expected behavior
I was expecting the log to be created but I don't see anything created and I'm enabling the default mods and not really seeing anything happening
Screenshots, UE4SS Log, and .dmp file
Is just a basic empty map,
Desktop (please complete the following information):

OS: win11

Additional context
Just wondering if I should package my game in an specific configuration for the mods to work?
I just want to add some simple debug command function to log a message from the console in the built game, with a C++ or Lua mod. Any help appreciated, cheers.

I guess a simpler way to put this is:
I'm making a new game in UE5.4, how can I ensure modding support with UE4SS?

Got it working with the FPS sampler, maybe this depends on having a player to start the system? Anyhow, made it working just adding a player to the content, thanks.

THe issue was I was building in not shipping mode


CC74711: Wrong reference to CONTRIBUTING.md

Hello, @ghogen, @Taojunshen, @corob-msft,
This proposed file change comes from https://github.com/MicrosoftDocs/cpp-docs.de-de/pull/9 .
Could you review this contribution and help to merge if agreed?
Many thanks in advance.

@v-makoud This seems to work in GitHub without the change. Is that not true in the language-specific repos?

@v-makoud It looks like the real issue is that the de-de repo does not have a localized copy of CONTRIBUTING.md, which is what the fix in MicrosoftDocs/cpp-docs.de-de#9 is trying to remedy by linking to the primary repo's English version. The right change would be to localize CONTRIBUTING, or if that's not possible, then to make a localized file-only change along the lines of the fix above.

Thanks @corob-msft


SAMZA-1498: Support arbitrary system clock timer in operators

This patch adds the capability to register arbitrary timers for both high-level and low-level api.
For high-level, InitableFunction will pass the TimerRegistry to user through the new OpContext, and user will implement the TimerFunction to get timer notifications.
For low-level api, user can register timer in the TaskContext, and then implements the TimerCallback for specific timer actions.

@xinyuiscool Can you provide some motivation for this new feature? My original intention was to let users get callbacks on a fixed container wide timer interval, similar to the behavior of WindowableTask. This feature seems a lot more general. Also, I'd prefer to not introduce a new context object/mechanism right now since @nickpan47 is working on cleaning up that part of the API.

@prateekm : there are two main motivations behind:

For BEAM to support processing time timer/trigger, this is a MUST. I need the capability of registering arbitrary timers anytime during processing. For example, for BEAM's trigger:
AfterProcessingTime
.pastFirstElementInPane()
.plusDelayOf(Duration.standardMinutes(5)
This requires registering the trigger after first element shows up in the window pane. On the BEAM side, I am waiting eagerly for this feature to be added so I can push the code.

For Samza, this allows the better windowing and trigger. It allows each window to register a different timer, instead of using the GCD hack we currently do (this fix is also requested by @cpettitt-linkedin ). It also allows the time trigger to use a real system timer, instead of approximating it by using the time of incoming message. I believe with this patch it will be straightforward to improve both existing features.

I think adding the context on the Operator level is inevitable to support timer on the operator level. This should also be needed if we need more context object on the operator level. I can talk to @nickpan47 about this, but I don't think the future refactoring should block current feature development.

@xinyuiscool Makes sense to implement if this is required for BEAM. If we're adding it to the Samza public API, that's fine too - its certainly a more flexible behavior than the current window implementation. Good point about timer triggers as well. If we do that, we should update and clarify the existing documentation, both javadoc and the introduction/architecture pages.
Re: adding the OpContext, not sure I understand why that is required if we can register the timers on the TaskContext directly. Let me take a look at the code again. Agree that refactoring shouldn't block feature development, but we also shouldn't add/change public APIs that may get refactored again. Let's check with @nickpan47 what the timeline looks like.


mark_visited function appears to be broken or using the wrong endpoint

Issue Description
I was tooling around in an interactive session so I don't have a super clean source code snippet, but I tried to mark a submission as visited and got this error instead.
In [44]: submi = reddit.submission(s.id)

In [45]: submi.mark_visited()
---------------------------------------------------------------------------
Forbidden                                 Traceback (most recent call last)
~/reddit_opioid_mining/scraper.py in <module>()
----> 1 submi.mark_visited()

/usr/local/lib/python3.5/dist-packages/praw/models/reddit/submission.py in mark_visited(self)
    181         """
    182         data = {'links': self.fullname}
--> 183         self._reddit.post(API_PATH['store_visits'], data=data)
    184
    185     def hide(self, other_submissions=None):

/usr/local/lib/python3.5/dist-packages/praw/reddit.py in post(self, path, data, files, params)
    463         """
    464         data = self.request('POST', path, data=data or {}, files=files,
--> 465                             params=params)
    466         return self._objector.objectify(data)
    467

/usr/local/lib/python3.5/dist-packages/praw/reddit.py in request(self, method, path, params, data, files)
    504         """
    505         return self._core.request(method, path, data=data, files=files,
--> 506                                   params=params)
    507
    508     def submission(  # pylint: disable=invalid-name,redefined-builtin

/usr/local/lib/python3.5/dist-packages/prawcore/sessions.py in request(self, method, path, data, files, json, params)
    183         return self._request_with_retries(
    184             data=data, files=files, json=json, method=method,
--> 185             params=params, url=url)
    186
    187

/usr/local/lib/python3.5/dist-packages/prawcore/sessions.py in _request_with_retries(self, data, files, json, method, params, url, retries)
    128                                   retries, saved_exception, url)
    129         elif response.status_code in self.STATUS_EXCEPTIONS:
--> 130             raise self.STATUS_EXCEPTIONS[response.status_code](response)
    131         elif response.status_code == codes['no_content']:
    132             return

Forbidden: received 403 HTTP response

System Information

PRAW Version: 6.0.0
Python Version: 3.5.2
Operating System: Ubuntu 16.04

Oh okay, thats it. Thanks.
On Tue, Nov 20, 2018, 8:27 AM nmtake<EMAIL_ADDRESS>wrote:

Submission.mark_visited() hits /api/store_visits
https://www.reddit.com/dev/api/#POST_api_store_visits and the endpoint
requires Reddit Premium https://www.reddit.com/premium (formerly known
as Reddit Gold).
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/praw-dev/praw/issues/981#issuecomment-440272872, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFiX7WkclgrHWnyNgcWT_OXkGNpYrK3Eks5uxAMzgaJpZM4Yp5dK
.


Add --fail-on-error option and fail exports on errors

This adds an OS error handler that optionally sets the exit code to EXIT_FAILURE. The option can be enabled with the --fail-on-error CLI flag, but it's also set when exporting a project. The error handling is a little funky to make sure that a script calling get_tree().quit() doesn't inadvertently set the exit code back to EXIT_SUCCESS. I'm also not really sure about the public  is_error_occurred/set_error_occurred handlers, but they were necessary for testing.
Fixes: #94957

However, this is always printed when --fail-on-error is used when SceneTree.quit() is called (since its default value is 0):
WARNING: Cannot set exit code to 0 after an error has occurred.
     at: set_exit_code (./core/os/os.cpp:221)

Code used:
func _ready() -> void:
	push_error("Error")
	get_tree().quit()

Should I drop the warning? It was my minor attempt at highlighting to users why quit() may not be doing what they expect, but I'm not sure it's adding any value. I could also change it to a WARN_VERBOSE or put it under DEV_ENABLED or something.

This does not occur if the engine is exited via other methods, e.g. --quit.

It still exit's unsuccessfully with --fail-on-error --quit, though, right?

Should I drop the warning?

Yes, I'd move it to WARN_VERBOSE().

I think all the comments are addressed. If it looks good to you, I'll squash the fixups.

@Calinou ping. Anything I can do to help get this merged?

Tested locally, it works great now 🙂

Thanks for testing! I squashed all the fixups now, but there's no change.


[FEA] Not sure but I think we may want to look at this in more detail

Is your feature request related to a problem? Please describe.
This is a part of the audit work
Not sure but I think we may want to look at this in more detail
Assoc Spark commit ID: 6c88d7c

Can we name this issue with some detail on what it is about

Agree with @kuhushukla the headline is not helpful, even after it was updated.  Please make the headline something much more descriptive about what this is tracking.  In general issue headlines with "this" and "we" in the title aren't very helpful because readers don't have the context to map the pronouns.

@razajafri Could you elaborate on why and how we think this needs to be covered and adjust the priority if needed?

Predicate push down for Orc seems to be broken regardless of the flag spark.sql.caseSensitive. I have tried the following query with Spark 3.0.0 and Spark 3.0.1 with 0.2.0 and 0.3.0 version of the plugin.
      val table = """CREATE TABLE `test_orc_data` (
      `_col1` INT,
      `_col2` STRING,
      `_col3` INT)
      USING orc"""

    spark.sql(table).collect

    spark.sql("insert into test_orc_data values(13, '155', 2020)").collect

    val df = """select _col2 from test_orc_data where _COL2 = 155"""
    spark.sql(df).collect

It works fine when using parquet.
Looks like this hasn't worked since 0.2.0 at least.

I was able to reproduce the issue, but it disappears when the fix from #1240 is applied.  I suspect this is related solely to the problem of ORC files without column names (or names that appear to be the default names, i.e.: using the _col prefix).


Enhancement of Dark mode functionality

Is your feature request related to a problem? Please describe.
When a user switch dark mode on the website and refreshes the page it instantly gets disabled which does not looks appealing.
I want to enhance the functionality of dark mode.
Describe the solution you'd like
I will enhance this feature using JavaScript to implement local storage at the backend so the mode remains the same after refreshing the website.
Describe alternatives you've considered
I have considered changing the whole functionality of the dark mode toggler.
Additional context
I am willing to work on the issue kindly assign this to me under GSSOC23.

can I work on this


Image classification doesn't work with GPU

System Information (please complete the following information):

Model Builder Version (available in Manage Extensions dialog):  17.14.4
Visual Studio Version: Visual Studio Community 2022

Describe the bug
I already founded that using GPU in Image classification does not work with previous versions of Model Builder.
Same with latest version 17.14.4 that does not requires GPU extension.
Since the bug I previously reported has been open for 3 months (here) with no progress, I hope that this time, by modifying the title of the  bug and shortening the details will help getting a fix asap.

Clear description of the problem:

To Reproduce
Steps to reproduce the behavior:
1.Select Image Classification Scenario
2.Select GPU as training environment

3. Press "Next" and start the training
4. Error comes (2 minutes after it starts)

5.Details of the log file
ID-GPU-833T4G.txt
Expected behavior
It should complete the training successfully
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.

Maybe try setting TF_FORCE_GPU_ALLOW_GROWTH to true to resolve cudnn_initialize_error?
source: TF_FORCE_GPU_ALLOW_GROWTH

Dear @LittleLittleCloud
the hint you suggested

did not change the result. Still same KO.
See the log:
error.txt
No I do not have cuda/cudnn more than that. In one of the images I posted above you see that recognizes another graphic card because I am running it on a remote session.

@IzzyHibbert
Glad to see your issue get resolved. And for sure it's possible to set TF_FORCE_GPU_ALLOW_GROWTH flag in ModelBuilder. We'll set that flag in the next release of ModelBuilder


version-related screen names in latest analytics

Not sure if this is intentional or not, but this is the latest top-screens:

@zanderso feel free to close if this is desired.

These should be suppressed after
https://github.com/flutter/flutter/commit/0bba472858c8f8b22b689c8bbb0ecc5080109638
See:
https://github.com/flutter/flutter/pull/37514#discussion_r310762170


Dashboard is failing on kubernetes 1.5

Rancher Version:
master build (14 Dec)
Docker Version:
1.12
OS and where are the hosts located? (cloud, bare metal, etc):
ubuntu 16.04
Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB)
single node rancher
Environment Type: (Cattle/Kubernetes/Swarm/Mesos)
Kubernetes 1.5 (vnext branch)
Steps to Reproduce:

Set the library catalog to rancher-catalog with branch vnext.
install kubernetes 1.5

Results:
dashboard container is failing with the following error message:
12/14/2016 2:36:17 AMUsing HTTP port: 9090
12/14/2016 2:36:17 AMUsing apiserver-host location: http://Kubernetes.kubernetes:80
12/14/2016 2:36:17 AMCreating API server client for http://Kubernetes.kubernetes:80
12/14/2016 2:36:47 AMError while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get http://Kubernetes.kubernetes:80/version: dial tcp: i/o timeout
12/14/2016 2:36:47 AMRefer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md

Steps to check -

Launch rancher/k8s:v1.5.1-rancher1-1
Choose kube-system namespace
Navigate to Service UI
Click on the port (link) mentioned in the row for "kubernetes-dashboard"
Ensure that the UI loads and you can see the various services and namespaces etc.

Rancher version: master build
Rancher-catalog - vnext branch
Kubernetes version: rancher/k8s:v1.5.1-rancher1-1
Tested kubernetes dashboard and i can see all namespaces, services, and pods.


Getting error while trying the cross validation

Thank you so much for this interesting tool & important paper.
I was using the GGS for the purpose of unsupervised clustering of multivatite time-series data. Everything is working perfect, except the cross-validation (I guess I have a problem with the multiprocessg.poll); I am getting the following error:
_File "~/anaconda/envs/python27/lib/python2.7/multiprocessing/pool.py", line 572, in get
raise self.value
TypeError: 'int' object has no attribute 'getitem'
I appreciate if you can help.
Thank you

Hi,
What type of setup are you running GGS on? Are you by chance using Windows?

No, I am working on Mac

Yes, you are right. The Cross-Validation code was missing a transpose in line 270 of ggs.py. It should work now, but let me know if you find any other errors - thanks for pointing it out!

Great! Now it works. Thank you for your quick action.


iron-nicotinamine transporter activity

Hi,
I need the following 2 terms to annotate few genes.
1. Iron-nicotinamine transporter activity:
Enables the directed movement of iron-nocotinamine
into, out of, within or between cells. Nicotinamine
is a metal chelator that is structurally similar to
phytosiderophores.
PMID: 15255869
2. copper-nicotinamine transporter activity
Enables the directed movement of copper-nocotinamine
into, out of, within or between cells. Nicotinamine
is a metal chelator that is structurally similar to
phytosiderophores.
PMID: 15255869
Let me know if these look ok.
thanx
Suparna
Reported by: smundodi
Original Ticket: "geneontology/ontology-requests/3307":https://sourceforge.net/p/geneontology/ontology-requests/3307

thanks www.movied.org


Assets editor in tutorial doesn't adapt size

Describe the bug
Assets editor in tutorial doesn't adapt size
Steps to reproduce the behavior
1.Navigate to https://arcade.makecode.com/beta.
2.Tutorial->Simple Extension-> Blocks Start Tutorial.
3.Drag set tilemap to block into on start.
4.Click the grey square to open editor and then observe.
Actual behavior:
Editor doesn't adapt size that there is no palette page arrow to switch palette.
Expected behavior:
There is palette page arrow.
Screen Recording:

Additional context:
1.OS: Windows 11
2.arcade version: 1.13.55
3.Microsoft MakeCode version: 9.3.10

This bug also reproduced in Multiplayer Tutorials->Pizza Party, shows incompletely color in Assets Editor.


AttributeError: 'Namespace' object has no attribute 'func'

I'm using python 3.6.9 to run rez. If I call rez with no selected sub parser I get an AttributeError
AttributeError: 'Namespace' object has no attribute 'func'

This is a know issue with python 3 and is triggered when no sub-parser is selected. https://bugs.python.org/issue16308

There is a proposed fix here https://github.com/nerdvegas/rez/pull/880


Mod i18n Support

Hi Tyler-IN team,
Thanks for sharing this awesome mod!
I want to contribute to do i18n for this mod.
Since this mod does not contains many texts now, (But there are some texts that would be shown via InformationManager.DisplayMessage), I want to start contribute i18n.
Do you have some workflow to do localization? Since this is C# we can use resx file in Properties.
Maybe we can make a list of text strings in resx files? Then do the translation.
We can define resx files in different culture info such as strings.fr.resx or strings.zh.resx
This can follow the 2 digits country code.
Thank you!

We'll probably use the game's localization system.
https://github.com/Tyler-IN/MnB2-Bannerlord-ModOptions/tree/master/ModuleData/Languages
We will probably use full english strings as Ids, sometimes just symbolic names like this, sometimes with full text including spaces and punctuation;
<string id="ModOptions" text="Mod Options" />
<string id="ModOptions" text="Mod Ayarları" />
<string id="ModOptions" text="模块选项" />

If you want to go ahead and add translations for the strings used, you can follow the pattern there, e.g.
<?xml version="1.0" encoding="utf-8"?>
<base xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" type="string">
  <tags>
    <tag language="简体中文" />
  </tags>
  <strings>
    <string id="CommunityPatch" text="社区补丁" />
    <string id="DisableIntroVideo" text="禁用介绍性视频" />
  </strings>
</base>

Thank you. @Tyler-IN  I will create a PR soon.

If you want to go ahead and add translations for the strings used, you can follow the pattern there, e.g.
<?xml version="1.0" encoding="utf-8"?>
<base xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" type="string">
  <tags>
    <tag language="简体中文" />
  </tags>
  <strings>
    <string id="CommunityPatch" text="社区补丁" />
    <string id="DisableIntroVideo" text="禁用介绍性视频" />
  </strings>
</base>

You can submit the files as PRs, the xmls can be freeform named or organized into folders as they are all loaded their file names are not used.
They just need to be made beneath CommunityPatch/ModuleData/Languages.

Thank you for explanation about i18n. Do you have a list of need-to-translated strings in our mods?
I found this file contains some strings to be translated.
https://github.com/Tyler-IN/MnB2-Bannerlord-CommunityPatch/blob/3b0d9ceab3e9eec174d099f60d654d3c384dfdce/src/CommunityPatch/CommunityPatchSubModule.cs
How about first translating all string such as nameof(RecordFirstChanceExceptions) and nameof(CommunityPatch)
Thank you.

And I want to translate those strings in file like this. https://github.com/Tyler-IN/MnB2-Bannerlord-CommunityPatch/blob/master/src/CommunityPatch/CommunityPatchSubModule.CopyDiagnosticsToClipboard.cs
There are a few strings which are hard-coded in the files.

I start a PR with a few translation here: https://github.com/Tyler-IN/MnB2-Bannerlord-ModOptions/pull/1
This is still WIP.

Hey @WuZhuoran I appreciate the changes over there - but seeing as developments have occurred that have pushed development away from ModOptions, we will need translations over here again.
After this coming RC, we'll be pushing for all strings to incorporate internationalization.

Hey @WuZhuoran I appreciate the changes over there - but seeing as developments have occurred that have pushed development away from ModOptions, we will need translations over here again.
After this coming RC, we'll be pushing for all strings to incorporate internationalization.
We'll use the same files, same pattern.

Thank you for your notice. I will work this this PR.


Add state machines implementation guidelines

Per discussions on the last 2 community calls
https://wiki.hyperledger.org/display/ARIES/2023-03-23+Aries-vcx+community+call
https://wiki.hyperledger.org/display/ARIES/2023-03-16+Aries-vcx+community+call
It will be important to stay aligned on the approach as we are preparing to rewrite state machines with the state pattern.

Codecov Report

Merging #781 (fb032b0) into main (945a67a) will decrease coverage by 24.66%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             main     #781       +/-   ##
===========================================
- Coverage   54.66%   30.01%   -24.66%
===========================================
  Files         381      379        -2
  Lines       36858    30617     -6241
  Branches     8123     6301     -1822
===========================================
- Hits        20150     9189    -10961
- Misses      10706    18702     +7996
+ Partials     6002     2726     -3276

Flag
Coverage Δ

unittests-aries-vcx
30.01% <ø> (-24.56%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.
see 212 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Probably the most notable takeaways would be:

Failure states -> Failed states could be represented as Err(StateMachine<FailedState>) variants of a Result. We can then treat the failed state types as errors and handle them however it seems appropriate. The success variant would thus look something like Ok(StateMachine<SuccessState>).
Reduced I/O -> Most state machines have states of the sort MessageSent. However, when thinking about it a bit, simply saying we sent something doesn't really mean that much since we cannot guarantee it was also received, especially in a transport abstract manner. Therefore, we could model the state machines so that we work on MessageGenerated states, which we can guarantee, and then the consumer code can retrieve the message and send it (as many times as they want). Then the state machine would advance when the reply to the sent message is received (if one is expected).


Standardise files with files in sous-chefs/repo-management

Signed-off-by: Dan Webb<EMAIL_ADDRESS>

Released as: 2.1.5


WIP: Add current state support to new chart resource

Towards https://github.com/giantswarm/giantswarm/issues/4949

Closing in favor of #173.


DocsArchive-<IP_ADDRESS>Auto archive content

Please don't merge this PR until target archive repo PR https://github.com/OPS-E2E-PPE/docs-archive-test-target_958b8dae-e75e-44e9-96ba-0159ae324b09/pull/1 is merged into live branch.
Auto archive content to https://github.com/VSC-Service-Account/docs-archive-test-target_958b8dae-e75e-44e9-96ba-0159ae324b09.git

Docs Build status updates of commit 07363e9:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error-ServiceAccountCannotAccessRepository] Your repository https://github.com/VSC-Service-Account/docs-archive-test-source has enabled Docs Build service account. It seems that at least one of the accounts, e2ebd2, does not have Read permission to the repo. Please ask your repo admin to grant correct permission to service accounts (refer to this document: https://review.docs.microsoft.com/en-us/engineering/projects/ops/engdocs/how-to-grant-service-account-permission-in-your-repository?branch=master#ops-e2e-ppe) and then retry the build. Don't know who to contact? Refer https://repos.opensource.microsoft.com/VSC-Service-Account/repos/docs-archive-test-source/permissions/ to get admin list of the repo if the repo is in one of the organizations Microsoft owns.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


Problems running the program

First of all I have to say the program is great. Easy to use and the output is very comprehensive.
However, recently I have been having this error:
/Users/JFF/opt/miniconda3/envs/cctyper/lib/python3.9/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
from pandas import MultiIndex, Int64Index
[2022-04-04 14:29:15] INFO: Running CRISPRCasTyper version 1.3.0
[2022-04-04 14:29:15] INFO: Predicting ORFs with prodigal
[2022-04-04 14:29:15] INFO: Running HMMER against Cas profiles
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 691/691 [00:14<00:00, 48.44it/s]
[2022-04-04 14:29:31] INFO: Parsing HMMER output
[2022-04-04 14:29:31] INFO: Subtyping putative operons
/Users/JFF/opt/miniconda3/envs/cctyper/lib/python3.9/site-packages/cctyper/castyping.py:294: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only.
single_effector_hmms = self.scores[self.scores['Hmm'].isin(list(specifics))].drop('Hmm', 1)
[2022-04-04 14:29:31] INFO: Predicting CRISPR arrays with minced
[2022-04-04 14:29:31] INFO: No CRISPRs found.
[2022-04-04 14:29:31] INFO: Plotting map of CRISPR-Cas loci
[2022-04-04 14:29:31] INFO: Removing temporary files
It's strange because I have run this same sequence on other occasions with the program and I am sure there are CRISPRs.
Do you know what might be causing this?
Kind regards,
Javier

Dear Javier
There should not be differences between runs of the same cctyper versions.
Could you have run it with another version of cctyper? I can see you're using an older CCTyper version.
Also, CRISPR arrays or cas operons that are different between occasions?
Best,
Jakob

Thank you, updating the program did the trick. I still get the message "FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead" but the program works without issue.
Best,
Javier

Great. Thank you for reporting this.

Jakob


Add github action to build Segment-mini container

Adds a GitHub action that runs when a GitHub release is created.
This action will build a Docker container using the Segment_mini
directory and push it to ghcr.io.
The Dockerfile is based on code from:

Segment_mini/Docker_SMP_env_v2/Dockerfile_smp_env_v2
Segment_mini/Singularity_def/Segment_mini_v3.def
The Dockerfile label annotations based on:

https://github.com/opencontainers/image-spec/blob/main/annotations.md.

@thibaulttabarin Can you review this PR? Thanks.


Cannot Login - Cookie Expired and Program Fails to Run

Prerequisites

[y] Did you read FAQ section?
[y] Did you test with the latest releases or commit ?

Description
On the last working PixivUtil version (pixivutil20191118), I got this error.

I did read the FAQ plus https://github.com/Nandaka/PixivUtil2/issues/505/, and pasted my cookie onto the config.ini but the issue still persists.
When trying to run the latest version (pixivutil20200125.-.CNY2020), I got this error instead.

Steps to Reproduce

I just start both programs.
Expected behavior: [Both of the programs run then ask for further user inputs]
Actual behavior: [Both programs encountered bugs and closes immediately]

Versions
You can get this information from executing PixivUtil2.py --help.
Last working version on my PC : pixivutil20191118
Other version I tried on my PC : pixivutil20200125.-.CNY2020

are using proxy? what is your windows version?

Sorry for the late reply.
I'm running 64-bit Win7. I don't use any proxies or VPNs.
After a bit of tinkering I'll run the program again soon, see if there's any other problem.

I have the same problem, happened just recently as i had used the program on the 18th.
Haven't had this despite using the cookie login for the better part of 2-3months without updating cookie.
Windows 10, same pixivutil20191118

ok so far for me using the same cookie. Maybe check if the useragent is different with your actual browser to login?

I tried, doesn't work. Not sure why it works for you. I even created a new pixiv account, verified it as well, nothing works.

updated python to 3.8, used the newest version, works alright. I guess it was just old and something new on the site broke the old version.

Running into the same thing, also with pixivutil20191118 on a Win8.1 machine. Had been working fine through the end of my last run on the 14th, but started throwing that error on log-in when I tried doing another run a few days ago. It might be worth noting that after that last successful run I visited some pixiv pages in my browser for the first time in a few weeks; don't know if that might've triggered some site change that blocked the downloader or not, but it's only proximate event I have between it working and not working. The site also had a notice about its new page header design when I visited after the error started, which I guess could be related.
Also, when I checked the cookies on a current pixiv page I found that my login cookie was unchanged, but I couldn't find anything that matched up with the 'refresh_token' listed under it in my config file. No cookie on the page with that name or one close to it, and nothing that matched up to the format of the one I had saved in my config.
No comment on how a newer version fares, as I actually haven't been able to get a 3.8-based one running yet. But that's a separate issue.

refresh token is used by oAuth to get additional information. It is based on the pixiv app application.

Looks like installing Phyton 3.8 solved the issue.
I just ran the newest PixivUtil release after installing Phyton and it worked.


Portainer-CE 2.0 is missing Stacks

Bug description
After upgrade to CE 2.0.0, Portainer is no longer displaying all existing stacks.
Also counts of Stacks are displayed incorrectly in multiple places. - however that's repro with 1.24.1 too.
Expected behavior
All stacks should be listed properly.
Steps to reproduce the issue:
Our setup:

Swarm with 1 manager and 1 worker
5 stacks with active services (incl Portainer stack)
2 stacks without services (placeholders created in Portainer but with services deleted)

Steps:

Have Portainer 1.24.1 on Docker Swarm with the aforementioned stack setup.
Upgrade to Portainer-CE 2.0.0
Log in and observe Home page - "primary" endpoint is listing 5 stacks:

Select the primary endpoint and observe the Dashboard - 7 stacks:

Select Stacks list - only 2 stacks are shown (Portainer stack and one of the empty placeholders):

Downgrade to Portainer 1.24.1 and go to Stacks - all 7 are visible now:

Technical details:

Portainer version:
Docker version (managed by Portainer): 19.03.13
Platform (windows/linux): Linux, CentOS 7
Command used to start Portainer (docker run -p 9000:9000 portainer/portainer): docker stack deploy -c portainer-agent-stack.yml portainer
2 Compose stacks (for 1.24.1 and 2.0.0): portainer-compose.zip
Browser: Firefox

Hi thanks for reaching us, kindly reintialize swarm and check

Hi kindly share the portainer and portainer agent logs after upgrading to 2.0.

Hi @balasu, thanks for prompt response. See below the logs I was able to get.
I tried to follow the logs link from Bug template but it's no longer valid...
Portainer log:
2020/11/02 16:14:55 Instance already has defined endpoints. Skipping the endpoint defined via CLI.
2020/11/02 16:14:55 server: Reverse tunnelling enabled
2020/11/02 16:14:55 server: Fingerprint 0c:94:ce:c8:90:3e:f6:31:37:48:53:80:52:3e:3a:4d
2020/11/02 16:14:55 server: Listening on <IP_ADDRESS>:8000...
2020/11/02 16:14:55 Starting Portainer 2.0.0 on :9000
2020/11/02 16:14:55 [DEBUG] [chisel, monitoring] [check_interval_seconds: 10.000000] [message: starting tunnel management process]
2020/11/02 16:14:58 [WARN] [docker,snapshot] [message: unable to snapshot engine information] [endpoint: primary] [err: Cannot connect to the Docker daemon at tcp://tasks.agent:9001. Is the docker daemon running?]
2020/11/02 16:14:58 [WARN] [docker,snapshot] [message: unable to snapshot containers] [endpoint: primary] [err: Cannot connect to the Docker daemon at tcp://tasks.agent:9001. Is the docker daemon running?]
2020/11/02 16:14:58 [WARN] [docker,snapshot] [message: unable to snapshot images] [endpoint: primary] [err: Cannot connect to the Docker daemon at tcp://tasks.agent:9001. Is the docker daemon running?]
2020/11/02 16:14:58 [WARN] [docker,snapshot] [message: unable to snapshot volumes] [endpoint: primary] [err: Cannot connect to the Docker daemon at tcp://tasks.agent:9001. Is the docker daemon running?]
2020/11/02 16:14:58 [WARN] [docker,snapshot] [message: unable to snapshot networks] [endpoint: primary] [err: Cannot connect to the Docker daemon at tcp://tasks.agent:9001. Is the docker daemon running?]
2020/11/02 16:14:58 [WARN] [docker,snapshot] [message: unable to snapshot engine version] [endpoint: primary] [err: Cannot connect to the Docker daemon at tcp://tasks.agent:9001. Is the docker daemon running?]
2020/11/02 16:15:52 http error: Invalid JWT token (err=Invalid JWT token) (code=401)
2020/11/02 16:15:52 http error: Unauthorized (err=Unauthorized) (code=401)
Agent Manager log:
2020/11/02 16:14:55 [INFO] [main] [message: Agent running on Docker platform],
2020/11/02 16:14:55 [INFO] [main] [message: Agent running on a Swarm cluster node. Running in cluster mode],
2020/11/02 16:14:58 [INFO] serf: EventMemberJoin: mgs-qa-harbor201.be.monster.com-8a224f45ef66 <IP_ADDRESS>,
2020/11/02 16:14:59 [INFO] [http] [server_addr: <IP_ADDRESS>] [server_port: 9001] [secured: true] [api_version: 2.0.0] [message: Starting Agent API server],
2020/11/02 16:15:03 [INFO] serf: EventMemberJoin: mgs-qa-harbor-worker201.be.monster.com-6d3a8de95c14 <IP_ADDRESS>,
Agent Worker log:
2020/11/02 16:15:00 [INFO] [main] [message: Agent running on Docker platform],
2020/11/02 16:15:00 [INFO] [main] [message: Agent running on a Swarm cluster node. Running in cluster mode],
2020/11/02 16:15:03 [INFO] serf: EventMemberJoin: mgs-qa-harbor-worker201.be.monster.com-6d3a8de95c14 <IP_ADDRESS>,
2020/11/02 16:15:03 [INFO] serf: EventMemberJoin: mgs-qa-harbor201.be.monster.com-8a224f45ef66 <IP_ADDRESS>,
2020/11/02 16:15:04 [INFO] [http] [server_addr: <IP_ADDRESS>] [server_port: 9001] [secured: true] [api_version: 2.0.0] [message: Starting Agent API server],

Hi @fkrivsky , pl let us know if u are using any extensions in 1.24

I just did an install using the scripts provided by the blog to depoy a stack to docker swarm and my setup seems to be having problems figuring out the numbers also.  It shows no endpoints available and the endpoints request doesn't seem to contain container details except for total number running. The payload response looks like this,
GET /api/endpoints?limit=100&start=0

[
    {
        "Id": 3,
        "Name": "primary",
        "Type": 2,
        "URL": "tcp://tasks.agent:9001",
        "GroupId": 1,
        "PublicURL": "",
        "TLSConfig": {
            "TLS": true,
            "TLSSkipVerify": true
        },
        "Extensions": [],
        "AzureCredentials": {
            "ApplicationID": "",
            "TenantID": "",
            "AuthenticationKey": ""
        },
        "TagIds": [],
        "Status": 1,
        "Snapshots": [
            {
                "Time":<PHONE_NUMBER>,
                "DockerVersion": "19.03.13",
                "Swarm": true,
                "TotalCPU": 16,
                "TotalMemory":<PHONE_NUMBER>8,
                "RunningContainerCount": 4,
                "StoppedContainerCount": 2,
                "HealthyContainerCount": 0,
                "UnhealthyContainerCount": 0,
                "VolumeCount": 0,
                "ImageCount": 37,
                "ServiceCount": 2,
                "StackCount": 1,
                "DockerSnapshotRaw": {
                    "Containers": null,
                    "Volumes": null,
                    "Networks": null,
                    "Images": null,
                    "Info": null,
                    "Version": null
                }
            }
        ],
        "UserAccessPolicies": {},
        "TeamAccessPolicies": {},
        "EdgeKey": "",
        "EdgeCheckinInterval": 0,
        "Kubernetes": {
            "Snapshots": [],
            "Configuration": {
                "UseLoadBalancer": false,
                "UseServerMetrics": false,
                "StorageClasses": [],
                "IngressClasses": []
            }
        },
        "AuthorizedUsers": null,
        "AuthorizedTeams": null,
        "Tags": null
    }
]

Running Docker version 19.03.13, build 4484c46, w/ Portainer-CE 2.0 on a 4 node swarm.  I've had some issues getting kubernetes installed so was reverting back to docker swarm.  I had it all working before, but seem to be having issues this time around.

Hi @balasu, not using extensions.

I am also having this same issue see screen shots attached.

Upgraded to 2.0

HI @Cpn-Neemo  could it possible to share more about environment its deployed. are you using NFS and extensions as well. pl share portainer logs

Ticket update by Portainer Support<EMAIL_ADDRESS>Confirm with Devs - do we have a bug here?

Hi there,
I'm having the same problem upgrading to Portainer-ce. My dashboard is showing me 8 stacks but when I'm going to the stacks page, only 4 of them are displayed.
After some research, here are helpful information:

I have 2 unmanaged stacks (outside Portainer), that are correctly shown
The API call to get stacks is returning 6 stacks (8 - 2 outside Portainer, seems OK)
In theses 6 stacks returned, only 2 of them have the Status prop set to 1 (= 2 portainer stacks that are correctly displayed) and the missing stacks are the ones that have the Status prop set to 0
I also looked in the data/compose folder my Portainer container, and all the stacks are present here
The missing stacks are well deployed (containers are alive)
I'm running Portainer in the Standalone mode (non swarm)
I don't have any logs about the update from Portainer to Portainer-ce (my bad)

It seems the Status prop of my stacks broke something. I don't know what was this prop value under Portainer 1.X
Do you have any idea what to do to get my stacks back?
Here is a screenshot of my Portainer + network console:

I can provide you more information if needed!
Thanks for your work!!

Hi there,
I'm having the same problem upgrading to Portainer-ce. My dashboard is showing me 8 stacks but when I'm going to the stacks page, only 4 of them are displayed.
After some research, here are helpful information:

I have 2 unmanaged stacks (outside Portainer), that are correctly shown
The API call to get stacks is returning 6 stacks (8 - 2 outside Portainer, seems OK)
In theses 6 stacks returned, only 2 of them have the Status prop set to 1 (= 2 portainer stacks that are correctly displayed) and the missing stacks are the ones that have the Status prop set to 0
I also looked in the data/compose folder my Portainer container, and all the stacks are present here
The missing stacks are well deployed (containers are alive)
I'm running Portainer in the Standalone mode (non swarm)
I don't have any logs about the update from Portainer to Portainer-ce (my bad)

It seems the Status prop of my stacks broke something. I don't know what was this prop value under Portainer 1.X
Do you have any idea what to do to get my stacks back?
Here is a screenshot of my Portainer + network console:

I can provide you more information if needed!
Thanks for your work!!

@chiptus could that be a bug related to stopped stacks? Is that what the Status property is used for?

@chiptus could that be a bug related to stopped stacks? Is that what the Status property is used for?

@chiptus could that be a bug related to stopped stacks? Is that what the Status property is used for?

That sounds like it, I'll try to repro locally and see if it's related. We do have a migration that sets the existing one to "1" (active), but I'll check it

@chiptus could that be a bug related to stopped stacks? Is that what the Status property is used for?

That sounds like it, I'll try to repro locally and see if it's related. We do have a migration that sets the existing one to "1" (active), but I'll check it

Status was introduced in 2.0 for active/stopped stacks. A short test confirmed that the migration did run.
@aleygues did it happen after upgrading 1.24.1 to 2.0?

Status was introduced in 2.0 for active/stopped stacks. A short test confirmed that the migration did run.
@aleygues did it happen after upgrading 1.24.1 to 2.0?

Hi there, thanks for the replies!
Indeed @chiptus stacks disappeared after migrating from 1.24.1 to 2.0.

Hi there, thanks for the replies!
Indeed @chiptus stacks disappeared after migrating from 1.24.1 to 2.0.

@chiptus is it possible that the migration did not run on some stacks? In this case Swarm stacks?
Thanks for the feedback @aleygues.

@chiptus is it possible that the migration did not run on some stacks? In this case Swarm stacks?
Thanks for the feedback @aleygues.

@deviantony I don't have any Swarm stacks, only compose ones (v2). All my stacks (the migrated ones and the missing ones) are pretty the same

@deviantony I don't have any Swarm stacks, only compose ones (v2). All my stacks (the migrated ones and the missing ones) are pretty the same

Ah I thought these were Swarm stacks as I was looking at the bug report context, sorry about that.

Ah I thought these were Swarm stacks as I was looking at the bug report context, sorry about that.

Ticket update by Portainer Support<EMAIL_ADDRESS>Hi, following up on the previous request - have you got more information you
can provide on this?

Ticket update by Portainer Support<EMAIL_ADDRESS>Hi, following up on the previous request - have you got more information you
can provide on this?

Unfortnetly I don't have anymore information, only the few I posted before. The bug is still present and the only way to update my stack is to open a displayed on, and change the ID is the URL by the ID of the stack I want to update (the one that disappeared from the list, Status 0).
I only can add that by using this workaround, when I force by the URL the view of a disappeared stack, services of the stack are not displayed in the stack summary. By doing this, only the compose editor work (including envs).

Unfortnetly I don't have anymore information, only the few I posted before. The bug is still present and the only way to update my stack is to open a displayed on, and change the ID is the URL by the ID of the stack I want to update (the one that disappeared from the list, Status 0).
I only can add that by using this workaround, when I force by the URL the view of a disappeared stack, services of the stack are not displayed in the stack summary. By doing this, only the compose editor work (including envs).

Same issue here, just happened today, all my swarm stacks are gone.

Same issue here, just happened today, all my swarm stacks are gone.

Just suddenly, or only after an upgrade?

Just suddenly, or only after an upgrade?

Was suddenly, but after service stop docker and service start docker everything is okay

Was suddenly, but after service stop docker and service start docker everything is okay

@deviantony any update on this? Is there at least a way to manually change stacks status to 1 to make them reappear? Maybe by updating the Portainer DB directly from Portainer Container?

@deviantony any update on this? Is there at least a way to manually change stacks status to 1 to make them reappear? Maybe by updating the Portainer DB directly from Portainer Container?

@aleygues unfortunately not as we're not able to reproduce, I'd advise against editing the database directly even though it is possible with tools such as https://github.com/br0xen/boltbrowser
Instead, I'd recommend using the Portainer HTTP API to update the status of each "missing" stack.

@aleygues unfortunately not as we're not able to reproduce, I'd advise against editing the database directly even though it is possible with tools such as https://github.com/br0xen/boltbrowser
Instead, I'd recommend using the Portainer HTTP API to update the status of each "missing" stack.

@deviantony, thanks for the answer! Do you know where I can find an example of HTTP call to update the status of a stack?

@deviantony, thanks for the answer! Do you know where I can find an example of HTTP call to update the status of a stack?

@aleygues unfortunately our API documentation is not really up to date but we're working on it at the moment.
You can find some examples (although some of them might be outdated) in https://gist.github.com/deviantony/77026d402366b4b43fa5918d41bc42f8

@aleygues unfortunately our API documentation is not really up to date but we're working on it at the moment.
You can find some examples (although some of them might be outdated) in https://gist.github.com/deviantony/77026d402366b4b43fa5918d41bc42f8

I've just hit the same issue. I was upgrading 1.24.1 to 2.1.1, I am using Portainer on single host with Docker socket.
I have had 10 stacks, but 6 stacks somehow disappeared. Containers were OK, networks were OK. I have checked API endpoints and stacks were there, but with status=0.

Here are my steps to recover, maybe it could help you:

Backup docker-compose.yml.

There is portainer data ({vps-path}/portainer/compose) folder on your VPS and structure looks like. Dump whole compose folder, just for sure.
# It's my path
tree /srv/portainer/data

# Tree output
.
├── bin
├── compose
│   ├── 1
│   │   └── docker-compose.yml
│   ├── 14
│   │   └── docker-compose.yml
│   ├── 15
│   │   └── docker-compose.yml
│   ├── 16
│   │   └── docker-compose.yml
│   ├── 17
│   │   └── docker-compose.yml
│   ├── 18
│   │   └── docker-compose.yml
│   ├── 4
│   │   └── docker-compose.yml
│   └── 8
│       └── docker-compose.yml
├── config.json
├── portainer.db
├── portainer.key
├── portainer.pub
└── tls

Stop & remove all containers from disappeared stacks.

Remove all networks from disappeared stacks.

[optionally] docker system prune

Create stacks again.

This is stricky. Portainer says there is a stack with given name. So, take a look at devtools a search for {domain}/api/stacks XHR request.  Open some other stack in browser and update ID in URL to some dissappeared stack ID, e.q. {domain}/#!/1/docker/stacks/name?id={ID}&type=2&external=false.  Optionally, take a look at editor tab and backup docker compose. Then, drop stack. After that, recreate stacks from your backups.

It works.

Ticket update by Bala Natarajan<EMAIL_ADDRESS>Hi @f3l1x
are the 6 stacks I.e disappeearing, deployed from portainer UI/API duirng 1.24 version. is it containers are running after upgrade but stack is not showing. could it possible to share the compose files as well. Assuming docker version is 19.03 or 20.10

Ticket update by Microsoft Outlook<EMAIL_ADDRESS>[https://products.office.com/en-us/CMSImages/Office365Logo_Orange.png?version=b8d100a9-0a8b-8e6a-88e1-ef488fee0470]
Your message to<EMAIL_ADDRESS>couldn't be delivered.
fkrivsky wasn't found at github.com.
hello   Office 365      fkrivsky
Action Required                 Recipient
Unknown To address
How to Fix It
The address may be misspelled or may not exist. Try one or more of the following:

Send the message again following these steps: In Outlook, open this non-delivery report (NDR) and choose Send Again from the Report ribbon. In Outlook on the web, select this NDR, then select the link "To send this message again, click here." Then delete and retype the entire recipient address. If prompted with an Auto-Complete List suggestion don't select it. After typing the complete address, click Send.
Contact the recipient (by phone, for example) to check that the address exists and is correct.
The recipient may have set up email forwarding to an incorrect address. Ask them to check that any forwarding they've set up is working correctly.
Clear the recipient Auto-Complete List in Outlook or Outlook on the web by following the steps in this article: Fix email delivery issues for error code 5.1.1 in Office 365https://go.microsoft.com/fwlink/?LinkId=389363, and then send the message again. Retype the entire recipient address before selecting Send.

If the problem continues, forward this message to your email admin. If you're an email admin, refer to the More Info for Email Admins section below.
Was this helpful? Send feedback to Microsofthttps://go.microsoft.com/fwlink/?LinkId=525920.

More Info for Email Admins
Status code: 550 5.1.1
This error occurs because the sender sent a message to an email address outside of Office 365, but the address is incorrect or doesn't exist at the destination domain. The error is reported by the recipient domain's email server, but most often it must be fixed by the person who sent the message. If the steps in the How to Fix It section above don't fix the problem, and you're the email admin for the recipient, try one or more of the following:
The email address exists and is correct - Confirm that the recipient address exists, is correct, and is accepting messages.
Synchronize your directories - If you have a hybrid environment and are using directory synchronization make sure the recipient's email address is synced correctly in both Office 365 and in your on-premises directory.
Errant forwarding rule - Check for forwarding rules that aren't behaving as expected. Forwarding can be set up by an admin via mail flow rules or mailbox forwarding address settings, or by the recipient via the Inbox Rules feature.
Mail flow settings and MX records are not correct - Misconfigured mail flow or MX record settings can cause this error. Check your Office 365 mail flow settings to make sure your domain and any mail flow connectors are set up correctly. Also, work with your domain registrar to make sure the MX records for your domain are configured correctly.
For more information and additional tips to fix this issue, see Fix email delivery issues for error code 550 5.1.1 in Office 365https://go.microsoft.com/fwlink/?LinkId=389363.
Original Message Details
Created Date:   2/16/2021 7:09:46 AM
Sender Address<EMAIL_ADDRESS>Recipient Address<EMAIL_ADDRESS>Subject:        Re: Portainer-CE 2.0 is missing Stacks #QUE00000073
Error Details
Reported error: 550-5.1.1 The email account that you tried to reach does not exist. Please try 550-5.1.1 double-checking the recipient's email address for typos or 550-5.1.1 unnecessary spaces. Learn more at 550 5.1.1 https://support.google.com/mail/?p=NoSuchUser a20si4908036lfk.226 - gsmtp
DSN generated by:       MEAPR01MB4566.ausprd01.prod.outlook.com
Remote server:  mx.google.com
Message Hops
HOP     TIME (UTC)      FROM    TO      WITH    RELAY TIME
1       2/16/2021
7:09:46 AM      ip-10-10-18-71.happyfox v2app8.mail.happyfox.com        ESMTPS  *
2       2/16/2021
7:09:53 AM      v2app8.mail.happyfox.com SJ0PR05CA0126.namprd05.prod.outlook.com Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384)    7 sec
3       2/16/2021
7:09:55 AM      MEAPR01MB2631.ausprd01.prod.outlook.com MEAPR01MB2631.ausprd01.prod.outlook.com mapi    2 sec
4       2/16/2021
7:09:55 AM      MEAPR01MB2631.ausprd01.prod.outlook.com MEAPR01MB4566.ausprd01.prod.outlook.com Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384)    *
Original Message Headers
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
b=AOv6RscTZq7l1e/8vnsFooJC9E/5D9i+HUNLSbogHuphjA9teraL3FpczZ3NPVr/jJTAu5vYjloWttE9Skyv0t8SMHxuZR1UnNWdvNUfZ/N2ExGIp1vY5Ji1aDuyqL20KAq2gU+WNaY7XWRAvk5CT0w9JozRNnPVMcIaMzKJ0J/7HSLq6uZB5wsbBGK6biMA/i/B6uISiqVPuxxZXD+xb/QUa1VXchchEkTfBw3s3Lpn+yzMK9uFTbI9XKCQ+W9qeQDXA7wgecMzVZJoZlDkuawP4fPxgm4v1vwnaWwMxnvZMONecVzWOgwZ17KnK66XxVPP/J2nCcY7EO18tmwtUg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
s=arcselector9901;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=pIfBm9mQK7mGbNqMOrkKPgGeVIXnW77oFftUrOl2yaA=;
b=SGdsOQP/x7g31xYkYPLRLWDA5WoAZODX0lVm02b1eAS232x4btQTNE2czqgZI/JIPh/2u5wraPWUerGi4DuExWu4TlGdu6i58lmPBVpyV0oZRhqvyFWZ1EnWBzTcacLOUUT/cqZJCrzWK37ofaoqjmM7r6gGCuwwQWagE55mzyC9EhwjcKRB5bwMBuABMfwC7cjmJGyO01XBNpGBXKpTB5Irz3NaqrJJN9qlBTRt/2o8J1RrD4p6AwgB0AyxwKYyx05Fm3Aye+JqK9MjBwJG9E4dt52RXTPFR/9ZNkj93AaGze9H6qvyH3BJpveZ0YHe4uX+YQBQGjcGavaEzoWmJA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
smtp.mailfrom=portainer.io; dmarc=pass action=none header.from=portainer.io;
dkim=pass header.d=portainer.io; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=portainer.io;
s=selector1;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=pIfBm9mQK7mGbNqMOrkKPgGeVIXnW77oFftUrOl2yaA=;
b=2baF6/BD4lI28laN1yi+hshFp9idcIAPWBAqXZVJwIDpaUJv5HCMF/nkhMeA7afr2IbZNnCoST0Uqr829JzCdT52ETAt+WRpq/NtOyvpy/NQIvi3hIy97SN84idwQ6/I7419VM/+pCGvhzz8tEuPf5m/5D/Rmj13Dso6AVyen7U=
Authentication-Results: github.com; dkim=none (message not signed)
header.d=none;github.com; dmarc=none action=none header.from=portainer.io;
Received: from MEAPR01MB2631.ausprd01.prod.outlook.com (2603:10c6:201:b::15)
by MEAPR01MB4566.ausprd01.prod.outlook.com (2603:10c6:220:15::20) with
Microsoft SMTP Server (version=TLS1_2,
cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3846.27; Tue, 16 Feb
2021 07:09:55 +0000
Received: from MEAPR01MB2631.ausprd01.prod.outlook.com
([fe80::cd2a:fba6:c086:163]) by MEAPR01MB2631.ausprd01.prod.outlook.com
([fe80::cd2a:fba6:c086:163%7]) with mapi id 15.20.3846.039; Tue, 16 Feb 2021
07:09:55 +0000
Content-Type: multipart/alternative;
boundary="===============4933024822907425458=="
Subject: Re: Portainer-CE 2.0 is missing Stacks #QUE00000073
From: "Bala Natarajan"<EMAIL_ADDRESS>To<EMAIL_ADDRESS>Date: Tue, 16 Feb 2021 07:09:46 -0000
Reply-To: "Bala Natarajan"<EMAIL_ADDRESS>References<EMAIL_ADDRESS>Message-ID<EMAIL_ADDRESS>X-Originating-IP: [<IP_ADDRESS>]
X-ClientProxiedBy: SJ0PR05CA0126.namprd05.prod.outlook.com
(2603:10b6:a03:33d::11) To MEAPR01MB2631.ausprd01.prod.outlook.com
(2603:10c6:201:b::15)
Return-Path<EMAIL_ADDRESS>MIME-Version: 1.0
X-MS-Exchange-MessageSentRepresentingType: 1
Received: from v2app8.mail.happyfox.com (<IP_ADDRESS>) by SJ0PR05CA0126.namprd05.prod.outlook.com (2603:10b6:a03:33d::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3868.15 via Frontend Transport; Tue, 16 Feb 2021 07:09:53 +0000
Received: from ip-10-10-18-71.happyfox (ip6-localhost [<IP_ADDRESS>])      by v2app8.mail.happyfox.com (Postfix) with ESMTPS id E79051E0380        for<EMAIL_ADDRESS>Tue, 16 Feb 2021 07:09:46 +0000 (UTC)
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 4de486f7-22b7-4f38-0bd6-08d8d249dc15
X-MS-TrafficTypeDiagnostic: MEAPR01MB4566:
X-Microsoft-Antispam-PRVS:
<EMAIL_ADDRESS>X-MS-Oob-TLC-OOBClassifiers: OLM:8273;
X-MS-Exchange-SenderADCheck: 1
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:
I4CyD/osm7uDJcif2LjpEcLQSpl9qtgSWocZrHoSJTWuJJ+SCQkNbjTB4DEZi4Ns3w/t1isD74TMcmdplAIyqYlAwIUW8mtUPQ3onbKOJryxuzvuwN0bL88yMptknRQDY1j/zbWEH+ZUmv56a6hrAQTRv6E6m1z4eGEBzPhO4iLfg8Ylh5OqDkY+ZpimV4PY89Zqtxc7UquB0YjGe6ke6RfPpOGaJ/a1H3zC7TF3Vecmmcgjp6kSypmyQgSu5DMCbMB6B+mwHaBTWO1wFgt3o9X/p0qEH3azOXEahs7KMU+OON9DSo3Q/1mEs63IQgNoM4XLS9FJkoi7iRmXzxo0e7TJiktZ6qrUxjPTeWAGREHBMozNICInR17JcXZeqen0sDIb47bdpU+6rHNTkI02mf23yJSEojvVrYzTugviJcUtZwox0PrJDhkN90rLtuZfI3zBr0Ig7LfELEsKQrr/pY9QXkGy8lXJnQJ9MMIiX6cR3WuixtHfhWCCxgx74Fq4hmz1RROTK01I36piIbtHzDcRX5fCxKolXPAzfykX8jmiFXm6cM6fcDpF9sso51suiCTkL4SWgCjKYoiES+HyZHNScx7ofglyVa/PRohFPDE=
X-Forefront-Antispam-Report:
CIP:<IP_ADDRESS>;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MEAPR01MB2631.ausprd01.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(346002)(136003)(396003)(366004)(39830400003)(376002)(33964004)(2906002)(66476007)(103116003)(186003)(83380400001)(478600001)(3450700001)(52116002)(86362001)(9686003)(6266002)(6916009)(8676002)(316002)(55236004)(8936002)(36756003)(26005)(166002)(66946007)(966005)(66556008)(5660300002)(7126003);DIR:OUT;SFP:1102;
X-MS-Exchange-AntiSpam-MessageData:
=?us-ascii?Q?5ipWZ/T+MqHch9/hIj8e3+63ylHv8nE6oZudVT7bRd1TweVXhoHAuMCgoZQg?=
=?us-ascii?Q?pV9miQre74S7ecmhxRLuMwVqJPdKv6tM6EdCAkm4oVd18zr+5FmzYEKMSfrT?=
=?us-ascii?Q?a18/hcmU4zFFNLXwGQzkjulNyyw2WackmYH9DYBhQ1d8OHcTxMQoqXjmxcTO?=
=?us-ascii?Q?Esbp41Jz70jBbqEWFIY5oz6MZ0/JOdYRkmrWoqOQ5Ow7HLKwhF3cbLpJ+dXN?=
=?us-ascii?Q?wNijawBpgNHOUyMRwKQ3zMUgNTnNvOFVvFlsu2P7UdW4WSAXNmdAqKn27Nvb?=
=?us-ascii?Q?9Q0wqD0jMoFTvaebVuhkN7qIl14QDDWuPnDkb5FwY3hZgq/RkNYN+ro2br8x?=
=?us-ascii?Q?d7rIMsQ9WnkiYllYG99S9ng0GtEQbKJ05lQXMjR+Mpjk1t51t9ksgmqGtuSD?=
=?us-ascii?Q?ZxVOreNOtgx/yD9KtXmF1NfnOyV9fu9JPnkHxcsVTBeEpqwTwgHuoHRT1pCQ?=
=?us-ascii?Q?McRK2efzufgktlQEVpExR0svdwgVmdxzLEcCcc+uCC5K+rtPZkBR4zpHqUje?=
=?us-ascii?Q?thO9d19tIyxWaWQjtrK68sV/Zcz+WjyiYE9/3WGNVgTXBrcj6LUeAv6lIwYk?=
=?us-ascii?Q?bk7OOCHt40ace6EfJBnTMXSHQN4FFPh98MNYrtNYdbMB1ATHjZYombmuVRYR?=
=?us-ascii?Q?S0X+7cijbYs4Rh8Il05qTvatR/pRTclwmKUHCAD7bUZdeSRkAXdwGHd/xIGP?=
=?us-ascii?Q?yQRRPfsdCtsMw77Qp9HBA7MjPGxTjqx8SLxOCiqg7t7w/iin0Mzza/7I70YH?=
=?us-ascii?Q?WGzmLIX7ncqZkZY09RljTUWdIxBmR+5zwzlIfZ2f2hCOqjEF47cCOea8iwxg?=
=?us-ascii?Q?29GSgZIWm/8egPJGX8VpbHr4CLWlL8mkILx0Jl5HnqCLwSp3SoB2aLOBrBJC?=
=?us-ascii?Q?OZNBn2MXJcD/3gzEezpXqcL7O6qNbdRsBlvS0lLNHFyVl4BLCbJDdGbscZXW?=
=?us-ascii?Q?2HdPB/8f7mD77TJ3X/AOtBWTIk2G8yMUnD4YDacd/LBTBlf4u1drxBg9ejAr?=
=?us-ascii?Q?ipkFI6fNJnrG2s8i5THV6zV56r18ukKmcfPkVjB1ptVOL0bmkMWL6gcfDEf5?=
=?us-ascii?Q?fh5zC6XWIuF/shmdhRVdo7lqQT983L41C0PpvcGBmo6+69K47hIwi0CTSFAe?=
=?us-ascii?Q?04xWfysMHyUkPrslxlQ0rIpQX0He53hb6Fa1qabguuVku95SW0Mxz9mqnIv2?=
=?us-ascii?Q?M405EOj+KiwEPp8/rmO2z+1CUdLWbqn31j4cX52V7e0NLBCvVNkcKoTAWtTt?=
=?us-ascii?Q?w3a0t2a1z4QPG6aybndKjIuuU7w+GF1f/URZvplKqv43PFECddawyO2+4Y11?=
=?us-ascii?Q?TN4oS6uajQfWPToXUeVXiRgM?=
X-OriginatorOrg: portainer.io
X-MS-Exchange-CrossTenant-Network-Message-Id: 4de486f7-22b7-4f38-0bd6-08d8d249dc15
X-MS-Exchange-CrossTenant-AuthSource: MEAPR01MB2631.ausprd01.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Feb 2021 07:09:55.1970
(UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 34d4c73d-4fff-4352-9ce2-df14c8d839f5
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 95nZWn800dYadCh8RyxD8h1Z77vRvWUgSNV7UBzKx5+HjM8bOU6/01XTKOLgCAYVYLyN87Ch6ifrTnbacwVrVQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MEAPR01MB4566
New reply for the ticket #QUE00000073
Hi @f3l1x
are the 6 stacks I.e disappeearing, deployed from portainer UI/API duirng 1.24 version. is it containers are running after upgrade but stack is not showing. could it possible to share the compose files as well. Assuming docker version is 19.03 or 20.10
Bala Natarajan
Reply to this email to submit an update.

I've tried and failed to replicate this in different scenarios.. Tested by creating stacks in and out of Portainer UI
Portainer 1.24.1 --> 2.0.0 on Docker 20.10.3 - Standalone and Swarm
Portainer 1.24.1 --> 2.1.1. On Docker 20.10.3 - Standalone and Swarm
Portainer 1.24.1 --> 2.0.0 on Docker 19.03.13 - Standalone and Swarm
Portainer 1.24.1 --> 2.1.1. On Docker 19.03.13- Standalone and Swarm
@deviantony :  Anything else I could try? Could we get either Devs or Product to look into this?

I've added it in our backlog for investigation by the development team.

My bad, in the DB, i can the some differences :
I have a new attribute "Status": 1 in new stacks entries.
Maybe, i can fix the DB entries ?

Hi Jean,
Can you try to start the missing stack using API
on the docker host using httpie
http POST :9000/api/auth Username="admin" Password="<admin password"
Copy the token
http GET :9000/api/stacks "Authorization: Bearer <JWT Token>
Copy the ID of the missing stack
'http POST :9000/api/stacks//start "Authorization: Bearer '
This should set the Status of the stack to 1 and should show up in the GUI.
Let me know how you get on.

Hi @samdulam ,
I thank you for this tip, i retrieved all my stacks. The api worked very well.
Its a big thank you.

We have created a new KB article with this workaround here

@chiptus is it possible that the migration did not run on some stacks? In this case Swarm stacks?

@deviantony this is the merge code:
func (m *Migrator) updateStacksToDB24() error {
	stacks, err := m.stackService.Stacks()
	if err != nil {
		return err
	}

	for idx := range stacks {
		stack := &stacks[idx]
		stack.Status = portainer.StackStatusActive
		err := m.stackService.UpdateStack(stack.ID, stack)
		if err != nil {
			return err
		}
	}

	return nil
}

The only reason it might happen is that it failed, but then the whole startup will fail, no?

it's either that some stacks are skipped, or that for some reason the status changes somewhere.
@jpoupee @fkrivsky did the migration fail for some reason? (maybe it failed, but the version was stored, and the next time it skipped this migration - shouldn't happen, but I'm trying :) )

I just had this upgrading too. Stacks are online, containers in it are up. But the stacks no longer show under Stacks in portainer. If I make new stacks in the GUI they show up fine.
Out of interest, do we all mount our portainer data dirs somewhere?
user@docker:/opt/portainer/compose$ ls -l
total 16
drwxr-xr-x 2 root root 4096 Mar 10 14:19 28
drwxr-xr-x 2 root root 4096 Mar 10 14:19 31
drwx------ 2 root root 4096 Apr 28 15:54 38
drwx------ 2 root root 4096 Apr 30 13:40 39

Here stacks 28 and 31 don't show, 38 and 39 do. Does portainer ignore if 755 is set or something weird?


Parsing apparently enters infinite loop

Using syn 2.0.77, the following reduced examples seems to never return (at least not before I kill it after several minutes). Notice that 2.0.61 returns, while 2.0.62 .. does not.
fn main() {
    syn::parse_str::<syn::File>("#[a=a=a<a<]");
}

fn main() {
    syn::parse_str::<syn::GenericParam>("a=<a<{a=a<a<}");
}

Using 09d020f5, #[r=6=r..1..] seems to run into the same problem.


Cant load dashboard for Target Drift

I've been trying to find out why the html with target drift dashboard always seems to crash. Till now I've figured out that it works only when my data is very less:
This works:
data_drift_report = Dashboard(tabs=[NumTargetDriftTab()])
data_drift_report.calculate(reference.head(10), week1.head(10), column_mapping=ColumnMapping(target="target",  datetime="datetime"))
data_drift_report.show()

This does not:
data_drift_report = Dashboard(tabs=[NumTargetDriftTab()])
data_drift_report.calculate(reference.head(100), week1.head(100), column_mapping=ColumnMapping(target="target",  datetime="datetime"))
data_drift_report.show()

Any insights will be helpful.

Hello,
Looks like it is a problem with specific data or depends on data size.
Can you describe what happened when the code does not work - and errors, warnings, or maybe just waiting without any result?
And maybe you can add some details about the dataset with week1.info()/reference.info() and week1.dtypes/reference.dtypes?

week.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2434 entries, 0 to 2433
Data columns (total 5 columns):
 #   Column                  Non-Null Count  Dtype
---  ------                  --------------  -----
 0   datetime         2434 non-null   datetime64[ns]
 1   target              2434 non-null   int64
 2   feature1          2434 non-null   float64
 3   feature2         2434 non-null   int64
 4   feature3         2434 non-null   float64
dtypes: datetime64[ns](1), float64(2), int64(2)
memory usage: 95.2 KB

reference.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 44466 entries, 0 to 44465
Data columns (total 5 columns):
 #   Column                  Non-Null Count  Dtype
---  ------                  --------------  -----
 0   datetime         44466 non-null  datetime64[ns]
 1   target              44466 non-null  int64
 2   feature1          44466 non-null  float64
 3   feature2         44466 non-null  int64
 4   feature3         44466 non-null  float64
dtypes: datetime64[ns](1), float64(2), int64(2)
memory usage: 1.7 MB

If I save as HTML I get this when I try to open it:

If i try to get the dashboard inline, my kernel crashes.

It looks like the code wants to get a lot of memory and crashes. We will try to investigate how to reduce the memory usage in Evidently.