The node SDK hfc command should support the "peer network/chaincode" commands

Description
The "hfc" command is installed by running
npm install -g hfc
The hfc command currently only supports a
hfc test <test-input.json>
command, which uses the node SDK to drive load against the fabric.
This command should also support the "peer network" and "peer chaincode" commands.
This means fabric/sdk/node/bin/main.js must be enhanced to support something similar to:
hfc login
hfc list
hfc deploy ...
hfc invoke
hfc query
Options:
-c                        (check for '$HOME/hfc.json' if not specified)
-ms            (default value is "grpc://localhost:50051")
-peer                             (default value is "grpc://localhost:30303", and this option can be used multiple times for failover)
The content of the  might be as follows:
{
"memberServicesUrl": "grpc://localhost:50051",
"peers": [ "grpc://localhost:30303" ],
}
Describe How to Reproduce
Not an issue

@andresgaragiola @srderson This was a feature for the node SDK but it looks like Andres added some behave tests.  Was this just associated with the wrong issue number?

Looks like that was the case. @andresgaragiola Could you close the correct issue?

@smithbk @srderson I have fixed the typo in the PR and the correct issue was closed. Thanks


add APIkey auth capability using headers

#10
Thanks to review this little addition :angel:

:up: :)

@BarthV Thank you for the PR.  I apologize for the delay although we're just in the process of establishing our Contributor License Agreement so it might take just a bit longer.  Once it's ready, i'll notify you so that you can sign the CLA before I accept/merge your PR.

🆙

up!

🆙


where is the python api : cv2.dnn.blobFromImage

Read README carefully first
Star :star: this project if you want to ask a question, no star, no answer
already star 😁
Question
I use yolov8 tflite model on flutter , the python code work well, so I just try to find opencv python api cv2.dnn.blobFromImage in this package

I just found it out 😓

dart doesn't support namespace or 'export as', so nearly everything w is under 'cv'.

I'm sorry, but I still have problems with it

I'am just from python script to make code logic .
the mat is read 250px * 250px with 3 channel image, call the blobFromImage function
return the nmat  with rows=-1,cols=-1, and channels = 1 ,
it should be 250px * 250px with 3 channel (rows=320,cols=320,channels=3) ?

Just go ahead, but I just find the Mat.size is not correct, will be fixed in the next release.

https://github.com/opencv/opencv/issues/13698

ok,got it,
so the return blob is tensor shape (1,3,320,320) , but found 1228800 element ,is not correct too ?

@einsitang Mat.data is Uint8List, which is a wrapper for https://docs.opencv.org/4.x/d3/d63/classcv_1_1Mat.html#a4d33bed1c850265370d2af0ff02e1564 , but the actuall data type of your mat is float32, so the length of data is not exactly match the number of pixels, if you convert data to float32 list (via data.buffer.asFloat32List()), I believe it will be 1*3*320*320.

thans
I try later

an unrelated question how do you run you yolo model, using tflite or pytorch and what do you think about the performance?

an unrelated question how do you run you yolo model, using tflite or pytorch and what do you think about the performance?

yolo model can export tflite file for tensorflow and that is official tool ,
my project with tflite_flutter and opencv_dart to rewrite python code to flutter .
I'am have not test performance yet , but yolo-flutter-app(yolo flutter example) detect work performance look like ok.
between tflite_flutter with opencv_dart and yolo-flutter-app , both use NativeLibrary  to work , the main difference is  preprocess and postprocess function working runtime. would be lower than yolo-flutter-app.
however, like 1~2 seconds to finish detect work (include preprocess and postprocess) is my expected
when logic done and basic work fine , I'll do more test tell you guys result

sorry, I have another problem that I need your help.
I've found that python and dart execute the same opencv function and get different results, and I'm not sure if it's my use or if the difference actually occurs across platforms
python script dump json files:
import json,codecs
url = 'https://i.mij.rip/2024/07/20/c43a166c11414cad2c019a3d814512dd.jpeg'
res = urllib.request.urlopen(url)
img = np.asarray(bytearray(res.read()),dtype="uint8")
img = cv2.imdecode(img,cv2.IMREAD_COLOR)

# #1 here is the origin image read from opencv (py_test_image.json)
print("json dump origin img")
json.dump(original_image.tolist(), codecs.open("py_test_image.json", 'w', encoding='utf-8'),
              separators=(',', ':'))

image = cv2.copyMakeBorder(original_image,0,length-height,0,length-width,cv2.BORDER_CONSTANT, value=(114, 114, 114))
blob = cv2.dnn.blobFromImage(image, scalefactor=1 / 255, size=(640, 640), swapRB=True)

# #2 here is the data with blobFromImage (py_test_blob.json)
print("json dummp (blob)")
json.dump(blob.tolist(), codecs.open("py_test_blob.json", 'w', encoding='utf-8'),
              separators=(',', ':'))

blob = blob[...,::-1].transpose((0,2,3,1)) # BCHW to BHWC

# #3 here is data with transpose((0,2,3,1)) BCHW to BHWC (py_test_blob_0231.json)
print("json dummp (blob_0231)")
json.dump(blob.tolist(), codecs.open("py_test_blob_0231.json", 'w', encoding='utf-8'),
          separators=(',', ':'))

flutter dump json files:
final String dir = (await getExternalStorageDirectory())!.path;

cv.Mat originImgMat = cv.imread(await _copy(imgPath)); // BGR
var oWidth = originImgMat.width;
var oHeight = originImgMat.height;
var length = oWidth > oHeight ? oWidth : oHeight;

// #1 here is the origin image mat data json (dart_origin_img_mat.json)
final matData = jsonEncode(ListShape(originImgMat.data).reshape([2339,1654,3]));
final writedFile = await File(dir + "/dart_origin_img_mat.json");
await writedFile.writeAsString(matData);

cv.Mat scaleImgMat = cv.copyMakeBorder(originImgMat, 0, length-oHeight, 0, length-oWidth, cv.BORDER_CONSTANT,
        value: cv.Scalar(114, 114, 114, 0));

var blobMat = cv.blobFromImage(scaleImgMat,
        scalefactor: 1 / 255,
        size: (IMG_WIDTH, IMG_HEIGHT),
        swapRB: true,
        ddepth: cv.MatType.CV_32F);

// #2 here is the data with blobFromImage (dart_blob_mat.json)
final blobMatData = jsonEncode(ListShape(blobMat.data.buffer.asFloat32List()).reshape(inputTensor.shape));
final blobMatDataFile = await File(dir + "/dart_blob_mat.json");
await blobMatDataFile.writeAsString(blobMatData);
log.d("dart_blob_mat.json write done");

the "py_test_image.json"(python.#1) and "dart_origin_img_mat.json"(dart.#1) files is same (with same md5)

but the "py_test_blob.json"(python.#2) and "dart_blob_mat.json"(dart.#2) files data look like different
py_test_blob.json:

dart_blob_mat.json:

Apart from the shape, the data order seems to be different

here is the files :
py_test_blob.json
dart_blob_mat.json
and here is the image file:

and here is the image on network:
https://i.mij.rip/2024/07/20/c43a166c11414cad2c019a3d814512dd.jpeg

latest discovery
the cv.blobFromImage params : swapRB is not working?
I change the swapRB = false , but the dump json file is no difference

pubspec.yaml lib version
opencv_dart: ^1.1.0+1

@einsitang hello.
I tested it with your code and every thing is ok, the saved json from your python code is wrong.
import 'package:opencv_dart/opencv_dart.dart' as cv;

void main(List<String> args) {
  final originImgMat = cv.imread("c43a166c11414cad2c019a3d814512dd.jpeg", flags: cv.IMREAD_COLOR);
  var oWidth = originImgMat.width;
  var oHeight = originImgMat.height;
  var length = oWidth > oHeight ? oWidth : oHeight;

  cv.Mat scaleImgMat = cv.copyMakeBorder(
    originImgMat,
    0,
    length - oHeight,
    0,
    length - oWidth,
    cv.BORDER_CONSTANT,
    value: cv.Scalar(114, 114, 114, 0),
  );

  final blob = cv.blobFromImage(
    scaleImgMat,
    scalefactor: 1 / 255,
    size: (640, 640),
    swapRB: true,
  );
  print(blob.data.buffer.asFloat32List());
}

dart run test.dart > dart_blob.json

 detailed images

I noticed
img = cv2.imdecode(img,cv2.IMREAD_COLOR)  // img
...
cv2.copyMakeBorder(original_image,0,length-height,0,length-width,cv2.BORDER_CONSTANT, value=(114, 114, 114)) // original_image

so the images you are processing are different, please check whether it's correct.

And ignore that why I use print and stream redirection > to create dart_blob.json, I am lazy 😆

no....
I cheked before,
the "py_test_image.json"(python.#1) and "dart_origin_img_mat.json"(dart.#1) files with same md5 value,
that mean originImgMat (dart) and original_image(python) is 100% the same data
img = cv2.imdecode(img,cv2.IMREAD_COLOR)  // img
...
cv2.copyMakeBorder(original_image,0,length-height,0,length-width,cv2.BORDER_CONSTANT, value=(114, 114, 114)) // original_image

img and original_image is same instance , just because I made a mistake copying the code,but only the show code , not the work code
here is the code with kaggle notebook , and work fine , this is why I'am follow python script to make flutter code logic

Allright, then the only suggestion I can give you is to check the value of blob in python.

As you can see, your image starts with 255, and the padding values of left and top are all zeros, then you scaled it with scalefactor=1 / 255, so blob should starts with 1.0, however,

this shows it's started with the values <1.0, which is definitely incorrect.

I just synchronization the python and flutter logic code , and ..... I'am make a mistake, the problem  precisely is not about copyMakeBorder and blobFromImage , the previously reported problem was due to an error in the debugging process.
The problems I'm struggling with really don't happen here.
we can skip the topic about copyMakeBorder and blobFromImage ...

the problem may on next step :

I don't know how do this on my flutter code ,  that could be the real problem of tensorflow with not truely input

this shows it's started with the values <1.0, which is definitely incorrect.

yes,I think my dart code has changed and output json logs many times before, but I took a cache and didn't notice it, so it's not actually the same logic when compared to python
var dx = (length - oWidth) ~/ 2;
var dy = (length - oHeight) ~/ 2;
cv.Mat scaleImgMat = cv.copyMakeBorder(originImgMat, dy,dy,dx,dx, cv.BORDER_CONSTANT,
    value: cv.Scalar(114, 114, 114, 0));
// dump json

cv.Mat scaleImgMat = cv.copyMakeBorder(originImgMat,0,length-height,0,length-width, cv.BORDER_CONSTANT,
    value: cv.Scalar(114, 114, 114, 0));
// dump json

I don't know how do this on my flutter code , that could be the real problem of tensorflow with not truely input

Yes, it could be the problem.
It's frustrating that dart has no usable multi-dimensional array libraries like numpy for dart, and this is out of the scope of this package. I am not sure whether Mat.reshape will do the right things, opencv_dart v1.2.0 added the wrapper for https://docs.opencv.org/4.x/d3/d63/classcv_1_1Mat.html#ab2e41a510891e548f744832cf9b8ab89 , you can upgrade to v1.2.0 and pass a list of dimensions to Mat.reshape to see whether it works, again, I am not sure.
And note that there are many breaking changes in v1.2.0, you may need to change your code.

thanks very much.
I found another lib tensor_dart , but just only can handle 2D matrix , that is why I skip this step to checked before step.

I don't know how do this on my flutter code , that could be the real problem of tensorflow with not truely input

Yes, it could be the problem.
It's frustrating that dart has no usable multi-dimensional array libraries like numpy, and this is out of the scope of this package. I am not sure whether Mat.reshape will do the right things, opencv_dart v1.2.0 added the wrapper for https://docs.opencv.org/4.x/d3/d63/classcv_1_1Mat.html#ab2e41a510891e548f744832cf9b8ab89 , you can upgrade to v1.2.0 and pass a list of dimensions to Mat.reshape to see whether it works, again, I am not sure.
And note that there are many breaking changes in v1.2.0, you may need to change your code.

reshape and transpose have different effects
anyway, thanks again for your help

Yes, but

You can reshape (H, W, C) to (HW, C) and then perform a transpose. Transpose would give (C, HW) which can then be reshaped to (C, H, W).`

https://answers.opencv.org/question/226929/how-could-i-change-memory-layout-from-hwc-to-chw/

yes , I try later , may have good news tell your guys

Rearrange the channels to match BHWC format

I don't think so , because I'am not good math for linear algebra or tensor something

@rainyl @abdelaziz-mahdy Thanks to your help, the logic of preprocess has been debugged and passed 🍾

here is the code BCHW to BHWC
_chw2hwc(cv.Mat mat){
    final size = mat.size;
    // final b = size[0];   // B , dont' need it
    final c = size[1];    // C
    final h = size[2];   // H
    final w = size[3];  // W
    // C,HW
    cv.Mat c_hw = mat.reshapeTo(0,[c,h*w]);
    // HW,C and reshape in fn outside
    return c_hw.transpose();
  }

// ListShape with lib : tensor_dart
var input = ListShape(_chw2hwc(blobMat).data.buffer.asFloat32List())
        .reshape(inputTensor.shape);

an unrelated question how do you run you yolo model, using tflite or pytorch and what do you think about the performance?

My iphone version is too high to use xcode debugging, so I found a MI PAD 4 for testing. In general, the pre-processing time is about 350~400ms, the inference time is about 500ms, and the post-processing time is the longest, generally higher than 2s, but the hardware performance of this device is average. So I'm happy with that performance.
the latest test run on mi pad 4:
preprocessTimes: 366.783 ms, inferenceTimes: 475.897 ms, postprocessTimes: 2113.832 ms
mi pad 4
CPU : Octa-core (4x2.2 GHz Kryo 260 Gold & 4x1.8 GHz Kryo 260 Silver)
RAM : 4G
kernel : Android 8.1.0
OS : MIUI <IP_ADDRESS>

I think I can close this issue , thank you very much

an unrelated question how do you run you yolo model, using tflite or pytorch and what do you think about the performance?

My iphone version is too high to use xcode debugging, so I found a MI PAD 4 for testing. In general, the pre-processing time is about 350~400ms, the inference time is about 500ms, and the post-processing time is the longest, generally higher than 2s, but the hardware performance of this device is average. So I'm happy with that performance.
the latest test run on mi pad 4 (no quantization): preprocessTimes: 366.783 ms, inferenceTimes: 475.897 ms, postprocessTimes: 2113.832 ms
mi pad 4 CPU : Octa-core (4x2.2 GHz Kryo 260 Gold & 4x1.8 GHz Kryo 260 Silver) RAM : 4G kernel : Android 8.1.0 OS : MIUI <IP_ADDRESS>

interested to know why post process takes too much time, did you do it in dart? also try in release or profile mode, since its much faster for dart code due to compilation

except inference part , all the code is in dart .
I review code again, is print log use wrong variable ... （another stupid mistake😂）
the real result is :
preprocessTimes:418.545 ms, postprocessTimes: 565.299 ms, inferenceTimes: 2020.973 ms
inferenceTimes takes most time , that well make sense

except inference part , all the code is in dart .
I review code again, is print log use wrong variable ... （another stupid mistake😂）
the real result is :
preprocessTimes:418.545 ms, postprocessTimes: 565.299 ms, inferenceTimes: 2020.973 ms
inferenceTimes takes most time , that well make sense

What do you inference with? Opencv? Dnn?
If you got open source I may help you with optimizing it
Also are these results on profile or release mode?

except inference part , all the code is in dart .
I review code again, is print log use wrong variable ... （another stupid mistake😂）
the real result is :
preprocessTimes:418.545 ms, postprocessTimes: 565.299 ms, inferenceTimes: 2020.973 ms
inferenceTimes takes most time , that well make sense

What do you inference with? Opencv? Dnn?

inference with tensorflow lite

If you got open source I may help you with optimizing it

is opensource project , just for fun.
if you want help me optimizing it , that is very good news for me
but this feature is not code done , so I'am not push remote yet. you can star my repository , when code update ,then you can review code and help.
but the repository from a long time ago ,  I'am chinese , english not good enough, comments is chinese and translate into english , may not good to read.
https://github.com/einsitang/sudoku-flutter

Also are these results on profile or release mode?

these results on debug mode , no try on release or profile yet.

https://www.kaggle.com/code/jianbintangelement/sudoku-digits-detect-predict-with-yolov8
this is I'am try to do on my project
use camera on real time scan picture with sudoku puzzle , recognize them and solve them

I have just finished the basic process of Sudoku detect. Thank you very much for your help. It's just that the current device I'm testing is not running very fast, a full model predicts that it will take 3s, I have two, so it will take at least 6s to complete a job. I also tried to quantify the optimization speed of the model, and the result was that the speed was doubled, but the accuracy was greatly reduced and basically unavailable, so I did not adopt the quantization scheme in the end. The problem that won't be quantified most likely is that my model is poorly trained

the latest code is committed and push , If you are interested, I would appreciate your input

I took a look at the impl of tflite_flutter, and I think their implementation is not efficient enough, https://github.com/tensorflow/flutter-tflite/blob/83a20133fc4faa0f75ae0ab3c23bbbcfac248453/lib/src/tensor.dart#L160-L176
if you pass a dart List to Interpreter.run, it will be copied 4 times at least.

2x: https://github.com/tensorflow/flutter-tflite/blob/83a20133fc4faa0f75ae0ab3c23bbbcfac248453/lib/src/util/byte_conversion_utils.dart#L36-L52
1x: https://github.com/tensorflow/flutter-tflite/blob/83a20133fc4faa0f75ae0ab3c23bbbcfac248453/lib/src/tensor.dart#L166
1x: https://github.com/tensorflow/flutter-tflite/blob/83a20133fc4faa0f75ae0ab3c23bbbcfac248453/lib/src/tensor.dart#L168

I didn't test the cost but I think it's unacceptable.
I suggest you to pass the Mat.data to interpreter.run() directly, this will avoid the first 2x copies, step further, I suggest you to open an issue in tflite_flutter to suggest them providing another api such as runWithTensor(Tensor tensor) to use the tensor created by users directly, then you can:
final Tensor t = interpreter.getInputTensor(0);
t.data = Mat.data; // only 1 copy, actually this will use setRange(), which will be faster than loops.
final result = interpreter.runWithTensor(t);

I took a look at the impl of tflite_flutter, and I think their implementation is not efficient enough, https://github.com/tensorflow/flutter-tflite/blob/83a20133fc4faa0f75ae0ab3c23bbbcfac248453/lib/src/tensor.dart#L160-L176
if you pass a dart List to Interpreter.run, it will be copied 4 times at least.

2x: https://github.com/tensorflow/flutter-tflite/blob/83a20133fc4faa0f75ae0ab3c23bbbcfac248453/lib/src/util/byte_conversion_utils.dart#L36-L52
1x: https://github.com/tensorflow/flutter-tflite/blob/83a20133fc4faa0f75ae0ab3c23bbbcfac248453/lib/src/tensor.dart#L166
1x: https://github.com/tensorflow/flutter-tflite/blob/83a20133fc4faa0f75ae0ab3c23bbbcfac248453/lib/src/tensor.dart#L168

I didn't test the cost but I think it's unacceptable.
I suggest you to pass the Mat.data to interpreter.run() directly, this will avoid the first 2x copies, step further, I suggest you to open an issue in tflite_flutter to suggest them providing another api such as runWithTensor(Tensor tensor) to use the tensor created by users directly, then you can:
final Tensor t = interpreter.getInputTensor(0);
t.data = Mat.data; // only 1 copy, actually this will use setRange(), which will be faster than loops.
final result = interpreter.runWithTensor(t);

@rainyl your code review is very careful, I only understand how these libraries are used, I do not know the specific implementation logic.
I followed the code you provided, and found that it is exactly what you said.
what I'm not sure about, though, is that doing this will bypass tflite's type conversion, which I don't know enough about to know if cv.Mat and tflite.Tensor are consistent at the byte level, which could be a very unsafe operation
however,I can try your way to test.
the code like :
final Tensor inputT = interpreter.getInputTensor(0);
inputT.data = Mat.data;  // not sure it work
interpreter.invoke();
final Tensor outputT = interpreter.getOutputTensor(0);
......

will try later and tell you result

what I'm not sure about, though, is that doing this will bypass tflite's type conversion, which I don't know enough about to know if cv.Mat and tflite.Tensor are consistent at the byte level, which could be a very unsafe operation

Sure, in this way you need to be careful to ensure the types and size of Mat are match to Tensor, but Mat.data are just raw pixel values, I am not sure whether the underlying data of Tensor are raw values, if so the this should work. https://github.com/tensorflow/flutter-tflite/blob/83a20133fc4faa0f75ae0ab3c23bbbcfac248453/lib/src/tensor.dart#L69-L79

great job ,
I originally thought that this part of the optimization would only provide a small performance improvement, but the modified test found that the speed was faster than I expected
this is recent performance logs,very high quality:
preprocessTimes:26.638 ms, postprocessTimes: 39.703 ms, inferenceTimes: 814.608 ms
preprocessTimes:20.872 ms, postprocessTimes: 77.102 ms, inferenceTimes: 816.108 ms

Sounds good.

我看了一下你的资料，你fork一些repo是全中文的，且姓liu，你应该是个中国人….我想我可以用国语向你致谢

哈哈哈哈哈是的，没事儿🤣

哈哈哈哈哈是的，没事儿🤣

方便问一下，你现在还是学生吗，就读武汉大学？


Iterators: Improve analysis

This PR is based on and therefore includes #539. It extends and cleans up the iterator analysis in preparation for #553. In that PR, we introduce an iterator op (a scan) than has a non-iterator operand -- something that doesn't occur with the ops we currently have. This PR makes the analysis correct for ops with non-iterator operands and cleans up a few other things.

Skimmed through the code in the WIP progress that includes this and it looked good modulo minor comments.
I can't see differences offhand in this PR so this LGTM.
If there are material differences, please ask me to review again.


Upgrade Terraform version for infra-security module.

Someone had run terraform apply with a much newer version of Terraform, which upgraded the state file in S3, which effectively forces us to upgrade:
Error refreshing state: state snapshot was created by Terraform v1.1.5,
which is newer than current v0.13.6; upgrade to Terraform v1.1.5 or
greater to work with this state

So I guess we'll upgrade then. This takes us up to the current version of Terraform and allows a newer (compatible) version of the AWS provider. Seemed safer to update the provider rather than continue to pin the same ancient minor release.
Also, since we're running this from Jenkins and potentially people's workstations, add the provider version lock file (with signatures for amd64 and arm64 Mac binaries and amd64 Linux binaries) to reduce the chance of Jenkins encountering version issues. Generated with terraform providers lock -platform=darwin_amd64 -platform darwin_arm64 -platform=linux_amd64
Tested: plans locally as expected

I guess at some point we should prioritise a set of govuk-aws terraform deployments which we should upgrade terraform in. Even with replatforming, some of them are likely to live for a long time.

Yeah, I'd put good money on at least a handful of these modules still being here in a year's time 🙈 The upgrade was surprisingly painless (so far), mostly just our dodgy lint checks getting in the way 🦬🦬🦬


Removed TODOs and created issues for them

Searched for "todo, to do, to be do, te do", case insensitive.
Created issues #423 to #457
Closes #389

31 issues fixes :) Nice.


add ports/taglib

TagLib Audio Meta-Data Library

@jbhelm,
Thanks for having already signed the Contribution License Agreement. Your agreement has not been validated yet. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot

Thanks for the PR! I made a few tweaks while merging:

Instead of using a checked-in license file, we can deploy both COPYING files from the sources.
Added vcpkg_copy_pdbs()
Always apply source patches -- this helps to reduce variability between builds.
The conversion std::wstring(s.begin(), s.end()) assumes the inner string is Latin-1 encoded. This is actually different behavior from the xyzA() Win32 functions, which use the current codepage. I've replaced the conversion with a call to MultiByteToWideChar which should handle alternate codepages better.


docs: expand run and gc commands

Will rebase before merging

Updated

lgtm

Whoops, I was gonna squash those, oh well!


Modify ci trigger in ci.yml

Codecov Report

Merging #7 (f9ad8f3) into main (2aaf527) will decrease coverage by 3.34%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main       #7      +/-   ##
==========================================
- Coverage   87.21%   83.87%   -3.35%
==========================================
  Files          10        9       -1
  Lines         532      521      -11
==========================================
- Hits          464      437      -27
- Misses         68       84      +16

Impacted Files
Coverage Δ

src/einsum.jl
38.77% <0.00%> (-37.50%)
:arrow_down:

src/Tensorial.jl

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2aaf527...f9ad8f3. Read the comment docs.

Codecov Report

Merging #7 (f9ad8f3) into main (2aaf527) will decrease coverage by 3.34%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main       #7      +/-   ##
==========================================
- Coverage   87.21%   83.87%   -3.35%
==========================================
  Files          10        9       -1
  Lines         532      521      -11
==========================================
- Hits          464      437      -27
- Misses         68       84      +16

Impacted Files
Coverage Δ

src/einsum.jl
38.77% <0.00%> (-37.50%)
:arrow_down:

src/Tensorial.jl

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2aaf527...f9ad8f3. Read the comment docs.


Pin kubernetes version to 29.x.x - 2114

[Auto generated]
Number: [#2114]

openshift-python-wrapper[pull_request][78b1-11ef-9833-39d67c34f210][PR 2114]: Closing issue for PR: Pin kubernetes version to 29.x.x.
PR was closed.


tests/drivers/counter/counter_basic_api  failed on mimxrt1050_evk board.

To Reproduce
Steps to reproduce the behavior:

west build -b mimxrt1050_evk   tests/drivers/counter/counter_basic_api
west flash
See error

Screenshots or console output
*** Booting Zephyr OS build zephyr-v2.1.0-246-g9602c651c220 ***
Running test suite test_counter
starting test - test_set_top_value_with_alarm
E: Wrap can only be set to 0xffffffff
Skipped for GPT1
PASS - test_set_top_value_with_alarm
starting test - test_single_shot_alarm_notop
Testing GPT1
Assertion failed at ZEPHYR_BASE/tests/drivers/counter/counter_basic_api/src/test_counter.c:216: alarm_handler: (now <= counter is false)
GPT1: Alarm (500084) too early now: 500106 (counting down).
FAIL - test_single_shot_alarm_notop
.....
Test suite test_counter failed.
PROJECT EXECUTION FAILED
Environment (please complete the following information):

OS: fedora28
Toolchain: zephyr-sdk-0.10.3
Commit ID; 9602c651c2

fix: #21477 21477


Bump react-native-network-client to 1.3.4

Summary
Bump react-native-network-client to 1.3.4
Ticket Link
None
Release Note
NONE

Cherry pick is scheduled.


How to get device motion status in traccar 3.7?

Hi,
I am currently using 3.5 version and i have made some changes to source code locally. Now i want to upgrade to version 3.7 and there is no motion status properties/methods anymore in Device model. Where can i get that in code?
Any help would be highly appreciated.
-Thanks

Motion is now reported as events.

Which file handles events? BaseEventHandler.java or ManEventHandler?

Figured it out. Thanks @tananaev .

How do I get the status of the device in version 3.7?

Same way as before. It's available in the API.


test: fix #957. change download response url

It looks like planet will eventually change the previous download link... we don't have any implementation of this endpoint, we can just change the test data regression.

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 95.90%. Comparing base (fa02412) to head (791ecf4).
Report is 2 commits behind head on main.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #959      +/-   ##
==========================================
- Coverage   96.02%   95.90%   -0.12%
==========================================
  Files          40       40
  Lines        4074     4079       +5
==========================================
  Hits         3912     3912
- Misses        162      167       +5

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests
JS Bundle Analysis - Avoid shipping oversized bundles


Feature 91666: User List Component

This implements the Users page. At the moment, I have not implemented the link to a profile page (it doesn't exist yet) or the ability to filter by "Copilot" or "Guest".

To test the PR, please run:
npm start

Enhancement: Alignment of "Allowance remaining" heading text
Bug: Profile images are not showing
Enhancement: Change to Sasha designed default profile icon (if non specified)

Can you please test @ediWeeks

@BenGWeeks please take a look
Enhancement 91711: Fix column title alignment

Will merge and keep enhancement in backlog.


fix return error instead of panic when slot and expr do not

match
Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall


Container ver 2 to ver 3

Hi,
we are having major difficulties upgrading from version 2 to version 3, it is really a different container with totally different API.
One part that is really unsolvable for us is the fact that the new League\Container\Container::share or League\Container\Container::add always adds a new definition of the service that one tries to add to the container and in ver2 the container would check if the definition already exists and if it does it would replace it with the new definition.
Can you bring back this functionality @philsturgeon ?
We would not mind if it is "locked" behind a param of the add/share method.

Closing due to lack of activity


Remove obsolete django-sslify package and replace with SECURE_SSL_REDIRECT setting

The django-sslify package was created for django versions <1.8. Django 1.8 or later doesn't need the package to force SSL.
Instead, we should set the SECURE_SSL_REDIRECT appropriately.

This was fixed in https://github.com/istrategylabs/mo-django/pull/53


Add angular list builder and table column customization modal, refactor entry points

This should be merged AFTER #2922
Needs to be updated to use the new moveColumn function introduced in #3088

Changes moved to #3265


Fix session check to be compatible with mojolicious/mojo@3594662

Fixes #4.

Thanks, merged and released.


Update UsingOptions sample app

Addresses #4208
:triangular_flag_on_post: Takes sample to 2.0

@guardrex,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by .NET Foundation. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot

Fixes #437


dotnet run command givs error.

Description

Configuration

Regression?

Other information

What error does it give? Can you provide a stack trace or a screenshot?


Kafka client collaboration

Hi there,
I see you are using rdkafka for your Kafka interaction. We (= InfluxDB IOx) implemented our own pure-Rust async client called rskafka which you might find useful. There is also an intro blog post: https://www.influxdata.com/blog/building-simple-pure-rust-async-apache-kafka-client/
I think you might be missing a feature or two, but nothing fundamental (to my knowledge).
Let me know if you have questions, or want to chat.
Cheers,
Marco
PS: Also feel free to close this ticket.

Thanks for sharing @crepererum ! I think it would be a cool idea to try rskafka to reduce C dependencies :)

@crepererum I have been looking over rskafka which seems like it might not be suitable for our use until a time when it supports consumer groups and offset tracking.
Based on the readme in the repository I cannot tell if such contributions would be welcome or not.

Based on the readme in the repository I cannot tell if such contributions would be welcome or not.

They are, it's just that we (InfluxData) won't implement them. We can help w/ reviews though.


Access a file uploaded in another page

I am using a simple polymer application with few pages in iron-pages. i am uploading a file in one page and then, i want to access this uploaded file in another page.
I tried several things but nothing seems to work, here is the sample code

page on which file is uploaded

`

        <my-input  file={{file}} id="sampleFileInput" btn-style="primary" max-files="1" accept=".xls, .xlsx" on-drop="fileUploadChangeListener"
                             label="[[localize('open_invoices_file')]]" help-text="[[localize('open_invoices_file_help')]]" no-auto required>
        </my-input>

`

the page on which file is accessed

`

`
The same code works in polymer1.0 with this
document.getElementById("sampleFileInput")
can somebody help what wrong am i doing in accessing this file in other page, and how can i handle this scenario in Polymer 2.0?

In Polymer 1 you were probably using shadydom. Polymer 2 uses shadow dom. Queryselectors on the document such as document.getElementById will therefore no longer work on elements inside shadow DOM. Instead, do something along the lines of
document.querySelector('file-upload-page').shadowRoot.getElementById('sampleFileInput')

Thanks Tim, It worked


Erro in Version 65.03298.36 64bits

typeerror: date.tolSOString is not a function
pagespeed.js:1076:36
at Array.forEach (anonymous>)
at pagespeed;RespirceAccumulator.onHAR_
pagespeed.js:1074:17

Thanks for reporting this issue. Sorry for the late response. It is fixed now in version <IP_ADDRESS>. I am rolling out the fix.


Update URL for Jasmine to reflect repository move on github (pivotal.…

…github.io/jasmin --> jasmine.github.io)
Update URL for NodeJS to use https.

Thanks! sorry for the delay merging!


Unable to pass tests - Missing dependencies?

I encountered an issue while running the tests. Unfortunately, the tests are not passing for me. I wanted to reach out to inquire if there are any missing dependencies or additional setup steps required to run the tests successfully.
File "/Users/eason/Desktop/github/smart_learn/user_service/serializers.py", line 74, in validate
    data = super().validate(attrs)
  File "/Users/eason/Desktop/github/smart_learn/venv/lib/python3.9/site-packages/rest_framework_simplejwt/serializers.py", line 72, in validate
    data["refresh"] = str(refresh)
  File "/Users/eason/Desktop/github/smart_learn/venv/lib/python3.9/site-packages/rest_framework_simplejwt/tokens.py", line 81, in __str__
    return self.get_token_backend().encode(self.payload)
  File "/Users/eason/Desktop/github/smart_learn/venv/lib/python3.9/site-packages/rest_framework_simplejwt/backends.py", line 113, in encode
    token = jwt.encode(
  File "/Users/eason/Desktop/github/smart_learn/venv/lib/python3.9/site-packages/jwt/api_jwt.py", line 67, in encode
    return api_jws.encode(json_payload, key, algorithm, headers, json_encoder)
  File "/Users/eason/Desktop/github/smart_learn/venv/lib/python3.9/site-packages/jwt/api_jws.py", line 153, in encode
    key = alg_obj.prepare_key(key)
  File "/Users/eason/Desktop/github/smart_learn/venv/lib/python3.9/site-packages/jwt/algorithms.py", line 186, in prepare_key
    key = force_bytes(key)
  File "/Users/eason/Desktop/github/smart_learn/venv/lib/python3.9/site-packages/jwt/utils.py", line 22, in force_bytes
    raise TypeError("Expected a string value")
TypeError: Expected a string value

Any assistance or suggestions would be greatly appreciated. Thank you for your time and support!

I have experienced this error but it is from a library. Give me a minute and I will walk you through it.

I found out where I talked about it.
https://github.com/chukaibejih/smart_learn/issues/13#issuecomment-1487430056

Nice! it's ok now, thanks!


ImvoxelNet相关

请问ImvoxelNet训练车路协同数据集是直接使用原始模型网络还是有些更改，请问有没有相关介绍和说明？

We have added the ImvoxelNet models for VIC3D late fusion at https://github.com/AIR-THU/DAIR-V2X/tree/main/configs/vic3d/late-fusion-image/imvoxelnet. Pease note that the results will be somewhat different from the paper due to the updated code. Everything is subject to the actual implementation results.


SDFSOF-50 - Added additional validation to not start anchor platform if custody integration is enabled and account_creation or claimable_balances are enabled. Added additional tests

PR Checklist
PR Structure

[ ] This PR has reasonably narrow scope (if not, break it down into smaller PRs).
[ ] This PR avoids mixing refactoring changes with feature changes (split into two PRs
otherwise).
[ ] This PR's title starts with name of package that is most changed in the PR, ex.
paymentservice.stellar, or all or doc if the changes are broad or impact many
packages.

Thoroughness

[ ] This PR adds tests for the most critical parts of the new functionality or fixes.

What
Added additional validation to not start anchor platform if custody integration is enabled and account_creation or claimable_balances are enabled. Added additional tests
Why
As for now, integration with custody servers doesn't support automatic account creation. That's why, there was added an additional validation to not start anchor platform if custody integration is enabled and account_creation or claimable_balances are enabled.

Reference Server Preview is available here:https://anchor-ref-pr914.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr914.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr914.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr914.previews.kube001.services.stellar-ops.com/


Disappearing Stake

Yesterday I staked 0.5 ETH to L1 liquidity pool. It still appears in the "Your Stake" column, however, "Available Balance" shows only 0.19 ETH and I cannot unstake more than 0.187 ETH. Where did my 0.5 ETH go?

Back to normal this morning: my stake is again less than the available balance.


Configure telescope to show file names

When paths are long, the telescope explorer is a bit difficult to navigate

@marcinjahn, good point, thank you for pointing this out.
I'm wondering what would be the best approach. I can show only the file names but I think paths can be helpful too.
I'll try a few things, but will love to hear more ideas.

@marcinjahn, I added support for Telescope's path_display option. Basically, you can use any option that Telescope supports.
The default in dotnet.nvim now is filename_first which looks like that:

Awesome, thanks


Scaling socket.io on multiple servers and processes

Hey all,
I'm using socket.io on multiple servers (several Azure machines) and processes (using cluster).
I found the regular answers that say I should use sticky sessions to make sure connections to an instance of socket.io will continue going to the same process between calls but I don't understand why sticky sessions are a good solution and how to solve this problem:

The was to implement sticky sessions is to have one "server" that is used as a kind of a proxy. This slows down the server and can create a load issue on some of the processes. It might take out the edge of having multiple processes.
How does sticky sessions solve the issue of having multiple different machines?
On earlier versions of socket.io I could set up a redis store that took care of that. Where did it go (and why?)?

Am I right in all my points here or am I completely off track?
Not sure what i'm missing.
Thanks in advance :)
We LOVE socket.io :heart:

@nkzawa can you provide a complete demo of socket.io with cluster ? i really need it, thanks very much ! because this #1944

FWIW @Hussion We moved away from using Cluster because you would need RedisAdapeter for multi-instance scaling anyways. Instead, on each instance we create 34 Node processes listening on incrementing ports, then use HAProxy to balance between them.


Bump standard and rubocop

https://github.com/rubyforgood/human-essentials/pull/3547 is failing lint - this one is not :)

Looks good to me. Also I learned about rubocop-performance which is nifty.


The new DQM GUI file management

Impact of the new feature
This request affects all systems that are responsible for harvested DQM data being uploaded to the DQM GUIs. This includes T0 processed DQM data and RelVal/reprocessing DQM data.
Is your feature request related to a problem? Please describe.
We're deploying a new, upgraded version of the DQM GUI tool. The procedure which notifies the DQM GUI about the new DQM files is different in a new version. We would like this new procedure to be used along side the the old, visDQMUpload based DQM file upload.
Describe the solution you'd like
Now, the DQM data is uploaded to the DQM GUIs using a tool called visDQMUpload. New procedure requires this process to be split into two stages:

Making a copy of the new DQM file to EOS.
Making an HTTP POST request to the new DQM GUI to notify it about the presence of a new file.

If required, a facade could be provided by us (DQM) that would have exactly the same interface as visDQMUpload. In such case, we would only like you to call the facade script (visDQMUpload_new) alongside the old one.
Describe alternatives you've considered
No viable, future proof alternatives were found.
Additional context
Bellow is a diagram that represents the current Offline DQM file movement:

Bellow is a diagram that represents the desired Offline DQM file movement, after the changes mentioned in this request:

For the records: link to the visDQMUpload tool
https://github.com/cms-sw/cmssw/blob/ba6e8604a35283e39e89bc031766843d0afc3240/DQMServices/FileIO/scripts/visDQMUpload.py

For the records 2: the POST should use the following API
https://cmsweb-testbed.cern.ch/dqm/offline-test-new/api/v1/register
HTTP request body:
[{"dataset": "/a/b/c", "run": "123456", "lumi": "0", "file": "/eos/cms/store/group/comm_dqm/DQMGUI_data/location/file.root", "fileformat": 1}]
For more information about this API endpoint (and others), please refer to:
https://github.com/cms-DQM/dqmgui#new-file-registering-endpoint

Hi @andrius-k , I'm very sorry for missing this GH issue.
Has this new DQM Gui server been deployed already?
Your proposal looks feasible to me, and it will make the DQMHarvesting process easier and more robust in the long run.
We are going to discuss this issue in the coming weeks and come back to you. Thanks

Hi @amaltaro
Unfortunately Andrius left CMS so, we take the baton:
the new GUI reading the new backend in eos is working since January on
https://cmsweb-testbed.cern.ch/dqm/offline-test-new/
which reads the following (temporary?) eos folder:
/eos/cms/store/group/comm_dqm/DQMGUI_data
Thanks

Hi @jfernan2, I'm working on this and have a question. Basically, the new replacement for visDQMUpload requires now more input parameters, is this right? Before, it needed only the file location. Now, it needs:

datasetName
RunNumber
Lumi
File Location
fileformat
I think I know how to get datasetName and runNumber, but I'm not sure about Lumi (I can't easily find that in the Workflow Management object). Is fileformat always 1 or 2 in this case? I'm reading there are 3 options

Legacy DQM TDirectory based ROOT files (1)
DQMIO TTree based ROOT files (2)
Protobuf based format used in Online live mode (3)

Is there an easy way to tell which one is the right one for a specific root file?

Hi @khurtado
Thanks for looking into this.
For the moment you can take Lumi=0 since this is reproducing the current per Run based root files. In the future we might have to upload single root files per LS where lumi will be declared somewhere
About file format, you can assume 1 since they are the ones uploaded to the GUI
Thank you

All DQM root files produced by Harvest processing are type 1 (plain ROOT), type 2 are DQMIO datasets in DAS not uploaded to the GUI
The name of the method could be changed now to visDQMRegister since it is not uploading files to any server but copying them to eos and registering them in DB instead.
Thanks
https://github.com/cms-DQM/dqmgui

@jfernan2 Thank you! That helps a lot. One more question from the diagrams. Right now we have:

visDQMUpload: Which uploads ROOT files to vocms0738,39,31

and we want:
2. Upload DQM ROOT files to EOS
3. To make this new HTTP post that does not upload files, only registers new files
Do we want to do 1, 2 and 3 and eventually get rid of 1, but not right now?
Is 3 dependent on 2? (E.g.: 1 worked, but upload of DQM root files to EOS failed for some reason, do we abort step 3?) Asking mainly because we are just trying to split this work in 2 pieces since there are 2 stages.

Since we want to have the old (legacy) DQM GUI (visDQMUpload) working on parallel until we end the commisioning of this new DQM GUI (visDQMRegister), I would vote for decouple both processes as much as possible.
When we are sure the new DQM GUI and the FW/WMcore workflow chain which makes it work is totally accepted by the Collaboration, we could start decommisioning the old visDQMUpload machinery.
On the visDQMRegister side:

If the copy of the file to eos does not succeed, step 3 should NOT be performed, since we don't want to register a file in the DB which cannot be accessed by the GUI
If step 3 fails for some reason, it should be retried. If you cannot register the file after several retries, we should log this somehow

Thanks

@jfernan2 : I'm testing some changes with the following workflow test:
https://cmsweb-testbed.cern.ch/reqmgr2/fetch?rid=wmagent_DQMHarvesting_LumiMask_khurtado_dqmup_v2_220308_174119_8053
For EOS:
It's basically using the WMAgent certificate and trying to write to:
/eos/cms/store/group/comm_dqm/DQMGUI_data/wmagent_DQMHarvesting_LumiMask_khurtado_dqmup_v2_220308_174119_8053/output/DQM_V0001_R000278175__NoBPTX__Run2016F-23Sep2016-v1__DQMIO_0002.root

But it cannot create the parent directory. Would the above be the expected path location to write though? Just double checking. I'm not sure if there is any voms DN mapping that need to be done in order to write to  /eos/cms/store/group/comm_dqm/DQMGUI_data. If so, please let me know.
2022-03-09 06:38:14,677:INFO:DQMUpload:Writing DQM root files to CERN EOS with retries: 3 and retry pause: 300
2022-03-09 06:38:14,677:INFO:StageOutMgr:==>Working on file: /wmagent_DQMHarvesting_LumiMask_khurtado_dqmup_v2_220308_174119_8053/output/DQM_V0001_R000278175__NoBPTX__Run2016F-23Sep2016-v1__DQMIO_0002.root
2022-03-09 06:38:14,677:INFO:StageOutMgr:===> Attempting 1 Fallback Stage Outs
2022-03-09 06:38:14,677:INFO:StageOutImpl:Creating output directory...
2022-03-09 06:38:14,681:INFO:StageOutImpl:Running the stage out...
2022-03-09 06:38:15,818:INFO:StageOutImpl:Command exited with status: 151
Output message: stdout: Local File Size is: 49421926
Remote File Size is:
ERROR: Size Mismatch between local and SE

stderr: Run: [ERROR] Server responded with an error: [3010] Unable to create parent directory /eos/cms/store/group/comm_dqm/DQMGUI_data/wmagent_DQMHarvesting_LumiMask_khurtado_dqmup_v2_220308_174119_8053/; Operation not permitted^@ (destination)

[ERROR] Server responded with an error: [3011] Unable to stat /eos/cms/store/group/comm_dqm/DQMGUI_data/wmagent_DQMHarvesting_LumiMask_khurtado_dqmup_v2_220308_174119_8053/output/DQM_V0001_R000278175__NoBPTX__Run2016F-23Sep2016-v1__DQMIO_0002.root; No such file or directory^@

[ERROR] Server responded with an error: [3011] Unable to remove /eos/cms/store/group/comm_dqm/DQMGUI_data/wmagent_DQMHarvesting_LumiMask_khurtado_dqmup_v2_220308_174119_8053/output/DQM_V0001_R000278175__NoBPTX__Run2016F-23Sep2016-v1__DQMIO_0002.root; No such file or directory^@

Thanks @khurtado
This is one of the key points of this request: that eos space has been granted to DQM group in the past as a temporary space for the project[1], but has a quota (at present) of 66TB out of which 43.56TB are being used. So, this space may not be a definite solution for the future once you set this workflow running in view of Run3.
Having said that, for your testing purpouses, I believe you should add yourself or Alan Malta if you are using his grid certificate to the following e-group which controls the write access[2], according to [3]. Or I can add you if you prefer, not sure if I should add you or Alan instead.
However, for the long term, it will be needed from WMCore or computing team another eos space to host all the DQM GUI root files, or this same space with larger quota.
Thank you very much
[1] https://twiki.cern.ch/twiki/bin/viewauth/CMS/T2CHCERNEosTeams
[2] https://e-groups.cern.ch/e-groups/Egroup.do?egroupName=cms-eos-PPD-DQM&tab=3
[3] eos root://eosproject.cern.ch attr ls /eos/cms/store/group/comm_dqm/
sys.accounting.vos.0="cms"
sys.acl="u:22014:rw,u:31275:rw,u:5410:rw,g:1399:!d,egroup:cms-eos-ppd-dqm:rw!d,egroup:cms-eos-ppd-dqm-cleaners:rw+d"
sys.forced.blockchecksum="crc32c"
sys.forced.blocksize="4k"
sys.forced.checksum="adler"
sys.forced.layout="replica"
sys.forced.nstripes="2"
sys.forced.space="default"
sys.recycle="/eos/cms/proc/recycle/"
user.acl=""

@jfernan2 Thank you! It would be @amaltaro, to keep consistency with the certs used in the test agent. He will be requesting access to it.

Hi @jfernan2 ,
So, Alan is part now of the egroup, but I can't still copy to the area. Is there anything else missing? I tried this interactively
[cmst1@vocms0192 ~]$ xrdcp test.txt root://eoscms.cern.ch//eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt
[0B/0B][100%][==================================================][0B/s]
Run: [ERROR] Server responded with an error: [3010] Unable to open file /eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt; Operation not permitted (destination)

[cmst1@vocms0192 ~]$ voms-proxy-info -all
subject   : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=amaltaro/CN=718748/CN=Alan Malta Rodrigues/CN=2073988181/CN=172652124/CN=2101403252/CN=334491060
issuer    : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=amaltaro/CN=718748/CN=Alan Malta Rodrigues/CN=2073988181/CN=172652124/CN=2101403252
identity  : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=amaltaro/CN=718748/CN=Alan Malta Rodrigues
type      : RFC3820 compliant impersonation proxy
strength  : 2048
path      : /data/certs/myproxy.pem
timeleft  : 163:56:40
key usage : Digital Signature, Key Encipherment
=== VO cms extension information ===
VO        : cms
subject   : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=amaltaro/CN=718748/CN=Alan Malta Rodrigues
issuer    : /DC=ch/DC=cern/OU=computers/CN=voms2.cern.ch
attribute : /cms/Role=production/Capability=NULL
attribute : /cms/Role=NULL/Capability=NULL
attribute : /cms/uscms/Role=NULL/Capability=NULL
timeleft  : 163:56:41
uri       : voms2.cern.ch:15002

Hi @khurtado
That is very strange, and honestly it scapes my knowledge of the system... Could you please try interactively
xrdcp -v -d 9  test.txt root://eoscms.cern.ch//eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt
?
I'd like to see the debug messages seeking for a possible mismatch between Alan's username and the one associated to his grid certificate, since for us (DQM) xrdcp command works fine, and a direct cp to eos as well, which for Alan's account (the one linked to his email registedred in the e-group) should work too.
Could you try a direct copy from lxplus using Alan's account or your account after registering to the e-group?
Are you xrdcpying from lxplus or somewhere lese? In my case, from lxplus and with my account lnked to the grid certificate the xrdcp -d 3  gives:
[2022-03-15 17:37:22.053623 +0100][Debug  ][XRootDTransport   ] [eoscms.cern.ch:1094.0] Sending out kXR_login request, username: jfernan, cgi: ?xrd.cc=ch&xrd.tz=1&xrd.appname=xrdcp&xrd.info=&xrd.hostname=lxplus789.cern.ch&xrd.rn=v5.4.1, dual-stack: true, private IPv4: false, private IPv6: false
[2022-03-15 17:37:22.057027 +0100][Dump   ][XRootD            ] [eoscms.cern.ch:1094] Got a kXR_ok response to request kXR_stat (path: /eos/cms/store/group/comm_dqm/DQMGUI_data/text.txt
Thanks

@jfernan2 This is from lxplus (I had to change 9 to 3, since xrdcp complains that's the max debug level):
It first tries krb5, then moves to gsi:
env -i X509_USER_PROXY=$PWD/myproxy.pem xrdcp -v -d 3 test.txt root://eoscms.cern.ch//eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt
2022-03-15 18:01:53.440219 +0100][Debug  ][XRootDTransport   ] [eoscms.cern.ch:1094.0] Logged in, session: 8889fd03deae0200a7420000ca4c5304
[2022-03-15 18:01:53.440224 +0100][Debug  ][XRootDTransport   ] [eoscms.cern.ch:1094.0] Authentication is required: &P=krb5,xrootd/eoscms.cern.ch@CERN.CH&P=gsi,v:10400,c:ssl,ca:5168735f.0|4339b4bc.0&P=sss,0.13:/etc/eos.keytab&P=unix
[2022-03-15 18:01:53.440238 +0100][Debug  ][XRootDTransport   ] [eoscms.cern.ch:1094.0] Sending authentication data
[2022-03-15 18:01:53.442406 +0100][Debug  ][XRootDTransport   ] [eoscms.cern.ch:1094.0] Trying to authenticate using krb5
[2022-03-15 18:01:53.442878 +0100][Debug  ][XRootDTransport   ] [eoscms.cern.ch:1094.0] Cannot get credentials for protocol krb5: Seckrb5: No or invalid credentials; No credentials cache found (p=xrootd/eoscms.cern.ch@CERN.CH).
[2022-03-15 18:01:53.445015 +0100][Debug  ][XRootDTransport   ] [eoscms.cern.ch:1094.0] Trying to authenticate using gsi
[2022-03-15 18:01:53.869336 +0100][Dump   ][AsyncSock         ] [eoscms.cern.ch:1094.0] Wrote a message:  (0x840084d0), 136 bytes
[2022-03-15 18:01:53.928655 +0100][Dump   ][XRootDTransport   ] [msg: 0x84007c80] Expecting 4385 bytes of message body
[2022-03-15 18:01:53.928701 +0100][Dump   ][AsyncSock         ] [eoscms.cern.ch:1094.0] Received message header, size: 8
[2022-03-15 18:01:53.928715 +0100][Dump   ][AsyncSock         ] [eoscms.cern.ch:1094.0] Received a message of 4393 bytes
[2022-03-15 18:01:53.928725 +0100][Debug  ][XRootDTransport   ] [eoscms.cern.ch:1094.0] Sending more authentication data for gsi
[2022-03-15 18:01:53.933586 +0100][Dump   ][AsyncSock         ] [eoscms.cern.ch:1094.0] Wrote a message:  (0x840b6780), 15180 bytes
[2022-03-15 18:01:53.943879 +0100][Dump   ][XRootDTransport   ] [msg: 0x84002190] Expecting 0 bytes of message body
[2022-03-15 18:01:53.943929 +0100][Dump   ][AsyncSock         ] [eoscms.cern.ch:1094.0] Received message header, size: 8
[2022-03-15 18:01:53.943936 +0100][Dump   ][AsyncSock         ] [eoscms.cern.ch:1094.0] Received a message of 8 bytes
[2022-03-15 18:01:53.944078 +0100][Debug  ][XRootDTransport   ] [eoscms.cern.ch:1094.0] Authenticated with gsi.
[2022-03-15 18:01:53.944105 +0100][Debug  ][PostMaster        ] [eoscms.cern.ch:1094] Stream 0 connected.
[2022-03-15 18:01:53.944120 +0100][Debug  ][Utility           ] Monitor library name not set. No monitoring
[2022-03-15 18:01:53.944164 +0100][Dump   ][AsyncSock         ] [eoscms.cern.ch:1094.0] Wrote a message: kXR_stat (path: /eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt, flags: none) (0x2225650), 74 bytes
[2022-03-15 18:01:53.944206 +0100][Dump   ][AsyncSock         ] [eoscms.cern.ch:1094.0] Successfully sent message: kXR_stat (path: /eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt, flags: none) (0x2225650).
[2022-03-15 18:01:53.944222 +0100][Dump   ][XRootD            ] [eoscms.cern.ch:1094] Message kXR_stat (path: /eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt, flags: none) has been successfully sent.
[2022-03-15 18:01:53.944228 +0100][Debug  ][ExDbgMsg          ] [eoscms.cern.ch:1094] Moving MsgHandler: 0x222dcb0 (message: kXR_stat (path: /eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt, flags: none) ) from out-queu to in-queue.
[2022-03-15 18:01:53.944241 +0100][Dump   ][PostMaster        ] [eoscms.cern.ch:1094.0] All messages consumed, disable uplink```

However, I get the permission denied:

[2022-03-15 17:59:35.423151 +0100][Dump   ][Utility           ] Path:      /eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt
[2022-03-15 17:59:35.423191 +0100][Debug  ][File              ] [0x1737bc0@root://eoscms.cern.ch:1094//eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt?oss.asize=10&xrdcl.requuid=da4053e1-383e-45e6-8f2c-19731aa88a1f] Sending an open command
[2022-03-15 17:59:35.423213 +0100][Dump   ][XRootD            ] [eoscms.cern.ch:1094] Sending message kXR_open (file: /eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt?oss.asize=10, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat )
[2022-03-15 17:59:35.423231 +0100][Debug  ][ExDbgMsg          ] [eoscms.cern.ch:1094] MsgHandler created: 0x1738c10 (message: kXR_open (file: /eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt?oss.asize=10, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ) ).
[2022-03-15 17:59:35.423243 +0100][Dump   ][PostMaster        ] [eoscms.cern.ch:1094] Sending message kXR_open (file: /eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt?oss.asize=10, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ) (0x1738570) through substream 0 expecting answer at 0
[2022-03-15 17:59:35.423283 +0100][Dump   ][AsyncSock         ] [eoscms.cern.ch:1094.0] Wrote a message: kXR_open (file: /eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt?oss.asize=10, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ) (0x1738570), 87 bytes
[2022-03-15 17:59:35.423308 +0100][Dump   ][AsyncSock         ] [eoscms.cern.ch:1094.0] Successfully sent message: kXR_open (file: /eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt?oss.asize=10, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ) (0x1738570).
[2022-03-15 17:59:35.423316 +0100][Dump   ][XRootD            ] [eoscms.cern.ch:1094] Message kXR_open (file: /eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt?oss.asize=10, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ) has been successfully sent.
[2022-03-15 17:59:35.423322 +0100][Debug  ][ExDbgMsg          ] [eoscms.cern.ch:1094] Moving MsgHandler: 0x1738c10 (message: kXR_open (file: /eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt?oss.asize=10, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ) ) from out-queu to in-queue.
[2022-03-15 17:59:35.423328 +0100][Dump   ][PostMaster        ] [eoscms.cern.ch:1094.0] All messages consumed, disable uplink
[2022-03-15 17:59:35.424149 +0100][Dump   ][XRootDTransport   ] [msg: 0xdc000ac8] Expecting 100 bytes of message body
[2022-03-15 17:59:35.424171 +0100][Dump   ][AsyncSock         ] [eoscms.cern.ch:1094.0] Received message header for 0xdc000ac8 size: 8
[2022-03-15 17:59:35.424185 +0100][Dump   ][AsyncSock         ] [eoscms.cern.ch:1094.0] Received message 0xdc000ac8 of 108 bytes
[2022-03-15 17:59:35.424191 +0100][Dump   ][PostMaster        ] [eoscms.cern.ch:1094] Handling received message: 0xdc000ac8.
[2022-03-15 17:59:35.424254 +0100][Dump   ][XRootD            ] [eoscms.cern.ch:1094] Got a kXR_error response to request kXR_open (file: /eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt?oss.asize=10, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ) [3010] Unable to open file /eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt; Operation not permitted
[2022-03-15 17:59:35.424284 +0100][Debug  ][XRootD            ] [eoscms.cern.ch:1094] Handling error while processing kXR_open (file: /eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt?oss.asize=10, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [ERROR] Error response: permission denied.
[2022-03-15 17:59:35.424293 +0100][Debug  ][ExDbgMsg          ] [eoscms.cern.ch:1094] Calling MsgHandler: 0x1738c10 (message: kXR_open (file: /eos/cms/store/group/comm_dqm/DQMGUI_data/test.txt?oss.asize=10, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ) ) with status: [ERROR] Error response: permission denied.

That's even stranger since now, I did a kdestroy from my secondary account and I was able to do xrdcp after gsi authentication ONLY :-S
So, indeed gsi is also working, may it be that Alan's grid certificate is not mapped to his cern account somehow so that e-group is not considering it?

The certificate is giving read access only for some reason. That means the mapping to the user is working, but the e-group portion is not recognized to give write permissions is not, it's hard to tell what is going on without knowing what the EOS configuration is. Is this something Service Desk at CERN is supposed to help with?  @amaltaro any ideas?

Hi @khurtado
I am not sure about your statement since I was able to write using my secondary account through gsi authentication only, but it is true CERN IT should be able to solve this
Thanks

@jfernan2
Thank you! I will just create a new directory for the tests inside, following Aln's suggestion: /eos/cms/store/unmerged/DQMGUI

@jfernan2 I'm getting closer here. I have one question regarding the API. In this format:
[{"dataset": "/a/b/c", "run": "123456", "lumi": "0", "file": "/eos/cms/store/group/comm_dqm/DQMGUI_data/location/file.root", "fileformat": 1}]

How do I pass a range of runs or lumis? This is what I see from WMCore:
'runAndLumis': {278175: [[70, 90]]}}

So, if we have a DQMHarvest workflow set to multiRun harvesting, it can have multiple runs and lumis associated with each run.

@khurtado
Current DQM GUI is only able to show DQM root files per RUN. In the future is expected to be able to handle per LS too.
This implies that, a single root file uploaded (now copied to eos and registered to the DB) can only contain a single run or a single LS (of a run). Hence, per RUN root files should be registered with Lumi=0, since this is reproducing the current per Run based root files. Once we were able to show in the GUI per LS data, runs will be registered with a single LS, not several, since plots per LS must be displayed.
Please note that same RUN (and LS) may be associated to severl datasets, but a single file each, like in: https://github.com/cms-DQM/dqmgui#api-documentation
Multirun harvesting root files are a special case of DQM root files; in this case, since all stats are harvested in a single root file and the GUI is not able to display the runs it contains (they are embedded in the dataset name or config which has produced it), for teh GUI runNumber is always forced/set to 999999 for data (and to 1 for MC as any MC). See: https://github.com/dmwm/WMCore/pull/9746 and https://github.com/dmwm/WMCore/issues/9690
Bottomline, in principle you should not copy and register the same file for more than one RUN or LS.
@ahmad3213  @emanueleusai @rvenditti  please consider to correct me at any point since you are the official DQM conveners, I am not DQM convener since 31st Dec 2021. Thanks

@jfernan2 Thank you!
Here is one workflow test with the current changes:
https://cmsweb-testbed.cern.ch/reqmgr2/fetch?rid=wmagent_DQMHarvesting_LumiMask_khurtado_dqm_v10_220320_210531_8322
ROOT files were uploaded here:
/eos/cms/store/unmerged/DQMGUI/wmagent_DQMHarvesting_LumiMask_khurtado_dqm_v10_220320_210531_8322

And the POST call to the register site looks like this:
2022-03-21 04:03:18,643:INFO:DQMUpload:HTTP Upload is about to start:
 => URL: https://cmsweb-testbed.cern.ch/dqm/offline-test-new/api/v1/register
 => Filename: /eos/cms/store/unmerged/DQMGUI/wmagent_DQMHarvesting_LumiMask_khurtado_dqm_v10_220320_210531_8322/output/DQM_V0001_R000277991__NoBPTX__Run2016F-23Sep2016-v1__DQMIO_0001.root

2022-03-21 04:03:18,643:INFO:DQMUpload:Using proxy file: /srv/myproxy.pem
2022-03-21 04:03:18,643:INFO:DQMUpload:Using CA certificate path: None
2022-03-21 04:03:18,643:INFO:DQMUpload:HTTP Register POST arguments: [{'file': '/eos/cms/store/unmerged/DQMGUI/wmagent_DQMHarvesting_LumiMask_khurtado_dqm_v10_220320_210531_8322/output/DQM_V0001_R000277991__NoBPTX__Run2016F-23Sep2016-v1__DQMIO_0001.root', 'dataset': '/NoBPTX/Run2016F-23Sep2016-v1/DQMIO', 'run': 277991, 'lumi': 0, 'fileformat': 1}]

2022-03-21 04:03:18,657:INFO:HTTPSAuthHandler:Found 149 default trusted CA certificates.
2022-03-21 04:03:18,657:INFO:HTTPSAuthHandler:SSL context manager created with the following settings:
2022-03-21 04:03:18,657:INFO:HTTPSAuthHandler:  check_hostname : True
2022-03-21 04:03:18,657:INFO:HTTPSAuthHandler:  options : Options.OP_ALL|OP_NO_SSLv3|OP_NO_SSLv2|OP_CIPHER_SERVER_PREFERENCE|OP_SINGLE_DH_USE|OP_SINGLE_ECDH_USE|OP_NO_COMPRESSION
2022-03-21 04:03:18,657:INFO:HTTPSAuthHandler:  protocol : _SSLMethod.PROTOCOL_TLS
2022-03-21 04:03:18,657:INFO:HTTPSAuthHandler:  verify_flags : VerifyFlags.VERIFY_X509_TRUSTED_FIRST
2022-03-21 04:03:18,657:INFO:HTTPSAuthHandler:  verify_mode : VerifyMode.CERT_REQUIRED
2022-03-21 04:03:18,886:INFO:DQMUpload:HTTP POST to register url finished succesfully with response:
  Status code: 201

I do see some DQM histograms associated with that Run and Dataset here:
https://cmsweb-testbed.cern.ch/dqm/offline-test-new/?folder_path=DQM%2FTimerService&dataset_name=%2FNoBPTX%2FRun2016F-23Sep2016-v1%2FDQMIO&run_number=277991&workspaces=Everything&overlay=overlay&normalize=true&lumi=0
but it's unclear to me how to tell if they came from what I registered in this test or if they were there before.
Anyway, does this look well to you?

@jfernan2 And this is for multiRun:
https://cmsweb-testbed.cern.ch/reqmgr2/fetch?rid=wmagent_DQMHarvesting_MultiRun_khurtado_dqm_v11_220321_111702_196
The run number will be 999999 for current multiRun workflows but should start showing 999999 or 1 accordingly for new workflows once  #9746 is applied.
2022-03-21 11:18:57,248:INFO:DQMUpload:HTTP Upload is about to start:
 => URL: https://cmsweb-testbed.cern.ch/dqm/offline-test-new/api/v1/register
 => Filename: /eos/cms/store/unmerged/DQMGUI/wmagent_DQMHarvesting_MultiRun_khurtado_dqm_v11_220321_111702_196/output/DQM_V0001_R000999999__NoBPTX__Run2016F-23Sep2016-v1-277932-278193__DQMIO_0001.root

2022-03-21 11:18:57,248:INFO:DQMUpload:Using proxy file: /srv/myproxy.pem
2022-03-21 11:18:57,248:INFO:DQMUpload:Using CA certificate path: None
2022-03-21 11:18:57,248:INFO:DQMUpload:HTTP Register POST arguments: [{'file': '/eos/cms/store/unmerged/DQMGUI/wmagent_DQMHarvesting_MultiRun_khurtado_dqm_v11_220321_111702_196/output/DQM_V0001_R000999999__NoBPTX__Run2016F-23Sep2016-v1-277932-278193__DQMIO_0001.root', 'dataset': '/NoBPTX/Run2016F-23Sep2016-v1/DQMIO', 'run': 999999, 'lumi': 0, 'fileformat': 1}]

2022-03-21 11:18:57,273:INFO:HTTPSAuthHandler:Found 149 default trusted CA certificates.
2022-03-21 11:18:57,273:INFO:HTTPSAuthHandler:SSL context manager created with the following settings:
2022-03-21 11:18:57,273:INFO:HTTPSAuthHandler:  check_hostname : True
2022-03-21 11:18:57,274:INFO:HTTPSAuthHandler:  options : Options.OP_ALL|OP_NO_SSLv3|OP_NO_SSLv2|OP_CIPHER_SERVER_PREFERENCE|OP_SINGLE_DH_USE|OP_SINGLE_ECDH_USE|OP_NO_COMPRESSION
2022-03-21 11:18:57,274:INFO:HTTPSAuthHandler:  protocol : _SSLMethod.PROTOCOL_TLS
2022-03-21 11:18:57,274:INFO:HTTPSAuthHandler:  verify_flags : VerifyFlags.VERIFY_X509_TRUSTED_FIRST
2022-03-21 11:18:57,274:INFO:HTTPSAuthHandler:  verify_mode : VerifyMode.CERT_REQUIRED
2022-03-21 11:18:57,392:INFO:DQMUpload:HTTP POST to register url finished succesfully with response:
  Status code: 201

Sorry @khurtado but I am not following you, at least not completely:

Here is one workflow test with the current changes: https://cmsweb-testbed.cern.ch/reqmgr2/fetch?rid=wmagent_DQMHarvesting_LumiMask_khurtado_dqm_v10_220320_210531_8322

What do you mean with current changes? Changes in the script you are creating to acomplish this task?

https://cmsweb-testbed.cern.ch/reqmgr2/fetch?rid=wmagent_DQMHarvesting_LumiMask_khurtado_dqm_v10_220320_210531_8322

This workflow is weird, CMSSW_8_0_20 ? And it has several runs in it, I understand there is a DQM root file per run

And the POST call to the register site looks like this:
The post looks OK, in principle

but it's unclear to me how to tell if they came from what I registered in this test or if they were there before. Anyway, does this look well to you?
They look OK, but bear in mind that since you did several tests, last upload/register is the one which will be displayed

@jfernan2 And this is for multiRun: https://cmsweb-testbed.cern.ch/reqmgr2/fetch?rid=wmagent_DQMHarvesting_MultiRun_khurtado_dqm_v11_220321_111702_196
The run number will be 999999 for current multiRun workflows but should start showing 999999 or 1 accordingly for new workflows once #9746 is applied.

I am confused here: PR #9746 is almost two year old, not yet merged (plans?) and it keeps syaing that run Number = 1 only will only be assigned to MC, but never to data. On the other hand, for data, run = 999999, info for harvested runs is lost, I would have expected to keep it on the dataset name somehow, otherwise another multirun harvesting (MRH) for the same dataset will overwrite this one.
For me MRH is a nasty task, in the sense that it may be subdetector dependent since the list of runs for a given dataset may vary from one DPG to another. It has been encouraged in the past that every DPG makes its own MRH for that reason. On the other hand, current DQM GUI will have problems displaying it since it was designed for single run uploads.
If you want to register brand new data which is not in the GUI yet, so that you can ensure that the plots you see come from your last registering, perhaps you could use current dataset: /Cosmics/Commissioning2022-PromptReco-v1/DQMIO
Thanks a lot

Sorry @khurtado but I am not following you, at least not completely:

Here is one workflow test with the current changes: https://cmsweb-testbed.cern.ch/reqmgr2/fetch?rid=wmagent_DQMHarvesting_LumiMask_khurtado_dqm_v10_220320_210531_8322

What do you mean with current changes? Changes in the script you are creating to acomplish this task?

Yes, in this PR: https://github.com/dmwm/WMCore/pull/11015

https://cmsweb-testbed.cern.ch/reqmgr2/fetch?rid=wmagent_DQMHarvesting_LumiMask_khurtado_dqm_v10_220320_210531_8322

This workflow is weird, CMSSW_8_0_20 ? And it has several runs in it, I understand there is a DQM root file per run

Yes, it has 1 DQM root file per run. This workflow is one of the WMCore ReqMgr templates we have for testing:
https://github.com/dmwm/WMCore/blob/master/test/data/ReqMgr/requests/Integration/DQMHarvesting_MultiRun.json

And the POST call to the register site looks like this:

The post looks OK, in principle

but it's unclear to me how to tell if they came from what I registered in this test or if they were there before. Anyway, does this look well to you?

They look OK, but bear in mind that since you did several tests, last upload/register is the one which will be displayed
That sounds good, thanks!

@jfernan2 And this is for multiRun: https://cmsweb-testbed.cern.ch/reqmgr2/fetch?rid=wmagent_DQMHarvesting_MultiRun_khurtado_dqm_v11_220321_111702_196
The run number will be 999999 for current multiRun workflows but should start showing 999999 or 1 accordingly for new workflows once #9746 is applied.

I am confused here: PR #9746 is almost two year old, not yet merged (plans?) and it keeps syaing that run Number = 1 only will only be assigned to MC, but never to data. On the other hand, for data, run = 999999, info for harvested runs is lost, I would have expected to keep it on the dataset name somehow, otherwise another multirun harvesting (MRH) for the same dataset will overwrite this one.
For me MRH is a nasty task, in the sense that it may be subdetector dependent since the list of runs for a given dataset may vary from one DPG to another. It has been encouraged in the past that every DPG makes its own MRH for that reason. On the other hand, current DQM GUI will have problems displaying it since it was designed for single run uploads.
If you want to register brand new data which is not in the GUI yet, so that you can ensure that the plots you see come from your last registering, perhaps you could use current dataset: /Cosmics/Commissioning2022-PromptReco-v1/DQMIO
Thanks a lot
Ah, okay. So, right now this is using the 999999 thing which apparently would be useless.
I can alternatively just NOT call the new DQMUI at all if the harvesting job is multiRun, since it is not supported anyway. How does that sound?

I can alternatively just NOT call the new DQMUI register or copy the file to EOS at all if the harvesting job is multiRun, since it is not supported anyway (so, we would just keep the old visDQMGUI call). How does that sound?

How is current visDQMGUI treating MRH files?

I can alternatively just NOT call the new DQMUI register or copy the file to EOS at all if the harvesting job is multiRun, since it is not supported anyway (so, we would just keep the old visDQMGUI call). How does that sound?

How is current visDQMGUI treating MRH files?

The HTTP post for visDQMGUI only asks for the full path filename in the worker node. No info on Run numbers or lumis.

Then, it is basing its DB registering on the Run Number from the DQM root file name, just as the new GUI. No info about the run numbers contained in dataset or file name

@jfernan2 Then, it seems MHR is not supported overall. Would you prefer for to skip upload to EOS and registration for MHR or to keep it in the 999999 (and 1 when using agents with #9746 merged)? The first makes more sense to me since the information is not useful for the reasons you exposed, the second sort of replicates the current behavior with the visDQMUpload that has been there for a long time.
I think any improvements to support MHR should be a different github issue.

@khurtado I would keep the behaviour we have in the old GUI to be coherent, otherwiser MRH files would not be accessible in the new GUI
On the other hand, I don't see why you keep claimng that MRH data will have runNumber=1 after #9746
RunNumber=1 is reserved for MC, 999999 for MRH data
See:
https://github.com/dmwm/WMCore/pull/9746/files#diff-3c13cdc9485083bb43b4e4d3d37f7310b878d36bc137ce2a7cf8f08de4e9daf0R176-R181
Do you agree?

@jfernan2 Ah, yes. What I meant is, without the PR, RunNumber is always 999999 in any case. The PR makes it either 999999 (Data harvesting) or 1 (MC). Is that correct?

Just to be clear, please correct me if I am wrong: right now, before #9746 is merged,

MRH files, either data or MC, have a parameter in WMcore "runLimits", "-%s-%s" % (minRun, maxRun))[1], which is used in the dataset name for DQM.

I am not sure how many of these have been uploaded to the DQM GUI, I can only find one of those in the development GUI, none in the Offline GUI. This one:  https://tinyurl.com/ycj7luc9
which has RunNumber forced as 999999 in the DQM search box despite there is a mismatch between this and the runNumber displayed in the Menu of the DQM GUI (278017, the longest one in the range?), but dataset name keeps the run range used in the harvesting: /NoBPTX/Run2016F-23Sep2016-v1-277932-278193/DQMIO
This would be the desired behaviour for MRH in DQM GUI, so that DQM user can trace back directly from dataset name, which runs (a range) it contains, despite the search is performed by run = 999999 in the DQM search.
I see several ALCAPROMPT datasets uploaded in this way into the Offline DQM GUI too, all of them with runNumber forced to 999999, but different dataset name and different run displayed in the header of the GUI.
E.g. /StreamExpress/Run2018A-PromptCalibProdSiStripGainsAAG-Express-v1-316702-316766/ALCAPROMPT
https://tinyurl.com/yaz6vfyt
So that they can be distinguished by dataset name (run range) and even by displayed Run Number (in the header of the GUI) despite all have 9999999

After #9746 is merged, we lose all the functionality defined above, and everytime a MRH root file is registered for an existing dataset name, it is overwritten no matter the range used in the harvesting

For single Run mode, always the run Number is kept

@ahmad3213 @emanueleusai @rvenditti please speak either if you agree or disagree
Thanks
[1] https://github.com/dmwm/WMCore/pull/9746/files#diff-3c13cdc9485083bb43b4e4d3d37f7310b878d36bc137ce2a7cf8f08de4e9daf0L181-R184

@jfernan2 That is a good point and I think you should make that comment on #9646 itself. But looking at the comments, it seems that PR is not finished and won't be merged right now, since Run dependent MC really has run number > 1, according to the comments.
For #11015, which is the PR that would solve this particular issue, we won't lose the dataset name tweak.

Hi @jfernan2 thanks for following this. We (DQMDC conveners) totally agree with you. @khurtado could you please let us know what is missing to complete the automation process? Moreover, let us know when you plan to do the next test, so that we can choose a suitable file (that is not present in the GUI).  FYI, we just finished to upload by hands the rootfiles of latest runs.

@rvenditti Yes, I think the basic functionality is there. The missing pieces are:

To get a permanent EOS storage space for this, rather than the unmerged we are using for testing. We talked with @amaltaro about getting a space that WMCore can manage. I'm following up with him on this.
Code review, before the PR can be merged.
More validation tests? This is connected to your comment, yes... please, let me know what dataset/files would be good to test in order to make a validation.

@rvenditti @jfernan2 On the EOS storage topic. How much space do you think would be needed for this?

@jfernan2 Do we need to support multi urls for the registration site? Or can we safely assume it will only be 1 single register url in the future? Current changes have it so that it supports multi url, but for the sake of simplicity, we could remove that if it's not going to be needed.

@khurtado
On the EOS space, the current quota we have in /eos/cms/store/group/comm_dqm  is 66TB out of which we have used 67% including this service (new DQM Offline GUI for Run2 and LS2 data) and others.
Run2 through all data takings in between specially in LS2 account so far for ~7TB
This is data taking, apart we would have  RelVals which so far up to CMSSW_12_3_X account for 3.1TB
Since this is a long term project where all DQM root files for Run3 and beyond may be stored, it is difficult to estimate a final need at this point, however a rough estimate could be 100TB at the moment. So, at present we could be using /eos/cms/store/group/comm_dqm once you solve the authentication problems, at least as a temporal solution
About urls, right now new OFFLINE GUI lives on: https://cmsweb-testbed.cern.ch/dqm/offline-test-new/
This url may change a bit once commisioning ends with this addition you are working on.
Another url/GUI should be created for RelVal and Development (used in Jenkins test from PR comparisons) GUI instances.

@jfernan2 Thank you for the details!
Regarding the URL, what I meant more is if we would ever be required to report to more than 1 register URL at a time.
For example, for a given ROOT files, report to  URL1 and URL2 in the form
url="URL1;URL2
We have a loop to support this below:
https://github.com/dmwm/WMCore/pull/11015/files/ee8c9fe76301997917e938b998393760893de7cf#diff-28545267cb6f40f2c39bdc92f389e86d92ddd36355037cc3df162b199606a16eR454
, but @amaltaro made the suggestion of removing this feature for simplicity if we only expect to work with 1 single URL (even if that single url changes over time)

ah ok, just a single url is fine then :-)
Thanks

ah ok, just a single url is fine then :-) Thanks

Sounds good, thanks!

@jfernan2 One question on this. Is this a single URL that is no workflow dependent?
Anything special in case of T0 workflows? See the question below from Alan:
While reviewing it, it just occurred to me that T0 also uses this code. So we need to verify whether T0 would need to use this same mechanism; or whether we need to work on workflow configuration that could enable/disable this new DQM feature. Can you please follow this up with the DQM team?

Hi @khurtado let me put it this way in order to clarify:

at present, the legacy DQM GUI in its 2 instances (Offline and RelVal) receive DQM root files (Prompt and ReReco data, and RelVal workflows, resp.) in a single url (https://cmsweb.cern.ch/dqm/offline   and   https://cmsweb.cern.ch/dqm/relva   resp.). I ignore the procedence, I guess Prompt data from Tier0, ReReco from Tier0 and others, and RelVal workflows from elsewhere (even Tier2s?). These GUIs should still live until the full commisioning of the new GUIs. The root files in this case are registered in the local db (index) of the GUI and stored in the machine local disk servers
new DQM GUIs (Offline already available in this temporal url https://cmsweb-testbed.cern.ch/dqm/offline-test-new/ , RelVal to be created once Offline is properly defined in terms of eos and this script you are developing) should replace at some point the legacy ones (Offline by Offline, RelVal by RelVal)
HTH

Hi @jfernan2, thank you for this clarification.
Changes proposed by Kenyi are almost ready to be merged, but there are two crucial details to be sorted before it goes in:

is T0 supposed to adopt to same "transient" DQM workflow? Meaning, i) upload the root file to the DQM Gui; ii) stage the root file to the EOS area; iii) register the dataset/run/lumi against the DQM register system?
we would like to avoid adding another configurable register URL to every single workflow in the system. So, I wonder if we could come up with endpoint names that we could derive from the current/old DQM Gui? For instance:

if we are uploading a root file to https://cmsweb.cern.ch/dqm/offline, then the register endpoint would be https://cmsweb.cern.ch/dqm/offline-register. If it's a RelVal against https://cmsweb-testbed.cern.ch/dqm/relval, then the new register endpoint would be https://cmsweb-testbed.cern.ch/dqm/relval.

Do you think it would be feasible? If not, then there is still a substantial development to be done to support/validate the new argument at the workflow level and propagate it all the way down to the job runtime code.

Hi @amaltaro

Yes, at least until we can decommission the old GUI, when step i) will be dropped
If using https://cmsweb.cern.ch/dqm/offline for i) and https://cmsweb-testbed.cern.ch/dqm/offline-test-new/api/v1/register for iii) causes you trouble, we can try to convince cmsweb to move https://cmsweb-testbed.cern.ch/dqm/offline-test-new/api/v1/register   to    https://cmsweb.cern.ch/dqm/offline-new/api/v1/register  or similar. I do not fully understand you why it makes a problem two different urls since the task are coupled but independent.

Bear in mind that the two GUIs (new and old) live in different machines, cmsweb crew offered this kind of provisional url for the new as a temporal place. So, i) is doing the job for the old GUI (which after receiving the file, the machine is registering it in its own internal database, that is why there is no API), while ii) + iii) is doing the same for the new GUI, hence the different urls
For RelVal GUI, since it has not been settled yet, we can for sure make it more legacy ompliant, since old RelVal GUI is on cmsweb (not in cmsweb-testbed).
Thanks

Thanks for these answers.
Okay, so once these changes take place, the T0 machinery will start adopting the new DQM Gui procedure as well. FYI  @germanfgv @jhonatanamado
Regarding the urls, the best way to deal with these multiple urls would be creating a new workflow/spec parameter, but that would require much more changes in WMCore, as well as in McM and PyReleaseValidation when creating workflows.
What we have right now is 3 instances of the DQM Gui:
a) offline
b) relval
c) dev
and EACH of those instances is available both in the cmsweb-testbed and the cmsweb production clusters.
The question we need to answer is, for any given workflow, which DQM register URL it has to use? Given that it's going to be something transient, I was considering in having a map between the OLD DQM url to the NEW DQM url. Can we agree on such convention/map? Maybe something like (where key is the old url, and value is the new one):
{"https://cmsweb.cern.ch/dqm/offline": "https://cmsweb.cern.ch/dqm/offline-new/api/v1/register",
 "https://cmsweb.cern.ch/dqm/relval": "https://cmsweb.cern.ch/dqm/relval-new/api/v1/register",
 "https://cmsweb.cern.ch/dqm/dev": "https://cmsweb.cern.ch/dqm/dev-new/api/v1/register"}

? We might have to loop in the CMSWEB team to see whether this would be possible as well.
In addition to that - and if we come up with a logic for the question above this comment becomes useless - I would not recommend to rely on cmsweb-testbed in production workflows. If anything happens to that cluster, we would be failing production workflows.
If you prefer, we could try to organize a zoom call between you/me/Kenyi tomorrow to discuss these details as well.

Thanks @amaltaro
From DQM side there is no problem to move from cmsweb-testbed to non-testbed. I guess it is just a matter of interacting with cmsweb crew.
I let DQM conveners to comment here

Hi, yes  I confirm that for the DQM-DC team, there is no problem in the migration of the offline gui from test bed to non-test bed. Is this migration already in the process to happen or do you still need to discuss the details with us?

@rvenditti If you could get the mapping mentioned by @amaltaro (also pasted below) then no additional discussion is necessary. We can modify the Pull Request accordingly to take this into account and avoid creating a new workflow/spec parameter.
{"https://cmsweb.cern.ch/dqm/offline": "https://cmsweb.cern.ch/dqm/offline-new/api/v1/register",
 "https://cmsweb.cern.ch/dqm/relval": "https://cmsweb.cern.ch/dqm/relval-new/api/v1/register",
 "https://cmsweb.cern.ch/dqm/dev": "https://cmsweb.cern.ch/dqm/dev-new/api/v1/register"}

@rvenditti @jfernan2 Looking at the JIRA ticket, we are close to getting the permanent EOS storage. Are there any news regarding the cmsweb mapping?

@rvenditti Awesome! Thank you.

@rvenditti I tested the new mapping, but I got an error while trying to register the files.
This link in particular:
https://cmsweb.cern.ch/dqm/offline-new/api/v1/register
shows expired SSL certs
Secure Connection Failed

An error occurred during a connection to cmsweb.cern.ch. SSL peer rejected your certificate as expired.

Error code: SSL_ERROR_EXPIRED_CERT_ALERT

Could that be the issue?

Hi @khurtado
The problem with the mapping you're trying to do is that there is no host on this link, it's just an empty link for when the new GUI moves to production.

Could that be the issue? I thought it could be my certs, but if I use the browser for example, I can see: https://cmsweb-testbed.cern.ch/dqm/offline-test-new/ but not: https://cmsweb.cern.ch/dqm/offline-new/

Right now the new GUI is being migrated from cmsweb testbed VM to cmsweb testbed Kubernetes cluster, but deployment is very recent and some bugs are still present. After the deployment in the cmsweb testbed Kubernetes cluster is a success and stable, deployment in cmsweb Kubernetes production cluster will start. Then "https://cmsweb.cern.ch/dqm/offline-new/" will be available, but for now only "https://cmsweb-testbed.cern.ch/dqm/offline-test-new/" is available.

@micsucmed @rvenditti @jfernan2 : Just pinging about this on what Alan asked last week.  Are there any news or time estimates on when the new url mappings will be available? We can't move forward with this until then.

@micsucmed Understood. Thank you for the update on this!

Hi @micsucmed ,  we had a Workflow Management meeting today and we were wondering if there was any progress on this, e.g.: on the cloud infrastructure + EOS issue

Hi @khurtado, we haven't got any response from the assignee to the EOS ticket in a while (it might appear empty as it is a private ticket but I can add you to the watchers list if you like), so it's very difficult to say when this problem will be solved. It may be best if we continue with this using the testbed endpoint for the offline GUI ( https://cmsweb-testbed.cern.ch/dqm/offline-test-new/ ) and once the EOS problem is solved we change to the production endpoints.

AFAWK, the main problem is solved and we are now finalizing some access issues, but @micsucmed can correct me and add a timescale for this.
What we really need before the data taking start-up is the automation of the rootfiles transfer on EOS (that was a part of the request), given that it is presently done by hand and we have other services that read from there.
So I would put this as  top priority.
Off course, we can still do the upload by hand, but given the amount of  incoming files, this could become a nightmare for us.
For the upload of the rootfiles in the new GUI, instead, we can survive like this for a couple of months indeed (we have the old GUI that is working).
Now I see here: https://github.com/dmwm/WMCore/pull/11015 that the transfer to EOS is failing still due to mapping. Is it possible to decouple the two parts of the problem (i.e. transfer to eos  and rootfile upload in the GUI) ?

@rvenditti Thanks for the update!
Regarding the EOS transfer, this is working.
Here is a status summary:

Root files upload to the old visDQM GUI works just fine, as always (this hasn't been touched).
Registering the files to the new GUI fails due to the new mapping not really hosting any services
Copy of DQM Root files to EOS works if we ignore the registration failures (but right now, we delete each file on EOS where the registration above fails, so both steps need to succeed).

Hi @khurtado I would expect to finish the testbed deployment sometime this week, so I would say that at the end of next week the production endpoints will be available.

Hi @micsucmed . Is there an update on this?
I still see e.g.: this service link unavailable:
https://cmsweb.cern.ch/dqm/offline-new/api/v1/register

@micsucmed Just pinging about this issue.

@khurtado sorry for the delay. The EOS issue has been fixed yesterday, it was fixed a couple weeks ago but a n update to the Kubernetes cluster reinstated the issue. I will deploy the testbed again and prepare for production deployment. At the end of this week or early next week I expect to have the production deployment ready for you to continue. Again sorry for the delays and my late response.

Hi @micsucmed. Thank you! Sorry I took 2 weeks to reply back, but I hope the deployment plans are going well. Please let me know once the services have been deployed so we can re-test this.

Hi @khurtado , I understood for @micsucmed that the update of the offline GUI to k8 is done, but there are some problems on the cmsweb side.  I don't have the details, but i think that problem can be solved in the time scale of some days. @micsucmed can you confirm?

Hi @khurtado, as @rvenditti say's there is some issue related to the frontend rules given by cmsweb so that the URL in production ( https://cmsweb.cern.ch/dqm/offline-new/ ) access the deployed pod in K8's. The issue is being solved by cmsweb team. I would like to think it's a simple issue that should be solved soon, nonetheless, as I am not the one with access to solving it I am not sure if this will be the case.

@rvenditti @micsucmed Thank you for the update. Is there a ticket or GH issue to track this from the cmsweb team side?

Hare it is: https://its.cern.ch/jira/browse/CMSKUBERNETES-145

@rvenditti how should we proceed on this? Are we supposed to continue working on this?
Is the new DQM GUI already working?


Verify manifests script

What this PR does / why we need it:

Adds verify manifests script

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #73

/approve

/lgtm

@amy now that you have the file diff 😄:
diffing hack/../config against freshly generated manifests
 .../bootstrap.cluster.x-k8s.io_kubeadmconfigs.yaml | 26 ++++++++++++++++++++++
 1 file changed, 26 insertions(+)
Manifests in hack/../config are out of date. Please run `make manifests`

Can you see if something truly is out of date?

Yah. Just did another rebase. Looks like running make manifests again removes this:
-            additionalUserDataFiles:
-              description: AdditionalUserDataFiles specifies extra files to be passed
-                to user_data upon creation.
-              items:
-                description: Files defines the input for generating write_files in
-                  cloud-init.
-                properties:
-                  content:
-                    description: Content is the actual content of the file.
-                    type: string
-                  owner:
-                    description: Owner specifies the ownership of the file, e.g. "root:root".
-                    type: string
-                  path:
-                    description: Path specifies the full path on disk where to store
-                      the file.
-                    type: string
-                  permissions:
-                    description: Permissions specifies the permissions to assign to
-                      the file, e.g. "0640".
-                    type: string
-                required:
-                - content
-                - path
-                type: object
-              type: array

ugh, apparently yesterday's rebase of #42 managed to eliminate part/all of #57 😦 cc @chuckha

/retest

I've fixed master so now we shouldn't fail manifest verification (#89)

oh nooo, did i hose a rebase -_-

/lgtm


Add function to get SVG BBox

I found that there is a resvg_get_image_bbox function, which seems to only be used in the c-api?
I hope that the bbox of SVG can also be obtained in the resvg API. This is similar to SVGGraphicsElement.getBBox(svg root) in the DOM API
Thanks.

tree.root().calculate_bbox()

The calculation of calculate_bbox() seems to have some problems
https://github.com/zimond/rusty-svg/pull/15/files#diff-b1a35a68f14e696205874893c07fd24fdb88882b47c23cc0e0c80a30c7d53759R482-R483
Test svg:
<svg viewBox="-40 0 150 100" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
  <!-- example from https://developer.mozilla.org/en-US/docs/Web/SVG/Attribute/transform -->
  <g fill="grey"
     transform="rotate(-10 50 100)
                translate(-36 45.5)
                skewX(40)
                scale(1 0.5)">
    <path id="heart" d="M 10,30 A 20,20 0,0,1 50,30 A 20,20 0,0,1 90,30 Q 90,60 50,90 Q 10,60 10,30 z" />
  </g>
  <use xlink:href="#heart" fill="none" stroke="red"/>
</svg>

This issue appears to have been fixed via https://github.com/RazrFalcon/resvg/pull/482


update simple-markdown to 0.4.4

0.4.4 fixes a Cross-Site Scripting vulnerability

Some more info here https://www.npmjs.com/advisories/815 and https://github.com/Khan/simple-markdown/pull/63

I think this can be merged, no ?

Requesting for this to be merged to fix a security alert


Build fix for the CPP example.

NotificationActivationCallback.h is not found because
the target platform is incorrectly put to 8.1 if you have
have the 8.1 SDK installed. It will not build as the header
is not included in that SDK. Enforcing to 10+ will make it compile.
Also renamed the include to follow
https://msdn.microsoft.com/en-us/library/windows/desktop/mt643711(v=vs.85).aspx
which uses a camelcase name : NotificationActivationCallback.h

@tfennel please take a look.

@MichaelGoulding can you take a look. Simple, seems fine, but would like you to double check.

Thanks!


fix: dependabot commit config

Why:
Closes:  #33996

What's being changed (if available, include any code snippets, screenshots, or gifs):

Check off the following:

[ ] I have reviewed my changes in staging, available via the View deployment link in this PR's timeline (this link will be available after opening the PR).

For content changes, you will also see an automatically generated comment with links directly to pages you've modified. The comment won't appear if your PR only edits files in the data directory.

[ ] For content changes, I have completed the self-review checklist.

@gagan-bhullar-tech Thank you for opening a PR! I'll get this triaged for review ✨

Yes, thanks @gagan-bhullar-tech — this now reflects my experience. Let's just wait for the triage to xcheck the wording, but this now fixes the issue for me. The code example matches its description below w/ expected results, and the allow table looks reasonably valid.
(Note for reviewer: The removed scope in example didn't need to be moved elsewhere, as the Composer example right above it already demonstrates that.)

thanks @janbrasna for reviewing the PR.


Add Support for the Swift 4 Serialization Mechanisms

Hey @davecom, have you put any thought to this yet? Otherwise I can start poking and post some thoughts. This'll be great for an ongoing project!

Hi @yoiang,
I haven't had time to look into this extensively. But I think it should be possible to make an extension of Graph (or perhaps separate ones for UnweightedGraph and WeightedGraph) that conditionally conforms to Codable when the vertices (and for WeightedGraph, weights) conform to Codable as well.
You could try that first... If you do let us know if it works or not.
Best,
Dave

That's the path I thought you'd meant and would prefer as well! I also meant if you'd had a chance to think about the nuts and bolts of correlating individual objects once they are serialized into their data representation.
I imagine we'll have to use some sort of hash reference system so that arbitrary vertice objects can refer to arbitrary edge objects and vice versa.

Because SwiftGraph uses a hybrid adjacency list model, I'm not sure any such extra machinery is actually necessary. As long as the Arrays representing all vertices and all edges are encoded and decoded in the same order, I think we should be good.


GH-355 Made user_dn_pattern of artifactory_ldap_setting optional

Fixes: #355

Notes on the fix:
From UI, Artifactory"user_dn_pattern" allows blank input for this "user_dn_pattern" field if the "search_filter" field is specified.
Provider uses "system/configuration" REST API call which allows both the fields to be blank and the call does not validate each field of the ldap config like in UI.
Currently our provider can not enforce rule ie: either "user_dn_pattern" or "search_filter" has to be provided / both can not be empty.