push_transaction RPC API, Invalid packed transaction [error unpacking]

Hi, I am trying to create a transaction and broadcast it to local node. I use push_transaction(signed_transaction) api but I was getting the error below:
{
   "code":500,
   "message":"Internal Service Error",
   "error":{
      "code":3010010,
      "name":"packed_transaction_type_exception",
      "what":"Invalid packed transaction",
      "details":[
         {
            "message":"Missing signatures",
            "file":"abi_serializer.hpp",
            "line_number":393,
            "method":"extract"
         },
         {
            "message":"Failed to deserialize variant",
            "file":"abi_serializer.hpp",
            "line_number":487,
            "method":"from_variant"
         },
         {
            "message":"Invalid packed transaction",
            "file":"chain_plugin.cpp",
            "line_number":455,
            "method":"push_transaction"
         }
      ]
   }
}

The signed transaction I created is like below:
{
   "actions":[
      {
         "account":"eosio.token",
         "name":"transfer",
         "authorization":[
            {
               "actor":"tester",
               "permission":"active"
            }
         ],
         "data":{
            "from":"tester",
            "to":"kris",
            "quantity":"1 EOS"
         }
      }
   ],
   "signatures":[
      "SIG_K1_KeH6wCsnUWRjrfGtqPpXFK4BJArfQ4FwuZARnpVyWkfgdrbrKMgH1msyvrvGyxABYj5UyNpMmipWhpyaruRaRFQbcuDBb4"
   ]
}

I am running node locally on dawn 4.0 version. I read the source code and I think I am giving the correct format of the signed transaction, wondering what the problem is. Thanks.

Likely due to quantity not being "1.0000 EOS" and/or missing "memo" in data section.

I added memo but the error still occurs. I don't think the problem is about the data section. The error message is "Missing signatures". The code below is how I get signature:
let signature = eosjs.modules.ecc.sign('signing tx', privateKey)

Should I convert the signed transaction json to hex? But if i use the abi_json_to_bin rpc api to convert the information of my transaction, signature is not included in the transfer params.

The precision of quantity HAS TO BE 1 / 10000, here should be 1.0000 EOS, not 1 EOS.
Maybe EOSDevHelper will help you.

ok, I use 1.0000 EOS and generate a hex of the transaction, and then put it as a packed_trx, I got the below error
{
   "code":500,
   "message":"Internal Service Error",
   "error":{
      "code":3010010,
      "name":"packed_transaction_type_exception",
      "what":"Invalid packed transaction",
      "details":[
         {
            "message":"Invalid packed transaction",
            "file":"chain_plugin.cpp",
            "line_number":455,
            "method":"push_transaction"
         },
         {
            "message":"read datastream of length 41 over by 0",
            "file":"datastream.cpp",
            "line_number":6,
            "method":"throw_datastream_range_error"
         },
         {
            "message":"error unpacking uint64_t",
            "file":"raw.hpp",
            "line_number":602,
            "method":"unpack"
         },
         {
            "message":"Error unpacking field value",
            "file":"raw.hpp",
            "line_number":361,
            "method":"operator()"
         },
         {
            "message":"error unpacking eosio::chain::name",
            "file":"raw.hpp",
            "line_number":602,
            "method":"unpack"
         },
         {
            "message":"Error unpacking field name",
            "file":"raw.hpp",
            "line_number":361,
            "method":"operator()"
         },
         {
            "message":"error unpacking eosio::chain::action",
            "file":"raw.hpp",
            "line_number":602,
            "method":"unpack"
         },
         {
            "message":"Error unpacking field context_free_actions",
            "file":"raw.hpp",
            "line_number":361,
            "method":"operator()"
         },
         {
            "message":"error unpacking eosio::chain::transaction",
            "file":"raw.hpp",
            "line_number":602,
            "method":"unpack"
         },
         {
            "message":"error unpacking eosio::chain::transaction",
            "file":"raw.hpp",
            "line_number":646,
            "method":"unpack"
         }
      ]
   }
}

The way I do it is as below:
➜  ~ curl  http://<IP_ADDRESS>:8888/v1/chain/abi_json_to_bin -X POST -d '{"code":"eosio.token", "action":"transfer", "args":{"from":"tester", "to":"user", "quantity":"1.0000 EOS", "memo":"transfer"}}'
{"binargs":"000000005c95b1ca00000000007015d6102700000000000004454f5300000000087472616e73666572"}%
➜  ~ curl  http://<IP_ADDRESS>:8888/v1/chain/push_transaction -X POST -d '{
"signatures":[                "SIG_K1_KeH6wCsnUWRjrfGtqPpXFK4BJArfQ4FwuZARnpVyWkfgdrbrKMgH1msyvrvGyxABYj5UyNpMmipWhpyaruRaRFQbcuDBb4"
],
"compression":"none",
"packed_context_free_data":"",
"packed_trx": "000000005c95b1ca00000000007015d6102700000000000004454f5300000000087472616e73666572"
}'

+1 getting the same error

+1

I think fc's unpack buffer is too small:
         {
            "message":"read datastream of length 41 over by 0",
            "file":"datastream.cpp",
            "line_number":6,
            "method":"throw_datastream_range_error"
         },

  // Stream adapter for use with the fc unpack functionality
  template <uint32_t buffer_len>
  class mb_datastream {
     public:
        mb_datastream( message_buffer<buffer_len>& m ) : mb(m) {}

        inline void skip( size_t s ) { mb.advance_read_ptr(s); }
        inline bool read( char* d, size_t s ) {
          if (mb.bytes_to_read() >= s) {
            mb.read(d, s);
            return true;
          }
          fc::detail::throw_datastream_range_error( "read", mb.bytes_to_read(), s - mb.bytes_to_read());
        }

        inline bool   get( unsigned char& c ) { return mb.read(&c, 1); }
        inline bool   get( char& c ) { return mb.read(&c, 1); }

      private:
        message_buffer<buffer_len>& mb;
  };

I'm getting this same error after deploying eosio.token then trying to read it back:
> api error => { message: 'read datastream of length 14 over by -12',
  file: 'datastream.cpp',
  line_number: 6,
  method: 'throw_datastream_range_error' } http://<IP_ADDRESS>:8888/v1/chain/get_code {"account_name":"inita"}

Got the same error when trying to read a contract
api error => { message: 'read datastream of length 14 over by -12',
  file: 'datastream.cpp',
  line_number: 6,
  method: 'throw_datastream_range_error' }

Any progress on this?

I have solve the problem, the struct of unbroadcast transaction has changed in dawn4.1/4.2
Like the struct below:
{
"transaction": transaction struct,
"signatures": xxxx,
"compression": "none"
}
For example:

@kristieguo

Does this problem still exist?

Im still having this issue. Below is the format I'm using on push transaction. Is this correct?
{
  "compression": "none",
  "signatures": [
    "SIG_K1_K2C2qKLbwQ3vGjrYQp89fTNkqHmnbiWPPw6Gg2AxD35o8s3PPrzfFdG7v8v9VpojHFwykt2rZze1BFfP3QQdjppsyQzHsD"
  ],
  "transaction": {
    "ref_block_prefix":<PHONE_NUMBER>,
    "net_usage_words": 0,
    "expiration": "2018-06-29T18:45:44.0000",
    "ref_block_num": 3300265,
    "delay_sec": 0,
    "context_free_actions": [

    ],
    "transaction_extensions": [

    ],
    "actions": [
      {
        "name": "transfer",
        "account": "eosio.token",
        "data": "10b0db269d43917990d45dd351042fe5010000000000000004535953000000000e746573742066726f6d206c796e78",
        "authorization": {
          "actor": "<username>",
          "permission": "active"
        }
      }
    ],
    "context_free_data": [

    ],
    "max_cpu_usage_ms": 0
  }
}

Always fails with:
{
  "code": 500,
  "message": "Internal Service Error",
  "error": {
    "code": 3010011,
    "name": "packed_transaction_type_exception",
    "what": "Invalid packed transaction",
    "details": [

    ]
  }
}

I was told I need to do a packed transaction if Im using signing offline and user the rpc interface. I think I've successfully packed the trx, but im still getting the same message. Its hard to tell by the error message. I guess its saying that the packed_trx is invalid? Is that the correct assumption?
{
  "packed_context_free_data" : [

  ],
  "compression" : "none",
  "signatures" : [
    "SIG_K1_Kida3zYdhCu2JcXzu7S27SgL89y4eJTbfEE6YQum4x2vzBfszbnzozByHxay5SBMHAoQ9f8569Mq1HexyYBboLm1e7uYAL"
  ],
  "packed_trx" : "1ce3365b00000b7320000032f763f0000000010a682343ea3055000572d3ccdcd110b0db269d4391790000a8ed32322f10b0db269d43917990d45dd35142fe5100000004454f530000e746573742066726f6d206c796e78100000000000000000000000000000000"
}

@ismyhc where can i find the "packed_trx"?

I packed the tx via rpc call 'v1/chain/abi_json_to_bin' but still getting the same error 3010010. Full response below
<Response [500]>
{'code': 500, 'message': 'Internal Service Error', 'error': {'code': 3010010, 'name': 'packed_transaction_type_exception', 'what': 'Invalid packed transaction', 'details': [{'message': 'Invalid packed transaction', 'file': 'chain_plugin.cpp', 'line_number': 1457, 'method': 'push_transaction'}, {'message': 'read datastream of length 37 over by -4', 'file': 'datastream.cpp', 'line_number': 6, 'method': 'throw_datastream_range_error'}, {'message': 'error unpacking uint64_t', 'file': 'raw.hpp', 'line_number': 616, 'method': 'unpack'}, {'message': 'Error unpacking field value', 'file': 'raw.hpp', 'line_number': 368, 'method': 'operator()'}, {'message': 'error unpacking eosio::chain::name', 'file': 'raw.hpp', 'line_number': 616, 'method': 'unpack'}, {'message': 'Error unpacking field account', 'file': 'raw.hpp', 'line_number': 368, 'method': 'operator()'}, {'message': 'error unpacking eosio::chain::action', 'file': 'raw.hpp', 'line_number': 616, 'method': 'unpack'}, {'message': 'Error unpacking field context_free_actions', 'file': 'raw.hpp', 'line_number': 368, 'method': 'operator()'}, {'message': 'error unpacking eosio::chain::transaction', 'file': 'raw.hpp', 'line_number': 616, 'method': 'unpack'}, {'message': 'error unpacking eosio::chain::transaction', 'file': 'raw.hpp', 'line_number': 660, 'method':
'unpack'}]}}

resolved.

I packed the tx via rpc call 'v1/chain/abi_json_to_bin' but still getting the same error 3010010. Full response below
<Response [500]>
{'code': 500, 'message': 'Internal Service Error', 'error': {'code': 3010010, 'name': 'packed_transaction_type_exception', 'what': 'Invalid packed transaction', 'details': [{'message': 'Invalid packed transaction', 'file': 'chain_plugin.cpp', 'line_number': 1457, 'method': 'push_transaction'}, {'message': 'read datastream of length 37 over by -4', 'file': 'datastream.cpp', 'line_number': 6, 'method': 'throw_datastream_range_error'}, {'message': 'error unpacking uint64_t', 'file': 'raw.hpp', 'line_number': 616, 'method': 'unpack'}, {'message': 'Error unpacking field value', 'file': 'raw.hpp', 'line_number': 368, 'method': 'operator()'}, {'message': 'error unpacking eosio::chain::name', 'file': 'raw.hpp', 'line_number': 616, 'method': 'unpack'}, {'message': 'Error unpacking field account', 'file': 'raw.hpp', 'line_number': 368, 'method': 'operator()'}, {'message': 'error unpacking eosio::chain::action', 'file': 'raw.hpp', 'line_number': 616, 'method': 'unpack'}, {'message': 'Error unpacking field context_free_actions', 'file': 'raw.hpp', 'line_number': 368, 'method': 'operator()'}, {'message': 'error unpacking eosio::chain::transaction', 'file': 'raw.hpp', 'line_number': 616, 'method': 'unpack'}, {'message': 'error unpacking eosio::chain::transaction', 'file': 'raw.hpp', 'line_number': 660, 'method':
'unpack'}]}}

Resolved. packed transction is not json_to_abi call but a separate action done on the local machine

@silvercondor Could you give me more detail on packed transaction?How could we make those packed_trx with rpc?
from what I know, packing of transaction will not be supported via rpc and has to be done on local machine. You may use a library like eosjs/eosjs2 to pack your transaction or study the library for your own implementation

I have solved this problem.
I do a packed transaction using signing offline, my problem is on the local signature with 'EosByteWriter'. If you need help, you can ask me questions.

I have solved this problem.
I do a packed transaction using signing offline, my problem is on the local signature with 'EosByteWriter'. If you need help, you can ask me questions.

How can I create a packed transaction?

I have solved this problem.
I do a packed transaction using signing offline, my problem is on the local signature with 'EosByteWriter'. If you need help, you can ask me questions.

How can I create a packed transaction?

What language are you using?

I have solved this problem.
I do a packed transaction using signing offline, my problem is on the local signature with 'EosByteWriter'. If you need help, you can ask me questions.

How can I create a packed transaction?

What language are you using?

python

Based on @choiceyou 's comment, it sounds like this can be closed.  If there is still a problem and wish to discuss further, please post it in https://eosio.stackexchange.com. To keep things organized, Github issues are intended for bugs reporting and features tracking, and Stack Exchange is the right place for any technical support discussions. Thanks!

need help kiddos


searching code

Very good work!!!  When will the searching code be released?

@WuJie1010 Same question

Same question

@chengaopro Great work, Hope to open source as soon as possible 👍

Any update on the search code?

impressive, want to view the search code  as keen as mustard


Consideration of Normalization on the Certified radius

Dear authors,
Hi! I went through your code for CIFAR10, and found that you applied a normalization with std=0.2 for it. However, there seems to be no adjustment of the certified radius with regard to it. Specifically, it seems that you certified 8/255 on the normalized data, which is equivalent to 8/255 * 0.2 = 1.6/255 certified radius for the original denormalized data. It could be that I overlooked the adjustment of radius, so could you please give some hints on where you consider this, and if not, why is this not necessary?
Thanks.

We have considered the normalization in our certification. See https://github.com/shizhouxing/Fast-Certified-Robust-Training/blob/281f6d15e0f2ea4208be57746d060b314949c615/certified.py#L68

Thanks. This is very helpful.


Add transactions to file writes and restore files to previous state when transactions fail

Addresses #91

[x] Add missing documentation and labelled params
[x] Update file codec to write to a temp file and move once completed
[ ] Add test cases

@xxfast Thanks for the fix any update on this when it was going to be in production thanks.

Hi @SultanArshad planning this for 0.9.0 release. Stay tuned

Hi @SultanArshad This is ready for review :)


Setting Header (Auth Token) within interceptors fails

When attempting to set the Authentication Header within the interceptor,
// Request Interceptor
axios.interceptors.request.use(function (config) {

  const token = Vue.auth.getToken();
  if(token){
    config.headers.common['Authorization'] = "eyJhbGciOiJkpXVCJ9.eyJhY2N6MTQ4OTA0Mzk5OX0.xRqxyWuFlvaIJl_W7r6OWk";
  }

  console.log("ok",config.baseURL)
  console.log("ok",config.headers.common)
  // Do something before request is sent
  return config;
}, function (error){
  // Do something with request error
  return Promise.reject(error);
})

The Auth Token is not appended to the request header.
Object
Accept : "application/json, text/plain, */*"
__proto__ :Object

However when i directly append the token to the header via
axios.defaults.headers.common['Authorization'] = "eyJhbGciOiJkpXVCJ9.eyJhY2N6MTQ4OTA0Mzk5OX0.xRqxyWuFlvaIJl_W7r6OWk"
it works
Object
Accept : "application/json, text/plain, */*"
Authorization : "eyJhbGciOiJkpXVCJ9.eyJhY2N6MTQ4OTA0Mzk5OX0.xRqxyWuFlvaIJl_W7r6OWk"
__proto__ : Object

Glossed over the if statement which was always evaluating to false in the test environment.


Linear8bitLt backward RuntimeError: size mismatch

Repro code:
import torch
import torch.nn as nn
from bitsandbytes.nn.modules import Linear8bitLt

class Linear(Linear8bitLt):
    def __init__(self, *args, device=None, threshold=6.0, **kwargs):
        super().__init__(*args, device=device, has_fp16_weights=False, **kwargs)

class SimpleNet(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, num_layers):
        super().__init__()
        self.num_layers = num_layers
        self.linears = nn.ModuleList([Linear(input_size, hidden_size)])
        self.nonlinearities = nn.ModuleList([nn.ReLU()])

        if num_layers > 1:
            for _ in range(num_layers - 1):
                self.linears.append(Linear(hidden_size, hidden_size))
                self.nonlinearities.append(nn.ReLU())

        self.output_linear = Linear(hidden_size, output_size)
        self.softmax = nn.Softmax(dim=1)

    def forward(self, x):
        for i in range(self.num_layers):
            x = self.linears[i](x)
            x = self.nonlinearities[i](x)
        x = self.output_linear(x)
        return self.softmax(x)

device = torch.device("cuda")
dtype = torch.float16

input_size = 16
model = SimpleNet(input_size, input_size * 2, 8, 1)
model.to(device)
x = torch.randn(10, input_size, device=device, dtype=dtype)

loss = model(x).sum()
loss.backward()

print(loss)

Traceback (most recent call last):
  File "/home/demo.py", line 43, in <module>
    loss.backward()
  File "/home/venv/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
    torch.autograd.backward(
  File "/home/venv/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/home/venv/lib/python3.10/site-packages/torch/autograd/function.py", line 288, in apply
    return user_fn(self, *args)
  File "/home/venv/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py", line 485, in backward
    .mul_(state.SCB.unsqueeze(1).mul(1.0 / 127.0))
RuntimeError: The size of tensor a (32) must match the size of tensor b (8) at non-singleton dimension 0

The same code works with the 4bit layers.

Hey @carmocca , I have the same error. Did you manage to resolve it?

Unfortunately no

Hi @carmocca, @edbeeching, I noticed the discussion on this closed issue. I'm facing a similar challenge and would greatly appreciate any insights or solutions you might have found. @TimDettmers, your expertise on this matter would also be incredibly valuable. Thanks in advance for any help you can provide!


Use of mulitple Shell DTI for ALPS Method

Dear Dr. Barisano,
Thank you very much for sharing this great code. I am still new to the topic of DTI ALPS. In the 2024 paper "Diffusion Tensor Image Analysis Along the Perivascular Space (DTI-ALPS): Revisiting the Meaning and Significance of the Method" by Taoka et al., it is stated that "The use of a single b-value instead of multiple shells makes it impossible to separate the velocity components of diffusion, and the information is considered to be a mixture of diffusion components of different velocities. This issue needs to be verified in the future."
As part of a study, we acquired a multiple shell DTI sequence with various b values  (5 2005 995 2000 1995 995 2000 2005 1000).
1.I successfully processed a subject yesterday, and the ALPS indices are about 1.5 on both the right and left sides. I wanted to ask if the script is designed to also process multiple-shell DTI, or would I need to make some adjustments?
2. What is your assessment of using multiple shell DTI for ALPS? Does it make sense to calculate DTI ALPS and ALPS Index this way?
Best regards,
Marcel

Hi Marcel,
Thank you for your message! And sorry for the delay in my response.
Here my answers to your questions:

No, I did not design the script to specifically process multi-shell DTI data. The tensor model which is fitted by my script is generally not a good model for multi-shell data: as the b-value increases, the diffusion signal becomes sensitive to more complex microstructural features of tissue, such as cell membranes, myelin, and other barriers that restrict or hinder water diffusion. These structures lead to non-Gaussian diffusion patterns, which the tensor model cannot capture accurately. The DTI model typically works with single-shell data, having b0 value and typically b-value = 1000s/mm^2 (maximum b-value 1200s/mm^2, less frequently up to 1500s/mm^2)
Despite the point above, even though by a theoretical point of view it might seem inappropriate to calculate DTI-ALPS on multi-shell data with multiple b-values above 1200s/mm^2, I did some tests on multi-shell data with my script and the ALPS results were reasonable. I also tried to re-run the script using only the b0 images and the b values close to 1000 or below 1000, but the DTI model did not fit very well.
Based on these tests, I think that as long as the FA maps look good, the ALPS results should be acceptable, but the actual values likely will not be directly comparable with DTI-ALPS values obtained from traditional single-shell DTI, so you would need to make sure that all the diffusion data included in your study have been acquired with the same multi-shell protocol.
I will try to run some additional tests and assess more systematically DTI-ALPS on multi-shell data.

I hope this helps. Let me know if you have any question or suggestion.
Giuseppe

One thing to keep in mind is that FSL's eddy in my script likely will not work for multi-shell data.


Crafty 4 Icon

crafty-controller

Sorry for the delay, I added this in 0.3.4.
https://github.com/loganmarchione/homelab-svg-assets/commit/7644aad45061f9bc31d53e800fdbd61dd4b9d3ba


METRON-1567: Large error message can't be written in Solr

Contributor Comments
This PR is against the feature branch.
There's a hard limit of ~32kb on string fields.  This migrates the raw_message fields to an unanalyzed TextField type, along with making it a dynamic field to handle the split into raw_messages_.
An integration test is added for this behavior which ensures no error is thrown on huge text, ensures the multiple values work as expected the say way, and that a plain string field does actually cause problems.
Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
For all changes:

[x] Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
[x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

[x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?

[x] Have you included steps or a guide to how the change may be verified and tested manually?

[ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh

[x] Have you written or updated unit tests and or integration tests to verify your changes?

[ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

+1 by inspection

Closing manually because feature branch


how to make datatable show custom field from joined table

i have payments datatable with field  'id', 'transaction_id',  'date','amount', 'payment_method_id', 'additional_note'
and payment_methods table with field 'id', 'name'
my question is in payments datatable how to show payment_methods.name instead of payment_method_id?
here the generated datatables https://pastebin.com/P2aVCmKD
nb: probably same with #412  but for me still fail to load joined field

@unitedlover0 I think this is what you are looking for https://github.com/InfyOmLabs/laravel-generator/issues/548


Segmentation fault

Hello,
I'm usign GPAT++ pFST function to get Fst Pvalues across populations.
I'have followed the instructions for installation and then used the program but an error ocurrs:
INFO: There are 11 individuals in the target
INFO: target ids: 0,1,2,3,4,5,6,7,8,9,10
INFO: There are 14 individuals in the background
INFO: background ids: 406,407,408,409,410,411,414,415,416,417,418,419,420,421
INFO: File: myvcf.vcf.gz
INFO: genotype likelihoods set to: PL
Segmentation fault (core dumped)
My SO is Ubuntu 14.10 64-bits
Hope you can help me
Best regards,
--Bernabé

Bernabé,
Happy to help.  I will need some additional information from you.  Can you show me the command line you ran?  Does the code produce any output besides the SDTERR?

Hi, Thanks for the quick answer:
This is the command line I'm using:
$ vcflib/bin/pFst --target 0,1,2,3,4,5,6,7,8,9,10 --background 406,407,408,409,410,411,414,415,416,417,418,419,420,421 --file myvcf.vcf.gz --type PL
The program do not create any kind of output, just stopped with the segmentation fault error.

Can you provide me a line of your VCF file or a minimum test example that generates the error? What variant caller did you use?
Thank you for reporting this issue by the way.

I think I've found the problem, I'm using a VCF file generated by PLINK1.9 recode option, using bim, bed and fam files. The likelihoods flags GP, GL or PL are not present. I tried to use just genotipe counts with the --counts options but the program still requires a Genotype likelihood parameter.
There is an example line of m vcf:
##fileformat=VCFv4.2
##fileDate=20150410
##source=PLINKv1.90
##contig=<ID=1,length=249216385>
##contig=<ID=2,length=243083604>
##contig=<ID=3,length=197875093>
##contig=<ID=4,length=190939083>
##contig=<ID=5,length=180715588>
##contig=<ID=6,length=171047509>
##contig=<ID=7,length=159122936>
##contig=<ID=8,length=146300230>
##contig=<ID=9,length=141102860>
##contig=<ID=10,length=135503941>
##contig=<ID=11,length=134945711>
##contig=<ID=12,length=133839028>
##contig=<ID=13,length=115107494>
##contig=<ID=14,length=107288378>
##contig=<ID=15,length=102471413>
##contig=<ID=16,length=90265726>
##contig=<ID=17,length=81151921>
##contig=<ID=18,length=78017074>
##contig=<ID=19,length=59097309>
##contig=<ID=20,length=62963204>
##contig=<ID=21,length=48099783>
##contig=<ID=22,length=51222053>
##contig=<ID=23,length=154930231>
##INFO=<ID=PR,Number=0,Type=Flag,Description="Provisional reference allele, may not be based on real reference genome"
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
#CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT  I1	I2	I3	I4
1       57952   rs189727433     C       A       .       .       PR      GT      0/0     0/0     0/0
1       69511   rs75062661      G       A       .       .       PR      GT      0/0     0/0     0/0
1       86018   rs142878000     C       G       .       .       PR      GT      ./.     0/0     0/0
1       356537  rs183318141     G       A       .       .       PR      GT      1/1     1/1     1/1
1       713977  rs74512038      C       T       .       .       PR      GT      0/0     0/0     0/0
1       752721  rs3131972       G       A       .       .       PR      GT      0/0     0/0     0/0
1       754334  rs3131967       C       T       .       .       PR      GT      0/0     0/0     0/0
1       754503  rs3115859       A       G       .       .       PR      GT      0/0     0/0     0/0
1       754894  rs184597775     G       A       .       .       PR      GT      0/0     0/1     0/0
1       754964  rs3131966       T       C       .       .       PR      GT      0/0     0/0     0/0
1       755274  rs78408995      C       T       .       .       PR      GT      0/0     0/0     0/0
I've calculated pairwise Weir and Cockerham Fst values for each SNP in PLINK in order to detect divergent locus. In this paper (http://dx.doi.org/10.1038/ng.78) they use a Fst cutoff of 0.65 to detetec significantly divergent SNPs, but reading other articles, is more common to do a statistical approach and calculate p values according to empirical Fst distributions to detect outliers, that's why I wanted to use your program. But I think maybe I'm not gonna be able to use it with this kind of input. Do you know other options to do this?.
Thanks again.

Ahh. The option "--type GT" should work for you.  pFst is a test for allele frequency differentiation between two populations.  If you want a strictly Fst based method try "wcFst" and bootstrap the output.

Thanks again, I really appreiciate your help!.
I'm running the program with that output, it's giving output already. So pFst is not an actual Fixation index statistic like WC ? I'ts like a chi-square statistic?. And the other thing, I'm introducing myself to this kind of analysis, when you say boostrap the output is to manually do permutations? is there a way to do this by a software?.

You are correct, it is not a fixation index as it does not use heterozygosity.  A good paper that describes the gist of pFst is :
"Design of association studies with pooled or un-pooled next-generation sequencing data."
So here is my typical workflow:
run wcFst.  plot the output using the GPAT++ R scripts.  run permutations to assign an emperical p-value to the Weir and Cockerham 1984 Fst values.  I haven't pushed the the permutation code into GPAT++, but if you are interested I can do that next week.
If you have a candidate region you can also run "bFst", which will give you a confidence interval around the Fst estimate.  WARNING: the code is very slow as it has to do MCMC.

That's very clear, thanks a lot.
It would be great if you can push the permutation code to GPAT++, It will be very useful to detect truly FST outliers, thus making a more accurate biological interpretation of the results.
I'll definitely look forward fot that upgrade.
Thanks again, have a great weekend!
Cheers
Bernabé

Hello!
I wanted to know if you have any news about implementing the boostraping option to GPAT++ as you said last week?
Cheers!
Bernabé

I'm almost finished.  The code will be ready by Monday.  Thank you for reminding me.

Thank you, I'll look forward to it!

Hello, I was reading about the outlier methods for detecting highly divergent SNPs and I've noticed the 95th percentile method following an empirical distribution is one of the most used one. Is this the method you are applying to GPAT?

Greetings,
The code is in place: https://github.com/jewmanchue/vcflib/wiki/permutation
You will need to re-install GPAT.
The method I implemented reports the "percentile - or - empirical p-value".

Thank you very much!, I'll will try it right away. So the input file is the output file from wcFST right?

Yes.  I will update the code and documents after I defend my thesis this Thursday.

Thanks again and good luck!

I'm downloading and reinstalling everything but this error appears:
make: *** No rule to make target 'src/wham-burden.cpp', needed by 'bin/vcfecho'.  Stop.
That file "wham-burden.cpp" is no more present to complie?

sorry.  Try now.

Thanks, but this appears now
Makefile:172: recipe for target 'bin/iHS' failed
make: *** [bin/iHS] Error 1

I'm not able to replicate this bug. Please try a clean install:
rm -rf vcflib
git clone --recursive https://github.com/jewmanchue/vcflib.git vcflib
cd vcflib
make

Yes, I did it again, this is the whole Error output...
src/iHS.cpp:126:18: error: call of overloaded ‘isnan(double&)’ is ambiguous
if(isnan(iHSA) || isnan(iHSR)){
^
src/iHS.cpp:126:18: note: candidates are:
In file included from /usr/include/features.h:374:0,
from /usr/include/x86_64-linux-gnu/c++/4.9/bits/os_defines.h:39,
from /usr/include/x86_64-linux-gnu/c++/4.9/bits/c++config.h:430,
from /usr/include/c++/4.9/bits/stl_algobase.h:59,
from /usr/include/c++/4.9/vector:60,
from src/Variant.h:4,
from src/iHS.cpp:1:
/usr/include/x86_64-linux-gnu/bits/mathcalls.h:234:1: note: int isnan(double)
__MATHDECL_1 (int,isnan,, (Mdouble __value)) attribute ((const));
^
In file included from /usr/include/c++/4.9/random:38:0,
from /usr/include/c++/4.9/bits/stl_algo.h:66,
from /usr/include/c++/4.9/algorithm:62,
from ./smithwaterman/SmithWatermanGotoh.h:4,
from src/Variant.h:19,
from src/iHS.cpp:1:
/usr/include/c++/4.9/cmath:626:3: note: constexpr bool std::isnan(long double)
isnan(long double __x)
^
/usr/include/c++/4.9/cmath:622:3: note: constexpr bool std::isnan(double)
isnan(double __x)
^
/usr/include/c++/4.9/cmath:618:3: note: constexpr bool std::isnan(float)
isnan(float __x)
^
src/iHS.cpp:126:33: error: call of overloaded ‘isnan(double&)’ is ambiguous
if(isnan(iHSA) || isnan(iHSR)){
^
src/iHS.cpp:126:33: note: candidates are:
In file included from /usr/include/features.h:374:0,
from /usr/include/x86_64-linux-gnu/c++/4.9/bits/os_defines.h:39,
from /usr/include/x86_64-linux-gnu/c++/4.9/bits/c++config.h:430,
from /usr/include/c++/4.9/bits/stl_algobase.h:59,
from /usr/include/c++/4.9/vector:60,
from src/Variant.h:4,
from src/iHS.cpp:1:
/usr/include/x86_64-linux-gnu/bits/mathcalls.h:234:1: note: int isnan(double)
__MATHDECL_1 (int,isnan,, (Mdouble __value)) attribute ((const));
^
In file included from /usr/include/c++/4.9/random:38:0,
from /usr/include/c++/4.9/bits/stl_algo.h:66,
from /usr/include/c++/4.9/algorithm:62,
from ./smithwaterman/SmithWatermanGotoh.h:4,
from src/Variant.h:19,
from src/iHS.cpp:1:
/usr/include/c++/4.9/cmath:626:3: note: constexpr bool std::isnan(long double)
isnan(long double __x)
^
/usr/include/c++/4.9/cmath:622:3: note: constexpr bool std::isnan(double)
isnan(double __x)
^
/usr/include/c++/4.9/cmath:618:3: note: constexpr bool std::isnan(float)
isnan(float __x)
^
Makefile:172: recipe for target 'bin/iHS' failed
make: *** [bin/iHS] Error 1

"should" be fixed.

Thanks again!, Its working now.
One last thing, I performed 1000 permutations on my wcFST output, Just to make sure that I'm interpreting right the results:
1   12354   0.0454545 0.142857  0.000859343   257   1000   0.257
1   12356   0.3636   0.3928   -0.0439   10   1000   0.01
The first result, column 5 shows a Fst value of 0.000085 then 1000 permutations with 257 successes, then yielding an empirical p value of 0.257. The second SNP has an empirical p-value of 0.031. When in papers they say:
"We calculated the global FST for each single SNP among the three populations. For autosomal SNPs, we considered a candidate selection if the SNP FST >= 0.58, which corresponds to a genome-wide empirical significance level of alpha 0.01."
Then, it means that if I want to emulate this in my data, I have to select all values from 0.01 and below?. How this empirical p value is matched with a Fst value like the example?.

yes, you want empirical p-values below 0.01 or the top 1% of all Fst values.

Ok, filtering all empirical p values by 0.01 and below wild yield a range of FST values from low to high, not only from a certain threshold like the example citation. Maybe I'm getting this wrong. I need to apply two filters?, first by the empirical p value and then the upper 95th percentile of the filtered Fst to see the mos divergent ones?.

I was doing the inverse permutation.  The code is updated and here is a plot vetting the code:

Please let me know if the permutation test is working as you desired.
--Zev

Thank you very much Zev for implementing the permutation tool.
I ran the tool several times now and I was thinking about the method of getting the empirical probability. The plots I'm having follows almost the same distribution shape of the plot you showed above. My question is that there is no high Fst values with poor pvalues, I mean, Fst values > 0.6 and I was thinking there sould be some snps with high Fst values but with low empirical palue not passing the 95th percentile treshold which I'm currently applying to get the significant results. I'm I correct with that thinking?.
Just to make myself clear with example:
Freq1 Freq2 S N P
0.8 0.1  321 10000 0.0321
The example says that doing 10.000 permutations on the frequency values of Freq1 and Freq2 I got 321 times the same Fst number as obtained with the fixed Freq values?.

  Just drop the image in the text box.

Maybe I'm just having troubles interpreting the results?

Would you like to arrange a video call?  It might be easiest.

Off course, we can use google talk, my email is<EMAIL_ADDRESS>

Hello Zev.
How can I cite your tools in a publication?

For now please cite the github page.  If you tell me which tools you used I can give you the appropriate citations.

I have used wcFST and permuteGPAT++. I would like to cite both, Thanks.

G'day Zev and Bibb,
I am currently implementing wcFst and then running the new permute function and everything is working well. This conversation has been extremely helpful for me to be able to use both wcFst and permute, however, I am having an issue interpreting the results as bibb expressed a couple of weeks ago.
I get extremely similar plots to the ones Bibb posted and am wondering whether it is appropriate to simply take those points that fall below an empirical p-value of 0.01 as being significantly divergent SNPs (or -log10(p) >= 2).
I ran 10,000 permutations in wcFst and have plotted the wcFst results from several different data treatments.

Thank you for your help.
Regards,
Adam

Adam,
I'm happy to hear that wcFst and the permutation are working for you.  That is really nice graph.  The catch with Fst is it is only a random variable.  Permutations will give you the empirical probability of each Fst value within a single dataset.  Comparing empirical probabilities across multiple treatments is not valid.  Have a look at (http://en.wikipedia.org/wiki/Empirical_probability)
I'm not sure what system you are working in, but I usually like to see a correlation between Fst values and genomic positions.  Do you see regions with elevated Fst values?  If there is a lot of noise, smoothing might be helpful (GPAT++ : smoother).   I could also build a permutation test for regions if that would be useful.
--Zev

bib - here are the citations:
Weir, B.S., and Cockerham, C.C. (1984). Estimating F-statistics for the analysis of population structure. Evolution (N. Y). 1358–1370.
how to cite a github repo:
http://academia.stackexchange.com/questions/14010/how-do-you-cite-a-github-repository

G'day Zev,
Thanks for the response and advice, I hadn't intending to compare probabilities across treatments, it was just easier to include everything in the same graph. If I am understanding correctly, the empirical probability is simply an indication of how likely you are to get the same Fst value (which is subject to variation due to chance) for a given SNP. So I can be more confident that the Fst value of SNPs with very low empirical probabilities.
I am using genotype-by-sequencing SNP data with no reference genome so unfortunately I don't have any information about genomic position. It's not ideal...
Thanks again,
Adam

Hi Zev,
I'm coming back to this analysis after a while. I'm having troubles understanding the permutation procedure of permuteGPAT++ (I'm not an statistician). I know that an empirical probability is the chance of finding a value within the data (as the software documentation says), so then what is being used in the permutation? I know that the software is not permuting individuals (e.g. target-background population individuals), so what is used?. I would really appreciate your help in this, I'll look forward to your answer, what do you mean with number of trials and successes?
Cheers
Bernabe

@bibb lets say you've run wcFst and then permuteGPAT++.  The output is the empirical probability of observing a value greater than the input.
example:
observed Fst value : 0.7
random sample from the same dataset: 0.1
random sample from the same dataset: 0.1
random sample from the same dataset: 0.1


Remove surrounding / from file filter

This placeholder is confusing since you're not actually supposed to
surround the regular expression with /.
Originally reported at https://phabricator.wikimedia.org/T188220.
Signed-off-by: Kunal Mehta<EMAIL_ADDRESS>

Thanks!


codecov.yml: move to .github

Codecov can't find its config file reporting:
warning - 2024-05-10 11:55:52,652 -- No config file could be found. Ignoring config.
debug - 2024-05-10 11:55:52,652 -- No codecov_yaml found

Stamp once more please.
@a-nogikh PTAL


Add support for RFC 8017

Add module and test for RFC 8017.

Codecov Report

Merging #76 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master      #76      +/-   ##
==========================================
+ Coverage   99.31%   99.32%   +<.01%
==========================================
  Files          72       73       +1
  Lines        5434     5500      +66
==========================================
+ Hits         5397     5463      +66
  Misses         37       37

Impacted Files
Coverage Δ

pyasn1_modules/rfc8017.py
100% <100%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e2f8882...7da5cc0. Read the comment docs.

Awesome, thank you!


Intune FullDisk Access mobileconfig does not enable Setting

I have installed the macos/mobileconfig/profiles/fulldisk.mobileconfig and deployed it via Intune.
Although it gets transfered to the device, the mac that has the profile installed shows both Defender and Extension with no FullDisk Access:

Is this profile still operational?
Is Defender for Endpoint able to access full disk for scanning and protection with these two settings disabled?
macOS 13.4 (22F66) Ventura on m1 max

The full disk access profile is operational.
See [https://community.jamf.com/t5/jamf-pro/macos-ventura-full-disk-access/td-p/276764](for explanation)


Apply missing header fields to drafts

X-CN-DRAFT-INFO, as well as the logic to mark a message answered if a draft with this info was sent; header field should be removedwhen message was sent
Message-ID (in client's draft "messageId" field) to make sure references is updated properly when replying to messages
"references" must be set
"in-reply-to" must be set

confident that header fields are treated properly since recent API changes


ability to skip paid tests on travis-ci

I haven't looked into this, so I don't know if there's an easy solution, but the cost of testing all the API services on every PR/merge is likely to get quite high over time. It's not really a concern for most of the APIs, which are in the $1/1,000 requests ballpark, but some are much more expensive--particularly Indico, which costs $6/1,000 requests. Although our usage will probably come down once we figure out the problem discussed in #272, aggregate costs will continue to rise as we add more APIs.
What we should probably do is add some triggering condition that allows us to only test paid APIs on request, or at least at regular intervals. It would be simple enough to probabilistically trigger the API tests within the travis config (e.g., randomly run them 10% of the time), but it would be nice to have something a little more predictable than that. The best-case scenario would be if there's an easy way to manually trigger only the API tests directly from a GitHub PR.

Note that this is partly independent of the issue raised in #237 (and solution in #270), as we'll still need some way to trigger/skip the API tests using the new py.test markers once #270 is merged.


Add test to ensure S3 truncated response is an error

[ ] I added CHANGELOG entry for this change.
[x] Change is not relevant to the end user.

Changes
From time to time we hear issues which looks be caused by a truncated response got from S3 or S3-compatible backends. An example is https://github.com/thanos-io/thanos/issues/2805.
I've added a unit test to S3 and GCS to check how the client behaves if the HTTP response body contains less data than the size reported in the Content-Length. Surprise surprise... GCS client returns io.ErrUnexpectedEOF while S3 client (Minio) returns no error.
I know this is not something we should fix in Thanos (I guess should be fixed in Minio) but this test should prove it.
I've tried to upgrade Minio to 7.0.8 (latest minor available) and the behaviour doesn't change. I've opened a Minio issue to hear some feedback from maintainers.
Verification
Unit test.

This sounds like an annoying bug, we cannot easily mitigate on our side. Let's see if we can have minio maintainers helping us this asap, otherwise, we can try our own investigation ):

@bwplotka Why did Thanos pick the Minio client instead of the official AWS SDK? I'm sure there's a good reason but I don't know which one.

This sounds like an annoying bug, we cannot easily mitigate on our side

Not sure about this. I think we can. We could wrap the reader returned by Get(), initialise the wrapper with the expected size (the GetObject() also include the object size) and count the read byte. If once we hit the EOF the number of read bytes != expected, then we return a io. ErrUnexpectedEOF.
WDYT?

So I tested with master branch and latest release with minio-go the test seems to return appropriate error for truncated body.
package minio

import (
        "context"
        "io"
        "io/ioutil"
        "net/http"
        "net/http/httptest"
        "testing"
)

func TestGetObjectReturnErrorIfServerTruncatesResponse(t *testing.T) {
        srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
                w.Header().Set("Last-Modified", "Wed, 21 Oct 2015 07:28:00 GMT")
                w.Header().Set("Content-Length", "100")

                // Write less bytes than the content length.
                w.Write([]byte("12345"))
        }))
        defer srv.Close()

        // New - instantiate minio client with options
        clnt, err := New(srv.Listener.Addr().String(), &Options{})

        obj, err := clnt.GetObject(context.Background(), "bucketName", "objectName", GetObjectOptions{})
        if err != nil {
                t.Fatal(err)
        }

        // We expect an error when reading back.
        if _, err = ioutil.ReadAll(obj); err != io.ErrUnexpectedEOF {
                t.Fatalf("Expected %v, got %v", io.ErrUnexpectedEOF, err)
        }
}

~  go test -v -run TestGetObjectReturnErrorIfServerTruncatesResponse
=== RUN   TestGetObjectReturnErrorIfServerTruncatesResponse
--- PASS: TestGetObjectReturnErrorIfServerTruncatesResponse (0.00s)
PASS
ok      github.com/minio/minio-go/v7    0.004s

My bad reproduced it, taking a look what can be done.

My bad reproduced it, taking a look what can be done.

@harshavardhana I upgraded to Minio 7.0.9, which was supposed to fix it. Unfortunately the unit tests I've added is still failing. I think the reason is because the fix done Minio affects only the first call to Read() while in Thanos we call Read() multiple times and only the last call has the opportunity to check if the total bytes read match the expected one. Am I correct / wrong?

@harshavardhana I upgraded to Minio 7.0.9, which was supposed to fix it. Unfortunately the unit tests I've added is still failing. I think the reason is because the fix done Minio affects only the first call to Read() while in Thanos we call Read() multiple times and only the last call has the opportunity to check if the total bytes read match the expected one. Am I correct / wrong?

Yes it is possible to fix this situation as well @pracucci - it may be slightly more involved. I will look into it.

@harshavardhana do I understand it correctly that this should have been fixed by https://github.com/minio/minio-go/pull/1453 and released in minio-go 7.0.10 ?

Yes it is possible to fix this situation as well @pracucci - it may be slightly more involved. I will look into it.

Thanks @harshavardhana for your hard work. Looks like Minio 7.0.10 is solving the issue 🤞

@bwplotka I've re-asked you the review because the purpose of the PR has changed from "add a test" to "fix it".


suwayomi-server: 0.7.0 -> 1.0.0

Description of changes
Updates Suwayomi-Server to v1.0.0.
Changes: https://github.com/Suwayomi/Suwayomi-Server/releases/tag/v1.0.0.
Adds also the option settings.server.extensionRepos in the freeform as it is now a key feature of this new version.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[x] (Package updates) Added a release notes entry if the change is major or breaking
[x] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/3667


Light adds support for EFFECT_OFF in HA 2024.2

HA 2024.2 now supports explicitly setting light effect to off.  We'll need to look at how to best handle this change.
https://developers.home-assistant.io/blog/2024/01/23/light-color-mode-changes

When effect is a separate dp, it should have an explicit "off" value, but if it is combined with color_mode (as is the case for many tuya bulbs supporting scene and music), we need a way to switch back out of the effect mode, so implicitly include EFFECT_OFF in that case.
Until now, it was a bit awkward to shift out of those modes, moreso when only one real color mode is supported, as the HA UI doesn't let you change to the same value it thinks it is, so you are forced to use a service call.


Add Python Type Hints

Justification:
-PEP 484 provided semantics for type hints/annotations in Python 3
-Implementing type hints for functions/methods will help greatly with code clarity and readability

can you assign this to me

@mgear2  Hey thanks for assigning this issue to me .I was going through the code and couldn't find the exact code where i have to add hints/annotations  .Could you please guide me in this .Thanks

Following #10 type hints are implemented for all applicable classes.


Updated API calls in plugin_loader.py to match IDA 7.4 API

Changed the idc.GetIdbPath calls in plugin_loader.py to .idc.get_idb_path to make The Sark Way of loading plugins work with IDA 7.4.

I recommend using https://github.com/tmr232/ida-plugin-loader instead, but this is a good pull request, thanks :)


How to set ENVIRONMENT VARIABLES per IIS AppPool

Fixes #2030
This should knock the last bits out on that one. The link to the AppCmd.exe example is relative now, too, so I go back to iis.md and fix that there.
cc/ @Rick-Anderson @tdykstra

@guardrex,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by .NET Foundation. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot

@tdykstra I try next with links that go back to the normal 'linking to title' approach. I mark the section in italics and leave the anchors on the links. ... Better? Worse?

Better

Yeah, I think so, too.


[Performance] replace Collection by FieldAccess[] array

This is a litte bit faster, because it does not use an iterator in for( ...) loops

So, what are the gains from these performance optimisations? Did you
measure them?
It's a bit weird to see pull requests like this without any rationale or
measurements, honestly. If there is a measurable gain surely you can inform
the maintainers about it, right? Nor do these PR's give them any background
into the why of it.
Op 8 jul. 2018 22:46 schreef "Roland Praml"<EMAIL_ADDRESS>This is a litte bit faster, because it does not use an iterator
You can view, comment on, or merge this pull request online at:
https://github.com/rzwitserloot/lombok/pull/1764
Commit Summary

replace Collection by FieldAccess[] array (saves iterator
instance)

File Changes

M src/core/lombok/core/AST.java
https://github.com/rzwitserloot/lombok/pull/1764/files#diff-0 (12)

Patch Links:

https://github.com/rzwitserloot/lombok/pull/1764.patch
https://github.com/rzwitserloot/lombok/pull/1764.diff

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/rzwitserloot/lombok/pull/1764, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKCRT0eNeE7ujZEgVArs_8kkLicJWqUks5uEm-3gaJpZM4VG3cG
.

Is that 179 ms per call or 176 ms across 160.000 calls?
That's quite a lot of calls. What were you compiling?
Would be nice to have a reproduction path to test performance patches
against.
On Mon, Jul 9, 2018 at 2:24 AM, Roland Praml<EMAIL_ADDRESS>wrote:

jprofiler showed a lot of Iterator instances, that were created by the
EclipseNode.drill method.
Thsi concrete change will mainly reduce the creation of Iterator
instances. It will also reduce the self time of the 'drill' method from
678ms to 399ms - 160k calls
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rzwitserloot/lombok/pull/1764#issuecomment-403329293,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKCRYXjSS-fj_xvNfSk-lELDVNfUD_rks5uEqK4gaJpZM4VG3cG
.

--
"Don't only practice your art, but force your way into it's secrets, for it
and knowledge can raise men to the divine."
-- Ludwig von Beethoven

Good work.
I think not detecting FQN's is not very safe. If a file is not parsed when
it should the compilation will break, which is not an option for a 'small'
performance optimisation.
So we need that alternative.
The second avenue of attack sounds promising. I don't know enough about the
internals to be sure it won't break things, but the only way to be
completely sure about that anyway is to try it.
On Mon, Jul 9, 2018 at 1:43 PM, Roland Praml<EMAIL_ADDRESS>wrote:

The performance improvements are for all 160k calls (and are probably not
measurable in the complete result)
I have compiled a test project https://github.com/ebean-orm/ebean with
~1200 classes that uses NO lombok. (I deliberately choiced this project, as
I do a lot of things in eclipse, which has the lombok agent by default and
tries to enhance every class, regardless if the project uses lombok or not)
To get a reproduceable result, I compiled the project with javac and ecj.
This are the results that 'mvn clean compile' takes
javac, no lombok: ~11.0s
ecj, no lombok: ~8.5s (ecj is faster than javac, probably because it uses
parallel threads)
javac and lombok 1.18.0: ~18 sec (build gets ~7 sec slower, this are ~70%)
ecj and lombok 1.18.0: ~13 sec (build gets ~4.5 sec slower, this are ~ 50%)
ecj and lombok 1.18.1: ~12.5 sec (so all performance improvements will
gain only 5%)
the same, but using the "-Dlombok.checkImport" switch, so only classes,
that have at least one lombok import will get enhanced: ~9sec (build is
slowed down only about .5 seconds)
I have also attached two screenshots of jprofiler:
ecj and lombok 1.18.0
[image: grafik]
https://user-images.githubusercontent.com/5530228/42447375-ec74abfc-8379-11e8-93d6-3ec4e1a217af.png
ecj and lombok 1.18.1
[image: grafik]
https://user-images.githubusercontent.com/5530228/42447400-02c1cc1e-837a-11e8-9395-b2f5152713e1.png
All in all, I see the following optimization potential:

Pre-check the file, if it should be enhanced. I currently do that
by checking the imports, but this does not detect FQNs. Alternative could
be to check the parser.scanner.source array for the occurence of the
"lombok." string.
Reduce the number of AST-traverses. In TransformEclipseAST there is
this method

public void go() {
for (Long d : handlers.getPriorities()) {
ast.traverse(new AnnotationVisitor(d));
handlers.callASTVisitors(ast, d, ast.isCompleteParse());
}
}
which traverses the complete AST tree for each priority. And then the
method
public void handleAnnotation(CompilationUnitDeclaration ast, EclipseNode annotationNode, org.eclipse.jdt.internal.compiler.ast.Annotation annotation, long priority) {
TypeResolver resolver = new TypeResolver(annotationNode.getImportList());
TypeReference rawType = annotation.type;
if (rawType == null) return;
  String fqn = resolver.typeRefToFullyQualifiedName(annotationNode, typeLibrary, toQualifiedName(annotation.type.getTypeName()));
  if (fqn == null) return;
  AnnotationHandlerContainer<?> container = annotationHandlers.get(fqn);
  if (container == null) return;
  if (priority != container.getPriority()) return;

which resolves each type for each AST run (there are > 3.7M TypeResolver
calls).
One strategy could be, if no handler was found, or a handler with a
certain priority was found, that all other priorities are skipped. e.g
change the method to this:
public void handleAnnotation(..., Set interestingPriorities) {
...
if (priority != container.getPriority()) {
interestingPriority.add(container.getPriority())
}
...
}
interestingPriority.add(handlers.getPriorities()[0]);for (Long d : handlers.getPriorities()) {
if (!interstingPriority.contains(d)) continue;
ast.traverse(new AnnotationVisitor(d));
handlers.callASTVisitors(ast, d, ast.isCompleteParse());
}
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rzwitserloot/lombok/pull/1764#issuecomment-403450989,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKCRWGoUAdP9fNkYNHS5hb9bvKeloqPks5uE0HGgaJpZM4VG3cG
.

--
"Don't only practice your art, but force your way into it's secrets, for it
and knowledge can raise men to the divine."
-- Ludwig von Beethoven

closing this, because it is included in #1771


#148 Manipulations with DICOM vue

Manipulations over DICOM such as scroll, window width / window centre, etc. series were requested by both #148 and #149. Were implemented by cornerstoneTools. It took a while to distinguish sustainable versions, compatible with this project:
    "cornerstone-core": " 0.13.0",
    "cornerstone-tools": " 0.10.0",
    "jquery": "^3.2.1"

I'd like to separate them from drawing features and made them part of OpenDICOM, still any suggestions are highly appreciated!
Description
In this PR scroll, window width / window centre DICOM manipulators were implemented.
Reference to official issue
This partially addresses #148
Screanshots
Scrolling:

WWWC:

P.S. there are errors in cornerstone-tools' README should be cornerstoneTools.external not in plural.
CLA

[x] I have signed the CLA; if other committers are in the commit history, they have signed the CLA as well

there are errors in cornerstone-tools' README should be cornerstoneTools.external not in plural.

Can you link to your PR that fixes this for an extra point? :)

Otherwise LGTM, ready to merge ^

@vessemer Hey, any update on this? :)

Hey, @lamby, was unavailable for a while, there already exists PR for it

@vessemer No problem. Should we therefore close this one then..?

Well, I guess that for now, it's better to take cornerstone-tools as is and merely restrict jquery usage by this case only. It'll be beneficial to end up with some minimal version of jquery required by cornerstone-tools or even get rid of it at all, but, perhaps it'll be an issue for the packaging milestone, isn't it?
@louisgv do you mind creating an issue for this?

(jquery slim?)

@vessemer issue created

Neat!
@louisgv What do you think here? :)

@lamby Jquery-Slim is still pretty sizable, according to their release note, it's "23.6k vs 30k.".
It would be best if we can use this pipeline: https://github.com/jquery/jquery/blob/master/README.md#how-to-build-your-own-jquery
And create an even slimmer version of jquery for cornerstone-tools. For now I think definitely let's use jquery-slim, restrict its usage, and have #215 tracking progress on the final jquery for corner-stone.
I haven't look closely at the module list of jquery-slim and whether if it has exactly what cornerstone need. But for now, I think we should be good to merge.

Pinging @vessemer To get his input :)

Guys @lamby, @louisgv, these changes are very important for #148 and #150. In fact, I can't proceed with #148 without it. Can we merge the current request and create a new issue for "slimmer version of jquery"?
P.S. great job, @vessemer!

Sure. I'd love to get another pair of eyes on this though.. @isms ? :)

Merged. This is fgreat - thanks! :confetti_ball:

Nice work! Awesome to see Cornerstone added to this project!
Sorry about the issues with the README. For what it's worth, we are working towards dropping jQuery (though it will be slow to remove it from Cornerstone Tools since we are using it all over the place for events).


crypto/tls: add new X25519Kyber768Draft00 code point

Point tls.X25519Kyber768Draft00 to the new 0x6399 identifier while the old 0xfe31 identifier is available as tls.X25519Kyber768Draft00Old.
Make sure that the kem.PrivateKey can always be mapped to the CurveID that was linked to it. This is needed since we now have two ID aliasing to the same scheme, and clients need to be able to detect whether the key share presented by the server actually matches the key share that the client originally sent.
Update tests, add the new identifier and remove unnecessary code.

Link: https://mailarchive.ietf.org/arch/msg/tls/HAWpNpgptl--UZNSYuvsjB-Pc2k/
Link: https://datatracker.ietf.org/doc/draft-tls-westerbaan-xyber768d00/02/

The current solution is the cleanest I could come up with. Alternatives considered:

Create a new implementation of the kem.Scheme interface, wrapping the hybrid.Kyber768X25519() scheme. I dropped this idea due to the amount of boiler plate code. In the sample below, I already wrapped the scheme, but I would also have to wrap the returned kem.PrivateKey and kem.PublicKey values since clientKeySharePrivateCurveID used their Scheme() method to get back the scheme name.
Alias to the same hybrid.Kyber768X25519() scheme. Fails for the same reason as above: I cannot get back to the original code point that was in use.

Incomplete wrapper code sample for kem.Scheme
package tls

import (
	"github.com/cloudflare/circl/kem"
	"github.com/cloudflare/circl/kem/hybrid"
)

// kemProxyScheme implements kem.Scheme, but aliasing it under a different name.
type kemProxyScheme struct {
	name   string
	scheme kem.Scheme
}

// Assert that kemProxyScheme implements kem.Scheme
var _ kem.Scheme = (*kemProxyScheme)(nil)

func (s *kemProxyScheme) Name() string { return s.name }

func (s *kemProxyScheme) GenerateKeyPair() (kem.PublicKey, kem.PrivateKey, error) {
	return s.scheme.GenerateKeyPair()
}
func (s *kemProxyScheme) Encapsulate(pk kem.PublicKey) (ct, ss []byte, err error) {
	return s.scheme.Encapsulate(pk)
}
func (s *kemProxyScheme) Decapsulate(sk kem.PrivateKey, ct []byte) ([]byte, error) {
	return s.scheme.Decapsulate(sk, ct)
}
func (s *kemProxyScheme) UnmarshalBinaryPublicKey(data []byte) (kem.PublicKey, error) {
	return s.scheme.UnmarshalBinaryPublicKey(data)
}
func (s *kemProxyScheme) UnmarshalBinaryPrivateKey(data []byte) (kem.PrivateKey, error) {
	return s.scheme.UnmarshalBinaryPrivateKey(data)
}
func (s *kemProxyScheme) CiphertextSize() int { return s.scheme.CiphertextSize() }
func (s *kemProxyScheme) SharedKeySize() int  { return s.scheme.SharedKeySize() }
func (s *kemProxyScheme) PrivateKeySize() int { return s.scheme.PrivateKeySize() }
func (s *kemProxyScheme) PublicKeySize() int  { return s.scheme.PublicKeySize() }
func (s *kemProxyScheme) DeriveKeyPair(seed []byte) (kem.PublicKey, kem.PrivateKey) {
	return s.scheme.DeriveKeyPair(seed)
}
func (s *kemProxyScheme) SeedSize() int { return s.scheme.SeedSize() }
func (s *kemProxyScheme) EncapsulateDeterministically(pk kem.PublicKey, seed []byte) (ct, ss []byte, err error) {
	return s.scheme.EncapsulateDeterministically(pk, seed)
}
func (s *kemProxyScheme) EncapsulationSeedSize() int { return s.scheme.EncapsulationSeedSize() }

var hybridKyber768X25519Old = &kemProxyScheme{
	"Kyber768-X25519Old",
	hybrid.Kyber768X25519(),
}

why do we need to maintain the old codepoint? Are there any clients that use it, perhaps Firefox or some internal service?

We've been advertising it in various places (blogs, etc.) and would like to keep the code samples working for some transitional period until the new kex is widely deployed.


Various texture fixes

Goal:
Associate all common objects with textures when possible/logical to do so.
Example; complex objects with many materials are not retextured as they require complex textures/uvs to be made and that is out of the scope of this PR.
Requirements:

[ ] All Ground Object Rocks up to mid game

[ ] All Game Object Rocks up to mid game

[ ] All trees have bark up to mid game

Morytania trees and rocks improvements:
before:

After:

Before:

After:

Misc location fixes:
Before:

After:

Al-Kharid
Before:

After:


implement the HANDSHAKE_DONE mechanism

See https://github.com/quicwg/base-drafts/pull/3145.

Codecov Report

Merging #2225 into master will decrease coverage by 0.06%.
The diff coverage is 55.17%.

@@            Coverage Diff             @@
##           master    #2225      +/-   ##
==========================================
- Coverage   86.32%   86.26%   -0.06%
==========================================
  Files         106      107       +1
  Lines        7700     7724      +24
==========================================
+ Hits         6647     6663      +16
- Misses        790      797       +7
- Partials      263      264       +1

Impacted Files
Coverage Δ

internal/handshake/crypto_setup.go
70% <0%> (-0.4%)
:arrow_down:

internal/wire/frame_parser.go
100% <100%> (ø)
:arrow_up:

internal/ackhandler/sent_packet_handler.go
75.95% <100%> (+0.15%)
:arrow_up:

internal/wire/handshake_done_frame.go
55.56% <55.56%> (ø)

session.go
74.94% <77.78%> (+0.03%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0552e7e...c1facf5. Read the comment docs.


How do I fill any value to a dask dataframe cell (cell having specific row number and column number)?

Thank you for reporting an issue.
Please read the following guidelines to ensure that your issue is handled
properly.  If you have a ...

General Question: like "How do I filter a dask.dataframe?" then
please consider asking the question on Stack Overflow using the #Dask tag.
Please do not ask a question in both locations.
If you think that you have run into a bug or have an unexplained exception
then please prefer to file a bug report here.

Bug report: then please include a self-contained copy-pastable
example that generates the issue if possible.  Please be concise with code
posted.  See guidelines below on how to provide a good bug report:

Minimal Complete Verifiable Examples
Craft Minimal Bug Reports

Bug reports that follow these guidelines are easier to diagnose,
and so are often handled much more quickly.

Question about a sub-project: like the distributed scheduler,
dask-kubernetes, s3fs, fastparquet, etc.  then please consider submitting a
report to that issue tracker instead.  A list of repositories can be
found at github.com/dask

Feature Request: then please do a quick search of existing issues to
make sure that this has not been asked before.

We know that asking good questions takes effort, and we appreciate your time.
Thank you again.

I can't implement to fill any value in a particular cell of a dask dataframe. However, It is implemented in pandas Dataframe with the help of 'loc' function. Here what I'm trying to do -
with pandas -
data.loc(row_index , column_name) = value (successfully run)
but in dask -
uniqueness_data.loc[unique_value, 'start_date'] = unique_count

TypeError                                 Traceback (most recent call last)
 in
----> 9 uniqueness_data.loc[unique_value, 'start_date'] = 11
TypeError: '_LocIndexer' object does not support item assignment
Kindly help me to resolve this. Thank you.

Correct, as the error message says, dask dataframe doesn't support direct item assignment.
There is no way to resolve this.  This is mentioned in the documentation.  Is there anything else we can help with here?  Otherwise I plan to close this.

So It means I have to do this with pandas only? Actually, I have a for loop inside it empty dataframe is passed , I calculate some value like mean, SD, max on a slice of data using rolling window. after it, I want to fill all these value row-wise in the dataframe with window start_date and end_date. Finally we can plot the graph of each column having rolling window basis.
Is there any other way to impliment this ?
Here is my pandas code snipt -
unique_value = final_data[hr_attribute].unique()
uniqueness_data = pd.DataFrame(columns = (['start_date', 'end_date'] + list(unique_value)))
while window_end <= window_moved_to_max_limit:
window_final_data = final_data[(final_data['effective_date'] >= window_start) &
(final_data['effective_date'] <= window_end)]
    uniqueness_count = window_final_data.groupby(hr_attribute)['effective_date'].count()
    #it gives a series of count for a unique hr_attribute combination

    uniqueness_data.loc[len(uniqueness_data), 'start_date'] = window_start
    uniqueness_data.loc[(len(uniqueness_data) - 1), 'end_date'] = window_end
    uniqueness_data.loc[(len(uniqueness_data) - 1), uniqueness_count.index] = uniqueness_count

    window_start = window_start + datetime.timedelta(days = window_jump_in_days)
    window_end = window_start + datetime.timedelta(days = window_size_in_days)
    #end loop here

Kindly help me. Thanks.

No, Dask doesn't support these kinds of operations.  They're not really parallelizable.

Kindly help me. Thanks.

This issue tracker is to report and solve bugs in the Dask project.  It isn't here to solve individual user problems.  Unfortunately I don't know of a good resource for you where you can get help for free for your particular problem.
I'm going to go ahead and close this because I don't see a resolution.  Good luck!

Ok Thanks Mrocklin.

I had a similar need in my analysis to replace some values in a dask dataframe column. My search ended up in this OS thread (https://stackoverflow.com/questions/54302694/updating-the-values-of-a-column-in-a-dask-dataframe-based-on-some-condition-on-s).  The second approach that uses mask works for me:
df['col4'] = df['col4'].mask(condition, df['col1'])

which replace values in col4 with those in col1 when those values meet the condition.

Thank you for making this note 3 years ago @pancodia, it pointed me in the right direction for a similar problem I'm having. And thank you dask maintainers for leaving these issues open and public for posterity! OSS ftw.


Add license and README file to SendTo+

Only binaries are included in the current release. License and readme file are needed.

Fixed from a source code point of view. It will be in the next release.


Add helicsPublicationGetTag and helicsPublicationSetTag

Codecov Report

Merging #45 (336ecc1) into main (0b9ca60) will decrease coverage by 0.15%.
The diff coverage is 20.00%.

@@            Coverage Diff             @@
##             main      #45      +/-   ##
==========================================
- Coverage   80.75%   80.60%   -0.16%
==========================================
  Files          20       20
  Lines        6703     6713      +10
==========================================
- Hits         5413     5411       -2
- Misses       1290     1302      +12

Flag
Coverage Δ

unit
80.60% <20.00%> (-0.16%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

helics/capi.py
80.60% <20.00%> (-0.18%)
:arrow_down:

helics/_build.py
41.66% <0.00%> (-4.17%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0b9ca60...336ecc1. Read the comment docs.


Cannot get sql query result to create a new table?

Hi,
I have tried to get the sql query result to create a new table, however when I run it, it outputs empty result?
here is the code:
`Import-Module Pode.Web
Import-Module dbatools
$sqlserver = 'xxx'
$database = 'xxx'
$siteserver = 'xxx'
$sitecode = 'xxx'
$tallyinterval = 'xxx'
$Query = "
code here
"
Start-PodeServer {
Add-PodeEndpoint -Address localhost -Port 8080 -Protocol Http
Use-PodeWebTemplates -Title 'SCCM' -Theme Dark
Add-PodeWebPage -Name 'Component Status' -Icon 'Settings' -ScriptBlock {
New-PodeWebCard -Content @(
New-PodeWebTable -Name 'Component Status' -ScriptBlock {
Invoke-DbaQuery -SqlInstance $sqlserver -Database $database -Query $query | ForEach-Object {
@{
MachineName = $.MachineName
ComponentName = $.ComponentName
Status = $.Status
Infos = $.Infos
Warnings = $.Warnings
Errors = $.Errors
}
} }
)
}}
`
I have also tried just using " Invoke-DbaQuery -SqlInstance $sqlserver -Database $database -Query $query" inside the script block. same result.
thanks!

Hey @qy2009,
Yes this makes sense, the variables created outside of Start-PodeServer won't be available to the page/table's scriptblock since they're in a different runspace while the server is running (scoping). I'd recommend looking at Pode's shared state functions, as these will let you set/get values that are accessible across the multiple runspaces.
It's on my list to get $using: syntax working with pages/elements, and I thought Add-PodeWebPage had an -ArgumentList but it's missing (this will be added from a different issue).
For the shared state, something like the below should work for you:
Import-Module Pode.Web
Import-Module dbatools

$sqlserver = 'xxx'
$database = 'xxx'
$siteserver = 'xxx'
$sitecode = 'xxx'
$tallyinterval = 'xxx'

$Query = "
code here
"

Start-PodeServer {
    Add-PodeEndpoint -Address localhost -Port 8080 -Protocol Http

    Use-PodeWebTemplates -Title 'SCCM' -Theme Dark

    Set-PodeState -Name 'SqlServer' -Value $sqlserver
    Set-PodeState -Name 'Database' -Value $database
    Set-PodeState -Name 'SiteServer' -Value $siteserver
    Set-PodeState -Name 'SiteCode' -Value $sitecode
    Set-PodeState -Name 'TallyInterval' -Value $tallyinterval
    Set-PodeState -Name 'Query' -Value $Query

    Add-PodeWebPage -Name 'Component Status' -Icon 'Settings' -ScriptBlock {
        New-PodeWebCard -Content @(
            New-PodeWebTable -Name 'Component Status' -ScriptBlock {
                $sqlserver = Get-PodeState -Name 'SqlServer'
                $database = Get-PodeState -Name 'Database'
                $query = Get-PodeState -Name 'Query'

                Invoke-DbaQuery -SqlInstance $sqlserver -Database $database -Query $query | ForEach-Object {
                    @{
                        MachineName = $_.MachineName
                        ComponentName = $_.ComponentName
                        Status = $_.Status
                        Infos = $_.Infos
                        Warnings = $_.Warnings
                        Errors = $_.Errors
                    }
                }
            }
        )
    }
}

@Badgerati  still got empty result :(
Is there a way to check the log and see what was wrong?
I have also tried moving all my variables inside the script block inside New-PodeWebTable. still nothing returned ;(

Hey @qy2009,
In case there are any errors, add the following under Add-PodeEndpoint:
New-PodeLoggingMethod -Terminal | Enable-PodeErrorLogging

If that doesn't show anything, with the server running, in the browser press F12 (or whatever opens devtools in your browser), open the console and force refresh the page - are there any table loading errors showing up? In the network tab does the ajax call to load the table result in a 200, or a different status code?
Other than that, try outputting the result of the database query to the powershell terminal instead - see if it's actually selecting any data:
Invoke-DbaQuery -SqlInstance $sqlserver -Database $database -Query $query | ForEach-Object {
    @{
        MachineName = $_.MachineName
        ComponentName = $_.ComponentName
        Status = $_.Status
        Infos = $_.Infos
        Warnings = $_.Warnings
        Errors = $_.Errors
    }
} | Out-Default

Or even just:
Invoke-DbaQuery -SqlInstance $sqlserver -Database $database -Query $query | Out-Default

Ooor, if it throws an error it should appear on the terminal with the able logging enabled; but there are some strange errors that seem to "fail successfully" and get missed, so try:
New-PodeWebTable -Name 'Component Status' -ScriptBlock {
    try {
        $sqlserver = Get-PodeState -Name 'SqlServer'
        $database = Get-PodeState -Name 'Database'
        $query = Get-PodeState -Name 'Query'

        Invoke-DbaQuery -SqlInstance $sqlserver -Database $database -Query $query | ForEach-Object {
           @{
                MachineName = $_.MachineName
                ComponentName = $_.ComponentName
                Status = $_.Status
                Infos = $_.Infos
                Warnings = $_.Warnings
                Errors = $_.Errors
            }
        }
    }
    catch {
        $_.Exception | Out-Default
    }
}

@Badgerati
From the console, I got this message:
jquery.min.js:2 POST http://localhost:8080/elements/table/table_componentstatus_componentstatus_table_componentstatus_componentstatus 400 (Bad Request)
On the Network tab, I got the same 400 from jquery.min.js
Request URL: http://localhost:8080/elements/table/table_componentstatus_componentstatus_table_componentstatus_componentstatus
Request Method: POST
Status Code: 400 Bad Request
Remote Address: <IP_ADDRESS>:8080
Referrer Policy: strict-origin-when-cross-origin
Content-Length: 1197
Content-Type: text/html; charset=utf-8
Date: Wed, 28 Jul 2021 15:44:50 GMT
Server: Pode
X-Pode-ContextId: 54585610-30d0-c059-587d-91dd5fa03533
Accept: /
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Content-Length: 0
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
x
Host: localhost:8080
Origin: http://localhost:8080
Referer: http://localhost:8080/pages/Component Status
sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"
sec-ch-ua-mobile: ?0
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36
X-Requested-With: XMLHttpRequest
From the Terminal, I can get all the results fine like the followings:
Name                           Value

ComponentName                  SMS_CLIENT_CONFIG_MANAGER
Infos                          5
MachineName                    xxx
Errors                         0
Warnings                       160
Status                         Critical
ComponentName                  AI_UPDATE_SERVICE_POINT
Infos                          0
Thanks so much for your help :)

Hey @qy2009,
Hmm, the 400 definitely suggest an error is being thrown 🤔 Which version of PowerShell, Pode, and Pode.Web are you using?
For the results on the terminal, are they from outputting the results of the query from within the table's scriptblock, or manually? Was there anything on the terminal from enable error logging?
Something like the following should hopefully show something in the terminal when you load the table 🤞 If there's an error in the table's scriptblock this should show it, if it shows the results of the query but is a 400 then something spooky is going on; or, if it shows nothing then the the function could be silently failing to get data 🤔:
Import-Module Pode.Web
Import-Module dbatools

$sqlserver = 'xxx'
$database = 'xxx'
$siteserver = 'xxx'
$sitecode = 'xxx'
$tallyinterval = 'xxx'

$Query = "
code here
"

Start-PodeServer {
    Add-PodeEndpoint -Address localhost -Port 8080 -Protocol Http
    New-PodeLoggingMethod -Terminal | Enable-PodeErrorLogging

    Use-PodeWebTemplates -Title 'SCCM' -Theme Dark

    Set-PodeState -Name 'SqlServer' -Value $sqlserver
    Set-PodeState -Name 'Database' -Value $database
    Set-PodeState -Name 'SiteServer' -Value $siteserver
    Set-PodeState -Name 'SiteCode' -Value $sitecode
    Set-PodeState -Name 'TallyInterval' -Value $tallyinterval
    Set-PodeState -Name 'Query' -Value $Query

    Add-PodeWebPage -Name 'Component Status' -Icon 'Settings' -ScriptBlock {
        New-PodeWebCard -Content @(
            New-PodeWebTable -Name 'Component Status' -ScriptBlock {
                try {
                    $sqlserver = Get-PodeState -Name 'SqlServer'
                    $database = Get-PodeState -Name 'Database'
                    $query = Get-PodeState -Name 'Query'

                    $queryResult = Invoke-DbaQuery -SqlInstance $sqlserver -Database $database -Query $query
                    if (!$?) {
                        throw "Query failed"
                    }

                    $results = $queryResult | ForEach-Object {
                        @{
                            MachineName = $_.MachineName
                            ComponentName = $_.ComponentName
                            Status = $_.Status
                            Infos = $_.Infos
                            Warnings = $_.Warnings
                            Errors = $_.Errors
                        }
                    }

                    $results | Out-Default
                    return $results
                }
                catch {
                    $_ | Out-Default
                }
            }
        )
    }
}

Also, from the example code the URL should be http://localhost:8080/elements/table/table_componentstatus_componentstatus (which I get locally), but you have http://localhost:8080/elements/table/table_componentstatus_componentstatus_table_componentstatus_componentstatus which suggests you might have double embedded table? 🤔 Is the example code the full code, or just a copy/paste issue?
The only other thing I can think of is: how many rows of data are being returned? The result could be too large, though I have tested a table with a lot of data and no pagination so I'd be surprised.

@Badgerati thanks so much for your update ;)
I ran your code, and it showed up the result successfully.
Then I removed the troubleshooting code, and it also ran successfully.
I guess the only difference from the my last code is this part:
return $results
btw, I am planning to use the "Select" from here:
https://badgerati.github.io/Pode.Web/Tutorials/Elements/Select/
However, I didn't see where I can get the value after the selection?
I am planning to let users to select an option, and then add the value inside the sql query to show the results.
thanks so much!

Then I removed the troubleshooting code, and it also ran successfully.
I guess the only difference from the my last code is this part:
return $results

Whaaat! Crazy!!
Aaah, yeah that's because a Select element is a form-input element, so needs to be within a New-PodeWebForm, and then the selected value will be available in the Form's scriptblock on submit.
(That being said, you've given me an interesting idea to look at maybe having an -OnChangeScriptBlock so you can do actions on a select outside of a Form, hmm 🤔)
In this instance though, you'd need an initial empty Table, then a Form with Select, and on Submit it runs the SQL and updates the table with the results:
Import-Module Pode.Web -DisableNameChecking
Import-Module dbatools -DisableNameChecking

Start-PodeServer {
    Add-PodeEndpoint -Address localhost -Port 8080 -Protocol Http
    Use-PodeWebTemplates -Title 'SCCM' -Theme Dark

    Add-PodeWebPage -Name 'Component Status' -Icon 'Settings' -ScriptBlock {
        # create an initial empty table with column headers
        # this will refresh every 10mins, based on the forms current selected value
        New-PodeWebCard -Content @(
            New-PodeWebTable -Name 'Component Status' -AutoRefresh -RefreshInterval 600 -ScriptBlock {
                return @{
                    DeviceName = ""
                    LastComplianceMessageDesc = ""
                    LastComplianceMessageTime = ""
                    LastEnforcementMessageTime = ""
                    LastMessage = ""
                }
            }
        )

        # this is the form with select, on submit the value will be retrieved.
        # The SQL then run and the results used to update the table
        New-PodeWebCard -Content @(
            New-PodeWebForm -Name FormData -Content @(
                New-PodeWebSelect -Name 'Single' -Options 'patchserver', 'interalpatch', 'externalpatch', 'serverbaseline', 'workstationbaseline' -SelectedValue 'patchserver'
            ) `
            -ScriptBlock {
                $value = $WebEvent.Data['Single']

                $sqlserver = 'xxx'
                $database = 'xxx'
                $siteserver = 'xxx'
                $sitecode = 'xxx'
                $tallyinterval = 'xxx'
                $query = "xxx where $value"

                $results = (Invoke-DbaQuery -SqlInstance "xxx" -Database "xxx" -Query $query | ForEach-Object {
                    @{
                        DeviceName = $_.DeviceName
                        LastComplianceMessageDesc = $_.LastComplianceMessageDesc
                        LastComplianceMessageTime = $_.LastComplianceMessageTime
                        LastEnforcementMessageTime = $_.LastEnforcementMessageTime
                        LastMessage = $_.LastMessage
                    }
                })

                $results | Update-PodeWebTable -Name 'Component Status'
            }
        )
    }
}

^ something like that should in theory work for what you're after :)

I know! Sometimes just a small thing....lol
Thanks so much for your code :)
I have tested it that this one is not correct:
$results | Update-PodeWebTable -Name 'Component Status'
After I changed to the following I can see the result
Update-PodeWebTable -Name 'Patching Status' -Data $results
However, the table is not showing the correct order of columns , do you know why? I put the DeviceName first, but the result is not.
also, the date is strange.  it should be like "2021-07-14 15:43:36.697" , but it is showing like "/Date(1627614141947)/"
Please see the screenshots :

On the chrome console, I see the errors here:
DevTools failed to load source map: Could not load content for http://localhost:8080/pode.web/libs/popperjs/popper.min.js.map: Connection error: net::ERR_CONNECTION_REFUSED
DevTools failed to load source map: Could not load content for http://localhost:8080/pode.web/libs/bootstrap/bootstrap.bundle.min.js.map: Connection error: net::ERR_CONNECTION_REFUSED
DevTools failed to load source map: Could not load content for http://localhost:8080/pode.web/libs/bs-stepper/bs-stepper.min.js.map: Connection error: net::ERR_CONNECTION_REFUSED
DevTools failed to load source map: Could not load content for http://localhost:8080/pode.web/libs/moment/moment.min.js.map: Connection error: net::ERR_CONNECTION_REFUSED
DevTools failed to load source map: Could not load content for http://localhost:8080/pode.web/min-maps/vs/loader.js.map: Connection error: net::ERR_CONNECTION_REFUSED

Hey @qy2009, The column order is a quick one, add [ordered] to the front of the hashtable 😄 ie:
$results = (Invoke-DbaQuery -SqlInstance "xxx" -Database "xxx" -Query $query | ForEach-Object {
    [ordered]@{
        DeviceName = $_.DeviceName
        LastComplianceMessageDesc = $_.LastComplianceMessageDesc
        LastComplianceMessageTime = $_.LastComplianceMessageTime
        LastEnforcementMessageTime = $_.LastEnforcementMessageTime
        LastMessage = $_.LastMessage
    }
})

The date is likely json/javascript trying to be clever 🤔 - try doing $_.LastComplianceMessageTime.ToString() and see what that does.
The map files are definitely there, it's not too bad that it oddly can't find them, but a strange one. Are you using Pode from the PSGallery or a locally built version from cloning the repo?

omg, you are genius!!!
thanks so much for all the help. now I got both of them working :)
I am using Pode & Pode.Web from the PSGallery.
2.4.0      Pode
0.5.1      Pode.Web
cannot say thank you enough!!!


Switch for tether

Description
Need an ethernet switch on topside of robot for tethering on practice field
Is your feature request related to a problem? Please describe.
This will be necessary for us to use the practice field
Describe the solution you'd like
An ethernet switch needs to be mounted on the bot with possible ethernet connection
Describe alternatives you've considered
Running ethernet directly to brainpan, but that would require taking bellypan off and leaving wiring exposed
Additional context
[Add any other context or screenshots about the feature request here. REMOVE THIS WHEN DONE]

Switch is now working and was installed in match 5


I2C timeout after light sleep (IDFGH-13075)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report
Hello !
I am a beginner in the world of ESP and zigbee and I am building a CO2 measurement system using the zigbee network for home automation. I use an ESP32-C6 and an SCD40 component which works in I2C. So far, everything is working perfectly. The component joins the zigbee network well and regularly reports the values measured by the sensor and read by I2C. Since I would like the system to run on battery power, I want to use "light sleep" mode. I was inspired by the example provided in ESP-IDF: "Sleepy End Device Example".
In the app_main function, I added the call to the function:
esp_zb_power_save_init();
With this function identical to the example:
static esp_err_t esp_zb_power_save_init(void)
{
 esp_err_t rc = ESP_OK;
#ifdef CONFIG_PM_ENABLE
 int cur_cpu_freq_mhz = CONFIG_ESP_DEFAULT_CPU_FREQ_MHZ;
 esp_pm_config_t pm_config = {
 .max_freq_mhz = cur_cpu_freq_mhz,
 .min_freq_mhz = cur_cpu_freq_mhz,
#if CONFIG_FREERTOS_USE_TICKLESS_IDLE
 .light_sleep_enable = true
#endif
 };
 rc = esp_pm_configure(&pm_config);
#endif
 return rc;
}

In the function static void esp_zb_task(void *pvParameters):
/* initialize Zigbee stack */
 esp_zb_cfg_t zb_nwk_cfg = ESP_ZB_ZED_CONFIG();
 esp_zb_sleep_enable(true);
 esp_zb_init(&zb_nwk_cfg);

In the function void esp_zb_app_signal_handler(esp_zb_app_signal_t *signal_struct):
case ESP_ZB_COMMON_SIGNAL_CAN_SLEEP:
 ESP_LOGI(TAG, "Zigbee can sleep");
 esp_zb_sleep_now();
 break;

My I2C initialization function :
static esp_err_t i2c_master_init(void)
{
    int i2c_master_port = I2C_MASTER_NUM;
    i2c_config_t conf = {
        .mode = I2C_MODE_MASTER,
        .sda_io_num = I2C_MASTER_SDA_IO,
        .scl_io_num = I2C_MASTER_SCL_IO,
        .sda_pullup_en = GPIO_PULLUP_ENABLE,
        .scl_pullup_en = GPIO_PULLUP_ENABLE,
        .master.clk_speed = I2C_MASTER_FREQ_HZ,
        .clk_flags = I2C_SCLK_SRC_FLAG_LIGHT_SLEEP,
    };
    i2c_param_config(i2c_master_port, &conf);
    return i2c_driver_install(i2c_master_port, conf.mode, I2C_MASTER_RX_BUF_DISABLE, I2C_MASTER_TX_BUF_DISABLE, 0);
}

The ESP switches to "light sleep" mode since I have the information on the console: "Zigbee can sleep" and I see that consumption is dropping. But the I2C communication no longer works and the reading functions return a timeout error code.
I use the old I2C driver (i2c.h) and not the new one (i2c_master.h). Is the new one likely to work better?
I must be missing something, but I don't see what... Any suggestions?
Thank you for your help !

yes,please try new one. new one support sleep retention..

https://github.com/espressif/esp-idf/commit/27b2f7a10b662fdbe4991e12ba9c04870083a453
Currently this commit only exist on the master branch and release/v5.3 branch.

Thank you for your help ! I will try to move towards the new i2c drivers. Until now, I haven't found some example. This will help for sure.
For now I managed to get I2C and light sleep mode to work by disabling the CONFIG_PM_POWER_DOWN_PERIPHERAL_IN_LIGHT_SLEEP option. But it's probably less effective, therefore.

@mycael Do you have any update on this issue? Could you close the issue if no further action required?


Fix xy angle sign

What
This PR fixes the wrong sign that was used for the xy_angle in the phase in the Instruction class.
Why
Makes rotations on the bloch sphere be in the correct direction. Closes #231
How
Removed minus sign.
Remarks
None
Checklist
Please include and complete the following checklist. Your Pull Request is (in most cases) not ready for review until the following have been completed. You can create a draft PR while you are still completing the checklist. Check the Contribution Guidelines for more details. You can mark an item as complete with the - [x] prefix

[x] Tests - Added unit tests for new code, regression tests for bugs and updated the integration tests if required
[x] Formatting & Linting - black and flake8 have been used to ensure styling guidelines are met
[x] Type Annotations - All new code has been type annotated in the function signatures using type hints
[x] Docstrings - Docstrings have been provided for functions in the numpydoc style
[x] Documentation - The tutorial style documentation has been updated to explain changes & new features
[x] Notebooks - Example notebooks have been updated to incorporate changes and new features
[x] Changelog - A short note on this PR has been added to the Upcoming Release section

The cases where the tests are failing seem to be because of fairly small differences in the resulting optimal parameters for various optimisations/calibrations (except for maybe test_flux_signal, where the differences are on the order of ~10%). What should I do to deal with this? Create new parameter files? @lazyoracle

The cases where the tests are failing seem to be because of fairly small differences in the resulting optimal parameters for various optimisations/calibrations (except for maybe test_flux_signal, where the differences are on the order of ~10%). What should I do to deal with this? Create new parameter files? @lazyoracle

In general, failing tests should not be modified to get them to pass. Do you possibly have some insight on why the optimal parameters changed? If however the change in the test optimal parameters is expected as a result of the code change in the PR, then feel free to just modify the parameters and move on. Make sure to note in your commit message the reasoning behind why the test was modified.

With optimization results, it can be tricky when the convention changed. An alternative to what Anurag suggested would be to change the gate definitions in those tests from phase -> -phase to recreate the previous behavior. Then add a note that the convention here goes the "wrong" way.
If this does not work, something else is wrong.


"unexpected error while defensively cleaning up"

Trying to import github.com/nightlyone/lockfile, which is an indirect dependency I have.
What version of dep are you using (dep version)?
v0.5.3
Also git version = git version 2.21.0.
What dep command did you run?
dep ensure -v
What did you expect to see?
Solved output.
What did you see instead?
Solving failure: No versions of github.com/nightlyone/lockfile met constraints:
	6a197d5ea61168f2ac821de2b7f011b250904900: unexpected error while defensively cleaning up after possible derelict nested submodule directories: Entering 'git-hooks'
error: unknown switch `x'
usage: git submodule--helper foreach [--quiet] [--recursive] [--] <command>

    -q, --quiet           Suppress output of entering each submodule command
    --recursive           Recurse into nested submodules

fatal: run_command returned non-zero status while recursing in the nested submodules of git-hooks
.
: command failed: [git submodule foreach --recursive git clean -x -d -f -f]: exit status 128

Huh. Can you run git help clean and tell me whether "-x" appears in the help text?

Ah, looks like https://github.com/golang/dep/pull/2168 fixes this.

Sadly I can't help you on this one as the project I was working on that
triggered this shut down :( I hope that PR fixes it.
Good luck!
On Sat, Jun 8, 2019 at 6:39 PM Kevin Burke<EMAIL_ADDRESS>wrote:

Ah, looks like #2168 https://github.com/golang/dep/pull/2168 fixes this.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/golang/dep/issues/2164?email_source=notifications&email_token=AAAKVPCDOLPUOFDLW5MYXKLPZRNNJA5CNFSM4HNQMQ52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXIBXBQ#issuecomment-500177798,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAKVPEH5LWZGVB5ABO6O4DPZRNNJANCNFSM4HNQMQ5Q
.

I ran into similar problem :\ (mac, git 2.22, dep 0.5.3)

same problem in mac.

Hi, yeah it seems the same issue that I fixed in https://github.com/golang/dep/pull/2168
I have the same git version. I inserted an example about how we could reproduce it directly with git command.
I hope for the merge.


Documentation unclear for simulateCells

simulateCells is a S4 dispatcher that calls simulateCells.celda_C, simulateCells.celda_G, and simulateCells.celda_CG depending on the given model argument. Although the individual methods are well documented, they do not show up in the help page of simulateCells. An update to the documentation of simulateCells is needed.

#73

Received the following error after updating the documentation for simulateCells:
"Documented arguments not in \usage in documentation object 'simulateCells':
‘S’ ‘C’ ‘C.Range’ ‘N.Range’ ‘G’ ‘K’ ‘L’ ‘alpha’ ‘beta’ ‘delta’
‘gamma’ ‘seed’"
Tried a couple of fixes on the function but none works. Need a trick or complete rewrite of the function to overcome this problem. The documentation of the function has been rewound for now for passing the check.