v.0.21.12

Version 0.21.12
Released March 15, 2023

Refactor NYSE holidays, #1018 (@KJhellico)
Fix the documentation build warning, #1026 (@arkid15r)
Optimize TestAllInSameYear::test_all_countries test, #1001 (@arkid15r)
Update Canada localization: fr, #1015 (@KJhellico)
Update Denmark holidays: remove "Store bededag" starting 2024, #1013 (@malthe)
Update PR template, #1019 (@arkid15r)
Update Portugal holidays, #1011 (@PPsyrius)
Update Ukraine holidays, #1014 (@KJhellico)
Update package build process, #1020 (@arkid15r)
Add multiple countries localization: en_US, es, uk, #1012 (@KJhellico)
Unify country imports order, #1003 (@arkid15r)
Use native apostrophe for uk locale holiday names, #1004 (@arkid15r)

Coverage: 99.961% (-0.04%) from 100.0% when pulling f42ed77cfd55fc011009e9b05e0f32f24943dac0 on release-0.21.12 into d39ac1ed39c34edc89c0eee0ca1598af91b9f5f5 on master.

https://youtu.be/5mcPOd28oN0


Make ops aware of rvalues: astype/as_strided/copy/full

Proposed changes
When compositing transforms lots of temporary of arrays will be created and passed to next primitive, and by making ops accepting args by value we can avoid lots of copies of temporary arrays.
Checklist
Put an x in the boxes that apply.

[x] I have read the CONTRIBUTING document
[x] I have run pre-commit run --all-files to format my code / installed pre-commit prior to committing changes
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have updated the necessary documentation (if needed)

I don't know how I feel about this change. I appreciate the possible efficiency benefits but I am not too sure it is worth having to pollute the code with std::move everywhere.
In the current state of the codebase, when passing an array to an op there is a copy into the inputs vector of the returned array. This would indeed cause a couple of copies that can be avoided when chaining operations. However, making pass by copy the only option results in a significantly worse experience when writing ops imho.
Current API
array x = ones(10, float32);
array a = array(2, float32);
array b = array(1, float32);
x = a * x + b;  // 2 copies when doing a*x and 2 more for + b
x = add(multiply(x, b), c);  // same as above

Proposed API
array x = ones(10, float32);
array a = array(2, float32);
array b = array(1, float32);
x = a * x + b;  // 2 copies when doing a*x and 1 more for + b
x = add(multiply(std::move(x), a), b); // 1 copy for a*x and 1 more for +b

So there is an improvement (maybe I missed one copy of a temporary there I am not sure how good the compiler would be to figure that out) but there needs to be a move on the outside and there need to be moves on the inside of the ops. Moving an and b would reduce more copies but would be even harder to call. Passing temporaries would auto move which is pretty cool though.

This change does not mean to reduce the copies from user's side, it means to remove the internal copies inside the mlx library as much as possible.
Using add as example:
array add(const array& a, const array& b, StreamOrDevice s /* = {} */) {
  ...
  inputs = broadcast_arrays(astype(a, out_type, s), astype(b, out_type, s), s);
  ...
  return array(..., std::move(inputs));
}

Calling add(a, b) has 4 copies of arrays at least: 2 caused by astype, and 2 caused by broadcast_arrays. With move semantics there will only be 2 copies left, i.e. 50% less copies.
Using your example add(multiply(x, b), c), with move semantics there is not only 1 copy saved, but 5: one implicit copy from user's side, 2 internal copies in add, and 2 internal copies in multiply.
There are of course lots of ops that don't have internal copies, like negative. But there are also ops that do deep compositions, like logsumexp, and move semantics will cut more than half of internal copies.
I didn't do any analysis of how frequently the ops are used in real worlds, but just by skimming the code, I believe the number of array copies will be at least halved in real code if all ops support move semantics.

The major downside is, we have to use lots of std::move in the implementations, which I don't like neither.
But no std::move does not mean clearer code, you still have to pollute the code with const array& everywhere, which makes code hard to read. And the API is deceiving: users would think there is no copy of argument because the API takes it by const reference, but in reality the arg is copied every time.

On the other hand, I totally get why you don't want this change, I don't like changes transforming coding styles in my projects neither.
I don't tend to force changing this library to use pass-by-value everywhere, I'm just studying this library and found there are places that can be improved and then spent some time tweaking things, which helps me understand the code better.

Fwiw array copy time is unfortunately non-trivial. It's really just an increment and decrement on a shared pointer which is unfortunately very slow (because it's an atomic). The move avoids that entirely since you don't need to touch the ref count. Just to know what it buys us, here's a little benchmark I ran on my machine:
  array a({1,2,3});
  auto fun = [&]() {
    for (int i = 0; i < 1000; ++i) {
      array b = a;
      a = b;
    }
  };

Versus with a move:
  array a({1,2,3});
  auto fun = [&]() {
    for (int i = 0; i < 1000; ++i) {
      array b = std::move(a);
      a = std::move(b);
    }
  };

The timings are:
0.017 ms for the copy vs 0.0000175 ms for the move.

Given the above benchmark I'm willing to pay the cost of using std::move inside some ops (as long as it doesn't make using the API slower without moves, which in this case it should not since we need to do the copy anyway).

Well that is only 1000x faster. My comment was mostly to avoid having to write the following
auto dt = y.dtype();
add(std::move(x), multiply(std::move(y), array(2.0, dt)));

instead of
add(x, multiply(y, array(2.0, y.dtype()));

Otoh I agree that this can be kept internal a bit so the user would still use the latter and pay 3 copies which would be paid anyway when these were put into the input vectors.

Another point is obviously favoring completely functional code (which I do believe it looks kind of nice) and the compiler will take care of moving the temporary for us. For instance
 auto mu = mean(x, /* axis= */ -1, /* keepdims= */ true, s);
 auto mu2 = square(mu, s);
 auto x2 = mean(square(x, s), /* axis= */ -1, /* keepdims= */ true, s);
 auto v = subtract(x2, mu2, s);

would have to be made into
auto mu = mean(x, /* axis= */ -1, /* keepdims= */ true, s);
auto v = subtract(mean(square(x, s), /* axis= */ -1, /* keepdims= */ true, s),
                  square(mu, s), s);

I agree that this can be kept internal a bit so the user would still use the latter and pay 3 copies which would be paid anyway when these were put into the input vectors.

Yea exactly. I don't even think we have to be dogmatic about it ourselve, even in a few of the fast ops that get used a lot (e.g. broadcast, astype, reshape, transpose) might be enough.

Another point is obviously favoring completely functional code (which I do believe it looks kind of nice) and the compiler will take care of moving the temporary for us. For instance

I like that style too. If we started using the stream context manager we could make it look even nicer with operator overloads.

So I tested to see how many copies / moves we do from different variations and its kind of interesting / unexpected.
The following does two copies (not one), one into the initializer list and the next into the vector for the array's inputs.
array op(const array& x) {
  return array({1}, float32, nullptr, {x});
}

void test_move() {
  array a(1.0);
  std::cout << "START " << std::endl;
  auto b = op(a);
  std::cout << "STOP " << std::endl;
}

This one does two moves and one copy (not 0): move into the argument x, move into the initializer list and copy into the vector of inputs:
array op(array x) {
  return array({1}, float32, nullptr, {std::move(x)});
}

void test_move() {
  array a(1.0);
  std::cout << "START " << std::endl;
  auto b = op(std::move(a));
  std::cout << "STOP " << std::endl;
}

This version does only two moves, no copies:
array op(array x) {
  std::vector<array> inputs;
  inputs.push_back(std::move(x));
  return array({1}, float32, nullptr, std::move(inputs));
}

void test_move() {
  array a(1.0);
  std::cout << "START " << std::endl;
  auto b = op(std::move(a));
  std::cout << "STOP " << std::endl;
}

It would be nice if there was a way to make the vector as an rvalue without incurring a copy, but I haven't been able to find it... possibly implementation dependent..

It would be nice if there was a way to make the vector as an rvalue without incurring a copy

There is a trick to eliminate the copy but it is not necessarily to be fast, see also: https://burnicki.pl/en/2022/08/14/initializing-with-std-initializer_list-involves-copying.html

There is a bigger problem with std::vector though: it stores things on heap so every time you create or copy a vector you are doing a heap allocation and it is slow, MLX does things like op({array({0})}, {1}) a lot and this is hurting.
One common solution is to use a custom vector type that stores small numbers of elements on stack and only use heap when the number of elements is large, i.e. the vector version of Small String Optimization. Projects like V8 and LLVM use the custom vector, and there are also implementations in absl/folly/boost.
And by using a custom vector type we can make its constructor accept rvalue arguments directly instead of initializer_list, which solves the problem of arrays being copied into vector.
I can give it a try if you are okay with it.

So even though std::vector cripples the benefits of move semantics, I think MLX should still use move semantics when feasible, because it is a solvable problem, and I think you probably want to solve it anyway to get rid of the overhead.

Regarding small vector type, indeed it occured to me to have one for shape and strides since we copy those around a lot as well. We are also considering moving to an explicit shape type rather than std::vector<int> so we can have more flexibility. But that is a big change..

I can give it a try if you are okay with it.

For passing arrays around it could be useful. But I think we would actually need to see some benchmarks to understand how much we gain so we can evaluate the pros and cons.
If you are interested in investigating that, the first thing I would do is set you up with some benchmarks so you can measure the time of graph construction and other overheads. And then see if using the small vector type improves anything.

Why do you say that? It should be faster than what we do now (?) which is both the allocation and copying from the initializer list.

It turns a constructor call into a series of emplace_back, which adds its own overhead, so I’m not really sure without benchmarking.

If you are interested in investigating that, the first thing I would do is set you up with some benchmarks so you can measure the time of graph construction overheads. And then see if using the small vector type improves anything.

Let me know if you'd like me to share some benchmarks.

If you are interested in investigating that, the first thing I would do is set you up with some benchmarks so you can measure the time of graph construction overheads. And then see if using the small vector type improves anything.
Let me know if you'd like me to share some benchmarks.

I have an idea that would allow us to convert the codebase to use small vector piece by piece instead of doing it all at once, I will experiment with the idea first. Some performance gain does not worth a change with thousands of lines, but might worth it if we can do it with small steps.

No more thoughts :-) . I 'm merging it, we can play it by ear on whether the code gets ugly with a lot of std::move calls everywhere.


Call a function between two blocks

I want to use a function between two blocks. This function will set the order of the stimulus that I want to present. The variables that are in the function are called from another script.
This is part of my code:
var trials = [];
      var timeline = [];
      var images = [
      "img/1.png", "img/2.png", "img/3.png", "img/4.png",
      "img/5.png", "img/6.png"
    ];

    var instructions = {
      type: "html-keyboard-response",
      stimulus: "
Waiting for the inputs
 ",
      choices: ['x'],
      post_trial_gap: 1000,
    };
timeline.push(instructions);

//-------------------------------------------------------------------------
// Here, I want to run this function, so trials can have this specific stimulus.
//After `"instructions"` and  before "block".
//The function:
function() {
        for (var i = 0; i < all_states.length;) {
          var tmppath = all_states[i];
          trials.push({
            stimulus: images[tmppath]
          });
          i++
        }
        }
//-------------------------------------------------------------------------

   var block = {
        type: 'image-keyboard-response',
        choices: jsPsych.NO_KEYS,
        prompt: '
Display of images.
',
        trial_duration: function(){
          return jsPsych.randomization.sampleWithoutReplacement([1000], 1)[0];
        },
        on_finish: function(data) {
          if (data.key_press == 40) {
            jsPsych.endCurrentTimeline();
          }
        },
        timeline: trials
      }
timeline.push(block);

      var after_block = {
        type: 'html-keyboard-response',
        stimulus: '
The trial is finished
',
        is_html: true
      }
timeline.push(after_block);

      jsPsych.init({
        timeline: timeline,
        on_finish: function() {
          jsPsych.data.displayData();
        }
      });

Any suggestions?
Thanks in advance of your attention and help.

I figure out a way to execute the function that I wanted. However, now "block" is not executed, it is just skipped.
This is my code so far:

    var trials = [];
    var timeline = [];
    var images = [
    "img/1.png", "img/2.png", "img/3.png", "img/4.png",
    "img/5.png", "img/6.png"
  ];

  var instructions = {
    type: "html-keyboard-response",
    stimulus: "
Waiting for the inputs
 ",
    choices: ['x'],
    post_trial_gap: 1000
  };
timeline.push(instructions);

      var evaluation = {
        type: 'call-function',
        func: function() {
          for (var i = 0; i < all_states.length;) {
            var tmppath = all_states[i];
            trials.push({
              stimulus: images[tmppath]
            });
            i++
          }
          }
      };
      timeline.push(evaluation);

    var block = {
      type: 'image-keyboard-response',
      choices: jsPsych.NO_KEYS,
      prompt: '
Display of images.
',
      trial_duration: function(){
        return jsPsych.randomization.sampleWithoutReplacement([1000], 1)[0];
      },
      on_finish: function(data) {
        if (data.key_press == 40) {
          jsPsych.endCurrentTimeline();
        }
      },
      timeline: trials
    }
timeline.push(block);

    var after_block = {
      type: 'html-keyboard-response',
      stimulus: '
The trial is finished
',
      is_html: true
    }
timeline.push(after_block);

    jsPsych.init({
      timeline: timeline,
      on_finish: function() {
        jsPsych.data.displayData();
      }
    });

Does anyone have any idea why "block" is no being executed?
Thanks for your attention.

I think the problem is that block doesn't have a stimulus parameter set.
Your function in the call-function plugin isn't going to achieve what you're aiming for. The timeline can't be modified on the fly in this particular way. There's one function that allows for timeline modification, but this form of timeline construction is something that we will add more support for in a future release.
The solution here is probably to add a stimulus parameter that is a function call to block. You can use that function to determine the correct order of stimuli and return the current stimulus. If you wrap the block inside a looping timeline, you should be able to get a dynamic order of stimuli.

Thank you jodeleeuw.
Now "block" is being executed. However, it is not showing the images dynamically, it is just displaying the first image, it looks like the loop is not working.
This is the new code:
var trials = [];
    var timeline = [];
    var images = [
    "img/1.png", "img/2.png", "img/3.png", "img/4.png",
    "img/5.png", "img/6.png"
    ];

    var instructions = {
    type: "html-keyboard-response",
    stimulus: "
Waiting for the inputs
 ",
    choices: ['m'],
    post_trial_gap: 1000
    };
    timeline.push(instructions);

    var block = {
      type: 'image-keyboard-response',
      choices: jsPsych.NO_KEYS,
      prompt: '
Display of images.
',
      stimulus: function (){
          for (var i = 0; i < all_states.length;) {
            var tmppath = all_states[i];
            return images[tmppath];
            i++
          }},
      trial_duration: function(){
        return jsPsych.randomization.sampleWithoutReplacement([1000], 1)[0];
      },
      on_finish: function(data) {
        if (data.key_press == 40) {
          jsPsych.endCurrentTimeline();
        }},
    }
    timeline.push(block);

    var after_block = {
      type: 'html-keyboard-response',
      stimulus: '
The trial is finished
',
      is_html: true
    }
    timeline.push(after_block);

    jsPsych.init({
      timeline: timeline,
      on_finish: function() {
        jsPsych.data.displayData();
      }
    });

Do you know how to fix it, to display an image for each cycle of the loop?
Thanks again for your attention!

In the end, this was my solution:
//-------------------Displaying image function-----
      var image_display = {
        type: 'image-keyboard-response',
        choices: jsPsych.NO_KEYS,
        prompt: '
Display of images.
',
        stimulus: function () {return images[all_states[s]];},
        trial_duration: function(){
          return jsPsych.randomization.sampleWithoutReplacement([1000], 1)[0];},
      }
//--------------------------------------------------------------------------------

//----------------Increase the value of images index (index of all_states) function----
      var increase_S = function (){
          s++
          console.log(s);
          }

      var increase_image_index = {
          type: 'call-function',
          func: increase_S
      }
//------------------------------------

//-----------Function that displays the whole secuence of images------
      var images_sequence = {
          timeline: [image_display, increase_image_index],
          repetitions: 5,
          randomize_order: false
      }
//-----------------------------------------------------------------------


update readme for template noSQL

Update readme from mentioning MongoDB to NoSQL
fix: https://github.com/Azure/azure-dev/issues/2527

Repoman Generation Results
Repoman pushed changes to remotes for the following projects:
Project: todo-csharp-cosmos-sql
Remote: azure-samples-staging
Branch: pr/2611
You can initialize this project with:
azd init -t Azure-Samples/todo-csharp-cosmos-sql -b pr/2611

View Changes | Compare Changes

/check-enforcer override


[CHANGE REQUEST] Dae is the new standard

Why are you making custom formats like G3DB since iOS/Unity is using DAE?
All developers are complaining about DAE to G3DB converters. They are not working!
 Why iOS are using them, doesn't ring the bell? Make a good converter then. G3db resulting file from maya->fbx>fbx-convert>g3db is always crashing my app at loading time with some exception, inside the lib. I'll post the error later, when I reach the mac
The blender 2.8 and 2.9 plugin import export libgdx g3db adon is not working.
Why are making custom formats and do not keep up with the new software? (Maya, 3dmax or at least Blender)

https://github.com/mgsx-dev/gdx-gltf

please don't open new issues for the same question as #6287
and please keep cool in your communication, thank you.

https://www.donmccurdy.com/2017/06/27/exporting-gltf-2-0-from-maya-lt-2/ all converters are or in alpha/beta or experimental, so a big NO

I'm just going to reply here and not on your other issue. Saying that DAE is the "new standard" is kinda dishonest. Just take a look at this blog post of the team behind the Godot Engine: there are a lot of issues with Collada/DAE and the adoption rate of FBX is probably higher. Also, Apple nowadays seems to lean towards using their own USDZ format.
If you don't want to use G3DB, the better format to use here is GLTF, which as @mgsx-dev pointed out can easily be used in libGDX via gdx-gltf. It is unfortunate that Autodesk is not supporting GLTF after all those years (and I guess that is intentional on their part), but that is more of an issue with Maya and not libGDX.

G3db resulting file from maya->fbx>fbx-convert>g3db is always crashing my app at loading time with some exception, inside the lib. I'll post the error later, when I reach the mac

This seems to be your real issue. If you need help figuring out, what is causing the crash, it would have probably been easier to just ask in the 3D channel on our Discord instead of going off on your rant here.


correct_negative_angle was taken out of the class.

correct_negative_angle is not an attribute: hiveplot.py #371, but it was being red from self.

Thanks @leotorr!


Latency Benchmarks

Have you ever, or would you consider, providing latency benchmarks? It'd be nice to know for sure that keyd doesn't add more than a handful of microseconds of typing latency in general.

You can run make test-io in the latest commit to get a sense of how much time is spent in the inner loop. It is on the order of single digit microseconds.
This does not account for any additional latency introduced by reading/writing to the kernel subsystems, which constitutes the bulk of time and is on the order of 30-40 us in my experiments (I haven't bothered to polish or publish these end to end tests). These will be hard limits for any remapping software.
The result should be under 1ms (by at least an order of magnitude), which is well below perceptual limits, even for those who are picky ;).

Good to hear! I'm satisfied with that


Auto update

@oliver-butterley @austinletson
Please do not merge this PR
This was implemented in an attempt to integrate lean-update actions into lean-action.
Summary of code

The update action is only set to be performed when executed as schedule and when debugging options are passed.
There is a part of the lean-update action that calls lean-action, but how to implement that part is difficult. A naive approach is likely to lead to more repetition of code, and commonisation would make it more complex.

My Opinion
I would suggest that the GitHub Action for automatic updates should be managed as a separate Action.Why not clone lean-update and create a repository under leanprover?
Reasons:

There is overlapping processing between lean-update and lean-action. (Run build and test on update, download mathlib cache, etc.) But this could be solved by calling lean-action from the lean-update side.
Combining lean-update and lean-action would complicate the management of if-conditions, make debugging difficult, and increase the scope of influence when changes are made.

Thanks. Please tell me your opinion.

@Seasawher, thank you for exploring this!
Your reasoning makes sense to me, especially regarding the maintainability of a complex action.yml. I have encountered maintainability issues as the action.yml of lean-action has become more complicated.
I like your recommendation to keep lean-action and lean-update separated but reuse as much functionality from lean-action as is practical within lean-update to avoid duplicated functionality. (This is already the case to some extent.)
@oliver-butterley what do you think?

Merge the existing version of lean-update into lean-action:question:  No

Would be unpleasant from the code maintainability side, would create a very complex action.yml :scream:. Also, currently lean-update uses lean-action in order to test if an update attempt works.
Currently lean-update isn't sufficiently tested for us to consider it v1 release material, confusing to the user if one functionality of the action were considered alpha whilst the rest is relatively solid.

I agree with @Seasawher's opinion on the current versions of the actions:

Combining lean-update and lean-action would complicate the management of if-conditions, make debugging difficult, and increase the scope of influence when changes are made.

Eventually merge lean-update into lean-action:question:
Keep separate because:

The tasks performed by lean-action and lean-update fit into different categories in the sense that the first is stuff that would run on a CI workflow, typically to confirm a PR works correctly, whereas the latter would run on cron and is likely to open a PR, open an issue or commit directly.
Reducing the functionality of each for easier maintainability of the code.

Merge because:

One action to rule them all, fits the pattern already established in lean-action to have a single action with multiple functionality selected by options.
There is functionality that can be shared by the two actions.

Next steps for lean-update:question:

[ ] Test lean-update in more places. @Seasawher has already been doing this for a while. @pitmonticone has just started testing it in various projects and getting valuable feedback.
[ ] Make sure that users can have it running on cron in their repo with minimal problems even if rather unaware of git and github technicalities. This includes making sure that it is hard to create a huge mess by accident and that feedback to the user is clear.
[ ] Convert lean-update to a typescript/javascript action to better deal with the logic, improve the feedback to the user (particularly when problems occur) and improve the way that PR and issues are opened/closed without relying on other actions.
[ ] Massively improve the feedback to the user when problems are detected.
[ ] Move development to a repo within lean-prover?
[ ] Prepare a roadmap to eventually merge with 'lean-action'?

@austinletson

Test lean-update in more places. @Seasawher has already been doing this for a while. @pitmonticone has just started testing it in various projects and getting valuable feedback.

I agree that lean-update should be introduced and tested in more repositories. What do you think?

@austinletson

Test lean-update in more places. @Seasawher has already been doing this for a while. @pitmonticone has just started testing it in various projects and getting valuable feedback.

I agree that lean-update should be introduced and tested in more repositories. What do you think?

I agree with the next steps for lean-update outlined above. To automatically add lean-update cron workflows to lake new, it should be tested on various projects as well as  iron out the  user experience and feedback, with a special focus on this point that Oliver made:

Make sure that users can have it running on cron in their repo with minimal problems even if rather unaware of git and github technicalities. This includes making sure that it is hard to create a huge mess by accident and that feedback to the user is clear.

Should we post in Zulip that we recommend keeping the actions separated for now and ask Kim if we can move the lean-update repo to lean-prover to prepare for a v1 release and then subsequently add a lean-update workflow when a new lean project is create with lake new?

It returns this error in all projects I've adopted lean-update: https://github.com/ImperialCollegeLondon/FLT/actions/runs/11176285474/job/31069319185#step:3:1

https://github.com/oliver-butterley/lean-update/issues/39

Solved.

Should we post in Zulip that we recommend keeping the actions separated for now and ask Kim if we can move the lean-update repo to lean-prover to prepare for a v1 release and then subsequently add a lean-update workflow when a new lean project is create with lake new?

nice!


Error in log for unrelated env variable in Metric server

Report
Keda metric server looks at unrelated environment variable on deployment & prints error message on log
Expected Behavior
Keda metric server should only be interested on the env variables on the deployment which are used by trigger-auth related to that Scaled Object/deployment
Actual Behavior
Keda metric server looks at unrelated environment variable on deployment & prints error message on log
Steps to Reproduce the Problem

Create a deployment which has env variable say ABCD  with value status.hostIP
Create Scalar (in my case SQS) and corresponding trigger-auth
Keda operator will start logging the error while running metric polling each time.

Logs from KEDA operator
Keda Metric server log, which prints each time the metric is polled.
I0826 04:32:07.903733       1 scale_resolvers.go:313] keda_metrics_adapter/scalehandler "msg"="cannot resolve env to a value. fieldRef and resourceFieldRef env are skipped" "name"="adapter" "namespace"="sbx" "type"="ScaledObject" "env-var-name"="ABCD"

KEDA Version
2.4.0
Kubernetes Version
1.21
Platform
Amazon Web Services
Scaler Details
SQS
Anything else?
No response

Are you able to test a Deployment without the potentially problematic ENV variable? SIGNALFX_AGENT_HOST ?
Would you mind pasting (redacted) envs section from the Deployment as it is defined, ie. not describe output, but kubectl get -oyaml ?
Thanks

Unfortunately i cant remove SIGNALFX_AGENT_HOST as it fail the deployment.
Below is the result of kubectl get deploy adapter -n sbx -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2021-08-26T03:02:40Z"
  generation: 31
  labels:
    app: adapter
  name: adapter
  namespace: sbx
  resourceVersion: "437585"
  uid: 84cdf64a-83ac-4dfc-ab31-6d180ab1916a
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: adapter
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: adapter
    spec:
      automountServiceAccountToken: true
      containers:
      - env:
        - name: SPRING_PROFILES_ACTIVE
          value: sbx
        - name: SPLUNK_ACCESS_TOKEN
          value: <redacted>
        - name: APP_NAME
          value: adapter
        - name: SIGNALFX_AGENT_HOST
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.hostIP
        image: <redacted>.dkr.ecr.us-west-2.amazonaws.com/adapter:sbx-testing-2
        imagePullPolicy: IfNotPresent
        name: adapter
        resources:
          requests:
            cpu: 75m
            memory: 1200Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      enableServiceLinks: true
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 65534
        runAsNonRoot: false
      serviceAccount: adapter
      serviceAccountName: adapter
      shareProcessNamespace: false
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2021-08-26T03:02:40Z"
    lastUpdateTime: "2021-08-26T03:02:55Z"
    message: ReplicaSet "adapter-85bb95ffb6" has successfully
      progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2021-08-26T23:00:59Z"
    lastUpdateTime: "2021-08-26T23:00:59Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 31
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

Okay, I can see the problem. It is a fieldRef which is not supported and this env is skipped.
@rangarb885 do you actually see error messages in the Metrics Adapter? Or just a log message? Do you see the the message with default log level setting?

@zroubalik  :
Log messages seen under DEBUG level. I have enabled DEBUG for our monitoring purpose.
I see this log entry on both keda-operator logs as well as keda-operator-metrics-apiserver logs.
2021-08-31T15:42:39.300Z	DEBUG	scalehandler	cannot resolve env to a value. fieldRef and resourceFieldRef env are skipped	{"type": "ScaledObject", "namespace": "sbx", "name": "adapter", "env-var-name": "SIGNALFX_AGENT_HOST"}

Okay, so this is expected, it is defined to be printed as Debug message.
https://github.com/kedacore/keda/blob/0e4a01f55cec12e65d11041f40f9d98d179ae2e5/pkg/scaling/resolver/scale_resolvers.go#L312-L315
What do you suggest?

IMO as keda is not going to use fieldRef and resourceFieldRef , we can safely ignore them from logging as well.


Viewer reports twice the number of expected results

I loaded a SARIF file with 11008 results. The VS viewer:

and the web viewer:

agree on this number. But the VS Code viewer reports 22016 results:

When I close and reopen VS Code and then load the SARIF file again, the number of issues is correct. The problem might be triggered by (for example) opening the same file twice, but I haven't been able to repro it. I'll reopen this if I see it again.


Feature Request: Swift Package Manager Support

Please provide support for Swift Package Manager

Hi, @ragzy15 Thank you for submitting the Feature Request. I will take this up for discussion with our engineering team and keep you updated here.

I want to +1 this. This would make integration 100x easier and less painful than manual integration for people who don't use Cocoapods.

Hi @Aditi3, is there any update regarding swift package support?

Please add support for SPM

Please add support for swift package manager

Hi All,
Thank you for your feedback 👍
We have taken up this request in our current quarter. The support for Swift Package Manager will be available in the first/ second week of October.
Best,
Aditi

Thank you!

Thanks for your support @Aditi3 👍

Any updated related to this @Aditi3 , seems like the latest version 3.9.1 still doesn't mention any support for SPM
Is it still planned for October release or pushed further back to November or later?
https://github.com/CleverTap/clevertap-ios-sdk/releases

Hi @baraeb92
Well, yes our latest release includes iOS 14.0 updates and some minor fixes. Due to the unexpected assignments, there's a delay in enabling swift package manager support.
Nevertheless, Swift Package Manager is next up in our iOS Roadmap. I will keep you posted with the timeline once we finalize internally.

Follow the progress at https://github.com/CleverTap/clevertap-ios-sdk/tree/feature/swift-pm. PRs and Suggestions welcome!

Hi @Aditi3, can you provide a timeline as when the version 3.9.2 will be released with swift package manager support?

Hey @ragzy15, Apologies for the delayed response. There’s a delay in the release of version 3.9.2 due to unforeseen issues and we're planning to append another feature request in the same version.
At present, we're aiming to release version 3.9.2 will include Swift Package Manager support by the end of December.

Hi @Aditi3 any updates regarding release of swift package manager support?

Hi @Aditi3 any updates regarding release of swift package manager support?

+1
I found clevertap uses SDWebImage package which I added in SPM, thus I have two SDWebImage libraries(one in cocoapods and the other in SPM) in the project.
Found this in Podfile.lock file.

CleverTap-iOS-SDK (3.9.1):

SDWebImage (~> 5.1)

And I'm getting this error:
Class SDAnimatedImage is implemented in both /private/var/containers/Bundle/Application/....app/Frameworks/SDWebImage.framework/SDWebImage (0x107f367f0) and /private/var/containers/Bundle/Application/....app/MyApp (0x1059f8248). One of the two will be used. Which one is undefined.

+1
I found clevertap uses SDWebImage package which I added in SPM, thus I have two SDWebImage libraries(one in cocoapods and the other in SPM) in the project.
Found this in Podfile.lock file.

CleverTap-iOS-SDK (3.9.1):

SDWebImage (~> 5.1)

And I'm getting this error:
Class SDAnimatedImage is implemented in both /private/var/containers/Bundle/Application/....app/Frameworks/SDWebImage.framework/SDWebImage (0x107f367f0) and /private/var/containers/Bundle/Application/....app/MyApp (0x1059f8248). One of the two will be used. Which one is undefined.

Hey @Aditi3, any news about the official (tag) SPM support? Is feature/swift-pm considered a stable release?

I have taken up Swift Package Manager release discussion internally with my team as some High priority task. Allow me some time to share a new timeline for the official release.

Hi All,
We will be releasing Swift Package Manage support by mid-March.

Hey @Aditi3, It's already April :)
Is it officially released yet?

How does this work somebody to talk to me I've been trying to learn this I can't figure it all out just got it￼

The official release will be out in a day or two. Thanks for your patience.

@Aditi3 Can't wait! Thanks :)


Replace native code with regular functions

Use numpy.einsum() for sumouter and sumprod functions.

Looks good. I wonder whether the native code was originally used because these functions were slow through python? I suspect not, and this is a worthwhile change.
I actually just found a bug with the native compiler today, as '-fopenmp' wasn't a defined flag on the OSX compiler I was using. But this fixes it in a nicer way :)
If I'm not mistaken, this is the only place the nutils are used, so why not remove ocrolib/nutils.py and ocrolib/native.py too?

Hi Nick!

I wonder whether the native code was originally used because these functions were slow through python?

I don't know... :-)
OpenMP
https://github.com/tmbdev/ocropy/blob/master/README_OSX.md
Actually, there is no actual OpenMP 'code' in nutils, so the '-fopenmp' flag is useless.

If I'm not mistaken, this is the only place the nutils are used, so why not remove ocrolib/nutils.py and ocrolib/native.py too?

Originally, I removed that code, but for this PR I decided to keep it. Later, it can be removed or moved to 'OLD'.

In my mini benchmarks the new functions are a few times faster than the native ones in nutils.
I ran ocropus-rtrain with '-N 1000'  on uw3-500.tgz and it looks like there is a ~20% improvement in speed for training. I think it's a good idea that someone else will verify this in his machine.

In lstm.py there are another sumouter and sumprod function.
The sumprod in lstm.py is not used in the code.
The sumouter is used in the softmax layer. For some reson it operates on lists instead on numpy ndarrays.

Thank you @amitdo.


Refactor AgentBatchTask.java

Reduce time complexity and avoid temporary object creation.

What's the purpose of this refactor? Could you give some message?

What's the purpose of this refactor? Could you give some message?

Done, PTAL

could you give the performance boost data in your test?


Updates for next release

Codecov Report

Merging #11839 (f6006ed) into main (1beeb8d) will decrease coverage by 0%.
The diff coverage is n/a.

@@          Coverage Diff           @@
##            main   #11839   +/-   ##
======================================
- Coverage     62%      62%   -1%
======================================
  Files        486      486
  Lines      34657    34657
  Branches    5636     5636
======================================
- Hits       21778    21735   -43
- Misses     10804    10819   +15
- Partials    2075     2103   +28

Impacted Files
Coverage Δ

src/kernels/execution/notebookUpdater.ts
88% <0%> (-8%)
:arrow_down:

src/kernels/jupyter/launcher/notebookProvider.ts
71% <0%> (-8%)
:arrow_down:

...s/extension-side/plotView/rendererCommunication.ts
55% <0%> (-7%)
:arrow_down:

...tform/interpreter/display/visibilityFilter.node.ts
85% <0%> (-6%)
:arrow_down:

...one/intellisense/notebookPythonPathService.node.ts
34% <0%> (-6%)
:arrow_down:

...latform/interpreter/workspaceInterpreterTracker.ts
85% <0%> (-5%)
:arrow_down:

...ls/raw/session/rawNotebookSupportedService.node.ts
86% <0%> (-5%)
:arrow_down:

...andalone/intellisense/intellisenseProvider.node.ts
77% <0%> (-5%)
:arrow_down:

...c/notebooks/telemetry/interpreterPackageTracker.ts
73% <0%> (-4%)
:arrow_down:

...otebooks/controllers/controllerPreferredService.ts
76% <0%> (-4%)
:arrow_down:

... and 17 more


Tests concerning PUTs can't coexist with other JUnit 5 tests with pact-jvm-consumer-junit5_2.12

It would appear that having consumer code that issue PUT requests on Pact's mock server cause issues if tests co-exist in the same JUnit test class.
I've created a simple exemplar project to replicate this issue. To replicate, follow these instructions:
git clone git@github.com:paulspencerwilliams/BugExampleWithPactAndPutRequests.git
./gradlew test -i
git checkout works-with-posts
./gradlew test -i

The second test on the works-with-posts branch would appear to prove this is PUT related.

Hi @paulspencerwilliams ,
I faced an issue with PUT request mixed with others as your case. And i find out that i must use a wrapper for RestTemplate (configure a timeout value) before calling it. IThe mock server maybe need a wait-time/timeout to proceed the PUT request, refer to #8 Configure Timeout of this link for more detailed info:
http://www.baeldung.com/rest-template
Hope it help@

Hey @minhdoan159, a colleague also discovered the timeout hack, and we've temporarily worked around this issue using     @BeforeEach     public void pause() throws InterruptedException {       Thread.sleep(5000L);     }

Thanks for the example project!
My guess is that this is related to the RestTemplate using a connection cache (see #342 for a similar issue). As each test gets a new mock server, any cached connection on the second test will be stale. A five second sleep might be enough time for the RestTemplate HTTP connection library to get a new connection.
Changing the test to use a different port for each test passes.

Hi guys, just had the same issue with HTTP PATCH, so not only PUT

It won't be dependent on the HTTP method, but on how the HTTP client library caches the connection.


Extension failing to recognise my settings.json

Describe the bug
I've got some code which includes non-standard symbols like +=, I have set it as a non-standard symbol to ignore in the extension settings of the lua language server and checked the settings.json of VSCode as well to make sure it's there, but it still errors out on them failing to recognise it's there
To Reproduce
Steps to reproduce the behavior:

Go to the extension settings
Add the += non-standard symbol to that option
Add some code in a .lua file including +=
See error

Expected behavior
It not erroring out on the code as it wasn't doing that a few days ago.
Screenshots
error in the code

extension settings

settings.json

Environment (please complete the following information):

OS: Windows 10
Is WSL remote? No
Client: VSCode

Additional context
It was working fine a few days ago, I haven't touched the code since last week
Provide logs
service.log

https://github.com/sumneko/lua-language-server/commit/25a78532b67032176b6c255019c203927cd004b4
Seemed to have fixed it


Distinct Query Continuation Token Support

Distinct Query Continuation Token Support
Description
Adding continuation token support for DISTINCT queries. The general idea is that users currently aren't able to run DISTINCT queries through the continuation token API without a matching ORDER BY clause. It's very limiting for these users. The solution was to add continuation token support for these types of queries. The general continuation token is like so:
{
    "SourceContinuationToken": <SourceContinuationToken>,
    "DistinctMapContinuationToken": <DistinctMapContinuationToken>
}

This means we are serializing both the base continuation token and serializing the distinct map. If it's a DISTINCT query + matching ORDER BY, then the token is just the most recent hash. Otherwise we dump the whole hash table which looks like this:
{
    "Numbers": [<Number64>, <Number64>, ..., <Number64>],
    "StringsLength4": [<uint32>, <uint32>, ..., <uint32>],
    "StringsLength8": [<ulong>, <ulong>, ..., <ulong>],
     .
     .
     .
     "StringsLength24Plus": [<UInt192>, <UInt192>, ..., <UInt192>],
     "Arrays": [<UInt192>, <UInt192>, ..., <UInt192>],
     "Object": [<UInt192>, <UInt192>, ..., <UInt192>],
}

Basically it's a dictionary of hashes which represents the state of the operator.
This dictionary is serialized using binary encoding, since the hashes are binary. The whole byte array is then Base64 encoded, so that it can go in a continuation token.
Type of change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[ x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Will CTLimitsinKB impact the completeness of CT for distinct's?

Lets please sync-up once what it means for other SDK's?
Replicating all of these including CosmosElements is a very expensive work.

There is no replication for other SDKs. As far as I know the other SDKs gave up on supporting continuation tokens a long time ago.

only shipping through TryGetContinuationToken.

In reply to: 535514974 [](ancestors = 535514974)


Match google.cloud.bigquery arguments in read_gbq

Passing project, dataset_id, table_id separately seemed like the right thing to do at first, but I've come around to thinking that it makes more sense to just mirror what the bigquery library does in most cases (see e.g. this fn): that is, accept a string project.dataset.table (or even dataset.table if a default project is set) or a table/table reference object. Regardless it seems like a good idea to keep things consistent with the upstream package, and personally I've found that passing around the single string or object just makes the code a lot less cluttered.

I wonder if this would work when you have an example like this one, where you are authenticated on "my_project" but reading from a public data set.
test = dask_bigquery.read_gbq(project_id="dask-bigquery",
                             dataset_id="bigquery-public-data.covid19_aha",    # this can be a bit confusing.
                             table_id="hospital_beds",
                             )

I wonder if this would work when you have an example like this one, where you are authenticated on "my_project" but reading from a public data set.

In retrospect, the project_id parameter in pandas-gbq should probably be named billing_project_id. I think you will need at least two parameters here, with the project_id/billing_project_id being optional, since it can sometimes be detected from the environment.

Going to close this since it seems like the 🚢  has sailed on a big top-level change like this


updated d.py to 1.4.1 and added voice as extra

"Closes #432 "
uninstalled old dpy and installed new d.py with voice as extra

[x] Join the Python Discord Community?
[x] If dependencies have been added or updated, run pipenv lock?
[x] Lint your code (pipenv run lint)?
[x] Set the PR to allow edits from contributors?

In Issue 432 reason is mentioned ,we require this for spooky sound command 😁


fixed homedir locations

Homedir in upstart script was fixed dir, made it parameter.

Set homedir to parameter when creating user.

thanks!


Fix #217: medium font not working. Adds disableDefaultFonts() method.

I'm not sure this is the best solution, as it requires the user to call the extra method. If you have a better idea, feel free to reject this pull request.

Could you possibly look at this too? https://github.com/afollestad/material-dialogs/issues/219
And thank you

I realized that it might've been easier to put the TTF files back in for Roboto and Roboto Medium, and always use them unless the disableCustomFonts() method is used. That way better backwards compatibility is maintained.

Should be in the latest commit now, but I gotta test it after class.


Problems with 24.11

I've been trying to update my system flake to 24.11 for the past week and been running into issues and I finally just now tracked down one of the issues to the Rust in flake-checker causing some compilation issues I think.
This is all the related error message I've got.
building '/nix/store/mkcqb9s5ah59vl8fx7w038ipl8vvbb1l-cargo-package-wasm-streams-0.4.0.drv'...
building '/nix/store/3m7d3ym0db8ndyxd3dqjlihgvk289nmv-hadoop-conf.drv'...
building '/nix/store/jjlyw8mqfx9dkd4k2pxri4k1lpx4ckrk-nixos-tmpfiles.d.drv'...
building '/nix/store/zm15jva0vn32pv2vw2z744pdns5yqn93-unit-qdrant.service.drv'...
error: builder for '/nix/store/sm56fw57zraqdpmwcbcsp5vwm80ipasl-flake-checker-0.2.0.drv' failed with exit code 101;
       last 44 log lines:
       > Running phase: unpackPhase
       > unpacking source archive /nix/store/4mn8pnmpsqx2d8nx0283vqcbxfmiqydr-kp6aq2vim1rg0qr6qmm4kdcfvizin22r-source
       > source root is kp6aq2vim1rg0qr6qmm4kdcfvizin22r-source
       > Running phase: patchPhase
       > Executing configureCargoCommonVars
       > decompressing cargo artifacts from /nix/store/qcraircjpr14qxpkhzy4phmdhzgxpmbi-flake-checker-deps-0.2.0/target.tar.zst to target
       > Running phase: updateAutotoolsGnuConfigScriptsPhase
       > Running phase: configurePhase
       > will append /build/kp6aq2vim1rg0qr6qmm4kdcfvizin22r-source/.cargo-home/config.toml with contents of /nix/store/19pg6lfvzd903h34b2g0kwbk5dks08a2-vendor-cargo-deps/config.toml
       > default configurePhase, nothing to do
       > Running phase: buildPhase
       > ++ command cargo --version
       > cargo 1.81.0-nightly (4dcbca118 2024-06-11)
       > ++ command cargo build --release --message-format json-render-diagnostics --target x86_64-unknown-linux-musl
       >    Compiling parse-flake-lock v0.1.1 (/build/kp6aq2vim1rg0qr6qmm4kdcfvizin22r-source/parse-flake-lock)
       >    Compiling flake-checker v0.2.0 (/build/kp6aq2vim1rg0qr6qmm4kdcfvizin22r-source)
       >     Finished `release` profile [optimized] target(s) in 3.64s
       > Running phase: checkPhase
       > ++ command cargo test --release --target x86_64-unknown-linux-musl
       >    Compiling flake-checker v0.2.0 (/build/kp6aq2vim1rg0qr6qmm4kdcfvizin22r-source)
       >     Finished `release` profile [optimized] target(s) in 1.13s
       >      Running unittests src/main.rs (target/x86_64-unknown-linux-musl/release/deps/flake_checker-897095c8ed7175de)
       >
       > running 5 tests
       > test flake::test::explicit_nixpkgs_keys ... ok
       > test flake::test::dirty_flake_locks ... ok
       > test flake::test::missing_nixpkgs_keys ... ok
       > test flake::test::clean_flake_locks ... ok
       > test flake::test::cel_conditions ... FAILED
       >
       > failures:
       >
       > ---- flake::test::cel_conditions stdout ----
       > thread 'flake::test::cel_conditions' panicked at src/flake.rs:193:17:
       > assertion failed: result.unwrap().is_empty()
       > note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
       >
       >
       > failures:
       >     flake::test::cel_conditions
       >
       > test result: FAILED. 4 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
       >
       > error: test failed, to rerun pass `--bin flake-checker`
       For full logs, run 'nix log /nix/store/sm56fw57zraqdpmwcbcsp5vwm80ipasl-flake-checker-0.2.0.drv'.
copying path '/nix/store/518rxhpb002g40rnm3x5iy4ykm5iqd5j-libpulseaudio-17.0' from 'https://cache.nixos.org'...
copying path '/nix/store/29axv68nmjdda61s0ll2pjvd31jgm2jq-ktextwidgets-5.116.0' from 'https://cache.nixos.org'...
copying path '/nix/store/3hv1xz8pls32hnncvfsk49xzsrkypcf4-python3.11-qtconsole-5.6.1' from 'https://cache.nixos.org'...
copying path '/nix/store/1m642r470vs78ryzdglj3lixx76k179q-python3.11-opentelemetry-instrumentation-fastapi-0.48b0' from 'https://cache.nixos.org'...
copying path '/nix/store/90z9i95sv3xiwdyfq4d7bk9kiqyr9lnd-qtmultimedia-5.15.15-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/b9gykb4vc0848icri1ffy0p92c3h4vc6-qttools-5.15.15-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/vh4vd8c0hskpqjfkgxpz7sixcyhsxq0w-solid-5.116.0-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/hxmkjh3fxp547glapmv4i93hvypj3zgc-sonnet-5.116.0-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/zzq649ji5ishj6b6bixlfa6732r1z6q7-x264-0-unstable-2023-10-01-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/jd47wyijsnaxwfqk4826kssa257pr2yj-x265-3.6' from 'https://cache.nixos.org'...
copying path '/nix/store/pldvs6gvbxm347x9vc087nxvpxq80l70-xavs-55' from 'https://cache.nixos.org'...
copying path '/nix/store/002wrzdz2bq6mqiwjf7aq91lnvc6sfxb-xcb-util-keysyms-0.4.1' from 'https://cache.nixos.org'...
copying path '/nix/store/by90jxqx8dg4ca594szv1m5pyjkl8bzw-xmlsec-1.3.5' from 'https://cache.nixos.org'...
copying path '/nix/store/d44hkx0x8jym9md4bvq6z9jp4fwjpj5l-xevd-0.5.0-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/zcwhzy3r7qm3ljcn3w1bd84hbnahkkcj-xeve-0.5.1-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/gi1nwzhsyxpmsfdm9nfw3fxsy2iy69fm-xvidcore-1.3.7' from 'https://cache.nixos.org'...
error: 1 dependencies of derivation '/nix/store/crw9s2wgvnxl3v0hxy4wmcva33sxf1v6-system-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/ak77dz3i7mm9mfp31ibfgf731vnp3l85-nixos-system-reckless-24.11.20241116.057f63b.drv' failed to build

--show-trace isn't super helpful.. I just removed it from my flake for the time being.
For reference my flake and the branch I am working the update on are at:
https://gitlab.com/usmcamp0811/dotfiles/-/commits/nixos24.11-update

I have the exact same issue with flake-checker v0.2.0 and  flake::test::cel_conditions in NixOS 25.05 (unstable).

@usmcamp0811 24.11 is not yet considered stable, so these errors aren't too surprising. I'd recommend waiting until it's stable and trying again.

For reference, nixos-24.11-small has finished the staging cycle and is the RC. It is not able to build flake-checker with same error as described above.

@usmcamp0811 Okay, I just verified that this wasn't a 24.11 issue at all. #145 should address this.

As a small update, the current main branch has resolved this issue.


Changing the binary message format?

Currently Ockam messages are encoded with BARE, a binary data format that optimises for compactness and simplicity. It does that by requiring external schema information which allows it to not encode a lot of structural information. Apart from actual data such as string values or numbers it only adds some length information where necessary, but not much else. Insofar the format has only minimal overhead. The flip side of this approach is that without the correct schema, BARE messages can not be decoded. This prevents generic message processing and makes it difficult to have support for forward and backward compatibility[^1] as it is not possible to skip over message parts without knowing their meaning. The ambiguities that may arise surface in functions like ockam_node::parser::message which retries message decoding should the first decoding attempt fail, by modifying the message. The appendix below contains further examples that highlight the importance of accurate schema information.
[^1]: Section 4 of the RFC draft of BARE mentions schema evolution in a backward and forward compatible way by using tagged unions with explicit tag numbers. While adding another tagged branch is backward compatible because newer software understands the messages of older software, the reverse is not true: a message using this new tag will be rejected by older software, hence forward compatibility is not realised.
Many formats work with schema information and it is a great way to save precious bytes, however it would be nice if in addition, messages have enough embedded structure so that forward and backward compatibility is possible and maybe also generic processing which can help with debugging or logging.
Are there alternatives to BARE which offer this without sacrificing any of the other important requirements, i.e.

a compact, simple and language agnostic format, with
simple codecs that have minimal resource requirements, which are capable of
safe message processing that handles malformed input gracefully without causing resource exhaustion or violating memory safety?

I would like to argue for CBOR here which is not as compact as BARE, but should otherwise offer all of this.
In contrast to BARE, CBOR does encode more structural information which increases message size. Every CBOR data item is prefixed with a byte that encodes the item type as well as additional information (mostly about how to load the data, e.g. length information). This enables generic processing of arbitrary data items, for example to map them to a human readable format. This also makes it possible to skip over message parts which in turn allows for forward and backward compatibility. The interesting question is how much overhead does this cost in practice? In https://github.com/twittner/encoding-size I attempt to do a comparison between BARE and CBOR. As data I use randomly generated trees of structs and enums with additional data items such as strings and numbers. Here is an example run:
>>> TINY
BARE = 53 bytes ( 82.81%)
CBOR = 64 bytes (120.75%)
>>> SMALL
BARE = 14754 bytes ( 92.56%)
CBOR = 15940 bytes (108.04%)
>>> MEDIUM
BARE = 800402 bytes ( 97.13%)
CBOR = 824074 bytes (102.96%)
>>> LARGE
BARE = 15972854 bytes ( 99.35%)
CBOR = 16077387 bytes (100.65%)

The labels "TINY", "MEDIUM", etc. classify the ratio of plain data sizes (e.g. the length of strings) to the overall structure, for example TINY means that the message contains very short strings and vectors. As expected the overhead of CBOR is higher the more structural information is encoded relative to plain data items. However I think the differences are fairly small, especially when considering absolute numbers.
More interesting than this is how this would work out for the ockam repository. In here I moved most of the message types from BARE to CBOR and printed the message lengths from ockam_core::message::Encodable::encode for the develop branch as well as for my CBOR branch while running cargo test --all -j 1 -- --test-threads=1 --nocapture. On average, BARE encoded messages are 256 bytes in size, whereas the CBOR encoded ones measure 285 bytes on average, confirming the above results (89%, 111%). (The CSV with all printed sizes can be found here).
If the increase in size is acceptable and there is agreement that it would be desirable to have generic message processing and the possibility of forward and backward compatibility, the  branch could be polished and become a pull request. However I am just exploring a possibility here and would like to discuss this first before investing more time.
For further information about CBOR see https://cbor.io. Forward and backward compatibility with application to CBOR is described in more detail here.
Appendix
Example 1

Flattened structs
use serde::{Serialize, Deserialize};

fn main() {
    // Structs in BARE are always flattened. Members are just concatenated and
    // the nesting is not reflected in the format. This makes struct values
    // potentially very very compact.

    #[derive(Debug, PartialEq, Eq, Serialize, Deserialize)]
    struct S(String);

    #[derive(Debug, PartialEq, Eq, Serialize, Deserialize)]
    struct T(S, bool);

    let t1  = T(S("hello".to_string()), true);
    let buf = serde_bare::to_vec(&t1).unwrap();
    let t2: T = serde_bare::from_slice(&buf).unwrap();
    assert_eq!(t1, t2);

    // The resulting byte sequence looks like this:
    // ┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐
    // │00000000│ 05 68 65 6c 6c 6f 01    ┊                         │•hello• ┊        │
    // └────────┴─────────────────────────┴─────────────────────────┴────────┴────────┘
    // Only the string value (prepended by its length) and the bool value are encoded.
    //
    // NB that the absence of schema information makes it possible to decode any struct
    // prefix on its own, e.g. we could parse this just as an `S` instead of a `T`:

    let s1: S = serde_bare::from_slice(&buf).unwrap();
    assert_eq!(t1.0, s1);

    // This breaks down of course when the suffix is not ignored, e.g. due to repetition:

    let buf = serde_bare::to_vec(&(&t1, &t1)).unwrap();

    // ┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐
    // │00000000│ 05 68 65 6c 6c 6f 01 05 ┊ 68 65 6c 6c 6f 01       │•hello••┊hello•  │
    // └────────┴─────────────────────────┴─────────────────────────┴────────┴────────┘

    let s2: (S, S) = serde_bare::from_slice(&buf).unwrap();
    assert_eq!(s2, (S("hello".to_string()), S(String::from(char::from(5)))));

    // Here, after the first string, the next byte `01` is read as the length of the
    // next string and therefore the `05` is read as a UTF-8 string value.
    //
    // This shows how important the schema information is. Without an accurate schema
    // the decoding can easily go astray.
}

Example 2

The void type
use serde::{Serialize, Deserialize};

fn main() {
    // Empty structs and the unit type are not encoded, i.e. in RFC terms their
    // schema is the void type. According to section 2.4, void types "MUST NOT
    // be used as an optional type, struct member, list member, map key, or map
    // value. Void types may only be used as members of the set of types in a
    // tagged union."
    //
    // When asked about this restriction the author answered with "To avoid
    // getting to futzy with the type system. I want BARE message specifications
    // to be reasonably close to what you're going to see on the wire. It's
    // designed to avoid getting too clever with types. And yes, some languages
    // don't have unit types, and this helps with broader compatibility." [1]
    //
    // Despite this, serde_bare accepts empty structs and treats them as void,
    // i.e. it does not encode them:
    #[derive(Debug, PartialEq, Eq, Serialize, Deserialize)]
    struct A;

    #[derive(Debug, PartialEq, Eq, Serialize, Deserialize)]
    struct B(A);

    #[derive(Debug, PartialEq, Eq, Serialize, Deserialize)]
    struct C(B);

    let c1  = C(B(A));
    let buf = serde_bare::to_vec(&c1).unwrap();
    assert!(buf.is_empty());
    let c2: C = serde_bare::from_slice(&buf).unwrap();
    assert_eq!(c1, c2);
}

// [1]: https://lists.sr.ht/~sircmpwn/public-inbox/%3CC106F0AA-BB7D-4B8C-A105-82C0F6BB4687%40taglialegne.it%3E#%3CC4HL2XIJK0XA.1OU5CCJ34OVTF@homura%3E

@twittner thank you for starting this discussion and writing such an insightful analysis ❤
I'll spend more time with your notes soon, but here's one idea that we should consider ...
Certain messages on the routing/messaging layer (e.g TransportMessage or SecureChannel's data) are wrapped inside each other multiple times .. to support tunneling.
For such messages ... overheads compound quickly, and compactness is more important .. especially to maintain support for low-power wireless protocols. It would be okay to compromise on ease of use at this low-level- if we have to. Safety is critical at this layer but the surface we will have to check is relatively small.
As we move up layers to protocols that are end-to-end, compactness is less important because the cost is not compounding. Ease of use is significantly more important. Safety critical surface is relatively large.
I think we should consider CBOR at these higher level layers - but keep BARE (or something similarly compact) at the routing/messaging layer.

Fascinating -- love that you actually approached this by measuring, and the change looks very nice.
I also wasn't aware of https://docs.rs/minicbor-derive/latest/minicbor_derive/index.html and impressed by the protobuf-style forward/backward compatibility -- very cool. (Honestly, I wasn't aware that you could replace the field names with indices at all -- that seems very useful).
Some questions I have are:

Does the relatively low size cost you demonstrate hold up with types different than what we currently use in Ockam. For example, Vec<i32> or Vec<String>?

There are cases where we wrap and re-wrap messages multiple times for serializaion. For bare, this, in it's plainest form of repeatedly applying compression to a Vec<u8> (e.g. just to measure the part of the size cost that isn't "our fault"), ends up being about 8 bytes of overhead each time. How does CBOR scale under repeated re-encoding.

I'm familiar with Packed CBOR, which I believe was motivated by a desire to reduce CBOR message sizes. While I have some... doubts about Packed CBOR (I think they should just use real compression), I had figured the motivation was genuine. Is the difference here just the use of field indices and not names? Is there something I'm missing?

Also, I must admit, I'm a little unsure what point you're getting at with the two examples. Can you explain them a bit more?
Some other points:

This prevents generic message processing and makes it difficult to have support for forward and backward compatibility as it is not possible to skip over message parts without knowing their meaning

Yeah, this is a big issue in practice -- as it's very easy to #[cfg(...)] away a field of a struct, and it's tedious to write serde deserializers that handle this. In #2492 I have this issue, and note it here.

This is a great analysis @twittner, I hadn't come across CBOR before.
I'd also be interested to see numbers for encoding efficiency with nested messages as @mrinalwadhwa mentioned!
My biggest outstanding question is around implementing CBOR on embedded targets.

Are there any codec implementation available that can build on no_std targets?
If there are, can any of them also be used in a heapless, statically allocated environment? (i.e. does not perform runtime memory allocation or require the alloc library)

Has anyone looked at RION? Their design goals are looking tempting
Also I could argue for protobuf. Having language-independent schema that can generate code for you is very useful. We already came across some problems serializing messages between Rust <-> Elixir. Imagine if we had 10 implementations - adding new message struct would require a lot of attention and testing.

Thank you all for your comments! I am going to reply in one go:

@mrinalwadhwa: Certain messages on the routing/messaging layer (e.g TransportMessage or SecureChannel's data) are wrapped inside each other multiple times .. to support tunneling. For such messages ... overheads compound quickly, and compactness is more important [...]
@thomcc: How does CBOR scale under repeated re-encoding.
@antoinevg: I'd also be interested to see numbers for encoding efficiency with nested messages as @mrinalwadhwa mentioned!

CBOR is less good than BARE as the structural overhead is repeated, but it depends on how much the structure dominates the message relative to plain data. To give a contrived example, let us consider this type:
#[derive(Encode, Decode, Serialize, Deserialize)]
struct S {
    #[cbor(n(0), with = "minicbor::bytes")] a: Vec<u8>
}

and use S to recursively encode itself:
fn wrap(rem: usize, to_vec: &impl Fn(S) -> Vec<u8>) -> S {
    if rem == 0 {
        S { a: Vec::new() }
    } else {
        S { a: to_vec(wrap(rem - 1, to_vec)) }
    }
}

BARE only encodes the vector length and its bytes whereas CBOR adds information about the structure itself. The byte lengths for the number of recursion steps are:
   0: BARE = 1 bytes (50.00%), CBOR = 2 bytes (200.00%)
   1: BARE = 2 bytes (50.00%), CBOR = 4 bytes (200.00%)
...
  10: BARE = 10 bytes (50.00%), CBOR = 20 bytes (200.00%)
...
 100: BARE = 100 bytes (33.56%), CBOR = 298 bytes (298.00%)
...
1000: BARE = 1872 bytes (48.02%), CBOR = 3898 bytes (208.23%)

The only thing we encode here is essentially (serialised) structural information and clearly BARE is more compact than CBOR. However I do maintain that types more commonly used, which also contain plain data such as strings are less pathological. If we change S for instance to also contain the string "hello world" as a second field the picture changes to the more familiar numbers I had reported previously:
   0: BARE = 13 bytes (92.86%), CBOR = 14 bytes (107.69%)
   1: BARE = 26 bytes (92.86%), CBOR = 28 bytes (107.69%)
...
  10: BARE = 130 bytes (87.84%), CBOR = 148 bytes (113.85%)
...
 100: BARE = 1390 bytes (87.97%), CBOR = 1580 bytes (113.67%)
...
1000: BARE = 13990 bytes (87.55%), CBOR = 15980 bytes (114.22%)

@thomcc: Does the relatively low size cost you demonstrate hold up with types different than what we currently use in Ockam. For example, Vec<i32> or Vec<String>?

In https://github.com/twittner/encoding-size/blob/master/src/main.rs#L13-L39 I tried to use a fairly general and recursive type that also uses a Vec<u16> and a Vec<String>, so the numbers I cited above should apply. Of course a Vec<T> with T not being a u8 takes up more space because in CBOR this will be an array of Ts and each element is prefixed with type information, whereas a Vec<u8> can be encoded more efficiently as a CBOR bytes type. It is also noteworthy that integers are always of variable length in CBOR, so even a u64 may end up occupying just one byte for very small values.

@thomcc: Packed CBOR [...] Is the difference here just the use of field indices and not names? Is there something I'm missing?

I think you understood correctly. I have not used packed CBOR here, but simply indices which in this case are mapped to array positions, but could also be used as keys in CBOR maps if a type is expected to have many gaps.

@thomcc: Also, I must admit, I'm a little unsure what point you're getting at with the two examples. Can you explain them a bit more?

Example 1 was meant to show how easy it is to be unaware of the boundaries between types. I stumbled across this when replacing BARE with CBOR because I got an error at https://github.com/ockam-network/ockam/blob/47a29d20081b52aca162aba9c3d19c88a69f6165/implementations/rust/ockam/ockam_transport_tcp/src/workers/sender.rs#L221 where the payload is decoded as a TransportMessage. IIUC the message is actually a LocalMessage and it surprised me at first that the decoding succeeded. Example 1 is really just an artefact of this investigation and for people familiar with BARE probably not very interesting.
Example 2 is about the void type and my confusion because I think serde_bare treats empty structs as void, but in places where the RFC states they must not be used (e.g. as struct fields, optional values etc). Again the example is more of a note and maybe I have misread the RFC.

@antoinevg: Are there any codec implementation available that can build on no_std targets? If there are, can any of them also be used in a heapless, statically allocated environment? (i.e. does not perform runtime memory allocation or require the alloc library)

Yes. The crate minicbor is meant to be used in a no_std environment and non-allocating.

@SanjoDeundiak: Has anyone looked at RION?

So far I have not but it seems to be similar to CBOR and MessagePack. I could not find a Rust implementation though. Do you know if there is one?

@SanjoDeundiak: Also I could argue for protobuf.

Yes, protocol buffers are nice in a way. I am not too fond of the code that is generated from the external schema and the schema language is quite restrictive compared to what is possible within programming languages, e.g. when it comes to generic types.

@SanjoDeundiak: We already came across some problems serializing messages between Rust <-> Elixir. Imagine if we had 10 implementations - adding new message struct would require a lot of attention and testing.

:+1: I did not focus on cross-language communication so far but it is absolutely important and with protocol buffers the schema usually comes first which takes care of that. With serialisation attributes on the other hand, be it serde or minicbor, the schema is embedded in the types and communicating this information correctly to another programming language is comparatively difficult. At least, even with formats that do not require a schema one can often still make use of one. For CBOR a schema description language CDDL has been standardised and there are tools to check that a CBOR item matches a schema. It still requires extra work compared to using protobuf which generates code that is guaranteed to conform to a schema.

Thank you @twittner for starting this discussion. We definitely have a bunch of foot-guns in the Rust code where we hack around weird BARE encoding mechanisms. We're also currently figuring out how to do encoding of metadata for higher level protocols so the timing for this discussion is impeccable :slightly_smiling_face:
I haven't worked with CBOR before but it reminds me of msgpack a lot. Having a self-describing encoding format imo only has upsides. The differences in size for simple datastructures are trivial in most cases. We will want to investigate the overhead this causes on embedded systems. But I also think that this is memory worth spending. Maybe instead we need to focus on optimising the runtime/ routing mechanisms themselves to reduce our memory footprint.

CBOR is less good than BARE as the structural overhead is repeated, but it depends on how much the structure dominates the message relative to plain data. To give a contrived example, let us consider this type:

While this is a pretty big limitation, I don't think we need to worry about it too much since a lot of our message nesting actually re-encodes payloads. We do this to create decoupling between workers to make the Rust type system happy, but it comes with the added benefit that, while we have deeply nested structures, encoding and decoding don't have to be fully aware of it.
Lastly: my experience with protobuf is that in most cases using it is overkill and causes more problems than it solves. We may want to investigate it nonetheless, but personally I think going with something like CBOR, which is well standardised and has implementations in many languages without relying on code-generators (meaning that writing a new implementation is also less daunting than writing a protobuf implementation), is the better way to go.

Having a self-describing encoding format imo only has upsides. The differences in size for simple datastructures are trivial in most cases. We will want to investigate the overhead this causes on embedded systems. But I also think that this is memory worth spending. Maybe instead we need to focus on optimising the runtime/ routing mechanisms themselves to reduce our memory footprint.

Absolutely, there are far larger embedded memory optimizations we can perform before we even begin to approach message format contributions! 👍

I think we will need to revive that discussion as we need to start implementing OpenTelemetry https://github.com/build-trust/ockam/discussions/2319 integration between Rust and Elixir as we need to accomodate telemetry metadata in the message format.


Servicemonitor create is not respecting the namespace

What did you do?
$ cat <<EOF | kubectl create -f -
apiVersion: monitoring.coreos.com/v1alpha1
kind: ServiceMonitor
namespace: logging
metadata:
  name: fluentd
  labels:
    app: fluentd
spec:
  jobLabel: fluentd
  selector:
    matchLabels:
      app: fluentd
  namespaceSelector:
    matchNames:
    - logging
EOF

What did you expect to see?
create service monitor in the namespace logging
What did you see instead? Under which circumstances?
Service monitor was created in the context namespace (default)
grubio@Giancarlos-MBP:~$ kubectl get servicemonitor  -n logging
No resources found.
grubio@Giancarlos-MBP:~$ kubectl get servicemonitor  -n default
NAME      KIND
fluentd   ServiceMonitor.v1alpha1.monitoring.coreos.com

environment
prom-operator v0.8.0
k8s 1.5.3
ps. if i run $ cat <<EOF | kubectl create -n logging -f - ... it works

Do the logs of the Prometheus Operator show anything? As there is no way to validate TPRs at creation at the apiserver, the error only occurs once they are processed by the Operator ... sadly.

Actually, re-reading the situation you described, this is intentionally not picked up by the Prometheus Operator in association with the Prometheus object, as the ServiceMonitors have to be in the same namespace as the Prometheus object that they should be selected by.

Sorry, third attempt ... it seems this might be a bug by third party resources in 1.5.x. I just tried this on a minikube cluster 1.6 and it doesn't seem to occur.

Sorry, third attempt ... it seems this might be a bug by third party resources in 1.5.x. I just tried this on a minikube cluster 1.6 and it doesn't seem to occur.

That's ok, I was suspecting this was related to k8s however I can't find an issue related. If in 1.6 is fixed I'm closing this issue. Thanks as always!

Always happy to help if I can! :slightly_smiling_face:


Azdo CI is failing but they are still green!

https://dev.azure.com/ms/vscode-python/_build/results?buildId=7329
Check every job that's green, they have all failed, but the status is still green!

Fixed as part of https://github.com/Microsoft/vscode-python/pull/4880


Add/Update/Remove for Hedgey Finance

Revision checks

[x] Used smart contracts were audited.
[x] Your Safe App include a manifest.json file at the root with the required data.
[x] The app can be loaded as a custom Safe App in the Apps section of https://app.safe.global.
[x] The app auto-connects to the Safe.
[x] You are able to trigger and execute one transaction with a Safe.
[x] RPC requests are optimized (not triggering many requests in a very short time period).

Name/Description
ADD
These are two new apps we'd like to get added.
1. Hedgey Investor Lockups
Investor lockups are designed for teams to allocate locked tokens to their investors. Tokens unlock linearly based on the terms, and claimed directly by investors
2. Hedgey Vesting Plans
Vesting plans designed for employees and contributors to vest their revocable token grants linearly over time
UPDATE
We'd like to update the name and description of the following two apps.
3. Hedgey Escrowless Token Swaps => Hedgey Treasury Swaps
Swap your token with any other address for any other token. You pick the parameters and execute it without an escrow
4. Hedgey Time-Locked Token Distributions => Hedgey Time Locks
Lock and distribute tokens to one or multiple people with custom parameters
REMOVE
We'd like this app completely removed.
5. Hedgey Token Distributions to Investors
Distribute time-locked tokens to your investors with custom cliffs and unlock schedules
Type

New additions x2
1. Hedgey Investor Lockups
2. Hedgey Vesting Plans
Update x2
3. Hedgey Escrowless Token Swaps => Hedgey Treasury Swaps
4. Hedgey Time-Locked Token Distributions => Hedgey Time Locks
Removal
5. Hedgey Token Distribution to Investors

Compatible Networks
- Mainnet
- Goerli
- Polygon
- Avalanche
- BSC
- Harmony
- Celo
- Boba
- xDai
- Arbitrum
- Optimism
- Evmos
- OEC

Audit document
https://github.com/hedgey-finance/NFT_OTC_Core/tree/main/audits
https://github.com/hedgey-finance/HedgeyDAOSwap/blob/Audit_Added/Hedgey_24082022_SCAudit_Report.pdf
https://github.com/hedgey-finance/StreamVestingNFT/blob/master/audit/Hedgey_SC Audit Report_06042023_[SA-1070].pdf (new)
Code for review
https://github.com/hedgey-finance/hedgey-app
App URL

https://app.hedgey.finance/investor-lockups
https://app.hedgey.finance/vesting
https://app.hedgey.finance/treasury-swaps
https://app.hedgey.finance/time-locks

Team information

Project logo: Hedgey Logo
Project landing page: https://hedgey.finance
Team or discord contact: https://discord.gg/7jvvgSgTdz
Twitter link: https://twitter.com/hedgeyfinance
Github link: https://github.com/hedgey-finance

Great catch, while we supported Celo through our interface I missed the announcement that it was added to Safe.
We've updated our app to now support the Celo safe, should be all ready now!
Thanks!

Good morning! (and sorry for the delay, I was off last week) Can confirm that it is working nicely on Celo now 🚀
I will let the team know and apps should be all good to be listed on the desired chains. We will post a confirmation message here when addition is completed.
Cheers!

All the apps should be available now


Cannot build using Carthage on Xcode 14 beta 5 - schema confusion

Integration Method: Carthage
Xcode Version: Xcode 14 beta 5
Library Version: 4.2.0
Platform: iOS
Language: Swift
Description:
Carthage correctly builds the iOS target when using Xcode 13. However, on Xcode 14 beta Carthage somehow only finds the "App Extension" schema as the only appropriate build target and that ends here:
A shell task (/usr/bin/xcrun xcodebuild -project /Users/REDACTED/ios/Carthage/Checkouts/mixpanel-iphone/Mixpanel.xcodeproj -scheme [App\ Extension]\ Mixpanel -configuration Release CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY= CARTHAGE=YES archive -showBuildSettings -skipUnavailableActions) failed with exit code 6:
2022-08-12 10:46:48.429 xcodebuild[93169:3580009] [MT] DVTAssertions: ASSERTION FAILURE in /System/Volumes/Data/SWE/Apps/DT/BuildRoots/BuildRoot8/ActiveBuildRoot/Library/Caches/com.apple.xbs/Sources/IDEFrameworks/IDEFrameworks-21330/IDEFoundation/Execution/RunDestinations/IDERunDestinationCLI.m:566
Details:  Unhandled/unexpected case where no run destinations were produced.
Object:   <IDERunDestinationCLI>
Method:   +resolveRunDestinationsWithWorkspace:scheme:buildAction:schemeCommand:schemeTask:destinationSpecifications:architectures:timeout:runDestinationManager:deviceManager:fallbackPreferredSDK:fallbackPreferredArchitectures:skipUnsupportedDestinations:shouldSkipRunDestinationValidation:didDisambiguate:disambiguatedMatches:disambiguatedMatchesDescription:error:
Thread:   <_NSMainThread: 0x6000033b0400>{number = 1, name = main}
Hints:

Backtrace:
  0   -[DVTAssertionHandler handleFailureInMethod:object:fileName:lineNumber:assertionSignature:messageFormat:arguments:] (in DVTFoundation)
  1   _DVTAssertionHandler (in DVTFoundation)
  2   _DVTAssertionFailureHandler (in DVTFoundation)
  3   +[IDERunDestinationCLI resolveRunDestinationsWithWorkspace:scheme:buildAction:schemeCommand:schemeTask:destinationSpecifications:architectures:timeout:runDestinationManager:deviceManager:fallbackPreferredSDK:fallbackPreferredArchitectures:skipUnsupportedDestinations:shouldSkipRunDestinationValidation:didDisambiguate:disambiguatedMatches:disambiguatedMatchesDescription:error:] (in IDEFoundation)
  4   -[Xcode3CommandLineBuildTool _resolveRunDestinationsForBuildAction:] (in Xcode3Core)
  5   -[Xcode3CommandLineBuildTool _resolveInputOptionsWithTimingSection:] (in Xcode3Core)
  6   -[Xcode3CommandLineBuildTool run] (in Xcode3Core)
  7   XcodeBuildMain (in libxcodebuildLoader.dylib)
  8   start (in dyld)

I ran a small comparison of the output of xcodebuild -showBuildSettings and there are a few concerning changed (lines with - in front belong to Xcode 14 Beta):
-    VALID_ARCHS = arm64
+    VALIDATE_WORKSPACE = NO
+    VALID_ARCHS = arm64 armv7

-    PLATFORM_PREFERRED_ARCH = undefined_arch
-    PLATFORM_PRODUCT_BUILD_VERSION = 20A5339d
+    PLATFORM_PREFERRED_ARCH = arm64
+    PLATFORM_PRODUCT_BUILD_VERSION = 19F64

Expected Behavior:
Carthage should build the iOS target of mixpanel-iphone.

I can confirm this issue with Xcode 14.1 and the Mixpanel v5.0.0 library.
Integration Method: Carthage
Xcode Version: Xcode 14.1
Library Version: 5.0.0
Platform: iOS
Language: Objective-C
Once in the main directory, I run the following command line argument carthage build --platform iOS --no-skip-current --use-xcframeworks and the Carthage compilation appears to fail with the following error:
A shell task (/usr/bin/xcrun xcodebuild -project /Users/USER/development/mixpanel-iphone-5.0.0/Mixpanel.xcodeproj -scheme [App\ Extension]\ Mixpanel -configuration Release CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY= CARTHAGE=YES archive -showBuildSettings -skipUnavailableActions) failed with exit code 6:
2022-11-07 11:40:33.136 xcodebuild[53430:1177183] [MT] DVTAssertions: ASSERTION FAILURE in /System/Volumes/Data/SWE/Apps/DT/BuildRoots/BuildRoot1/ActiveBuildRoot/Library/Caches/com.apple.xbs/Sources/IDEFrameworks/IDEFrameworks-21534.1/IDEFoundation/Execution/RunDestinations/IDERunDestinationCLI.m:566
Details:  Unhandled/unexpected case where no run destinations were produced.
Object:   <IDERunDestinationCLI>
Method:   +resolveRunDestinationsWithWorkspace:scheme:buildAction:schemeCommand:schemeTask:destinationSpecifications:architectures:timeout:runDestinationManager:deviceManager:fallbackPreferredSDK:fallbackPreferredArchitectures:skipUnsupportedDestinations:shouldSkipRunDestinationValidation:didDisambiguate:disambiguatedMatches:disambiguatedMatchesDescription:error:
Thread:   <_NSMainThread: 0x600000184240>{number = 1, name = main}
Hints:

Backtrace:
  0   -[DVTAssertionHandler handleFailureInMethod:object:fileName:lineNumber:assertionSignature:messageFormat:arguments:] (in DVTFoundation)
  1   _DVTAssertionHandler (in DVTFoundation)
  2   _DVTAssertionFailureHandler (in DVTFoundation)
  3   _sortDevicesForDisplay (in IDEFoundation)
  4   -[Xcode3CommandLineBuildTool _resolveRunDestinationsForBuildAction:] (in Xcode3Core)
  5   -[Xcode3CommandLineBuildTool _resolveInputOptionsWithTimingSection:] (in Xcode3Core)
  6   -[Xcode3CommandLineBuildTool run] (in Xcode3Core)
  7   XcodeBuildMain (in libxcodebuildLoader.dylib)
  8   start (in dyld)

It appears that Carthage is indeed only locating the Xcode project scheme [App Extension] Mixpanel, and not the [iOS] Mixpanel xcscheme.

@m-keller @csaper This is fixed in v5.0.1. The Carthage builds attached to the Releases are fixed (and changed to XCFrameworks). You can see the Carthage command we use here: https://github.com/mixpanel/mixpanel-iphone/blob/master/scripts/carthage.sh#L2


Add some @inbounds

This PR sprinkles some @inbounds in some loops.

This is mostly an improvement based on the benchmark.

bors r+


MA refuses playing to renderer

What version of Music Assistant has the issue?
2.3.3
What version of the Home Assistant Integration have you got installed?
2024.11.4
Have you tried everything in the Troubleshooting FAQ and reviewed the Open and Closed Issues and Discussions to resolve this yourself?

[X] Yes

The problem
I have 2 players: MusicBox running on a Pi Zero and a Technisat 143 radio/media player. The musicbox works, but the Technisat doesn't. It DOES work when playing a single song from Media, but MA seems to start playing and then stops after 2 seconds or so.
How to reproduce
Choose Music Assistant from the side bar, select DIgitradio as renderer, select album and hit Play
Music Providers
Music Assistant, Filesystem (Local Disk)
Player Providers
UPnP/DLNA Player Provider
Full log output
d5369777_music_assistant_2024-11-28T16-10-37.125Z.log
Additional information
No response
What version of Home Assistant Core are your running
2024.11.3
What type of installation are you running?
Home Assistant OS
On what type of hardware are you running?
Raspberry Pi

The Technisat works with all my other streamers: BubblePnP and Universal Media Serveri.

Sorry, accidentally closed...

Please follow the troubleshooting page closely. You have not listed what you have tried to resolve this yourself particularly the requirements on https://music-assistant.io/player-support/dlna/

I tried every possible combination of settings. Currently, what works (more or less) is Disable Queue flow mode, automatically play,  disable volume normalization, disable (lossy) mp3 stream, Profile 3 - forced content length, disable Pre-announce TTS announcements. Now in most cases, MA starts playing but there are 3 issues: sometimes MA stops streaming after the first track, sometimes it starts the next track on 80% of the previous track (so, without letting it finish playing), or it will crash and the Technisat renderer/radio switches back to the last selected source before MA took over. Media provider is Local Disk (an SMB mount to a Linux server containing the mp3's).

OK thanks. You probably saw this then

Unfortunately due to the difficulty in resolving these issues they are low priority. If your device supports a different protocol then use that instead of raising an issue

If you can't use any other provide type with the device then there is not much more we can do at this time.
When you say "it will crash" do you mean the whole MA server goes down and has to be restarted or do you just mean the stream stops?

@OzGav I am also experiencing an issue with DLNA on a specific device. Tried all profiles and tbh, may just abandon that device altogether since DLNA seems to have issues, unless I am missing something. The only error it logs related to the stream is
2024-12-02 20:33:05.048 ERROR (MainThread) [music_assistant.dlna] Error during call cmd_pause: 'Error during async_call(), action: Pause, status: 500, upnp error: 501 (Action Failed)'
Not really sure how to interpret. Does that look like something related to the profiles or do you think I have a problem elsewhere?

Yes probably a device which requires a particular tweak to fix which we don't have the capacity to do.

okie dokey

Hi, knowing this has low prio, just wanted to add some more info.
I am having a very similar problem as waarsenburg describes with a Technisat Multyradio 4.0. I think in my case the problem is only linked to the queue flow mode option. If I enable it, DLNA streaming works (but stops after one song of course), if I enable it, it will not play.
Funny enough, using the Home Assistant player provider for this device (DMR Integration) it works fine even continuing to play the songs of an album (so apparently the device does support queuing although in the MA logs it says otherwise).

@jonathanarchergithub have you tried each of the different http profiles with queue flow mode on?

@OzGav yes. There's no difference in behaviour. Btw. I notice the stereo displays "buffering" with a progress bar when I hit play but the progress bar stays at 0%.

@jonathanarchergithub then unfortunately I direct your attention to the docs and these points

Due to the very inconsistent implementation of this protocol by manufacturers, some players will work great and others will simply not work at all or need workarounds.

Unfortunately due to the difficulty in resolving these issues they are low priority. If your device supports a different protocol then use that instead of raising an issue

I know, that's okay. Just wanted to add the info. I noticed a typo in my post however. I meant that it works with flow mode disabled, it doesn't play at all (and shows the 0% progress bar) with flow mode enabled. The HA player provider version (via DMR integration) works well via MA and plays all songs of an album.


Changing the send_interval causes wildly different latency results

I've spent much of my day debugging my Unifi Dream Machine Pro's latency metrics which are not in line with reality on my Quantum Fiber 2/1Gbps internet. Ultimately this lead me here.
I'll let these 3 outputs do most of the talking, the dpinger binary I am using for this test is 3.3 compiled locally from https://github.com/dennypage/dpinger/commit/193f83db0abfba98e3b139b4a72968605ffebe6c. But I was first experiencing this on Unifi's unversioned copy of dpinger.
ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS>: icmp_seq=1 ttl=56 time=4.65 ms
64 bytes from <IP_ADDRESS>: icmp_seq=2 ttl=56 time=3.68 ms
64 bytes from <IP_ADDRESS>: icmp_seq=3 ttl=56 time=4.46 ms
64 bytes from <IP_ADDRESS>: icmp_seq=4 ttl=56 time=4.18 ms
64 bytes from <IP_ADDRESS>: icmp_seq=5 ttl=56 time=3.91 ms
64 bytes from <IP_ADDRESS>: icmp_seq=6 ttl=56 time=3.64 ms
64 bytes from <IP_ADDRESS>: icmp_seq=7 ttl=56 time=3.72 ms
64 bytes from <IP_ADDRESS>: icmp_seq=8 ttl=56 time=4.02 ms

sudo ./dpinger -f <IP_ADDRESS>
send_interval 500ms  loss_interval 2000ms  time_period 60000ms  report_interval 1000ms  data_len 0  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr <IP_ADDRESS>  bind_addr (none)  identifier ""
135168 36226 0
257879 153114 0
195028 154650 0
169796 146250 0
150217 136557 0
144098 129654 0
125740 128211 0
137954 139087 0
154539 147717 0

sudo ./dpinger -s 1 -f <IP_ADDRESS>
send_interval 1ms  loss_interval 4ms  time_period 60000ms  report_interval 1000ms  data_len 0  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr <IP_ADDRESS>  bind_addr (none)  identifier ""
4152 1076 0
4154 1087 0
4128 1059 0
4104 1039 0
4100 1034 0
4108 1086 0
4109 1079 0

Am I wrong in assuming that send_interval should have basically no impact on those numbers aside from the frequency?
Or would this be expected especially with a network misconfiguration either locally or on my ISP's side? Obviously not everyone's hitting this, so I imagine there's /some/ network configuration involvement.
Here's the primary thread on this issue on the Unifi side: https://community.ui.com/questions/High-latency-network-issues-on-new-UDM-Pro/512170f8-cad0-48db-ae43-22e9fb518b90?page=1 but there's a few dozen smaller threads scattered around.
The solution in that thread is basically to end up doing dpinger localhost which defeats the purpose entirely.
Thanks,
Riley

Edit:
I got to try on my work machine in the office; and there was a tiny difference but not what I was seeing here.
$ sudo ./dpinger -s 500 -f <IP_ADDRESS>
send_interval 500ms  loss_interval 2000ms  time_period 60000ms  report_interval 1000ms  data_len 0  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr <IP_ADDRESS>  bind_addr (none)  identifier ""
2509 22 0
2485 61 0
2543 151 0
2514 145 0
2540 165 0
2544 142 0
2531 144 0

$ sudo ./dpinger -s 1 -f <IP_ADDRESS>
send_interval 1ms  loss_interval 4ms  time_period 60000ms  report_interval 1000ms  data_len 0  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr <IP_ADDRESS>  bind_addr (none)  identifier ""
2264 75 0
2259 129 0
2258 123 0
2258 114 0
2272 294 0
2272 282 0

Several things...
Throughput and latency are not the same thing. A 2Gb/s connection doesn't intrinsically have lower latency than a 100Mb/s connection.
The first and foremost issue would be running dpinger with a 1ms send rate. Sending 100 ICMP packets per second across the internet to Cloudflare is a bad idea. With these kind of rates, either your ISP or Cloudflare is going to start shaping (rate limiting) or dropping your ICMP packets. Also, the loss interval is around your minimum round trip time which is going to mess everything up quite a bit. It needs to be much longer.
I highly recommend that you do not exceed a 250ms send rate. Here's a reasonable command to try: dpinger -f -s 250m -l 5s <IP_ADDRESS>.
Another issue when comparing with ping will be the payload size. By default dpinger uses a payload of size 0 (option -d), whereas ping usually defaults to a payload size of 56 bytes (option -s to ping).
If you need to compare ping with dpinger, you want to either increase the size of the dpinger payload or decrease the size of the ping payload. 16 would be a reasonable value. Examples:
dpinger -f -s 1s -d 16 <IP_ADDRESS>
ping -s 16 <IP_ADDRESS>
HTH

This has been super helpful.
Looks like it is indeed the payload size that makes a big difference.
$ ping -s 16 <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 16(44) bytes of data.
24 bytes from <IP_ADDRESS>: icmp_seq=1 ttl=56 time=3.91 ms
24 bytes from <IP_ADDRESS>: icmp_seq=2 ttl=56 time=3.67 ms
24 bytes from <IP_ADDRESS>: icmp_seq=3 ttl=56 time=4.62 ms
24 bytes from <IP_ADDRESS>: icmp_seq=4 ttl=56 time=3.53 ms
24 bytes from <IP_ADDRESS>: icmp_seq=5 ttl=56 time=3.68 ms

$ sudo ./dpinger -f -s 1s -d 16 <IP_ADDRESS>
send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 1000ms  data_len 16  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr <IP_ADDRESS>  bind_addr (none)  identifier ""
4369 0 0
4227 155 0
4049 286 0
4144 293 0
4171 271 0
4101 297 0
4015 343 0

$  sudo ./dpinger -f -s 1s <IP_ADDRESS>
send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 1000ms  data_len 0  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr <IP_ADDRESS>  bind_addr (none)  identifier ""
331027 0 0
192448 138579 0
281640 169451 0

$ sudo ./dpinger -f -s 1s -d 6 <IP_ADDRESS>
send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 1000ms  data_len 6  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr <IP_ADDRESS>  bind_addr (none)  identifier ""
246542 0 0
318470 71930 0

$ sudo ./dpinger -f -s 1s -d 7 <IP_ADDRESS>
send_interval 1000ms  loss_interval 4000ms  time_period 60000ms  report_interval 1000ms  data_len 7  alert_interval 1000ms  latency_alarm 0ms  loss_alarm 0%  alarm_hold 10000ms  dest_addr <IP_ADDRESS>  bind_addr (none)  identifier ""
4371 0 0
4225 159 0
4392 275 0

Seems like the data length needs to be at least 7 bytes for me to be seeing reasonable/expected numbers. Now to start investigating whether this is my home network causing this or my ISP doing packet shaping or batching of packets smaller than the ICMP header + a 7 byte payload. One oddity is that below 16 bytes iputils ping stops returning the time for some reason.
By going down to a send rate of 1ms I was probably hitting that batch rate immediately or overflowing something.
Feel free to close this, I might keep posting in here and keeping this updated with my findings though.
It would be nice if Ubuiqiti would just let us set some dpinger parameters other than the dest_addr; though I guess ideally this sounds like it really shouldn't be necessary and the defaults really should just be working fine.

One oddity is that below 16 bytes iputils ping stops returning the time for some reason.

ICMP Echo Reply reflects the same data payload that was in the ICMP Echo Request. Ping puts the original timestamp in the data payload, and depends upon the timestamp being returned by the target. If there is no room for a timestamp in the packet, ping cannot calculate the elapsed time.
Dpinger keeps track of time without using the data payload.

Hokay I figured this out!
My Q1000K ONT which was in transparent bridge mode was apparently still doing /something/. Reconfiguring the ONT by explicitly turning off the firewall, UPNP, etc, and setting up my unifi hardware to do the VLAN tagging instead of the ONT has 100% completely resolved my issues. I basically never see >4ms out of dpinger or ping with any settings now!
Thanks for the pointers. What a weekend.