systemd: stop reading from /etc/systemd-mutable

Spun out of https://github.com/NixOS/nixpkgs/pull/111786.
This will remove code once added into our systemd fork looking for system units in /etc/systemd-mutable/system and user units in /etc/systemd-mutable/user.
This was initially added as a place for dysnomia/disnix to drop imperative units, which has been removed from nixpkgs, as its module was broken for over a year. In that PR, there has been some discussion about re-introducing dysnomia, but even with it getting re-introduced, I don't think we should read from that folder by default, due to the following reasons:

We already have too many patches on top of systemd.
Adding support for a location that's not read on any other distribution (nor documented) is confusing.
We link applications that might run on non-NixOS systems against the nixpkgs systemd. systemd has some path lookup methods implemented, and we shouldn't return paths that the systemd running on another distro doesn't read from.

IMHO, /run/systemd/system is the right place to drop imperative unit files, which is already supported in vanilla systemd. If unit files need to be persisted across reboots, there could be a script copying them over from another place (something like /var/lib/name-of-tool/system-units for example).
Motivation for this change
https://github.com/NixOS/nixpkgs/issues/80038

What I don't understand about the whole discussion is that today you're perfectly able to also write units to /etc/systemd/system. the directory isn't immutable on NixOS. It's just that things will get hairy when a NixOS service conflicts with one that you manually put there.

What I don't understand about the whole discussion is that today you're perfectly able to also write units to /etc/systemd/system. the directory isn't immutable on NixOS. It's just that things will get hairy when a NixOS service conflicts with one that you manually put there.

Probably safe with some namespace prefix.

@arianvp /etc/systemd/system is a symlink to a store path:
realpath /etc/systemd/system
# => /nix/store/7gs...-system-units
namei /etc/systemd/system

How is it possible to mutably write to /etc/systemd/system?

Oh darn it.
I was under the assumption each unit file would be its own derivation but
the entire unit directory is.
Ignore what I said.
Perhaps we could change that though...
On Sat, 13 Feb 2021, 11:50 erikarvstedt<EMAIL_ADDRESS>wrote:

@arianvp https://github.com/arianvp /etc/systemd/system is a symlink to
a store path:
realpath /etc/systemd/system
=> /nix/store/7gs...-system-units
namei /etc/systemd/system
How is it possible to mutably write to /etc/systemd/system?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/pull/112891#issuecomment-778598844, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEZNI7DVRE5GTFA6VODEFLS6ZKQNANCNFSM4XQSRSMQ
.

@arianvp @erikarvstedt  I raised exactly the same concern. So if we have no mutable folder to store systemd units in, then managing systemd units is a purely Nix-driven approach. Of course, for most use cases this is not a problem -- since most people deploy from a configuration.nix file anyways.
However, I also tend to sometimes use a mutable systemd folder for development projects. For example, in a project that ships with a systemd unit file, it is convenient to test it locally by putting it in that folder. (I typically want to use a systemd file directly, not deploy it from a Nix expression)
With the removal of that folder, managing persistent systemd units is something you can only do by deploying Nix packages (either as part of a the global Nix profile or in a configuration.nix).
@flokli /run/systemd/system is not a good alternative, because unit files are not stored in that folder persistently. After a reboot, the content of this directory is wiped. Before I knew of the existence of the systemd-mutable folder (Eelco has added this to the patch for his own testing purposes), I was using the /run/systemd/system folder instead, but it was not very practical.
@arianvp @flokli An alternative to the mutable folder is making the ordinary /etc/systemd/system location not a symlink to a Nix profile, but a mutable directory in which you can put custom files as well, e.g. by creating symlinks for each NixOS provided unit. In very old releases of NixOS (roughly late 2012, this is exactly how we used to manage that folder. I don't know the reason why that was changed). Then you also support both use cases, without the need to patch systemd.

So while making /etc/systemd/system writable seems like an "easy" route to go, I want to raise the concern, that this might block the route to making /etc readonly (actually does it / will systemd follow symlinks, when looking for unit files?). I think viewed from that angle using something below /var is the better approach.
I recently rewrote the setup-etc and update-users-groups perl scripts in rust, so I know there is a lot of complexity there, which could be significantly reduced by a read only /etc, which just gets symlinked in place (and mutable things moved below /var).

@xaverdh, could you share your rust version of setup-etc.pl? I tried to estimate the effects on performance and implementation complexity of the "mutable /etc/systemd/system" approach, but I couldn't figure out the Perl source in a reasonable amount of time.

@xaverdh, could you share your rust version of setup-etc.pl? I tried to estimate the effects on performance and implementation complexity of the "mutable /etc/systemd/system" approach, but I couldn't figure out the Perl source in a reasonable amount of time.

It currently lives  here. Note that it is completely untested yet though.

I agree with @svanderburg that /run/systemd/system is no adquate replacement for systemd-mutable.
Making /etc/systemd/system mutable isn't directly compatible with the current environment.etc architecture:
Mutable directories are only created for parents of environment.etc entries (defined at eval time), but the contents of systemd/system can only be determined at build time.
This can be fixed, although we lose some of the simplicity of the current design.
The impact on activation script performance (with an empty system config) is the following:
In addition to the ~100 /etc leaf files, the etc setup would have to traverse ~200 extra unit files in systemd/system, both below /etc (for cleanup) and below /nix/store/...-system-units (for linking).
Although this is might be negligible in itself, system activation is on the hot path for booting and updating and should be optimized where possible.
What do you think about just re-enabling /usr/lib/systemd as a config dir, instead of systemd-mutable?

I agree with @svanderburg that /run/systemd/system is no adequate replacement for systemd-mutable.

Why is it not possible to have something in the activation script / on-boot-script restoring from /var/lib/… into /run/systemd/system?

Making /etc/systemd/system mutable isn't directly compatible with the current environment.etc architecture:
Mutable directories are only created for parents of environment.etc entries (defined at eval time), but the contents of systemd/system can only be determined at build time.
This can be fixed or worked around, although we lose some of the simplicity of the current design.

Yeah, I wouldn't want to make the design inside /etc/systemd/system more complicated.
Suddenly, NixOS activation would need to care about something else creating files in there, which is calling for trouble.
Assembling the contents at eval time, and being able to easily switch to the next generation without having to worry about other files that might have been created by other tools sounds like an advantage to me.

What do you think about just re-enabling /usr/lib/systemd as a config dir, instead of systemd-mutable?

We don't have anything in /usr except /usr/bin/env - which is only there as shebangs usually expect /usr/bin/env - I don't think we should/can introduce more in /usr.
Also, /usr/lib/systemd/system would in priority come after /etc/systemd/system.
Traditionally, /run/systemd/system is used for units created at runtime, /etc/systemd/system by the administrator/config management tool, and /usr/lib/systemd/system by the distribution package manager.
In NixOS, the package manager also /is/ your config management tool, so having /etc/systemd/system managed by NixOS (and no  /usr/lib/systemd/system) makes kinda sense.
Dysnomia and other "imperative service management tools" like https://github.com/erikarvstedt/extra-container sidestep some of this. IMHO, having these tools write to a more ephemeral location (like /run/systemd/system), accompanied by a script restoring it from a well-documented state location on disk (such as /var/lib/dysnomia) feels more predictable and canonical to me.
It's certainly preferrable over having the systemd shipped with NixOS (and other nix-built projects linking against systemd) having a undocumented path baked in.
If you really don't want to implement such a restore-on-startup logic, you could maintain an alternative systemd.package reading from this path too (and NixOS modules enabling dysnomia etc. can set this), but I don't want vanilla systemd having these quirks baked in by default.

WRT.

However, I also tend to sometimes use a mutable systemd folder for development projects. For example, in a project that ships with a systemd unit file, it is convenient to test it locally by putting it in that folder. (I typically want to use a systemd file directly, not deploy it from a Nix expression)
With the removal of that folder, managing persistent systemd units is something you can only do by deploying Nix packages (either as part of a the global Nix profile or in a configuration.nix).

A systemd user service might be appropriate for that use case.
As long as home-manager do not get merged to NixOS, you pretty much have a strong guarantee ~/.config/systemd/user/ won't be managed by Nix.

Considering the :+1:'s and approvals of recent systemd maintainers, let's merge this.
As previously elaborated, that path is nowhere documented, there's nothing in nixpkgs using it, and considering that systemd is also what all our nix-built packages link to, it shouldn't be there.
As expressed in the comments, using /run/systemd/system (and a serialization script on bootup) should make this relatively easy for tools to adapt (and could even unify code with non-NixOS distributions).
If it's really not possible/too complicated to use that path, Dysnomia or other "imperative process management" tools could maintain their own flavor of systemd, and set it via systemd.package.

Too bad that this is already pushed forward -- I was hoping that we could still somehow retain the ability to manage persistent mutable systemd units (the directory in which these files are stored is not so important, it could also very well be /usr/lib/systemd or another folder)
Of course, I can provide a patched systemd version and override the systemd package so that this functionality is restored. Still, I don't see this really as a step forward.
Anyway, since the majority believes it is not something we should keep, then I'll have to investigate how I can best support my use cases, without relying on the availability this feature anymore.

I just found out that you can actually configure systemd to read from
alternative paths with SYSTEMD_UNIT_PATH

When the variable $SYSTEMD_UNIT_PATH is set, the contents of this
variable overrides the unit load path. If $SYSTEMD_UNIT_PATH ends with an
empty component (":"), the usual unit load path will be appended to the
contents of the variable

https://www.freedesktop.org/software/systemd/man/systemd.unit.html
perhaps we can add that to the systemd module to support the dysonmia
usecase?
This way we can reintroduce the behaviour without having to patch or
recompile systemd
On Thu, 18 Feb 2021, 19:26 Sander van der Burg<EMAIL_ADDRESS>wrote:

Too bad that this is already pushed forward -- I was hoping that we could
still somehow retain the ability to manage persistent mutable systemd units
(the directory in which these files are stored is not so important, it
could also very well be /usr/lib/systemd or another folder)
Of course, I can provide a patched systemd version and override the
systemd package so that this functionality is restored. Still, I don't see
this really as a step forward.
Anyway, since the majority believes it is not something we should keep,
then I'll have to investigate how I can best support my use cases, without
relying on the availability this feature anymore.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/pull/112891#issuecomment-781546478, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEZNI3ZQPWYOKXVSZIJAI3S7VLVNANCNFSM4XQSRSMQ
.

@arianvp Nice pointer! I'd propose setting this in the dysonmia NixOS module though, not unconditionally ;-)

Yes but the systemd module should support starting the systemd binary for a
set of environment variables.  That's something we'd have to add to allow
the dysonmia module to use this.
On Thu, 18 Feb 2021, 20:19 Florian Klink<EMAIL_ADDRESS>wrote:

@arianvp https://github.com/arianvp Nice pointer! I'd propose setting
this in the dysonmia NixOS module though, not unconditionally ;-)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/pull/112891#issuecomment-781576806, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEZNI6DAVSV5LOJICI3TR3S7VR4LANCNFSM4XQSRSMQ
.

Awesome catch @arianvp!

Apply this patch to your nixpkgs checkout:
diff --git a/nixos/modules/system/boot/stage-2-init.sh b/nixos/modules/system/boot/stage-2-init.sh
index 936077b9df1..f4a99eca7dd 100644
--- a/nixos/modules/system/boot/stage-2-init.sh
+++ b/nixos/modules/system/boot/stage-2-init.sh
@@ -169,4 +169,5 @@ exec {logOutFd}>&- {logErrFd}>&-
 echo "starting systemd..."
 PATH=/run/current-system/systemd/lib/systemd:@fsPackagesPath@ \
     LOCALE_ARCHIVE=/run/current-system/sw/lib/locale/locale-archive \
+    SYSTEMD_UNIT_PATH=/cust: \
     exec @systemdExecutable@

Build a VM image against your nixpkgs checkout and boot it. From here you can place .service files in /cust and they'll be accessible after systemctl daemon-reload:
[root@nixos:~]# systemctl cat test.service
# /cust/test.service
[Unit]
Description=example systemd service unit file.

[Service]
ExecStart=/bin/sh /example.sh

[Install]
WantedBy=multi-user.target

If I were to make a PR out of this does anyone have any specific requests? Seems pretty straight forward...

@arianvp Awesome, this is a great discovery! I'll investigate how this can be used.
Maybe this can also help us to get rid of the other workarounds that we normally need to patch.

@svanderburg so far I haven't been able to get this to update after calling systemctl daemon-reexec (ie. it only works on boot), but I haven't dug in deep enough to know if that is really true or not.
Please ping back here if you discover anything. Thanks!

@aanderse I took me a while to respond, but recently I just ran locally some tests.
For me it seems to work fine for my use cases. I followed your instructions and it worked for me. The only thing I did different is configuring SYSTEMD_UNIT_PATH to read from /etc/systemd-mutable/system instead of /cust
Dysnomia as well as the Nix process management framework, basically do the following to deploy a systemd unit:

Put units file in a mutable directory, e.g. /etc/systemd-mutable/system
Reload the systemd daemon with: systemctl daemon-reload (this is strictly not always required, but you'll see warnings)
Then start all the units with: systemctl start

I have never really used daemon-reexec much. I don't know if it used to work with the patch enabled, I can't really tell. My gut feeling says, that the behaviour with the patch and the SYSTEMD_UNIT_PATH is exactly the same.
Anyway, I think we can move a PRt forward that uses the SYSTEMD_UNIT_PATH functionality.
@aanderse To make a patch like this work, it would be nice if we can use a NixOS configuration property that allows you to set multiple paths (as a list) that automatically get translated to a SYSTEMD_UNIT_PATH environment variable. Then each application that requires mutable systemd units can have its own directory.

@aanderse There is one caveat -- I can indeed not load anything if the directory in the SYSTEMD_UNIT_PATH does not exist yet. So after creating the directory, I had to reboot to make it work. But after the reboot, I could add and remove things, do a daemon-reload and then start or stop the stuff I want.
So we must probably make sure that if other paths have been configured, the directories already exist before starting systemd.
Have you already started working on a pull request? If not, I was actually planning to pick this up myself soon.

@svanderburg that sounds great. Please go ahead. I haven't started a PR, I was just playing around a bit.
@flokli said he would take a look to see if we could reexec to avoid reboot, but I have a feeling we can't...

I didn't check, and am somewhat busy with other stuff currently. I propose simply giving this a try, and if it doesn't work, start diving into the source code ;-)

I think the environment variable has to be set when calling the systemd executable. For us this happens in modules/system/boot/{stage-2-init.sh,stage-2.nix}
# Start systemd.
echo "starting systemd..."
PATH=/run/current-system/systemd/lib/systemd:@fsPackagesPath@ \
    LOCALE_ARCHIVE=/run/current-system/sw/lib/locale/locale-archive \
    exec @systemdExecutable@

We could change this to:
# Start systemd.
echo "starting systemd..."
SYSTEMD_UNIT_PATH=@systemdUnitPath@ PATH=/run/current-system/systemd/lib/systemd:@fsPackagesPath@ \
    LOCALE_ARCHIVE=/run/current-system/sw/lib/locale/locale-archive \
    exec @systemdExecutable@

and adding a boot.systemdUnitPath option
Obviously systemctl daemon-reexec won't restart stage2-init.sh so indeed reboot is required for it to take effect. But I think the fact we put the option under the boot. namespace signifies this well to to the user.

@arianvp I have already created such a thing. The only thing I still need to do is write/do a small test.
I'll make a PR later today!

@arianvp @aanderse https://github.com/NixOS/nixpkgs/pull/116455


when is support eloquent available?

Hello,
when is support eloquent available?
thank!

Hi @ytake
This is My question too.


UnsupportedDataExtension error on trying to get json data from Google Spreadsheet via GAS

I tried to get json data from Google Spreadsheet via Google Apps Script, but failed.
what I did is:
1.publishd spreadsheet data (here's data: spreadsheet data)

2.wrote Google Apps Script code and publishd (here's sciprt and execution result, success.  )
ref. Google Apps Script で spreadsheet のデータを JSON として読み込む / Getting spreadsheet data as json format via Google Apps Script
[{"id":1,"name":"apple"},{"id":2,"name":"beer"},{"id":3,"name":"cookie"}]

3.middleman server up, and encounterd UnsupportedDataExtension error like below.
/Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/middleman-data_source-0.5.0/lib/middleman/data_source/extension.rb:71:in `decode_data': Middleman::DataSource::Extension::UnsupportedDataExtension (Middleman::DataSource::Extension::UnsupportedDataExtension)
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/middleman-data_source-0.5.0/lib/middleman/data_source/extension.rb:36:in `block (2 levels) in initialize'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/middleman-core-3.3.12/lib/middleman-core/core_extensions/data.rb:149:in `call'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/middleman-core-3.3.12/lib/middleman-core/core_extensions/data.rb:149:in `data_for_path'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/middleman-core-3.3.12/lib/middleman-core/core_extensions/data.rb:163:in `method_missing'
        from /Users/hi/Sites/lavida.co.jp/v2/dev/config.rb:88:in `initialize'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/middleman-core-3.3.12/lib/middleman-core/core_extensions/extensions.rb:162:in `instance_eval'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/middleman-core-3.3.12/lib/middleman-core/core_extensions/extensions.rb:162:in `initialize'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/middleman-core-3.3.12/lib/middleman-core/core_extensions/data.rb:33:in `initialize'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/middleman-core-3.3.12/lib/middleman-core/application.rb:204:in `initialize'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/middleman-core-3.3.12/lib/middleman-core/core_extensions/request.rb:56:in `new'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/middleman-core-3.3.12/lib/middleman-core/core_extensions/request.rb:56:in `inst'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/middleman-core-3.3.12/lib/middleman-core/preview_server.rb:107:in `new_app'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/middleman-core-3.3.12/lib/middleman-core/preview_server.rb:24:in `start'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/middleman-core-3.3.12/lib/middleman-core/cli/server.rb:81:in `server'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/middleman-core-3.3.12/lib/middleman-core/cli.rb:72:in `method_missing'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/command.rb:29:in `run'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/command.rb:126:in `run'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/middleman-core-3.3.12/lib/middleman-core/cli.rb:20:in `start'
        from /Users/hi/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/middleman-core-3.3.12/bin/middleman:18:in `<top (required)>'
        from /Users/hi/.rbenv/versions/2.1.2/bin/middleman:23:in `load'
        from /Users/hi/.rbenv/versions/2.1.2/bin/middleman:23:in `<main>'
        from /Users/hi/.rbenv/versions/2.1.2/bin/ruby_executable_hooks:15:in `eval'
        from /Users/hi/.rbenv/versions/2.1.2/bin/ruby_executable_hooks:15:in `<main>'

here's my setting on config.rb
activate :data_source do |c|
  c.files = {
    "https://script.google.com/macros/s/AKfycbyzNbrKRfz5c-4ya4-ScyIyUqr4GKUeDXFda6-CFlQ9x13yRxP-/exec)" => "Sheet1"
  }
end

I'm so excited becourse middleman can handle data from other API data, but it failed...
By the wáy, I tried the JSON data on local json file (that is, copy and paste the JSON data to local file), then I found suceed without any errors. so the json data is right.
anyone has ideas?
thanks.

Code for converting spreadsheet data to JSON (Google Apps Script)
var URL_BOOK = 'https://docs.google.com/spreadsheet/ccc?key=1nlrukb7TZme3iUsJCG2lLJDsmYtFSbbYOxa0UZu50xo';
var SHEET_NAME = 'Sheet1';

function doGet(e) {
  var url = URL_BOOK;
  var sheetName = SHEET_NAME;

  var book = SpreadsheetApp.openByUrl(url);
  var sheet = book.getSheetByName(sheetName);

  // {hoge : 'fuga'};
  var json = convertSheet2JsonText(sheet);
  return ContentService.createTextOutput(JSON.stringify(json)).setMimeType(ContentService.MimeType.JSON);
}

function convertSheet2JsonText(sheet) {
  // first line(title)
  var colStartIndex = 1;
  var rowNum = 1;
  var firstRange = sheet.getRange(1, 1, 1, sheet.getLastColumn());
  var firstRowValues = firstRange.getValues();
  var titleColumns = firstRowValues[0];

  // after the second line(data)
  var lastRow = sheet.getLastRow();
  var rowValues = [];
  for(var rowIndex=2; rowIndex<=lastRow; rowIndex++) {
    var colStartIndex = 1;
    var rowNum = 1;
    var range = sheet.getRange(rowIndex, colStartIndex, rowNum, sheet.getLastColumn());
    var values = range.getValues();
    rowValues.push(values[0]);
  }

  // create json
  var jsonArray = [];
  for(var i=0; i<rowValues.length; i++) {
    var line = rowValues[i];
    var json = new Object();
    for(var j=0; j<titleColumns.length; j++) {
      json[titleColumns[j]] = line[j];
    }
    jsonArray.push(json);
  }
  return jsonArray;
}

This comes up because I was really lazy w/ the data type detection (https://github.com/stevenosloan/middleman-data_source/blob/master/lib/middleman/data_source/extension.rb#L65-L73), and your endpoint doesn't end with .json.
I think it'd be good to add a feature for specifying the content-type and/or giving a proc for doing your own decoding instead of just depending on the extension.

Hey @ikedhidenori took me a little bit but I believe I came to a good solution for this (some nice added features :+1: )
Could you try out on this feature branch:
# Gemfile
gem 'middleman-data_source', git: 'git@github.com:stevenosloan/middleman-data_source.git',
                             branch: 'feat/set_types_and_define_decoders'

Then to pull the data:
# config.rb
activate :data_source do |c|

  c.sources = [
    {
      alias: "Sheet1",
      path:  "https://script.google.com/macros/s/AKfycbyzNbrKRfz5c-4ya4-ScyIyUqr4GKUeDXFda6-CFlQ9x13yRxP-/exec",
      type:  :json
    }
  ]

end

If you could let me know if the new docs in the readme make sense as well that'd be great https://github.com/stevenosloan/middleman-data_source/tree/feat/set_types_and_define_decoders#custom-data-types
thanks!

Surprising!  That's exactly what I want!
It works and seems perfect for me.
Thanks, steven!

Awesome, glad that works for you. I think it's a nice added feature set.
I'll get a new version cut & pushed later today. Thanks!

Ok, 0.6.0 is out so you can use that instead of the branch now.


SPARQLConnection remove(st, contexts) method cannot handle null values in statement object

Problem is in SPARQLConnection::createDataBody
The method does not check if st.getSubject() is null and so fails on .stringValue(line:886) with a nullpointer exception.

How did you get a statement with a null subject?

Honestly cannot remember.
I tested a lot of different rdf4j store implementations so maybe some store's ValueFactory let me create such a statement.
I will take another look tomorrow and let you know.

Re-open (or create a new) issue when you have more information about this.


Compress logs to use less disk space

On the chat.zulip.org server, /var/log/zulip/ takes up a total of 4.8 GB of space (*). The bulk of that disk usage would end up looking the same on any other busy Zulip server with our default configuration. That's kind of a lot, and we can do better.
The space breaks down as (approximately)

1.5GB server.log and older versions
1.0GB django.log and older versions
1.0GB tornado.log and older versions
1.0GB other (some of which may not come from default Zulip behavior.)

The reason this is so much is

For server.log, we have rotation, but we let the file grow past 500MB before rotating it, and we compress only the .2 and up. So two 500MB files, plus 9 more files (up to .10) which are compresed to about 50MB. This is configured in /etc/logrotate.d/zulip, which comes from puppet/zulip/files/logrotate/zulip. We should reduce the size we let the file grow to before rotation, and perhaps keep more files in order to keep the same total amount of log information.

For django.log and tornado.log, we have rotation but no compression. We keep 10 old files for each, and they're 100MiB in size. This is configured through supervisord, from puppet/zulip/templates/supervisor/zulip.conf.template.erb. Unfortunately supervisor doesn't support log compression, which is the real solution for this. See #3385 and the supervisor issue Tim linked to from there. One solution would be to disable supervisor's log rotation, and use logrotate -- see a blog post -- though that post's solution involves using the copytruncate option, which isn't great. Following more breadcrumbs, there's this suggestion on a supervisor PR: use logrotate, and "send[] SIGUSR2 to Supervisor to signal it to reopen the logs." Maybe that'd work.

(*) Actually 5.3 GB, but at least 0.5GB are in some files it doesn't look like we currently generate, namely tornado.log*gz.

zulip apparently comes with logrotate.d configuration file - but it appears to be imcomplete as it only compresses some of the logs but not all relevant ones. at least django.log and tornado.log are quickly growing besides server.log

that is by design, in that the log files managed by Supervisor (e.g. django.log and tornado.log) can't also use logrotate easily, see the discussion above.  It'd be nice to improve this, but it's not trivial and could easily break things, so we'd need to carefully test one of the proposals mentioned above.

We now have log rotation for all files that Zulip generates since 93d661d11934d91bdfe49aa4a83c71f711277991.  So the only remaining issue is the supervisord thing.  Sadly https://github.com/Supervisor/supervisor/issues/322 remains unresolved.

I believe with e6c64e78e61cc5207f4e99817a838d9bec9e4b3e, we've reworked all large log files to use logrotate for compression. Closing as fixed.


[ Nicer to have ] Optimize webpack build time

removing unnecessary loaders
minimize the scope of loaders

https://webpack.js.org/guides/build-performance/

It's also worth to add PNP. I'm working on it but unfortunately, it requires a lot of effort :)

I dived into this a bit: The babel-loader could be improved a bit by adding a cacheDirectory flag to it:
loader: 'babel-loader?cacheDirectory',

Disabling the following line speeds up a bit as well (and obviously disables source mapping, but maybe this is not needed by everyone):
devtool: 'source-map',

Maybe the devtool source-map should be replaced with something else from this list with faster build time:
https://webpack.js.org/configuration/devtool/


High cpu usage and freeze

Ok so i have added a item that cant be placeable in itemframe from Config.. and after i uploaded this plugin my server started freezing often and high cpu usage so i checked timings and it shows your plugin let me show you pic. I cant ss the timings so i will attach the lonk
https://timings.pmmp.io/?id=158503

Pls optimize your plugin

Ya esta echo. ;)

I just rewrote the plugin, I hope and I no longer have problems of exaggerated CPU usage :)


cargo-bitbake: Add an option to trust lockfiles

Description
When using Cargo Bitbake in combination with lockfiles, I was a little surprised that cargo-bitbake didn't just read from the lockfile directly as that has the resolution information I'd like to use.
--require-lockfile can now be used to blindly trust an existing lockfile without performing resolution. This is done so that users can avoid performing resolution if they already have a lockfile checked in.
I could see a fair argument here that the right way of doing this is likely staying more similar to the rest of the family of cargo tools, and accepting something like a --locked and --offline flag, and if we attempt to access the network or modify the Lockfile while performing resolution, we panic.
That being said, this was the easiest thing for my company to do without thinking about it too much, so we went with this 🤷 . Would love to discuss how cargo bitbake is intended to interact with lockfiles in the ecosystem as a whole if anyone has ideas!
Caveat: this won't work with workspaces / virtual manifests correctly, but this tool doesn't seem to handle them, so I don't think this is really a regression.
Test Plan
I tested this by running it on all of my companies recipes, but can do some more testing / testing in a public way if desired!

LGTM


Error during compilation

Hello, so far I have been running CoastalApp in the MIT-supercloud cluster. Recently they changed the environment, so I compiled again the esmf library in the new environment. After that, I tried to compile again the CoastalApp, but I got this error, right at the beginning of the compilation process:

Do you know what the problem might be?

Try to clone the application as usual:
git clone  --recurse-submodules https://github.com/noaa-ocs-modeling/CoastalApp.git
I built the application from scratch using:
build.sh --component "atmesh ww3 adcirc" --compiler gnu --tp parmetis
and everything compiled just fine.

Thank you for your response Takis! I am trying to compile a code that has worked in the past, so the folders are all there. Also, the compilation of ADCIRC and WW3 as standalone models inside the CoastalApp works fine. The only thing that changed between the time that I was able to compile and run the CoastalApp, was that they changed the system's compilers. And somehow this change has made the CoastalApp compilation crushing at the very beginning of the process.
Angelos

Note also, that I was trying with a previous version of CoastalApp. When I tried to compile the latest version, the compilation got through this initial stage, ADCIRC and WW3 were compiled successfully, but then just before the end of the compilation process, I got this error:

Do you know what could be the reason for this error?
@pvelissariou1
Angelos

Angelos,
What version of ESMF are you using to compile the application.
Make sure before each compilation that you clean the system completely.
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell:<PHONE_NUMBER>
email: @.***
On Sat, Aug 26, 2023 at 12:28 AM sarriegi @.***> wrote:

Note also, that I was trying with a previous version of CoastalApp. When I
tried to compile the latest version, the compilation got through this
initial stage, ADCIRC and WW3 were compiled successfully, but then just
before the end of the compilation process, I got this error:
[image: image]
https://user-images.githubusercontent.com/77630268/263441550-8ff3285b-c207-4b3f-a27c-0e4ce53c4960.png
Do you know what could be the reason for this error?
@pvelissariou1 https://github.com/pvelissariou1
Angelos
—
Reply to this email directly, view it on GitHub
https://github.com/noaa-ocs-modeling/CoastalApp/issues/179#issuecomment-1694173711,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APC7TP2B4DBL53E6S7QRH5DXXGCPLANCNFSM6AAAAAA37C63SI
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hello Takis, I am using version 8.0.1, the same version I was always using, which was working fine before they changed the compilers.
Angelos

What is the version of the compiler you are using?
Make sure that the ESMFMKFILE variable points to the correct installation
of ESMF.
I have tested CoastalApp/ESMF with GNU and Intel compilers (up to 2022)
without any issue.
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell:<PHONE_NUMBER>
email: @.***
On Sat, Aug 26, 2023 at 10:44 AM sarriegi @.***> wrote:

Hello Takis, I am using version 8.0.1, the same version I was always
using, which was working fine before they changed the compilers.
Angelos
—
Reply to this email directly, view it on GitHub
https://github.com/noaa-ocs-modeling/CoastalApp/issues/179#issuecomment-1694391702,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APC7TP6IK3FSASNWQMOPSY3XXIKVDANCNFSM6AAAAAA37C63SI
.
You are receiving this because you were mentioned.Message ID:
@.***>

I am using the new compilers that they installed in MIT Supercloud, intel 2023.1. With the previous compiler which was intel 2019.5, it was compiling fine.
Angelos

Could you please send me the full compile log from CoastalApp and possibly
from ESMF?
I cannot determine what is wrong with your compilations.
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell:<PHONE_NUMBER>
email: @.***
On Sun, Aug 27, 2023 at 10:43 PM sarriegi @.***> wrote:

I am using the new compilers that they installed in MIT Supercloud, intel
2023.1. With the previous compiler which was intel 2019.5, it was compiling
fine.
Angelos
—
Reply to this email directly, view it on GitHub
https://github.com/noaa-ocs-modeling/CoastalApp/issues/179#issuecomment-1694958095,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APC7TP7VH7XKLEPRE3GOKGTXXQHURANCNFSM6AAAAAA37C63SI
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hello Takis, I am attaching the coastalapp compilation and the esmf build log files. The error was this (for some reason was not printed on the log file:

coastalapp_compile.log
esmf_make.log
Thank you for looking into this!
Angelos

Angelos,
I'll try to look into this as soon as possible. I'll keep you posted
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell:<PHONE_NUMBER>
email: @.***
On Mon, Aug 28, 2023 at 12:34 PM sarriegi @.***> wrote:

Hello Takis, I am attaching the coastalapp compilation and the esmf build
log files. The error was this (for some reason was not printed on the log
file:
[image: image]
https://user-images.githubusercontent.com/77630268/263777704-2ae368d1-22d4-4380-8936-e1665d3b1b58.png
coastalapp_compile.log
https://github.com/noaa-ocs-modeling/CoastalApp/files/12456883/coastalapp_compile.log
esmf_make.log
https://github.com/noaa-ocs-modeling/CoastalApp/files/12456884/esmf_make.log
Thank you for looking into this!
Angelos
—
Reply to this email directly, view it on GitHub
https://github.com/noaa-ocs-modeling/CoastalApp/issues/179#issuecomment-1696080382,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APC7TP4O4NLE65FFGUAQUUTXXTJDLANCNFSM6AAAAAA37C63SI
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hi Angelos,
I didn't find any apparent issues in the compilations of ESMF/CoastalApp.
(I am wondering though why the ESMF compilation is stored in an openmpi
folder: /home/gridsan/apapandreou/esmf-ESMF_8_0_1_openmpi)
So in the CoastalApp/NEMS/src/module_MEDIATOR.F90 we have the two code
blocks:
!---------------------------

! validate all data by default
!---------------------------

#if ESMF_VERSION_MAJOR >= 8

call NUOPC_SetTimestamp(NState_AtmExp, clock, rc=rc)

#else
call NUOPC_UpdateTimestamp(NState_AtmExp, clock, rc=rc)
#endif
if (ESMF_LogFoundError(rcToCheck=rc, msg=ESMF_LOGERR_PASSTHRU, &
line=LINE, file=FILE)) return  ! bail out

!---------------------------
! COLDSTART:
! invalidate all data on timestep 1
! invalidate SST on all timesteps
!---------------------------

if (coldstart) then

  if (is_local%wrap%fastcntr == 1) then

#if ESMF_VERSION_MAJOR >= 8
call NUOPC_SetTimestamp(NState_AtmExp, clock_invalidTimeStamp,
rc=rc)
#else
call NUOPC_UpdateTimestamp(NState_AtmExp, clock_invalidTimeStamp,
rc=rc)
#endif

which means that NUOPC_SetTimestamp is called despite the fact that your
ESMF version is greater than 8.
Can you check the esmf.mk pointing to by the ESMFMKFILE env. variable for
the variables:
ESMF_VERSION_MAJOR, ESMF_VERSION_MINOR, ESMF_VERSION_REVISION
to see if these variables are set correctly?
Can you share this esmf.mk file?
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell:<PHONE_NUMBER>
email: @.***
On Mon, Aug 28, 2023 at 3:38 PM Panagiotis Velissariou - NOAA Affiliate <
@.***> wrote:

Angelos,
I'll try to look into this as soon as possible. I'll keep you posted
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell:<PHONE_NUMBER>
email: @.***
On Mon, Aug 28, 2023 at 12:34 PM sarriegi @.***>
wrote:

Hello Takis, I am attaching the coastalapp compilation and the esmf build
log files. The error was this (for some reason was not printed on the log
file:
[image: image]
https://user-images.githubusercontent.com/77630268/263777704-2ae368d1-22d4-4380-8936-e1665d3b1b58.png
coastalapp_compile.log
https://github.com/noaa-ocs-modeling/CoastalApp/files/12456883/coastalapp_compile.log
esmf_make.log
https://github.com/noaa-ocs-modeling/CoastalApp/files/12456884/esmf_make.log
Thank you for looking into this!
Angelos
—
Reply to this email directly, view it on GitHub
https://github.com/noaa-ocs-modeling/CoastalApp/issues/179#issuecomment-1696080382,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APC7TP4O4NLE65FFGUAQUUTXXTJDLANCNFSM6AAAAAA37C63SI
.
You are receiving this because you were mentioned.Message ID:
@.***>

Thank you for looking into this Takis, I think that they are set correctly. I am attaching the esmf.mk file
Angelos
esmf.txt

I don't understand either. I think I will move on by trying to run CoastalApp in a different cluster. Thank you again!
Angelos

Let me know how it goes.
Panagiotis Velissariou, Ph.D., P.E.
UCAR Scientist
National Ocean and Atmospheric Administration
National Ocean Service
Office of Coast Survey CSDL/CMMB
Physical Scientist - Project Lead
cell:<PHONE_NUMBER>
email: @.***
On Tue, Aug 29, 2023 at 12:22 PM sarriegi @.***> wrote:

I don't understand either. I think I will move on by trying to run
CoastalApp in a different cluster. Thank you again!
Angelos
—
Reply to this email directly, view it on GitHub
https://github.com/noaa-ocs-modeling/CoastalApp/issues/179#issuecomment-1697852485,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APC7TP2472D6APFNR4LATMLXXYQNNANCNFSM6AAAAAA37C63SI
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hi Takis, I was able to compile it after in MIT supercloud (by adding -DESMF_VERSION_MAJOR=8 to CPPFLAGS in NEMS/src/GNUmakefile) after all. However, when I try to run I get these errors:

Have you ever seen errors like that? Do you have any idea of what the problem might be?
Angelos

@sarriegi , @saeed-moghimi-noaa Angelos it seems to me that in your cloud cluster the environment variables are not propagating properly. CoastalApp and its components during the compilation stage source and/or, include the file pointed by the ESMFMKFILE variable which, contains the definitions of all the ESMF related variables. We never had any of these issues with ESMF. May be your bash login is not configured properly. You should never have to modify the NEMS/src/GNUmakefile for this.
Regarding the errors, I think these are coming from the library libucs which is part of the ucx package in linux. In the cluster run the command: rpm -q ucx ucx-devel (I guess the cluster uses some version of RedHat/CentOS) to see what version of the ucx package is installed in your system. I believe, you should have ucx >= 1.12 installed for things to work. I had this issue before and it was all due to the ucx library. I compiled a newer version of ucx in one of our HPC clusters and all these issues were resolved. See what you can find and let us know.

Thank you for this information Takis, this is the related information about Supercloud:

Since it is 1.12 I assume this is not the problem. Also everything was working fine with the old version of the compiler. It's the new compiler that has messed things up. I think I will move on and try to run CoastalApp in the Unity cluster (UMass).
Angelos

Hello Takis, I compile CoastalApp in a new cluster. However, when I tried to run it I received these errors in on of the PET* files:

Do you have any idea about what I could be doing wrong?
Angelos
@pvelissariou1 @saeed-moghimi-noaa

Have you tried this configuration before?

@sarriegi are the atm. fields given on triangular mesh? Same mesh as the ocean model? Can you try to use: remapMethod=bilinear:extrapMethod=nearest_stod to see what happens?

Thank you for the suggestion, Takis! I have used this configuration successfully before. I am using atmesh.F90 and atmesh_mod.F90 files that are not the original ones in CoastalApp. These files allow for the wind forcing file to be given on a regular (structured) grid. I am attaching these files here:
atmesh.txt
atmesh_mod.txt
Angelos

@sarriegi I guess the atmospheric grid overlaps the ocean mesh completely.


[Request]: sidebar collapsed

It will be very useful to add an option for the scene viewer to load the panel with the sidebar collapsed.
Indeed, if the scene viewer is in a dashboard with other panels on the right, then the sidebar reduces to much the 3D scene.

Hi @raffistef, configuration options for scene viewer aren't supported yet but we will take this feedback into consideration, thanks!

Hi @johnnyw-aws thank you for your answer.
Please, do not hesitate to contact me if you need more feedbacks for this request!

@mumanity This is related to the sidebar enhancement you are working on. Please update this ticket once done.

Supported in v1.6.2


Vulnerability roundup 103: hdf5-1.10.7: 4 advisories [5.5]

search, files

[ ] CVE-2020-10809 CVSSv3=5.5 (nixos-21.05)
[ ] CVE-2020-10810 CVSSv3=5.5 (nixos-21.05)
[ ] CVE-2020-10811 CVSSv3=5.5 (nixos-21.05)
[ ] CVE-2020-10812 CVSSv3=5.5 (nixos-21.05)

Scanned versions: nixos-21.05: 0b8b127125e.

See also: #116881

NixOS 21.05 is EOL, but this package was downgraded to 1.10.6. Still an issue.


Fixed an issue with contextual type for intersection properties (take 2)

fixes #48812
This PR is another take on what already has been approved and merged: https://github.com/microsoft/TypeScript/pull/48668 . It later got reverted to give time for further investigation of https://github.com/microsoft/TypeScript/issues/49307 since that PR broke Redux Toolkit's types.
This PR adds adjusted test cases that were discovered in #49307. The minimal repro case posted by @andrewbranch was this:
type Validate<T> = T & { [K in keyof T]: object }
declare function f<S, T extends Record<string, (state: S) => any>>(s: S, x: Validate<T>): void;

f(0, {
  foo: s => s + 1, // contextual type for `s` provided, no implicit any reported
})

At the moment RTK uses an index signature for their SliceCaseReducers type (here) and this PR breaks that. @andrewbranch has reasonable doubts (starting from here) if that should ever work.
However, we can adjust the RTK types to use Record instead of the indexed signature and they still work today, and with this fix. I've "spiked" such a change in a PR to the RTK project here
and all tests pass - on all tested TS version 🚀
I propose to change the RTK types and land this PR in its current form. It seems to me like a solution that would give us the best of both worlds.
Note that to fix #48812 we could potentially still pull the contextual type from index signatures, even in presence of "concrete" properties. However, I agree with @andrewbranch - this seems like an inconsistent thing to do. So I left it out of this PR - while making sure that the RTK case can work with slightly adjusted types.
The only practical difference in this PR, that makes it different from the original #48668, is that now the getTypeOfConcreteLikePropertyOfContextualType pulls the "concrete" properties of non-generic mapped types (such as Record):
https://github.com/microsoft/TypeScript/pull/52095/files#diff-d9ab6589e714c71e657f601cf30ff51dfc607fc98419bf72e04f6b0fa92cc4b8R28459-R28461
cc @andrewbranch @RyanCavanaugh @phryneas @markerikson

At the moment RTK uses an index signature for their SliceCaseReducers type (here) and this PR breaks that. @andrewbranch has reasonable doubts (starting from https://github.com/microsoft/TypeScript/issues/49307#issuecomment-1176562527) if that should ever work.

From experience with the larger TS community, I can say with some confidence that IndexSignature & SpecificProps is a very common workaround in the wild for https://github.com/microsoft/TypeScript/issues/17867.  I have to grit my teeth every time I encounter it and have actively tried to steer people away from it, but most people tend to insist they need it anyway despite the footguns.  So any change to the behavior of that pattern is likely to break people, unfortunately.

Also aren't Record<string, T> and { [x: string]: T } the exact same type?

One is { [K in string]: T } and the other is { [x: string]: T } - while they describe the same thing (in the case of the full string type, but not in any other case), they have always behaved differently.

@phryneas Hmm, if they behave differently then that makes this rather misleading:

That indeed seems to be a bit misleading. The definition of Record definitely uses a mapped type though.
type Record<K extends keyof any, T> = {
    [P in K]: T;
};

Yeah, I was aware that was the implementation.  I just thought that { [K in string]: T } automatically reduced to an index signature, thanks mostly to the above tooltip.
@matheusiacono I'm... not sure why you 👎'd my comment?  That the hover tip displays an index signature when the actual type behaves differently from what's displayed is factual and not just my opinion.

From experience with the larger TS community, I can say with some confidence that IndexSignature & SpecificProps is a very common workaround in the wild for https://github.com/microsoft/TypeScript/issues/17867 (as a matter of fact, it's the very first reply in that issue!). I have to grit my teeth every time I encounter it and have actively tried to steer people away from it, but most people tend to insist they need it anyway despite the footguns. So any change to the behavior of that pattern is likely to break people, unfortunately.

Note that this PR doesn't necessarily have to break patterns like this. It's hard to tell though - I would have to examine a concrete example, with a concrete use case in mind (contextual typing, completions, etc).
As to the difference between Record<string, T> and { [x: string]: T }. Those are 2 different types, type displays can sometimes be confusing like that - but TS at times tracks how a given type originated and might utilize this information. For example PlainFormFields and MappedFormFields display exactly the same here but yet they behave differently.
It's helpful to check out the TS AST Viewer and the flags contained on those types (link here). Also note that their types have different IDs - which is the ultimate no answer to the question if they are the same.
// flags: Object (2 ^ 19) | DefinitelyNonNullable | StructuredType | StructuredOrInstantiable | ObjectFlagsType | Narrowable | IncludesMask | NotPrimitiveUnion
// objectFlags: Anonymous (2 ^ 4) | ObjectTypeKindMask
type A = { [key: string]: number }
// flags: Object (2 ^ 19) | DefinitelyNonNullable | StructuredType | StructuredOrInstantiable | ObjectFlagsType | Narrowable | IncludesMask | NotPrimitiveUnion
// objectFlags: Mapped (2 ^ 5) | Instantiated (2 ^ 6) | CouldContainTypeVariablesComputed (2 ^ 19) | CouldContainTypeVariables (2 ^ 20) | ObjectTypeKindMask
type B = Record<string, number>
// flags: Object (2 ^ 19) | DefinitelyNonNullable | StructuredType | StructuredOrInstantiable | ObjectFlagsType | Narrowable | IncludesMask | NotPrimitiveUnion
// objectFlags: Mapped (2 ^ 5) | ObjectTypeKindMask
type C = { [K in string]: number }

We might notice here that all of them have the same flags but they are all different when it comes to objectFlags. B and C are roughly the same though, B just have some extra objectFlags since it's coming from an instantiated type alias.
That being said - I can't be sure if the proposed change is how this whole fix should look like. It's a conversation starter, maybe I will have to tweak this based on the TS team feedback.

Does this PR create an observable difference between Record<string, T> and { [x: string]: T } or did you just discover an existing one that lets you work around the other effects of the PR?

@andrewbranch it creates an observable difference in getting the type of the property of a contextual type. Some of the the new test cases wouldn't infer correctly with the generic constraint of { [x: string]: T } but they infer OK with Record<string, T>. You could change the definition of this SliceCaseReducers back and forth to observe the difference
However, I'm not saying - by any means - that this is what I want to land. This whole PR is meant to restart the conversation around this since the original PR landed and got reverted later.
When discussing the regression in https://github.com/microsoft/TypeScript/issues/49307, both you and @RyanCavanaugh said that the original PR had merit and that contextual properties coming from index signatures are dubious (when a concrete property is available) and that cases like this were handled inconsistently by TS. The original PR was reverted to give you more time to think about this - it has been half a year and I just hope to revive this conversation.
There are some important things to note down here - we don't need to create any observable difference between both of those. This is not required to fix what this PR is fixing.
So to move this PR forward we can either:

ignore index signatures in presence of concrete properties. This would likely reintroduce the regression from #49307. It's even possible that wouldn't necessarily be a bad thing (I'm not super clear on that though) but it's unknown to me how RTK could rewrite their types today to fix their types. I proposed a solution here but it was dismissed because it doesn't work in older TS versions. I just rechecked this playground and it seems that it doesn't work on TS 3.5 (!) but it already works with TS 3.6. IMHO, at this point, this is a fair game for RTK to break compat with TS 3.5 if they don't want to hussle with the typesVersions complexity. Of course, this doesn't even mean that this solution would work for them - as I only validated it with that slimmed down repro. I just created a PR to RTK with this change to check out the CI results. It also doesn't necessarily mean that their current types shouldn't work - they are certainly unusual but maybe they don't abuse the implementation and are aligned with TS design. That's something that the TS team would have to decide on.
never ignore those index signatures. This PR would still fix what it is fixing. I just didn't do it because you had doubts about index signatures behavior in https://github.com/microsoft/TypeScript/issues/49307
use a mixed approach - but I'm not sure what exact heuristics should be used. Perhaps we should check if the contextual type is part of the generic constraint? If it is - then it would make sense to use its "index signature", when the contextual type would be concrete then the index signature could be ignored. I'm not sure what the exact condition for such a check would be preferred but I could experiment with something if you decide that this is what your team wants to have implemented.

For the record we already only support TS 4.2+ as of RTK 1.9.

Your CI job just reported that the proposed solution doesn't work for you 🤣 it seems that there is more to this there than what was captured by the slimmed-down repro

@andrewbranch @RyanCavanaugh would there be a chance to put this on 5.2 agenda since the work on that has just started? I would gladly tweak the content of this PR however you see fit - but right now it's not entirely obvious how this should behave when index signatures are involved. This probably warrants a design meeting or something.

To get the ball rolling I made some changes to this PR. I decided to treat indexed mapped type substitutions within intersections differently. Now they are intersected with either concrete property types or with applicable index info types.
They are not treated as "concrete" - so they don't take precedence over applicable index info types. They are always used though, if we find any concrete property types then we intersect them with those. If we don't have any concrete property types then we find applicable index info types and we intersect with those.
It is somewhat strange... but it maintains backward compatibility.

@ahejlsberg @gabritto friendly 🏓

@jakebailey would u mind creating a playground for this one?

@typescript-bot pack this

@ahejlsberg @gabritto @RyanCavanaugh any chance that this could be considered for 5.5? It fixes one of the "Help wanted" issues - let me help you! 😉

5.6 development has begun...

@typescript-bot test it

@typescript-bot test it

@typescript-bot test top800

@typescript-bot pack this

@jakebailey @gabritto I'd appreciate a user tests + top 800 rerun here :)

@typescript-bot test it

The latest simplification revealed new issues in the extended tests. I went back to my previous approach and added more tests. Could I ask for a rerun? :)

@typescript-bot test it

The break above seems to be related to how contextual any is eliminated from intersections here. On main contextual this gets reduced to any (see the TS playground here) and it doesn't happen with the code in this PR.
this is a contextual parameter and regular contextual parameters are not ignoring any so it also shouldn't be ignored for this. I'll fix this soon.

@jakebailey would you be so kind to rerun the tests? :)

@typescript-bot test it

@typescript-bot test it

@typescript-bot test it
@typescript-bot pack this


Travis build fails for the kabanero-pipelines repo

Something in the deploy phase with building an image is failing.

Not really sure what happened but seems to have fixed itself this morning after not working for a week 🤷♂


Cronjob not running

I posted this also in the emby forums, but wanted to put it here in case this is better.
I'm running v1.51 on a mac.  I can get the docker to run and pull and create the .strm files, but noticed that the cron jobs do not run.  In the logs I can see this: "no cronjob for root". Also, the cron.log file is empty.
I also noticed that the cron-movies.log only shows "...Starting Download...", but it is working.
Below is my sanitized compose yaml:
version: "3"
streamgen:
image: ghcr.io/hooray4me/ustreamgen:v1.51-dev
container_name: ustreamgen
command: /m3u2strm/initialize_cron.sh
environment:
- UID=501 #UID for file ownership
- GID=20 #GID for file ownership
- UNAME=aaron #Username for file ownership
- GROUP=staff #Group name for file ownership
- SINGLELIST=false #when true a single list of movies, series and events is used. flase for multiple lists
- MOVIES=true #set to false if movies are not desired
- TVSHOWS=true #set to false if tv shows are not desired
- EVENTS=true # set to false is live sporting events are not desired
- CRONHOUR=0 #0-23 # sets the hour of the day the script will run again for all content in 1 list **ALL must be true
- CRONMINUTE=10 #0-59 # sets the minute of the day the script will run again for all content 1 list **ALL must be true
- TVCRONHOUR=0 #0-23 # sets the hour of the day the script will run again for tv shows **TVSHOWS must be true
- TVCRONMINUTE=10 #0-59 # sets the minute of the day the script will run again for tv shows **TVSHOWS must be true
- MOVIECRONHOUR=0 #0-23 # sets the hour of the day the script will run again for movies **MOVIES must be true
- MOVIECRONMINUTE=20 #0-59 # sets the minute of the day the script will run again for movies **MOVIES must be true
- EVENTCRONHOUR=0 #0-23 # sets the hour of the day the script will run again for events **EVENTS must be true
- EVENTCRONMINUTE=30 #0-59 # sets the minute of the day the script will run again for events **EVENTS must be true
- TZ=America/New_York
- SINGLELISTURL=https://tvnow.best/api/list/redacted/m3u8/movies/ # Full M3U Provider URL for all content in one list **SINGLELIST must be true
- MOVIEURL=https://tvnow.best/api/list/redacted/m3u8/movies/ # Full M3U Provider URL for Movies **MOVIES must be true
- TVSHOWURL=https://tvnow.best/api/list/redacted/m3u8/tvshows/ # Full M3U Provider URL for TV Shows **TVSHOWS must be true
- EVENTURL=https://tvnow.best/api/list/redacted/m3u8/events/ # Full M3U Provider URL for Events **EVENTS must be true
- APOLLO=true #Optional change to true if your Provider is Apollo
volumes:
- $PWD/tv:/tv
- $PWD/movies:/movies
- $PWD/events:/events
- $PWD/logs:/logs
I tried running the released version 1.42 but as soon as I started it up it would give me a permissions error, but the v1.51 seems to work working except for the cronjobs.

I spoke to soon.  After checking this over a couple of days it seems to be updating without issue.


build: Add rust compiler to charms

Context
Hit during #1156 and #1177, we hit the issue where charms fail to be built due to the absence of a rust compiler during build time.

[ ] argo-controller
[ ] kfp charms apart from metadata-writer
[ ] knative charms
[ ] kserve-controller
[ ] kubeflow-profiles-operator
[ ] volumes-operator
[ ] metacontroller-operator
[ ] minio-operator
[ ] namespace-node-affinity-operator
[ ] ngc-integrator
[ ] training-operator

What needs to get done
Add appropriate dependencies as we previously did https://github.com/canonical/kubeflow-dashboard-operator/commit/c31941bfe338b5913809741722aeb94b8c03a176 or follow what mysql-k8s team is doing seen here https://github.com/canonical/charmcraft/issues/1722#issuecomment-2259780356. The latter proved to be building faster the charm ( 12 minutes compared to 18 minutes for argo-controller)
Definition of Done
Charms can be built with dependencies requiring a rust compiler.

KFP
KFP tests fail due to space in the runner as seen here https://github.com/canonical/kfp-operators/actions/runs/12162787081/job/33958507596?pr=631#step:5:468 (same as https://github.com/canonical/kfp-operators/issues/602)
FAILED tests/integration/test_kfp_functional_v2.py::test_create_and_monitor_run - AssertionError: assert 'FAILED' == 'SUCCEEDED'

  - SUCCEEDED
  + FAILED

and here https://github.com/canonical/kfp-operators/actions/runs/12162787081/job/33958506393?pr=631#step:5:442
unit-kfp-profile-controller-0: 08:30:35 ERROR juju.worker.uniter.operation hook "update-status" (via unknown/invalid hook handler) failed: starting jujuc server: listening to jujuc socket: mkdir /var/lib/juju/agents/unit-kfp-profile-controller-0/426642296: no space left on device

This means that adding rust compiler to the charms needs first this feature to land https://github.com/canonical/kfp-operators/issues/627.

Note that as done in mysql-k8s (see https://github.com/canonical/charmcraft/issues/1722#issuecomment-2259780356), the build packages pkg-config, libffi-dev, libssl-dev are needed. Otherwise, building fails with the following error:
...
:: Running external command ['python3', '-m', 'venv', '/root/parts/charm/build/staging-venv']
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--force-reinstall', 'pip@https://files.pythonhosted.org/packages/c0/d0/9641dc7b05877874c6418f8034ddefc809495e65caa14d38c7551cd114bb/pip-24.1.1.tar.gz']
::    :: Collecting pip@ https://files.pythonhosted.org/packages/c0/d0/9641dc7b05877874c6418f8034ddefc809495e65caa14d38c7551cd114bb/pip-24.1.1.tar.gz
::    ::   Downloading pip-24.1.1.tar.gz (1.9 MB)
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::     Preparing wheel metadata: started
::    ::     Preparing wheel metadata: finished with status 'done'
::    :: Building wheels for collected packages: pip
::    ::   Building wheel for pip (PEP 517): started
::    ::   Building wheel for pip (PEP 517): finished with status 'done'
::    ::   Created wheel for pip: filename=pip-24.1.1-py3-none-any.whl size=1824282 sha256=e8294edf16c52b89a0eccd34036d30218ae3bdac7a2acf323b6dfe8d3d8e88e1
::    ::   Stored in directory: /root/.cache/pip/wheels/a2/93/80/a8fa207c102102e4c7f0ea7965db2509466086aa06a927a5ce
::    :: Successfully built pip
::    :: Installing collected packages: pip
::    ::   Attempting uninstall: pip
::    ::     Found existing installation: pip 20.0.2
::    ::     Uninstalling pip-20.0.2:
::    ::       Successfully uninstalled pip-20.0.2
::    :: Successfully installed pip-24.1.1
:: Installing Python pre-dependencies from source.
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=:all:', 'wheel', 'setuptools', 'pip']
::    :: Collecting wheel
::    ::   Downloading wheel-0.45.1.tar.gz (107 kB)
::    ::      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 107.5/107.5 kB 19.5 MB/s eta 0:00:00
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::   Preparing metadata (pyproject.toml): started
::    ::   Preparing metadata (pyproject.toml): finished with status 'done'
::    :: Requirement already satisfied: setuptools in ./staging-venv/lib/python3.8/site-packages (44.0.0)
::    :: Requirement already satisfied: pip in ./staging-venv/lib/python3.8/site-packages (24.1.1)
::    :: Building wheels for collected packages: wheel
::    ::   Building wheel for wheel (pyproject.toml): started
::    ::   Building wheel for wheel (pyproject.toml): finished with status 'done'
::    ::   Created wheel for wheel: filename=wheel-0.45.1-py3-none-any.whl size=72494 sha256=708e7481cc80179af0e556bbf0cc00b8444c7321e2700b8d8580231d13017248
::    ::   Stored in directory: /root/.cache/pip/wheels/1b/00/e6/216671cc7f33efcb5fc4e2b16b86f829cc57d2e0e28d9b90e8
::    :: Successfully built wheel
::    :: Installing collected packages: wheel
::    :: Successfully installed wheel-0.45.1
::    ::
Notice: ::    :: [notice] A new release of pip is available: 24.1.1 -> 24.3.1
Notice: ::    :: [notice] To update, run: python3 -m pip install --upgrade pip
:: Installing packages from requirements files and charm lib dependencies.
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=:all:', '--requirement=requirements.txt']
::    :: Collecting annotated-types==0.7.0 (from -r requirements.txt (line 7))
::    ::   Downloading annotated_types-0.7.0.tar.gz (16 kB)
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::   Preparing metadata (pyproject.toml): started
::    ::   Preparing metadata (pyproject.toml): finished with status 'done'
::    :: Collecting anyio==4.5.2 (from -r requirements.txt (line 9))
::    ::   Downloading anyio-4.5.2.tar.gz (171 kB)
::    ::      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 171.3/171.3 kB 20.4 MB/s eta 0:00:00
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::   Preparing metadata (pyproject.toml): started
::    ::   Preparing metadata (pyproject.toml): finished with status 'done'
::    :: Collecting attrs==24.2.0 (from -r requirements.txt (line 11))
::    ::   Downloading attrs-24.2.0.tar.gz (792 kB)
::    ::      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 792.7/792.7 kB 96.0 MB/s eta 0:00:00
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::   Preparing metadata (pyproject.toml): started
::    ::   Preparing metadata (pyproject.toml): finished with status 'done'
::    :: Collecting backports-strenum==1.3.1 (from -r requirements.txt (line 13))
::    ::   Downloading backports_strenum-1.3.1.tar.gz (7.3 kB)
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::   Preparing metadata (pyproject.toml): started
::    ::   Preparing metadata (pyproject.toml): finished with status 'done'
::    :: Collecting bcrypt==4.2.1 (from -r requirements.txt (line 15))
::    ::   Downloading bcrypt-4.2.1.tar.gz (24 kB)
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::   Preparing metadata (pyproject.toml): started
::    ::   Preparing metadata (pyproject.toml): finished with status 'done'
::    :: Collecting cachetools==5.5.0 (from -r requirements.txt (line 17))
::    ::   Downloading cachetools-5.5.0.tar.gz (27 kB)
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::   Preparing metadata (pyproject.toml): started
::    ::   Preparing metadata (pyproject.toml): finished with status 'done'
::    :: Collecting certifi==2024.8.30 (from -r requirements.txt (line 19))
::    ::   Downloading certifi-2024.8.30.tar.gz (168 kB)
::    ::      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 168.5/168.5 kB 40.9 MB/s eta 0:00:00
::    ::   Preparing metadata (setup.py): started
::    ::   Preparing metadata (setup.py): finished with status 'done'
::    :: Collecting cffi==1.17.1 (from -r requirements.txt (line 25))
::    ::   Downloading cffi-1.17.1.tar.gz (516 kB)
::    ::      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 516.6/516.6 kB 53.3 MB/s eta 0:00:00
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::   Preparing metadata (pyproject.toml): started
::    ::   Preparing metadata (pyproject.toml): finished with status 'done'
::    :: Collecting charmed-kubeflow-chisme==0.4.3 (from -r requirements.txt (line 29))
::    ::   Downloading charmed_kubeflow_chisme-0.4.3.tar.gz (43 kB)
::    ::      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.3/43.3 kB 13.4 MB/s eta 0:00:00
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::   Preparing metadata (pyproject.toml): started
::    ::   Preparing metadata (pyproject.toml): finished with status 'done'
::    :: Collecting charset-normalizer==3.4.0 (from -r requirements.txt (line 31))
::    ::   Downloading charset_normalizer-3.4.0.tar.gz (106 kB)
::    ::      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 106.6/106.6 kB 29.3 MB/s eta 0:00:00
::    ::   Preparing metadata (setup.py): started
::    ::   Preparing metadata (setup.py): finished with status 'done'
::    :: Collecting cosl==0.0.45 (from -r requirements.txt (line 33))
::    ::   Downloading cosl-0.0.45.tar.gz (67 kB)
::    ::      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.5/67.5 kB 16.9 MB/s eta 0:00:00
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: still running...
::    ::   Installing build dependencies: still running...
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::   Preparing metadata (pyproject.toml): started
::    ::   Preparing metadata (pyproject.toml): finished with status 'done'
::    :: Collecting cryptography==44.0.0 (from -r requirements.txt (line 35))
::    ::   Downloading cryptography-44.0.0.tar.gz (710 kB)
::    ::      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 710.7/710.7 kB 91.0 MB/s eta 0:00:00
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: still running...
::    ::   Installing build dependencies: still running...
::    ::   Installing build dependencies: finished with status 'error'
::    ::   error: subprocess-exited-with-error
::    ::
::    ::   × pip subprocess to install build dependencies did not run successfully.
::    ::   │ exit code: 1
::    ::   ╰─> [89 lines of output]
::    ::       Collecting maturin<2,>=1
::    ::         Downloading maturin-1.7.8.tar.gz (195 kB)
::    ::            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 195.7/195.7 kB 2.5 MB/s eta 0:00:00
::    ::         Installing build dependencies: started
::    ::         Installing build dependencies: finished with status 'done'
::    ::         Getting requirements to build wheel: started
::    ::         Getting requirements to build wheel: finished with status 'done'
::    ::         Preparing metadata (pyproject.toml): started
::    ::         Preparing metadata (pyproject.toml): finished with status 'done'
::    ::       Collecting cffi>=1.12
::    ::         Using cached cffi-1.17.1.tar.gz (516 kB)
::    ::         Installing build dependencies: started
::    ::         Installing build dependencies: finished with status 'done'
::    ::         Getting requirements to build wheel: started
::    ::         Getting requirements to build wheel: finished with status 'done'
::    ::         Preparing metadata (pyproject.toml): started
::    ::         Preparing metadata (pyproject.toml): finished with status 'done'
::    ::       Collecting setuptools!=74.0.0,!=74.1.0,!=74.1.1,!=74.1.2
::    ::         Using cached setuptools-75.3.0-py3-none-any.whl
::    ::       Collecting tomli>=1.1.0 (from maturin<2,>=1)
::    ::         Using cached tomli-2.2.1-py3-none-any.whl
::    ::       Collecting pycparser (from cffi>=1.12)
::    ::         Downloading pycparser-2.22.tar.gz (172 kB)
::    ::            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 172.7/172.7 kB 9.1 MB/s eta 0:00:00
::    ::         Preparing metadata (setup.py): started
::    ::         Preparing metadata (setup.py): finished with status 'done'
::    ::       Building wheels for collected packages: maturin, cffi, pycparser
::    ::         Building wheel for maturin (pyproject.toml): started
::    ::         Building wheel for maturin (pyproject.toml): still running...
::    ::         Building wheel for maturin (pyproject.toml): still running...
::    ::         Building wheel for maturin (pyproject.toml): finished with status 'done'
::    ::         Created wheel for maturin: filename=maturin-1.7.8-cp38-cp38-linux_x86_64.whl size=5995658 sha256=e9cbdec65db8641ca6bfcfc3c1f7c89ee91357d80a058e10f54fdbf57c44c[549](https://github.com/canonical/argo-operators/actions/runs/12199502801/job/34033534048#step:4:550)
::    ::         Stored in directory: /root/.cache/pip/wheels/f1/2a/cc/94408ea6aec8dad6addae6c4bb4fb04597715b4315e43aca2d
::    ::         Building wheel for cffi (pyproject.toml): started
::    ::         Building wheel for cffi (pyproject.toml): finished with status 'error'
::    ::         error: subprocess-exited-with-error
::    ::
::    ::         × Building wheel for cffi (pyproject.toml) did not run successfully.
::    ::         │ exit code: 1
::    ::         ╰─> [35 lines of output]
::    ::             running bdist_wheel
::    ::             running build
::    ::             running build_py
::    ::             creating build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/error.py -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/vengine_gen.py -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/recompiler.py -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/model.py -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/vengine_cpy.py -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/verifier.py -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/setuptools_ext.py -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/_shimmed_dist_utils.py -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/__init__.py -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/cffi_opcode.py -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/ffiplatform.py -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/cparser.py -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/backend_ctypes.py -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/lock.py -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/pkgconfig.py -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/api.py -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/_imp_emulation.py -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/commontypes.py -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/_cffi_include.h -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/parse_c_type.h -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/_embedding.h -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             copying src/cffi/_cffi_errors.h -> build/lib.linux-x86_64-cpython-38/cffi
::    ::             running build_ext
::    ::             building '_cffi_backend' extension
::    ::             creating build/temp.linux-x86_64-cpython-38/src/c
::    ::             x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -DFFI_BUILDING=1 -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/root/parts/charm/build/staging-venv/include -I/usr/include/python3.8 -c src/c/_cffi_backend.c -o build/temp.linux-x86_64-cpython-38/src/c/_cffi_backend.o
::    ::             src/c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
::    ::                15 | #include <ffi.h>
::    ::                   |          ^~~~~~~
::    ::             compilation terminated.
::    ::             error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
::    ::             [end of output]
::    ::
::    ::         note: This error originates from a subprocess, and is likely not a problem with pip.
::    ::         ERROR: Failed building wheel for cffi
::    ::         Building wheel for pycparser (setup.py): started
::    ::         Building wheel for pycparser (setup.py): finished with status 'done'
::    ::         Created wheel for pycparser: filename=pycparser-2.22-py3-none-any.whl size=117532 sha256=0c29a4cd2b8e5899651b460a8dfa20524f1a3ead0f26e7b24cbfee87b84bad9d
::    ::         Stored in directory: /root/.cache/pip/wheels/9d/ac/d9/e5fa4cb20ee9c45d0664af14d6a9da2664[558](https://github.com/canonical/argo-operators/actions/runs/12199502801/job/34033534048#step:4:559)3dc723e3ab868
::    ::       Successfully built maturin pycparser
::    ::       Failed to build cffi
::    ::       ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (cffi)
::    ::
Notice: ::    ::       [notice] A new release of pip is available: 24.1.1 -> 24.3.1
Notice: ::    ::       [notice] To update, run: python3 -m pip install --upgrade pip
::    ::       [end of output]
::    ::
::    ::   note: This error originates from a subprocess, and is likely not a problem with pip.
::    :: error: subprocess-exited-with-error
::    ::
::    :: × pip subprocess to install build dependencies did not run successfully.
::    :: │ exit code: 1
::    :: ╰─> See above for output.
::    ::
::    :: note: This error originates from a subprocess, and is likely not a problem with pip.
::    ::
Notice: ::    :: [notice] A new release of pip is available: 24.1.1 -> 24.3.1
Notice: ::    :: [notice] To update, run: python3 -m pip install --upgrade pip
:: Traceback (most recent call last):
::   File "/snap/charmcraft/4914/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 483, in <module>
::     main()
::   File "/snap/charmcraft/4914/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 478, in main
::     builder.build_charm()
::   File "/snap/charmcraft/4914/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 93, in build_charm
::     self.handle_dependencies()
::   File "/snap/charmcraft/4914/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 354, in handle_dependencies
::     self._install_dependencies(staging_venv_dir)
::   File "/snap/charmcraft/4914/lib/python3.10/site-packages/charmcraft/instrum.py", line 152, in _f
::     return func(*args, **kwargs)
::   File "/snap/charmcraft/4914/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 284, in _install_dependencies
::     _process_run(
::   File "/snap/charmcraft/4914/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 414, in _process_run
::     raise RuntimeError(f"Subprocess command {cmd} execution failed with retcode {retcode}")
:: RuntimeError: Subprocess command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=:all:', '--requirement=requirements.txt'] execution failed with retcode 1
:: error:
:: Traceback (most recent call last):
::   File "/snap/charmcraft/current/lib/python3.10/site-packages/craft_parts/ctl.py", line 110, in main
::     ret = CraftCtl.run(cmd, args)
::   File "/snap/charmcraft/current/lib/python3.10/site-packages/craft_parts/ctl.py", line 44, in run
::     _client(cmd, args)
::   File "/snap/charmcraft/current/lib/python3.10/site-packages/craft_parts/ctl.py", line 97, in _client
::     raise RuntimeError(message)
:: RuntimeError: Failed to run the build script for part 'charm'.
'override-build' in part 'charm' failed with code 1.
Recommended resolution: Review the scriptlet and make sure it's correct.
Failed to execute charmcraft in instance.
Full execution log: '/home/runner/.local/state/charmcraft/log/charmcraft-20241206-132340.499833.log'


Performance problem using actix-web with sqlx

Recently I tried actix-web 4.0-beta with sqlx 0.5 features = [ "runtime-actix-native-tls" ], and it appears that the query speed of this combination performs really bad than other framework like tide or roa.
I've got a simple API interface quering several columns in database.
If you want to get more detail, it's the tfb-framework benchmark-multi-queries.
I run the service in a docker container, and actix+sqlx will cost more than 20s to request 500 rows in the database.
However others like tide+sqlx will not(only cost around 300ms).
Also if it is a simple request without querying the database, actix will run fast as normal.
The relying is my code. I'm willing to believe that this is caused by myself reasons.
I tried a lot of ways to optimize it, but none of them work.
I tried connecting into different database, but the postgres version runs slower.
So I'm wondering if it is a problem related with the runtime.
Maybe in the progress something blocks server from establishing connection in the connection pool?
Actually I also tried runtime-tokio, but the result is the same.
I use for iteration to run sql for queries of times:
pub async fn get_world_by_random_id(pool: &MySqlPool, queries: usize) -> ServiceResult<Vec<World>> {
    let mut result = Vec::with_capacity(queries);
    for _ in 0..queries {
        let random_id = rand::get_random_u32(1, 10000);
        let column: World = sqlx::query_as("SELECT * FROM world WHERE id = ?")
            .bind(random_id)
            .fetch_one(pool).await?;
        result.push(column);
    }
    Ok(result)
}

And the query function is called by the handler in actix-web:
#[get("/queries")]
pub async fn get(
    params: web::Query<Params>,
    pool: web::Data<MySqlPool>,
) -> Result<HttpResponse, ServiceError> {
    let queries: usize =
        lexical::parse(params.queries.clone().unwrap_or("1".to_owned())).unwrap_or(1);
    // ...
    db::get_world_by_random_id(&pool, queries)
        .await
        .map(|res| HttpResponse::Ok().json(&res))
}

I'm really sorry for bothering you by this issue.
I'm not sure if this issue is apporiate to be put into sqlx's repository, maybe I should also leave one in Actix-web.
You can close this issue if it is not the problem with sqlx.
But I want to know is there any ways that can help me go futher into this problem? Can anyone give me some clues with this issue?
A lot thanks. ❤❤❤

You seem to be using the pool directly in a loop, could you try to first .aquire() a connection before the loop and using &mut acquired_connection instead of pool as the executor (argument to fetch_one) to see if that improves performance at all?

You seem to be using the pool directly in a loop, could you try to first .aquire() a connection before the loop and using &mut acquired_connection instead of pool as the executor (argument to fetch_one) to see if that improves performance at all?

Thanks! Thanks a lot! ❤
It works, at least the time cost looks reasonable now.
A little bit curious about the different between using a connection and using pool.
Anyway I will close this issue, for it's currently been solved.

When you use the pool "directly", a new connection is acquired from the pool and released back to it in every iteration of your loop. The pool performance could probably be perf-tuned, but idk for sure (I've never looked at that part of the code).

I think I've got a better solution, the mistake I made actually was that I wait query to finishing in every loop in the iteration.
So the code ran in a serialized way, and using pool "directly" caused a lot of release and acquire things.
We should use Future Object to make the query run in a true async way:
use crate::custom_type::{pool::MySqlPool, result::ServiceResult};
use crate::models::benchmark::World;
use crate::utils::rand;
use futures::stream::{FuturesUnordered, TryStreamExt};

pub async fn query_worlds_by_random_id(pool: &MySqlPool, queries: usize) -> ServiceResult<Vec<World>> {
    let worlds = FuturesUnordered::new();
    for _ in 0..queries {
        worlds.push(get_world_by_random_id(&pool));
    }
    worlds.try_collect().await
}

pub async fn get_world_by_random_id(pool: &MySqlPool) -> ServiceResult<World> {
    let random_id = rand::get_random_u32(1, 10000);
    let res: World = sqlx::query_as("SELECT * FROM world WHERE id = ?")
        .bind(random_id)
        .fetch_one(pool)
        .await?;
    Ok(res)
}

Actually, the much easier solution and fastest solution would be changing your query to get all of the worlds at once:
use std::iter;
use itertools::Itertools;

pub async fn query_worlds_by_random_id(pool: &MySqlPool, queries: usize) -> ServiceResult<Vec<World>> {
    let random_ids: Vec<_> = iter::repeat_with(|| rand::get_random_u32(1, 10000)).take(queries).collect();
    let query = format!("SELECT * FROM world WHERE id IN ({}), iter::repeat("?").take(queries).format(", "));
    sqlx::query_as(&query).bind(&random_ids).fetch_all(pool).await.map_err(Into::into)
}

Actually, the much easier and fastest solution would be changing your query to get all of the worlds at once:
use std::iter;
use itertools::Itertools;

pub async fn query_worlds_by_random_id(pool: &MySqlPool, queries: usize) -> ServiceResult<Vec<World>> {
    let query_s = format!(
        "SELECT * FROM world WHERE id IN ({})",
        iter::repeat("?").take(queries).format(", "),
    );

    let mut query = sqlx::query_as(&query_s);
    for _ in 0..queries {
        query = query.bind(rand::get_random_u32(1, 10000));
    }

    query.fetch_all(pool).await.map_err(Into::into)
}

It's a good choice in actual use, but I'm not allow to use SQL like SELECT ... WHERE id IN (...) according to Framework benchmark requirements for multi-query 😢

This test is designed to exercise multiple queries with each resulting row selected individually. It is not acceptable to execute multiple SELECTs within a single complex query. It is not acceptable to retrieve all required rows using a SELECT ... WHERE id IN (...) clause. However, note General Requirement #7 which permits pipelining of network traffic between the application and database.

Oh, I didn't see this is about a web framework perf comparison.

I think I've got a better solution, the mistake I made actually was that I wait query to finishing in every loop in the iteration.
So the code ran in a serialized way, and using pool "directly" caused a lot of release and acquire things.
We should use Future Object to make the query run in a true async way:
use crate::custom_type::{pool::MySqlPool, result::ServiceResult};
use crate::models::benchmark::World;
use crate::utils::rand;
use futures::stream::{FuturesUnordered, TryStreamExt};

pub async fn query_worlds_by_random_id(pool: &MySqlPool, queries: usize) -> ServiceResult<Vec<World>> {
    let worlds = FuturesUnordered::new();
    for _ in 0..queries {
        worlds.push(get_world_by_random_id(&pool));
    }
    worlds.try_collect().await
}

pub async fn get_world_by_random_id(pool: &MySqlPool) -> ServiceResult<World> {
    let random_id = rand::get_random_u32(1, 10000);
    let res: World = sqlx::query_as("SELECT * FROM world WHERE id = ?")
        .bind(random_id)
        .fetch_one(pool)
        .await?;
    Ok(res)
}

I'm reopening the issue for new problem appears.
The above code met PoolTimedOut error in high concurring(512 connection, 1 request for each) situation.

There's been some reoccurring issues with pools, would be nice if you could test whether #1211 fixes your issue.

There's been some reoccurring issues with pools, would be nice if you could test whether #1211 fixes your issue.

Yeah, I have also seen this fix!
Confirmed, it works for me ❤


Binary interpreter API improvements

This changes the binary interpreter's API to be a bit easier to work with. Instead of supplying a module when reading an item from a ReadScope, the module is loaded up when the interpreter context is initially constructed. The ReadScope is then passed as an argument to Context::read_item.

Incidentally, that's a lot of changes!  Did your editor help you out, or did you have to go and change all of that stuff manually?

Yeah I lean a bunch on multiple cursors and find and replace and such. Can be a bit of a double edged sword though as it makes me accept a bit of verbosity early on. Really think we could have some way of making the binary tests easier with some sort of common setup stuff, but I haven't got around to it yet!

Yeah I lean a bunch on multiple cursors and find and replace and such. Can be a bit of a double edged sword though as it makes me accept a bit of verbosity early on. Really think we could have some way of making the binary tests easier with some sort of common setup stuff, but I haven't got around to it yet!

Yep interesting.  I just wanted to know whether i was missing out on some new updated LSP Rust-specific magic or anything.  I'm "glad" to hear that's how you do it :wink:

Refactoring support is getting loads better in recent updates to the rust-analyzer at least - I'm finding I can rely lots more on smart renaming etc. Haven't tried out the other stuff though.

Yeah I lean a bunch on multiple cursors and find and replace and such. Can be a bit of a double edged sword though as it makes me accept a bit of verbosity early on. Really think we could have some way of making the binary tests easier with some sort of common setup stuff, but I haven't got around to it yet!

Yep interesting. I just wanted to know whether i was missing out on some new updated LSP Rust-specific magic or anything. I'm "glad" to hear that's how you do it 😉

I try not be too annoying about about this but IntelliJ + Rust plugin (or CLion + Rust) have excellent refactoring tools. The recent renames I did in https://github.com/yeslogic/prince/pull/773 were a few seconds each with perfect accuracy (including comments, and code examples in doc comments).

IntelliJ + Rust plugin (or CLion + Rust)

Out of curiosity, these tools aren't Libre/FOSS, right?

CLion isn't but Intellj CE is:

IntelliJ CE (Apache 2)
IntelliJ Rust plugin (MIT)

Cool, great news!
On 2020-12-09 at 12:25 ACST+0930, quoth Wesley Moore
<EMAIL_ADDRESS>
CLion isn't but Intellj CE is:

IntelliJ CE
(Apache 2)
IntelliJ Rust
plugin (MIT)


[Video Support][Android] Video keeps playing even when out of sight.

Describe the bug
Once a video starts it never stops, even when you go into Full-Screen and pause it, once you quit the full-screen mode it will start playing from where it was paused.
To Reproduce
Steps to reproduce the behavior:

Let a video load and play by itself
Scroll up and down
It doesn't stop

OR

Let a video load and play by itself
Pause it from the full-screen mode
Quit from the full-screen
It doesn't stop

Expected behavior
Whenever a video starts playing, it should pause itself once it's out of the viewer's sight or remain paused if it was paused from the full-screen mode.
Note: it would be nice to pause the video without the need to enter full-screen mode. (Tap and Hold to enter full-screen mode and just a tap to pause/play the video)
Details

Platform: Android
Platform version: Android 10
App version: 1.91.0 (Latest)

I was just about to open an issue asking for video to stop playing while you compose a repost, but I think it's a case which falls under this.

I was just about to open an issue asking for video to stop playing while you compose a repost, but I think it's a case which falls under this.

Yeah, i totally forgot of this kind of event, it should pause the video whenever the video post is not focused (cases such a modal appearing, tapping the post to go to the comment section (unpauses it once the post page load again) and many others).

Same here, it keeps playing even after locking the screen.


Enable/disable pieces of the auto discovery & installation from the CLI, based on user selected checkboxes

The appropriate command options within the CLI need to be developed to support this User Experience. Specifically this includes the following scenarios:

Provide command line argument to install Infrastructure Agent
Provide a command line argument to execute the process discovery and send recommendations to the UI
Provide command line argument to install the recommenced OHIs. In order to perform the installation of the on-host integrations, the discovery step above will be a pre-requisite. In the UI, the user will not be able to select the installation stand-alone without also enabling the check box for discovery.
Provide a command line argument to install & configure logs. Note: This includes the auto-discovery of log files, there will be no separate option for doing only log discovery.

These arguments should be able to be used in combination with one another, so the UI can just append the necessary arguments based on what the user selects. Acceptance Criteria:
Scenarios mentioned above are tested to be working, and UI team confirms that it satisfies the necessary use case.
Aha! Link: https://newrelic.aha.io/features/VIRTUOSO-94

Comment added by Rishi Mukhopadhyay in Aha! - View
Populated description in this @Joel Worrall and @Sheila Krishnan.


Remove (move off) the fallback styling on the "map" directive

I mean that you want to yield the user a ready-to-use component, but in my case the actual size (height) is dynamically calculated and it isn't ready at the moment of the "map" creation (i.e. directive linking).
The result is that there's no (easy) way to set the desired "height" style (100%) in my case, because the calculated result is always "0px", then set to "300px" by your patch.
My suggestion is to remove, as the original maps are, and/or add a small CSS document with the default styles.
At the moment I'm using a modified version of your source, but I don't like this solution!

That's a good suggestion.
I do not usually style element in JS, however this directive was for a
beginner, it would be convenient for a user without .css.
How about this? Having an extra attribute not to write style for "map" tag.
e.g., no-style
Allen
On Wed, Jul 15, 2015 at 6:15 AM, Mario Vernari<EMAIL_ADDRESS>wrote:

I mean that you want to yield the user a ready-to-use component, but in my
case the actual size (height) is dynamically calculated and it isn't ready
at the moment of the "map" creation (i.e. directive linking).
The result is that there's no (easy) way to set the desired "height" style
(100%) in my case, because the calculated result is always "0px", then set
to "300px" by your patch.
My suggestion is to remove, as the original maps are, and/or add a small
CSS document with the default styles.
At the moment I'm using a modified version of your source, but I don't
like this solution!
—
Reply to this email directly or view it on GitHub
https://github.com/allenhwkim/angularjs-google-maps/issues/240.

Well, bear in mind I'm not an expert of the web dev.
The "no-style" could be a solution, but (question) would be feasible to inspect the "style" content attribute and look for the "height" prop? Just peeking at the presence: if present, the default styling should be skipped. I mean, whereas someone explicitly sets the style, probably do know what to do.
Anyway, again, feel free to choose what's better for your viewpoint.
In the meantime, I have other problems to solve (info-windows), but that's off this thread.
Mario

added an attribute default-style to override it. you can find example at testapp/map_no_default_style.html


partysocket: add subpath support

This patch lets you add path: string to new PartySocket({...}) and point to a subpath in a room. This is a client side analog to the recent subpath support we added to multiparty .fetch()/.socket()

Fix for the id thing in https://github.com/partykit/partykit/pull/503