ExecutorSSH init's signature was changed

Now to connect using sshkeys not login/passwd, .authenticate(pubkey, pushkey) must be called explicitly.
This obviously breaks anything that uses an ssh executor with sshkeys.
Super Urgent.

Should be fixed in core.


cubicsdr: fix for liquid-dsp v1.50

Pulling in a patch to fix CubicSDR to work with liquid-dsp v1.50, since there is not yet a release with this commit.
The issue I was experiencing is when zooming in to the waterfall spectrum view beyond a certain level, the spectrum view stops working and turns a solid color:
cubicsdr.webm
This may quite possibly also resolve the crash described in #219206, though I was unable to reproduce that myself.

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

@mattmelling Any interest in reviewing this? It's simple and has been out here for awhile. 😁

Was able to repro in the current version, and confirm that this patch fixes the zooming bug.
Result of nixpkgs-review pr 254649 run on x86_64-linux 1

1 package built:

cubicsdr

Thanks!


Is bolt-micronaut compatible with Micronaut 3.x?

I am trying bolt-micronaut, but I keep getting 400 (Bad Request) errors. I took a quick look at the source code - I don't see anything that should break from Micronaut 2.4 to 3.2, but maybe I am missing something.

Hi @volgin, thanks for writing in!
The bolt-micronaut package does not support Micronaut 3.x yet: https://github.com/slackapi/java-slack-sdk/blob/v1.14.0/bolt-micronaut/pom.xml#L13-L15
We'll update the module to be compatible with 3.x in the future, but it may take at least a few weeks (possibly a few months).
If you need an immediate solution for it, please consider modifying the module in your app. The module consists of only two source files and they are fairly simple ones: https://github.com/slackapi/java-slack-sdk/tree/v1.14.0/bolt-micronaut/src/main/java/com/slack/api/bolt/micronaut

Which is the best dependency to use then instead of bolt-micronaut?

Do you mean you're fine with not using Micronaut? If so, many developers tend to choose Spring Boot for Bolt app development. If you are fine with the framework, the only necessary optional dependency is bolt-servlet.
Refer to the following resources:

https://slack.dev/java-slack-sdk/guides/supported-web-frameworks#spring-boot
https://github.com/slackapi/java-slack-sdk/blob/v1.14.0/bolt-spring-boot-examples/pom.xml#L21-L30

I hope this helps!

@seratch I just copied these 2 files directly into my Micronaut project, and everything worked. The only change was replacing @Inject and @Singleton annotations with Jakarta. I am still working on the complete code, but it looks like migrating bolt-micronaut should take very little effort.
Cheers and thank you for great work and quick responses!

@volgin Thanks for the info! I just submitted a PR that resolves the issue here: https://github.com/slackapi/java-slack-sdk/pull/903


Code blocks are formatted correctly

Category

[x] Content fix

Related issues:

fixes https://github.com/SharePoint/sp-dev-docs/issues/4767

What's in this Pull Request?
All code blocks are formatted correctly
Guidance
All code blocks are formatted correctly

Nevermind on the review... this issue was assigned to me so I'll fix this.


Make ProtectionProfile methods public

Description
Make ProtectionProfile methods public. This allows to get values like key length and use them with key management protocols like MIKEY, instead of creating own constants for this.
Reference issue
Fixes #258

I have changed names as requested and added comments to make linter happy.
Please take a look on test pipeline, it failed with following error:
[2024-02-24T04:11:55.282Z] ['error'] There was an error running the uploader: Error uploading to https://codecov.io: Error: There was an error fetching the storage URL during POST: 404 - {'detail': ErrorDetail(string='Unable to locate build via Github Actions API. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}

@sirzooro Great!
Can you squash those commits into one.
I added you to the GitHub org so the tests should run (without our intervention). When it goes green we can merge :)

Done :)

Perfect! Thank you so much @sirzooro
Feel free to push a tag if you need one. git tag v3.0.1 && git push --tags is all the process you need :)


Offset grid setup somehow missing bottom right corner

Hi, I followed your tutorial from YouTube (https://www.youtube.com/watch?v=eUUDxg5MU3w) regarding dual grid setup.
I'm using latest release from AssetLib.
Here are my terrain pairing bits:

But when painting terrain, the bottom-right corner sometimes doesn't show up and I have no idea how this is possible:

I've tried replicating it with other terrain shapes and it's always the bottom-right corner that has this issue.

That's weird. Looks like it should work, and it definitely shouldn't be placing an empty cell. Can you upload an example project for this?

Now that's interesting, I cannot reproduce in a minimal setup project, I'll investigate further and report my findings here

Was due to changes from https://github.com/Portponky/better-terrain/issues/62, closing this but maybe it'll help someone with the same needs (I don't need cleanup anymore with dual tilemap setup tho)


Labeled events

Fixes #13

Might want to move syllables to their own interface, since they get added as acquisition rather than in the behavior processing module.  OTH, they are behavioral syllables.  @CodyCBakerPhD lmk what you think.

Might want to move syllables to their own interface

Up to you - I always personally prefer things to be as modular as possible
Can you share an example file with this events data type?

Up to you - I always personally prefer things to be as modular as possible
Yeah, I think it's probably the right call here.

Can you share an example file with this events data type?
Done!

Looks good overall
Only thing I wonder, is it possibly to cast the integer syllable_indices field as np.uint8 instead of np.uint64? Would save 8x space before compression

Might want to move syllables to their own interface, since they get added as acquisition rather than in the behavior processing module.

Done,

Only thing I wonder, is it possible (as in, does ndx-events allow it?) to cast the integer syllable_indices field as np.uint8 instead of np.uint64?

and done,

Can you share an example file with this events data type?

and resent!


Postgres: remove duplicates from New Relic plugin inventory

Hello,
I have completed setup of newrelic-plugin-agent for our Postgres servers without name parameter in .cfg file. Later, I added this parameter to YAML file and restarted plugin. Instead of displaying only one "changed" instance it shows 2 server monitoring stats with both old (IP name) and new actual server name. Could you please assist in resolving this?
postgresql:
          name: <ServerName>
          host: localhost
          port: 5432
          user: postgres
          dbname: postgres
          superuser: True

Thank you in advance,
Dmitry

Never mind, I was impatient and haven't waited for 15+ min.
This is resolved.


Update licensing files

Hi,
this patch adds the "Gradle License Report" plugin in order to generate the "NOTICE.md" file in the root directory.
Along the lines, it also cleans up the current versions of the "LICENSE" and "NOTICE" files.
With kind regards,
Andreas.

thanks for preparing this patch, @amotl
however, I don't approve of the changes made to NOTICE.txt. the purpose of that file is collect mandatory notices that you are required to include by third-party packages
the correct place for enumerating all third-party code along with their licensing information is in LICENSE.txt
ideally, NOTICE.txt is very small. and it usually should not contain all the third-party code used by the project. it should ONLY contain third-party code with an instruction to reproduce a specific notice (not a copyright line or the name of the license)
all Apache projects have a NOTICE file, and usually there is at least one notice that you must reproduce in your own top level NOTICE file:
This product includes software developed by The Apache Software Foundation
(http://www.apache.org/).

there is no need to reproduce this sentence multiple times for every project that includes it. once is enough
some projects may have more notices that we have to reproduce. for example, notices about the export of cryptography, and so on

for the license file, I am not sure the format you have chosen works
for each piece of bundled third-party software, we must:

reproduce the copyright information in full (unmodified)
reproduce the license in full (unmodified)

it's usually a good idea to separate each one of these sections with horizontal rules (as can be found in the existing file, with double new lines above)
in the case of multiple components using the same license, it is fine to refer to a previous license in the file as long as the text is identical (be careful with BSD licenses, which often contain differences)

follow-up:
LICENSE.txt and NOTICE.txt are legal documents. I don't think it's a good idea to generate them with a script. perhaps a better approach would be to write a linter. i.e., a linter that enumerates all the third-party software found and for each piece of software, performs heuristics to see if it can find the necessary information in both files. then, leave it up to a human to satisfy those checks

just caught this:
the changeset introduces a NOTICE.md alongside the NOTICE file. we should only have one file, and it should be named NOTICE

something else comes to mind:
the only software we need to consider when updating LICENSE and NOTICE is software that is actually bundled in our release artifact (tarball)
I am not experienced with Gradle or the plugin you are using, but the thought occurred to me that maybe some of these items are not included in the release artifact (i.e., fetched at install time, at run-time, or at some other point after the user unpacks the tarball)
you can see this is mentioned in the current notice file:
Third-party components bundled with the CrateDB source code are indicated in
the LICENSE file along with their respective licenses.  Any additional
mandatory notices pertaining to those dependencies will be reproduced in this
file.

Crate provides pre-built releases for convenience, also known as convenience
releases.  Building CrateDB from source pulls in a number of additional
third-party dependencies.  Hence, convenience releases will contain a mix of
CrateDB source code and third-party build-time dependencies not listed in the
LICENSE file or in this file.

Crate is committed to only using permissively licensed third-party code.
However, for the time being, if you make use of convenience releases, it is
your responsibility to check the licensing status of the bundled third-party
build-time dependencies.

Isn't this superseded by https://github.com/crate/crate/pull/11334?

Isn't this superseded by #11334?

#11334 takes care about license headers in all source files. This one (#11220) adds a Gradle plugin to automatically generate license information for all runtime dependency packages.

Hi Naomi,
thanks for your feedback. The intention of this patch was primarily to add a mechanism to automatically generate a list of dependencies from the current state of the software configuration defined by gradle/version.properties.
I just amended some details of this, postponed some changes to the main license files to #11605 in order to separate this into another iteration and also renamed the automatically generated file to 3RD-PARTY-NOTICES.md.
With kind regards,
Andreas.

Hi again,
after the de-duplication issue has been fixed now, I would consider this patch ready to go. Can I humbly ask you for another review, @mfussenegger and @seut?
With kind regards,
Andreas.

@mfussenegger: I explicitly excluded JMH through ef6d99c5e3ff. Let me know if you spot other anomalies where modules should not be included into this report.


Tensorflow-probability version

Hi Danijar,
The setup instructions do not contain a pointer to the right tensorflow-probability version. It's probably not the case that bugs are induced by new versions but just in case, do you know the version you used?
Thanks!

I ran into the same issues and was able to get things running with tensorflow-probability==0.8.0
Here is a full list from pip in my conda env:
Package                Version

absl-py                0.11.0
astor                  0.8.1
atari-py               0.2.6
cached-property        1.5.2
cachetools             4.1.1
certifi                2020.6.20
chardet                3.0.4
cloudpickle            1.1.1
cycler                 0.10.0
decorator              4.4.2
dm-control             0.0.322773188
dm-env                 1.3
dm-tree                0.1.5
ffmpeg                 1.4
future                 0.18.2
gast                   0.2.2
glfw                   2.0.0
google-auth            1.23.0
google-auth-oauthlib   0.4.2
google-pasta           0.2.0
grpcio                 1.33.2
gym                    0.17.3
h5py                   3.1.0
idna                   2.10
importlib-metadata     2.0.0
Keras-Applications     1.0.8
Keras-Preprocessing    1.1.2
kiwisolver             1.3.1
labmaze                1.0.3
lxml                   4.6.1
Markdown               3.3.3
matplotlib             3.3.3
numpy                  1.19.4
oauthlib               3.1.0
opencv-python          <IP_ADDRESS>
opt-einsum             3.3.0
pandas                 1.1.4
Pillow                 8.0.1
pip                    20.2.4
protobuf               3.14.0
pyasn1                 0.4.8
pyasn1-modules         0.2.8
pyglet                 1.5.0
PyOpenGL               3.1.5
pyparsing              2.4.7
python-dateutil        2.8.1
pytz                   2020.4
requests               2.25.0
requests-oauthlib      1.3.0
rsa                    4.6
scipy                  1.4.1
setuptools             50.3.1.post20201107
six                    1.15.0
tensorboard            2.1.1
tensorflow-estimator   2.1.0
tensorflow-gpu         2.1.0
tensorflow-probability 0.8.0
termcolor              1.1.0
tqdm                   4.52.0
urllib3                1.26.2
Werkzeug               1.0.1
wheel                  0.35.1
wrapt                  1.12.1
zipp                   3.4.0

Thanks! It also works with 0.9.0 so I just worry there might be bugs in between tf-probability versions since silent breaking changes between versions happens ALL THE TIME with tensorflow


Missing FMA instrinsics

Current Ruffle master (https://github.com/ruffle-rs/ruffle/commit/fa05c66e2a7d08c695628705c5beb550030efbc8) shows the following with cargo-clif test -p tests:
warning: unsupported x86 llvm intrinsic llvm.x86.fma.vfnmadd.pd; replacing with trap

warning: unsupported x86 llvm intrinsic llvm.x86.fma.vfnmadd.ps; replacing with trap

warning: unsupported x86 llvm intrinsic llvm.x86.fma.vfmaddsub.pd; replacing with trap

warning: unsupported x86 llvm intrinsic llvm.x86.fma.vfmaddsub.ps; replacing with trap

warning: unsupported x86 llvm intrinsic llvm.x86.fma.vfmsubadd.pd; replacing with trap

warning: unsupported x86 llvm intrinsic llvm.x86.fma.vfmsubadd.ps; replacing with trap

warning: unsupported x86 llvm intrinsic llvm.x86.fma.vfnmadd.pd.256; replacing with trap

warning: unsupported x86 llvm intrinsic llvm.x86.fma.vfnmadd.ps.256; replacing with trap

warning: unsupported x86 llvm intrinsic llvm.x86.fma.vfmaddsub.pd.256; replacing with trap

warning: unsupported x86 llvm intrinsic llvm.x86.fma.vfmaddsub.ps.256; replacing with trap

warning: unsupported x86 llvm intrinsic llvm.x86.fma.vfmsubadd.pd.256; replacing with trap

warning: unsupported x86 llvm intrinsic llvm.x86.fma.vfmsubadd.ps.256; replacing with trap

warning: unsupported x86 llvm intrinsic llvm.x86.avx2.gather.d.pd; replacing with trap

warning: unsupported x86 llvm intrinsic llvm.x86.avx2.gather.q.pd; replacing with trap

warning: unsupported x86 llvm intrinsic llvm.x86.avx2.gather.d.pd.256; replacing with trap

warning: unsupported x86 llvm intrinsic llvm.x86.avx2.gather.q.pd.256; replacing with trap

warning: unsupported x86 llvm intrinsic llvm.x86.avx.vzeroupper; replacing with trap

It throws a lot of
[ERROR swf::read] Error decompressing SWF: corrupt deflate stream
[WARN  swf::read] SWF length doesn't match header, may be corrupt

but doesn't actually crash at any point on an unimplemented intrinsic on my laptop despite it having AVX2 and FMA support.

I've tracked down the 'corrupt deflate stream' error to a miscompilation of simd-adler32: https://github.com/rust-lang/rustc_codegen_cranelift/issues/1426

Thanks for tracking this down! I will fix it once I have time.

The miscompilation has been fixed. It now crashes on _mm256_fmaddsub_ps in rustfft.

rustfft now works on the even_more_simd_intrinsics branch. Next issue:
thread '<unnamed>' panicked at tests/framework/src/options.rs:101:38:
assert_relative_eq!(actual, expected, epsilon = epsilon)

    left  = 286.65
    right = 358.25

The failing test is avm2/edittext_tab_stops. I found several miscompilations in the stdarch test suite. Hopefully one of them is the cause of this test failure.

@bjorn3 The avm2/edittext_tab_stops panic is legitimate (it occurs under LLVM). However, the ruffle test suite normally catches that panic.

I see. cg_clif currently only supports panic=abort, so test failures can't be caught. For regular libtest tests, -Zpanic-abort-tests is enabled by default which puts every test in a separate process to make panics abort just a single test process rather than the test runner, but ruffle uses libtest-mimic which doesn't support this feature.

Is from_avmplus/as3/Types/Int/wraparound also expected to fail?

thread '<unnamed>' panicked at core/src/avm2/activation.rs:2297:25:
attempt to add with overflow
stack backtrace:
   0: rust_begin_unwind
             at /rustc/6b771f6b5a6c8b03b6322a9c77ac77cb346148f0/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/6b771f6b5a6c8b03b6322a9c77ac77cb346148f0/library/core/src/panicking.rs:72:14
   2: core::panicking::panic
             at /rustc/6b771f6b5a6c8b03b6322a9c77ac77cb346148f0/library/core/src/panicking.rs:127:5
   3: ruffle_core::avm2::activation::Activation::op_increment_i
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2/activation.rs:2297:25
   4: ruffle_core::avm2::activation::Activation::do_next_opcode
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2/activation.rs:1043:35
   5: ruffle_core::avm2::activation::Activation::run_actions
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2/activation.rs:861:26
   6: ruffle_core::avm2::Avm2::run_script_initializer
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2.rs:297:25
   7: ruffle_core::avm2::script::Script::globals
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2/script.rs:569:13
   8: ruffle_core::avm2::domain::Domain::get_defined_value
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2/domain.rs:229:23
   9: ruffle_core::display_object::movie_clip::MovieClip::symbol_class
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/display_object/movie_clip.rs:877:32
  10: ruffle_core::display_object::movie_clip::MovieClip::handle_bytecode_tag
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/display_object/movie_clip.rs:4210:41
  11: ruffle_core::display_object::movie_clip::MovieClip::run_frame_internal::{{closure}}
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/display_object/movie_clip.rs:1559:21
  12: ruffle_core::tag_utils::decode_tags
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/tag_utils.rs:464:26
  13: ruffle_core::display_object::movie_clip::MovieClip::run_frame_internal
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/display_object/movie_clip.rs:1567:17
  14: <ruffle_core::display_object::movie_clip::MovieClip as ruffle_core::display_object::TDisplayObject>::enter_frame
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/display_object/movie_clip.rs:2599:17
  15: <ruffle_core::display_object::DisplayObject as ruffle_core::display_object::TDisplayObject>::enter_frame
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/display_object.rs:1050:1
  16: <ruffle_core::display_object::stage::Stage as ruffle_core::display_object::TDisplayObject>::enter_frame
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/display_object/stage.rs:836:13
  17: ruffle_core::frame_lifecycle::run_all_phases_avm2
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/frame_lifecycle.rs:79:5
  18: ruffle_core::player::Player::run_frame::{{closure}}
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/player.rs:1661:17
  19: ruffle_core::player::Player::update::{{closure}}
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/player.rs:2022:24
  20: ruffle_core::player::Player::mutate_with_update_context::{{closure}}
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/player.rs:1966:23
  21: gc_arena::arena::Arena<R>::mutate
             at /home/bjorn/.cargo/git/checkouts/gc-arena-f613d25074b8ab45/efd89fc/src/gc-arena/src/arena.rs:155:13
  22: ruffle_core::player::Player::mutate_with_update_context
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/player.rs:1882:9
  23: ruffle_core::player::Player::update
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/player.rs:2021:20
  24: ruffle_core::player::Player::run_frame
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/player.rs:1659:9
  25: ruffle_core::player::Player::tick
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/player.rs:525:17
  26: ruffle_test_framework::runner::run_swf
             at ./framework/src/runner.rs:127:13
  27: ruffle_test_framework::test::Test::run
             at ./framework/src/test.rs:61:26

And from_avmplus/ecma3/TypeConversion/e9_3_1_3_rt?

thread 'main' panicked at core/src/avm2/value.rs:394:17:
attempt to multiply with overflow
stack backtrace:
   0: rust_begin_unwind
             at /rustc/6b771f6b5a6c8b03b6322a9c77ac77cb346148f0/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/6b771f6b5a6c8b03b6322a9c77ac77cb346148f0/library/core/src/panicking.rs:72:14
   2: core::panicking::panic
             at /rustc/6b771f6b5a6c8b03b6322a9c77ac77cb346148f0/library/core/src/panicking.rs:127:5
   3: ruffle_core::avm2::value::string_to_f64
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2/value.rs:394:17
   4: ruffle_core::avm2::value::Value::coerce_to_number
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2/value.rs:742:17
   5: ruffle_core::avm2::activation::Activation::op_negate
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2/activation.rs:2339:22
   6: ruffle_core::avm2::activation::Activation::do_next_opcode
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2/activation.rs:1048:31
   7: ruffle_core::avm2::activation::Activation::run_actions
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2/activation.rs:861:26
   8: ruffle_core::avm2::function::Executable::exec
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2/function.rs:194:17
   9: <ruffle_core::avm2::object::function_object::FunctionObject as ruffle_core::avm2::object::TObject>::call
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2/object/function_object.rs:191:9
  10: <ruffle_core::avm2::object::Object as ruffle_core::avm2::object::TObject>::call
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2/object.rs:149:1
  11: ruffle_core::avm2::object::TObject::call_property
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2/object.rs:486:17
  12: <ruffle_core::avm2::object::Object as ruffle_core::avm2::object::TObject>::call_property
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2/object.rs:149:1
  13: ruffle_core::avm2::activation::Activation::op_call_property
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2/activation.rs:1326:21
  14: ruffle_core::avm2::activation::Activation::do_next_opcode
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2/activation.rs:967:21
  15: ruffle_core::avm2::activation::Activation::run_actions
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2/activation.rs:861:26
  16: ruffle_core::avm2::Avm2::run_script_initializer
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2.rs:297:25
  17: ruffle_core::avm2::script::Script::globals
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2/script.rs:569:13
  18: ruffle_core::avm2::domain::Domain::get_defined_value
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/avm2/domain.rs:229:23
  19: ruffle_core::display_object::movie_clip::MovieClip::symbol_class
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/display_object/movie_clip.rs:877:32
  20: ruffle_core::display_object::movie_clip::MovieClip::handle_bytecode_tag
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/display_object/movie_clip.rs:4210:41
  21: ruffle_core::display_object::movie_clip::MovieClip::run_frame_internal::{{closure}}
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/display_object/movie_clip.rs:1559:21
  22: ruffle_core::tag_utils::decode_tags
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/tag_utils.rs:464:26
  23: ruffle_core::display_object::movie_clip::MovieClip::run_frame_internal
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/display_object/movie_clip.rs:1567:17
  24: <ruffle_core::display_object::movie_clip::MovieClip as ruffle_core::display_object::TDisplayObject>::enter_frame
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/display_object/movie_clip.rs:2599:17
  25: <ruffle_core::display_object::DisplayObject as ruffle_core::display_object::TDisplayObject>::enter_frame
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/display_object.rs:1050:1
  26: <ruffle_core::display_object::stage::Stage as ruffle_core::display_object::TDisplayObject>::enter_frame
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/display_object/stage.rs:836:13
  27: ruffle_core::frame_lifecycle::run_all_phases_avm2
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/frame_lifecycle.rs:79:5
  28: ruffle_core::player::Player::run_frame::{{closure}}
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/player.rs:1661:17
  29: ruffle_core::player::Player::update::{{closure}}
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/player.rs:2022:24
  30: ruffle_core::player::Player::mutate_with_update_context::{{closure}}
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/player.rs:1966:23
  31: gc_arena::arena::Arena<R>::mutate
             at /home/bjorn/.cargo/git/checkouts/gc-arena-f613d25074b8ab45/efd89fc/src/gc-arena/src/arena.rs:155:13
  32: ruffle_core::player::Player::mutate_with_update_context
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/player.rs:1882:9
  33: ruffle_core::player::Player::update
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/player.rs:2021:20
  34: ruffle_core::player::Player::run_frame
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/player.rs:1659:9
  35: ruffle_core::player::Player::tick
             at /home/bjorn/Projects/cg_clif2/ruffle/core/src/player.rs:525:17
  36: ruffle_test_framework::runner::run_swf
             at ./framework/src/runner.rs:127:13
  37: ruffle_test_framework::test::Test::run
             at ./framework/src/test.rs:61:26

Looks like it. Seems like ruffle fully works as expected then.


[release-1.6] Fix tombstone recovery bug in Galley

This is an automated cherry-pick of #25358

@bison Can you please accept the CLA on this? It needs an @googlebot I consent


Shipping Security Updates

Motivations
The way of handling security updates today involves a lot of humans efforts, going from reading the mailing list, to adding lines to the configuration files.
More over, this implies having control over the top-level derivation, which is not always possible with nix-env, and nix-shell.  This has to be patched inside the current pipeline, as what got made for NixOS.
Also, as the current method for applying security updates is complex, we are only restricting them to a few packages, such as glibc, bash and openssl which are used a lot, but we do not such mechanism for all other packages which have minor security / usability updates such as firefox.
As a work-around, today, we offer the option of having a small channel, to get updates from, but this is not a good solution for end-users, as the small channel basically involves recompiling any large project.
More over the current mechanism cannot safely be composed as explained in #4336.
Goals
This work aims at making it transparent, such as security updates are cannot be distinguished from any other updates by the end user.
This work aims at making it easy, such that packagers only have to cherry-pick a commit made on master into a security/master branch to let the buildfarm compile and distribute the package against the latest stable branch.
This work aims at making it fast, such that the buildfarm in charge of producing pre-compiled versions of the packages will only recompile the packages which are being fixed, against the latest stable branch.  When a user updates, Nix will download from the latest stable channel, with the addition of the recompiled packages, and Nix will patch all the packages depending on the fixed packages.
This new approach is sane, as it constructs the patches which have to be apply, without any knowledge of the runtime dependency.  Thus, all the packages patches can be build up-front without having to realize any of the derivations.
Approach
Nixpkgs as a pure function
This approach aims at constructing patches based on the Nix expressions of the packages. Thus we have to change the way we evaluate Nixpkgs to reflect the patching mechanism.
This approach is based on the recent modification of #14000 (#9400), which make Nixpkgs behave as a single function with no external inputs.  As the Nix language is pure, the following property holds for Nixpkgs.
    fix f == f (fix f)

This property means that we can call f one extra time on the result of the fix-point without changing the result of the fix-point.  As a consequence, all the packages will keep the same hashes.
Recompile only fixed packages
This patching mechanism relies on one hypothesis.  This hypothesis is that all packages derivations are expressed in the last evaluation of Nixpkgs, and that dependencies are taken from the previous evaluation of Nixpkgs.
This hypothesis does not hold, as of today, but with the help of a static analysis, we can identify how many times we jump through the Nixpkgs argument before we settle on a derivation, and before we settle on its inputs.
Once this hypothesis is guaranteed, we can change the above expression to
    g (fix f)

Where g is a slightly different version of f.  With the above hypothesis, this means that all derivations of packages are taken from g, but the dependencies of g are taken from f.
If f is the stable version of Nixpkgs, and g is a stable version of Nixpkgs with extra patches.  This implies we will recompile any patched packages of g, while keeping the same hashes as f for non-patched packages.
At this stage, we have what we expect to let Hydra compile.  Still, this stage does not apply any patches to any of the packages.
Building patches
To be able to apply patches to the dependencies, we need the equivalent of a fix-point, to evaluate the dependencies, and the dependencies of the dependencies, and so on … In addition to a fix-point, we want to iterate from the maybe-recompiled packages of g (fix f), and iterate over g to settle on the patched version of the packages.  We would name this patched version of packages h.
To construct patches, we need to know the dependencies of the packages, and which one got updated.
To list the dependencies of a package, we have to rely on some form of introspection, encoded as part of the derivations.  As a first approach I relied on callPackage, which would give us a super set of the dependencies, but this has a lot of issues, and left-over packages.  As a second approach, I now rely on the buildInputs which are exposed by the derivation, but this has the issue of being a sub-set of the dependencies.  Still, I am confident that we can instrument the default builder to ensure that buildInputs are a super set of the runtime dependencies.
To check for updates, we can compare the outPath of 2 derivations.  If the derivations have a different outPath, this means that the derivations has a different inputs.  (Note: hash collisions are unlikely, but if this happens we can easily add a dummy environment variable to get a different hash)
Patches are composed of the hashes of the old versions, and the hashes of the new versions.  Making a patch for one derivation, is being able to list the old hashes and associate them with the new hashes.  The hashes are the hashes of the dependencies.
Based on the last hypothesis, looking for the dependencies of g (fix f), will gives us hashes of derivations from fix f.  These are our old hashes.
Looking for the new hashes, implies that we have to build packages based on patched versions of the packages, i-e h.  Thus if we look for the dependencies of g h, we would have our new hashes.
Therefore h definition should look like:
  h = patch (g (fix f)) (g h);

Where patch is a function which on a derivation, will take the value of g (fix f), and apply a patch for any of its dependencies which got updated, by substituting the dependencies from fix f by dependencies from h.
In practice, we also need to feed fix f to this patch function, as we have to make sure that patches can be applied safely, by ensuring that the outPath of the derivation still have the same length.
Status

[x] Implement this approach on top of Nixpkgs
[x] Add test case to ensure that, when the one-application hypothesis holds, we can only recompile fixed packages, and apply patches on the rest of the packages.
[x] Make a static analysis which reports all violations of the one-application hypothesis.
[x] nix-env -f ./. -qaP --drv-path works when quickfixPackages is equal to defaultPackages, and does not cause recompilation of existing packages. (Note: nox-review fails to see that we have no recompilation because of violation of the one-application hypothesis)
[x] Add a test case to verify that nix-env -f ./. -qaP --drv-path always works, when quickfixPackages is equal to defaultPackages.
[x] Hook the test cases as release blockers of Nixpkgs.

This mechanism is disabled by default, as no quickfixPackages are provided yet.  Also, when disabled, this work has no impact.
To avoid to bloat and delay these changes, I would recommend that we merge them as soon as possible, even if the one-application hypothesis does not hold yet, and address any issues reported by the static analysis as follow-up issues.
References
NixCon slides: http://nbp.github.io/slides/NixCon/2015.ShippingSecurityUpdates/
NixCon video: https://www.youtube.com/watch?v=RhcKXS00zEE

Thank you! This is much needed. I'm excited to see how this will turn out :)

@Mathnerd314 :

There's some existing work in https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/replace-dependency.nix if you haven't seen it.

I saw it, and this is exactly why I am not using it, and why I am making this work.  I will explain more next Sunday in my talk.
@Mathnerd314 :

I guess this is a hack until the promised "Optimize this function by only using runtime dependencies of the original package set" is implemented?

No, in fact, this is exactly the opposite.  The hack is to rely on runtime dependencies, as runtime dependencies are not known statically (without building packages).  The current implementation is a conservative approximation of runtime dependencies.  There is still one issue with the current function, as it does not consider the closure of dependencies, but only explicit dependencies.

This extra patches are changing the design to fit better with the ideal model shown in the slide of my presentation. [1][2]
So far, while evaluating packages with nix-env -f ./. -qaP --drv-path I saw multiple issues.

Aliases are shown in the list of packages, while they are not listed without the quickfix stage applied.
So far, the code is unable to retrieve the list of dependencies of 3190 packages, which means that for these packages we would not patch them if one of their dependencies got modified.

[1] http://nbp.github.io/slides/NixCon/2015.ShippingSecurityUpdates/
[2] https://www.youtube.com/watch?v=RhcKXS00zEE&list=PL_IxoDz1Nq2Y7mIxMZ28mVtjRbbnlVdmy&index=13

It won't render all-packages.nix for me to do a line comment, but the documentation the of self (the parameter) says self is affected by user overrides, but since it is fixed within pkgFun I don't think it is.

lib is let-bound in all-packages.nix, but since it is also inherited in the body of the function this could be made more concise by doing just lib  = import ../lib;. Since with pkgs; comes after the let-binding of lib, either way were a user to override lib that would take priority.

@nbp I've only glanced at your slides, so can't yet speak to that. But just in terms of refactoring things to make sense (rather than policy), this is a HUGE improvement. Thanks so much!

Oh and @Mathnerd314 got a good portion of my comments too, hah, but they didn't show up on the diff.

Aliases are shown in the list of packages, while they are not listed without the quickfix stage applied.

This is because nix-env uses pointer equality for skipping duplicate derivations, rather than something sane like output path or derivation path: https://github.com/NixOS/nix/blob/master/src/libexpr/get-drvs.cc#L193.

I rebased the patches; still testing it.

from @Ericson2314

If we remember the hashes of things we are going to rebuild, and we find something we wouldn't rebuild that has the same hash as something we are rebuilding, we could "rebuild" it anyways. That will work for simply rebound aliases, but not for alises that involve overriding (an example of introduction >= elimination, but introduction > 0).

I intended to do something similar to ensure that we do not add such aliases to nixpkgs.
I do not think this is a good choice to do this automatically as this effectively means that for installing any sotfware you will have to evaluate all of them.
from @Mathnerd314

Aliases are shown in the list of packages, while they are not listed without the quickfix stage applied.

This is because nix-env uses pointer equality for skipping duplicate derivations, rather than output path or derivation path: https://github.com/NixOS/nix/blob/master/src/libexpr/get-drvs.cc#L193. Personally, I would remove that check; outputting all possible aliases is consistent with the behavior of command-not-found.

That's what I thought at first, but then I realized that these differences are related to aliased packages, and that for each newly reported entry, we have a potential security issue (as described in https://github.com/NixOS/nixpkgs/pull/10851#discussion_r45405883)
So even if this behaviour of nix-env is not consistent, we should still fix these issues.

@nbp Sorry for taking a while to respond, but I think that all makes sense now. Thanks for explaining it.
I wonder if we should revive #10307 after all as this a diff this big goes stale so quick.

Also https://github.com/Ericson2314/nixpkgs/commit/03486c8c6559b25510d122099b5ad315c647e814 here is a commit just with the improved documentation for overridePackages you might want to add to this.

I got codetriaged here. What's the status of this?

I'm not sure about the status of the main security part, but the improved fix-point part has been good to go for a while, hense my advocacy to split this in two.

Wait, so if the improved fix-point is already finished, what do we have to do to make it work and patch security updates? I'd really like to help on this, but I have no idea how.

To be clear, the fix-point I mean just a saner way to do overrides with self and super like the Haskell package sets. The "hotfxing part" that replaces a dependency without causing rebuilds downstream uses that but is separate.

The improved fixpoint is #9400, I have rebased and updated it so it should be ready to merge. After that, I would look at reopening #10307, it is the kind of change that bitrots very quickly.

@nbp @Mathnerd314 #9400 is finally merged if you want to give another stab at this. It surely would have been useful to avoid re-compiling the world after the glibc CVE.

Bumping milestone.

IIUC a rebased/fixed #10307 shouldn't change any hashes either. Should we try to squeeze it in before 16.03 to make backports easier?

Eh, so waiting to resolve #9400 (again).

Sorry for not answering for a long while, I got discouraged by a contributor claiming that they were not going to improve their patches because this work is not going to happen any time soon.
Thus, I changed my approach to make sure we can blame every package which is adding additional security issues.
I continued the branch that I started and from where it was based (back in november).  I stepped down compared to my initial attempt, which was to fix the world.  Now I am aiming at making sure we can have a working infrastructure, and incrementally improve the state of security.
So far I managed to make 2 important steps forward:

Generate a list of insecure packages, which can be checked in Nix.
Add a test case which verifies that security fixes are working as expected.

The test case is working as expected, and caught a minor flaw in my current implementation, which should not be too hard to fix.
I will update the current pull request over the week-end.

I have updated the first comment, to better describe the motivations, goals, and approach taken in this pull-requets.
I will make a test case to ensure that we have do not introduce issues similar to the chicken package.

Hmm, I wonder if the security workflow could be tweaked so that self.chicken vs. chicken doesn't matter. It's a bit non-intuitive, as I think the set should feel as "recursive attrset that can have parts overrided". And IMO for normal evaluation it is the same as we have a fixed point (tested with chicken at least).

To list the dependencies of a package, we have to rely on some form of introspection, encoded as part of the derivations. ... I now rely on the buildInputs which are exposed by the derivation, but this has the issue of being a sub-set of the [runtime] dependencies.

It is still not clear to me why you approximate the runtime dependencies, instead of reading them from the Nix store. This is for hotfixes, so it is safe to assume that the old (vulnerable) derivation is already present; nix-store -q --references is just a database query so it is quite quick.

@Mathnerd314: in some cases you wouldn't know for sure if a package depends on something vulnerable until you finished the build and scanned the references – that's not always so quick.

@Mathnerd314 because relying on the nix-store implies that you have to build the packages first.  Thus you nix expression is as deterministic as your build, and the result of your nix expression depends on the result of your builds.
By computing the set of dependencies based on the information contained in the nix expression only, I can predict the outPath of all the patched packages ahead of the build.
This is a sane approach, because you do not need to build all the packages to know their hashes, and it is predictable.

But the packages have to be built regardless, otherwise you have nothing to run the hash-rewrite script on.

True, I see, the "not always so quick" point is invalid. It will still be practical if evaluation time is independent of running the builds.

@Mathnerd314
Reading the dependencies from the nix-store should at best be a validation phase, not something which is part of the nix expression.
If reading from the nix-store is part of the building process, then you are not able to predict which output path you will have ahead of time.  Which means that we no longer have one phase to register the derivations, and one phase the realize the derivations.
This branch does that statically, thus we can predict the outPath ahead of the build, and even for the patched packages.
Now image the following scenario:

We have 2 computers, A and B.
Computer A compute the hashes, build and patch all the packages.
Computer A runs a garbage collector, and remove all the old packages.
Computer B compute the hashes, ...

build the packages, and patch all packages.
download the patched packages from A.

If you are unable to predict the outPath ahead of time, you cannot request other computers about the packages that you have already build.  On the other hand, if you are able to compute the outPath ahead of time, then you can just pull all the patched packages from another computer.

I'm a bit confused on this too. From 30,000 feet, I'd think we only need to swap out run-time dependencies on packages that are already built prior to the security update. For packages that have never been built, might as well replace all run-time and build-time deps.
My mental algorithm is: Keep the vanilla and secure fix-points around. When building a secure fix-point, check if it's vanilla versions is built, and if so swap out the known runtime deps and use that as the official secure build. Otherwise just build normally (aka this algo is used for building of deps, so transitive builds may be ninja'd as appropriate).

@Ericson2314

I'd think we only need to swap out run-time dependencies on packages that are already built prior to the security update.

How would you know that packages are already built, without relying on the content of the nix-store?
How would you guarantee to have the same hashes if you are not building the same things?
Note, the patching mechanism is a not used on derivation where none of the dependency changed.

Keep the vanilla and secure fix-points around. When building a secure fix-point, check if it's vanilla versions is built, and if so swap out the known runtime deps and use that as the official secure build.

How would you check that if you intent to deploy this new configuration on another computer?  Would you have to specify the remote, such that we can inspect its nix store?

@Ericson2314

When building a secure fix-point, check if it's vanilla versions is built

Note if you are always expecting to recompile from scratch, then there is no point of making security updates this way.  But, one of the force of the Nix package manager is the fact that you can download pre-compiled binary packages by computing the hashes of the outPath.  Thus, as the vanilla version is supposed to already be in the binary cache.  A large number of the packages from the vanilla version are effectively built by the buildfarm.

How would you know that packages are already built, without relying on the content of the nix-store?

By "secure fix-point package" [n.b. I accidentally left out the word package in the comment], I meant formally a function from "Any -> derivation" which is applied to each of the vanilla and secure package sets.
That said, that's not perfect and I am starting to see what you mean :).

Note, the patching mechanism is a not used on derivation where none of the dependency changed.

Sure I guess I left out the base case in my algorithm, but that's what I meant. It just occurred to me this could work by overriding mkDerivation (or plain derivation) like cross compiling and it's multiple derivations.

How would you check that if you intent to deploy this new configuration on another computer? Would you have to specify the remote, such that we can inspect its nix store?

But, one of the force of the Nix package manager is the fact that you can download pre-compiled binary packages by computing the hashes of the outPath.

I don't understand what your asking---I don't see how local vs remote is implicated in what I am proposing? Under my plan, binary substitution is completely transparent as it is today: when I say "check for build" I mean locally and from remote binary caches.

@Ericson2314

Under my plan, binary substitution is completely transparent as it is today: when I say "check for build" I mean locally and from remote binary caches.

How would you check for downloading the patched packages under the binary caches if you need the result of the non-patched builds to figure out what has to be patched, and thus the hash of the patched binaries?
By only relying on the Nix expression, this branch is capable of predicting the outPath of the patched packages.  Which means that we can query other computers for the patched packages, instead of query the non-patched packages and fix them locally on every computer.

How would you check for downloading the patched packages under the binary caches if you need the result of the non-patched builds to figure out what has to be patched, and thus the hash of the patched binaries?

Well, technically one could query the binary cache for references relatively cheaply, as *.narinfo does contain that information. (See e.g. curl http://cache.nixos.org/la5imi1602jxhpds9675n2n2d0683lbq.narinfo.) But we would probably have to change the way nix evaluation works; it seems cleaner if we stay within the nix language, as depending on anything remote during evaluation is problematic.

Note, the source of the dependencies is one tiny part of the algorithm.  Using callPackage, buildInputs, or pre-computed runtime dependencies (*), is irrelevant to how this patching mechanism works, and we can be debated about it later.
The important point here, is that this patching mechanism does not involve querying anything out-side the Nix language.  Which makes the generation of the compilation rules deterministic, and this determinism is reflected by a stable hash for the outPath of the patched packages.
Another important point, is that this approach provides a nice way to handle security updates, as simple as maintaining another branch of Nixpkgs, which we already do for releases.  Thus, this approach would be a tiny burden for package maintainers, while being transparent for all users.
(*) As I mentioned during my talk, we can optimize this mechanism to let hydra help us filter the list of build time dependencies, by using pre-computed runtime dependencies, as part of the security channel that we would distribute.

@vcunat

Hmm, I wonder if the security workflow could be tweaked so that self.chicken vs. chicken doesn't matter. It's a bit non-intuitive, as I think the set should feel as "recursive attrset that can have parts overrided". And IMO for normal evaluation it is the same as we have a fixed point (tested with chicken at least).

The reason why it fails is because the patch function is a zipAttrs, which will create the result for each attribute set once it has been computed.
So, the laziness issue is located at the resolution of the .override name.  We could work around this issue, if we knew what names were to be expected ahead of time, by doing something like:
  { inherit (merge) outPath drvPath override; }

Currently, the patching function use the merge operator, and as I complained many times before, this merge operator is not lazy enough, as a // b has to resolves the names of a, even if they are overwritten by b.
In the case of the chicken package.  Not using self.chicken here would be an aliased-package issue, as the bootstrap version would be taken from the non-fixed set of packages.  So, as this is the only issue which breaks the patching mechanism, I think the current patch is the best approach.  Now, if a valid use-case of that show-up to not be a security issue, then we can reconsider the patching mechanism to make it a bit more lazy.

This pull request is now complete and good for review, with 3 added test cases, and 1 static analysis:
Test cases are checking:

That the quick-fix does not change any of the hashes when no fixes are applied.  This is useful to guarantee that hydra would not recompile the world, if we are to build a security branch on top of the stable branch.
That the aib-fix does not change any of the hashes when no fixes are applied.  This is useful to guarantee that users will not iterate over unpatched packages.
That when the static analysis does not report any issues, then the security model is working as expected.  This ensure that we will only recompile the packages which have fixes, and that we will patch the packages which are depending on one of the fixed packages.

The static analysis is checking:

That all Nixpkgs is in a good state to be patched with the algorithm present in this pull request.  It reports any violation to the hypothesis on which this model is based on.

cc @edolstra, @domenkozar and @peti.

This pull request is now complete and good for review

Note, I will work on the documentation and the change-log, as soon as we agree on the technical aspect of the solution.
This pull request is now in a clean-enough state, so I am going to remove the "needs: clean-up" label.

I'll be able to review a week after 16.03 release (~10 days).

Thanks for the answer, in the mean time I will start looking at the static analysis issues, and reference this pull request in all the up-coming changes.

@vcunat
Ok, it seems that I would have to handle the override case any way, because of the almost package loop with:
  libkrb5 = krb5Full.override { … };
  krb5Full   depends on openldap.
  openldap   depends on cyrus_sasl
  cyrus_sasl depends on libkrb5

Not having a lazy solution for the override keyword make the current solution loop as soon as I am trying to fix the static analysis, as the resolution of the override function is made on the result of the patch and not on the result of fix f.

@vcunat commit a400adb fixes the issue, so I removed the modification on the chicken package form this branch.

After fixing some static analysis reports, I had to change the override mechanism as mentioned above, and I also had to revisit the test cases to use foldTree (added as part of commit 9fe5a47) instead of the fold function, because this was causing a stack overflow (~15k packages to compare).

Well, looking at this bit lib.foldTree lib.and true:

computing conjunction over a big list is best done strictly and from the left, i.e. foldl', IMO;
we have lib.all that should be redirected to a builtin ;-).

foldl in other language, but not without tail recursion.
https://github.com/NixOS/nixpkgs/blob/master/lib/lists.nix#L52
Good call for lib.all, I will use that instead.

I expect our buildins.foldl' only needs constant space, but I hadn't verified that in any way.

I modified the current set of patches, to include @vcunat suggestion of using lib.all instead of a fold function.
I also modified the test case which verifies that security updates are applied correctly, by checking the length of the outPath instead of checking the equality of the name attribute.  This is closer to the restrictions that we have with patching compiled programs.

@nbp could you explain how and when in python-packages.nix we're supposed to use self. We now use self to refer to all Python packages, expect some functions like buildPythonPackage and buildPythonApplication.

I have reviewed the patch. I think I understand the high-level design and implications, but I clearly didn't dig into fixed-point magic since my mind doesn't handle that abstraction (if anyone wants to draw that logic out, many of us will be happy). I'll trust test to prove it works.
Here's my general overview (from user perspective - I hope @nbp can confirm I'm not mistaken):

given two nixpkgs checkouts A and B, take packages A as a base
build all packages that changed in B - but skip all dependencies.
using similar logic as in pkgs.replaceDependencies, replace dependencies in B using A hashes

So we'd ship two channels, A being stable and B being stable-security.
So far I have only one nitpick, these changes share some of the functions in pkgs/build-support/replace-dependency.nix

@nbp Sorry missed your response a few days ago. Yeah my plan does unavoidably involve reflecting at eval time whether a derivation is built, and if we want to count available-to-substitute as built (which is distinct from treating substitution as building), that means querying remotes at eval time do.
I see that always building the vanilla pkg and then swapping run-time out runtime deps for unpatched packages avoids the needs for that reflection.

If you are unable to predict the outPath ahead of time, you cannot request other computers about the packages that you have already build. On the other hand, if you are able to compute the outPath ahead of time, then you can just pull all the patched packages from another computer.

@shlevy has a patch to Nix that allows import-from-derivation to not hurt substitution like it does today.

@FRidh
Not using self for the function implies that you will use the building rules coming from the old packages for patching, which is fine.  What matters is essentially the generation of the input of the function, because this one would be the one which matters for making security patches.
Also note that we should not always use self.  We should only use self for ensuring the each package definition is made out of the last generation.  Using self instead of pkgs is used to ensure that override are using the compilation rules of the patched packages.  For the build inputs, we actually want to use pkgs instead of self.  Failing to do so implies that any patched package will cause the recompilation of all dependent packages which is using self instead of pkgs.

@domenkozar

Here's my general overview (from user perspective - I hope @nbp can confirm I'm not mistaken):

given two nixpkgs checkouts A and B, take packages A as a base

Correct.

build all packages that changed in B - but skip all dependencies.

Build all packages that changed in B. Correct.
With dependencies from A.

using similar logic as in pkgs.replaceDependencies, replace dependencies in B using A hashes

using similar logic as in pkgs.replaceDependecnies. Correct.
Replace dependencies in B, by replacing A hashes by the fixed-with-patch hashes.

So we'd ship two channels, A being stable and B being stable-security.

This is one option.
I would prefer to merge both channels by putting the stable-security directory under the stable channel.  Thus, one secured nixpkgs channel will have 2 copy of nixpkgs.  The base A as of today, and in a sub-directory, the fixes B.

@nbp Could you provide a little walkthrough on how to use this? I mean concretely - what commands to run to produce a security update? (I.e., what should Hydra do to produce a fast channel update given a Glibc security patch, and what would the resulting channel look like.)

@edolstra

Could you provide a little walkthrough on how to use this?

Sure, I will first detail a few arguments that I added to top-level/default.nix.  This file has 3 new arguments:

defaultPackages: This argument describes the top-level files used to build the default set of packages.  This is an internal detail which is only used for making test cases.
quickfixPackages: This argument describes top top-level files of another Nixpkgs tree which contains a custom version of the current tree of Nixpkgs, where packages are patched with ABI compatible changes.  This option expects the same layout as defaultPackages.  If this option is null, then no quick-fix / patches are created.
doPatchWithDependencies: This argument describes that if quickfixPackages is not null, then the packages should be patched (sed) to have their dependencies updates.  Otherwise this will only recompile packages from the quickfix set, but not any of their dependencies.

In Hydra, we would have 2 channels, the base channel that may take time to catch-up with due to re-compilations, as we have today, and a new secure channel which will correspond to a new job.
The base channel will have no quickfixPackages input (= null).
The secure branch should be based on the commit of the latest base channel, thus lagging behind. The secure job should use the the base channel as input for the defaultPackages argument of the top-level/default.nix file, and use a Nixpkgs input taken from the secure branch of Nixpkgs repository for the quickfixPackages argument of the top-level/default.nix file.  The job input should also set doPatchWithDependencies to false, to avoid patching dependencies.
The content of the secure channel should look like:
secure channel
 |- default.nix (set the a default `quickfixPackages` argument)
 |- pkgs/ (from the `base` channel)
 |- lib/
 |- …
 |- quickfix/
    |- pkgs/ (from the `secure` Nixpkgs branch input)
    …

To locally test quickfixes, with patched dependencies, you can do the following:
$ git-new-workdir ./nixpkgs ./nixpkgs/quickfix
$ cd nixpkgs
$ <edit pkgs/top-level/default.nix to give the quickfixPackages argument>
$ cd ./quickfix
$ <edit any package such as the glibc>
$ cd - # back in nixpkgs
$ nix-build -A foobar -o foobar --dry-run

The modified top-level/default.nix should look something like:
, quickfixPackages ? {
    adapters = import ../../quickfix/pkgs/top-level/../stdenv/adapters.nix;
    builders = import ../../quickfix/pkgs/top-level/../build-support/trivial-builders.nix;
    stdenv = import ../../quickfix/pkgs/top-level/stdenv.nix;
    all = import ../../quickfix/pkgs/top-level/all-packages.nix;
    aliases = import ../../quickfix/pkgs/top-level/aliases.nix;
  }

If you want to see what hydra would build, you will have to toggle the default value of the doPatchWithDependencies argument inside top-level/default.nix.

@nbp Hm, I can't get it to work. I have a nixpkgs repo with the following change:
diff --git a/pkgs/top-level/default.nix b/pkgs/top-level/default.nix
index f9c81df..29b8b9e 100644
--- a/pkgs/top-level/default.nix
+++ b/pkgs/top-level/default.nix
@@ -44,7 +44,13 @@
   # Additional list of packages, similar to defaultPackages, which is used
   # to apply security fixes to a few packages which would be compiled, and
   # to patch any of their dependencies, to have a fast turn-around.
-, quickfixPackages ? null
+, quickfixPackages ? if builtins.pathExists ../../quickfix then {
+    adapters = import ../../quickfix/pkgs/stdenv/adapters.nix;
+    builders = import ../../quickfix/pkgs/build-support/trivial-builders.nix;
+    stdenv = import ../../quickfix/pkgs/top-level/stdenv.nix;
+    all = import ../../quickfix/pkgs/top-level/all-packages.nix;
+    aliases = import ../../quickfix/pkgs/top-level/aliases.nix;
+  } else null

   # When a set of quickfix packages is defined, this flag is used to enable
   # patching packages which are depending on the packages which are updated

And a quickfix subdirectory with this change:
diff --git a/pkgs/development/libraries/glibc/builder.sh b/pkgs/development/libraries/glibc/builder.sh
index 85f27c7..7804c4b 100644
--- a/pkgs/development/libraries/glibc/builder.sh
+++ b/pkgs/development/libraries/glibc/builder.sh
@@ -1,6 +1,8 @@
 # Glibc cannot have itself in its RPATH.
 export NIX_NO_SELF_RPATH=1

+echo PATCHED
+
 source $stdenv/setup

 postConfigure() {

However, building a package in the top-level nixpkgs gives an unchanged package straight from the binary cache:
$ nix-build -A hello
/nix/store/1i31yv0ik8s4h9fc4pqainrlsmrgckgs-hello-2.10

So, is this still going anywhere? Can we actually patch and deliver security updates quickly, or is there something missing in the implementation? Or are we simply lacking a packaging team that seeks and patches packages quickly?

What's stopping this from merging?

I wish I knew. Even Guix seems to have fixed the update delivery problem.

@C0DEHERO Guix uses Grafting, which is what we already have.

But I think Guix doesn't expect the enduser to do it but actually grafts the security updates in the update channel. I'm not sure though.

Important security updates tend to get a full rebuild of nixos small channels within 24h. Most CVEs don't even get noticed/patched in nixpkgs that fast.

Also, @fpletz and me constantly watch oss-security and usually push updates or patches in a timely manner. (mostly @fpletz beats me)

@spacekitteh wrote:

What's stopping this from merging?

@edolstra approval on the fact that this is the way to go, even if not all packages are working perfectly yet.
Ideally I wanted to go through all packages, but I probably too slow, and hardly fully motivated for periods as long as this would required, but this is not mandatory as we can merge this as a disabled feature first, and start experiment with it.
Unfortunately, fixing glibc issue noted by @edolstra would probably require some mass-rebuild changes.
I yet have to investigate it deeper, but I fear that this is related to our bootstraping mechanism of the stdenv, which builds all packages against an unpacthed version of the glibc coming from the last stage of the bootstrapped stdenv.
Otherwise, I am glad to see that this change can still merge without conflicts.
Note, this is disabled by default, and only enabled by the nixpkgs tests, so there is no risk at merging this branch right now.
@vcunat wrote:

Important security updates tend to get a full rebuild of nixos small channels within 24h. Most CVEs don't even get noticed/patched in nixpkgs that fast.

… and let everybody recompile their systems for ages?

@nbp

Stdenv bootstrapping woes

My PR #15043 reworks this. Perhaps it might help? [Also it will syntactically conflict with this, so some advice on how to fix after this is merged would be much appreciated].

It seems like we're only waiting for Eelco's approval. @edolstra do you have some time to investigate this PR?

@jgeerds I did, and I couldn't get it to work: https://github.com/NixOS/nixpkgs/pull/10851#issuecomment-214683948

We've had a longer debate on IRC today. @edolstra major "concern" is to have a fully working prototype that proves this PR works (together with two channels, one stable and another one shipping security updates).
I agree with that, we'll need that eventually anyway to use this work.
Like we did with multiple outputs, let's work on that after 16.09 and get it merged early for 17.03

Nicolas was kind enough to walk me through the details over several hours last Sunday. I've written up my understanding of the scheme here: https://theshortlog.com/2016/10/01/Nix-security-updates-update/
For me I think I'm missing the following pieces:

Explain what the output of the static analysis means, and how we can fix it
Add some docs how this can be used by end-users
A working demo

Other than that it's a mind-bender but it's a hard problem, and I couldn't come up with a simpler solution that fits into the constraints.

@teh, Thanks for the blog post. As you mention, I should document what are the reports of the static analysis.
Before documenting what these errors are, I will briefly explain how these are collected and why these are real issues.  In the description, you can find that the final evaluation looks like h = patch (g (fix f)) (g h).  This is made such that we can compare the dependencies of packages listed in g, by comparing (g (fix f)) with (g h).  (g (fix f)) would recompile packages which have updated sources.  (g (fix f)) would be post-compilation patched by replacing input changes of (fix f) by h.
The static analysis works on the original function of the fix-point g (nixpkgs function).  To make this analysis, we create an annotate function which will give a depth number to every derivation produced by the package set, unless it is already numbered.  The static analysis runs something similar to:
let
  pkgs =
    annotate 0 (pkgsFun (
      annotate 1 (pkgsFun (
        annotate 2 pkgs))));

  annotate = …;
  displayAnalysis = …;
in
  displayAnalysis pkgs

The depth number is equivalent to unrolling the loop of the fix-point made is top-level/default.nix.
The displayIssues function collect the numbered derivations, and reports errors and warnings.

alias (error): An alias is a derivation which is numbered with a non-zero depth.  An alias is a security issue as the patching mechanism is based on the hypothesis that all packages got evaluated in the last step of the iteration of nixpkgs.  This means that aliases are taken verbatim from (fix f).

unpatched (error): An unpatched package means that the dependencies of are taken from depth larger than 1 above the depth of the package it-self.  What this error implies is that if one of the dependency got changed, then we would not notice this change, and we would not patch the post-compilation output of the package.  Aliases are often the source of unpatched packages, when they are used as input of other packages, but this might not be the only one.

recompile (warning): A recompile package means that one of its dependencies has a zero depth difference with the package.  This implies that this is part of the same evaluation, and that the package will add extra work-load to Hydra.  This is a warning as a few recompilation might be acceptable under specific conditions, such as statically-compiled packages.  If the number of recompilation is high, then this would defeat the purpose of this work, as this adds extra packages in the way of shipping security updates faster.

To run the static analysis, you can use the following commands:
$ nix-instantiate ./pkgs/test/security/static-analysis.nix --eval-only --show-trace 2>&1 | \
     tee analysis.log | less

A summary can be obtain with the shell script ./pkgs/test/security/static-analysis-stats.sh, which accept either one log file or two, to provide a summary or a difference of the statistics.

@teh:

Explain what the output of the static analysis means,

Done (see above)

and how we can fix it

Unfortunately highly depends on each of the Nix expression.  But as soon as you understand how the static analysis works, you can probably figure your-self how to fix it.  Otherwise fill free to ask questions here.

Add some docs how this can be used by end-users

This is made to be transparent, so technically the only documentation needed to to point people to the right channel / repository-branch, and that would be all.  But for that we need prototype to distribute the content of security-updates branch.

A working demo

@domenkozar and I had a discussion about that almost a month ago.  On my side, I have to fix enough of these issues such that we can reduce the number of recompilation.

@nbp what sort of fixes are holding up a working demo? Anything that actually breaks the mechanism? Or is it that you're trying to fix all of nixpkgs of these errors?

@spacekitteh ,

what sort of fixes are holding up a working demo?

There is no mandatory fixes but this would be a nicer experience to demo a larger set of fixable packages.
From the infrastructure, the main blocker for getting a working demo is to get the security-updates branch built in Hydra, i-e making rules to build the base branch and merge it with the security branch.
Then, we need some task-force to get some fixes (or dummy fixes) applied in the security updates branch.
The static analysis is only a tool made to highlight the parts which would not work properly.  I have not yet measured, but I would expect that we have already have a large number of packages with no issues.  Among the prerequisite are that I should write some tests to ensure that we can successfully update real packages such as openssl.

Anything that actually breaks the mechanism?

The mechanism can already be tested locally.  For that you need to check out the security-updates branch twice, in 2 different directories, and call the default.nix file from one with a quickfixPackages argument which imports the pkgs/top-level/index.nix file of the other. The other directory contains all the program which have security patches.
Currently, we could merge the security-updates branch as-is, as the patching mechanism is disabled unless the quickfixPackages argument is provided.

Or is it that you're trying to fix all of nixpkgs of static analysis errors?

That is something that I would hardly be able to complete alone.  Writing documentation to explain how to fix them would be hard as people tend to innovate every-time they write new packages.
One of the restriction of the security-updates branch is that all packages should use callPackage, and that callPackage should provide all the input of the package.  We should not use callPackages, as this suggest that people can use rec to get other packages, instead of getting them from the arguments of callPackage.

This pull request has been mentioned on Nix community. There might be relevant details there:
https://discourse.nixos.org/t/in-overlays-when-to-use-self-vs-super/2968/5

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/2022-10-19-nixpkgs-architecture-team-meeting-14/22578/3

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/timely-updates-for-nixos/29231/7

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/correct-way-to-export-functions-in-overlays-and-use-them-in-others/45016/2


New version: PressureDrop v1.0.1

Registering package: PressureDrop
Repository: https://github.com/jnoynaert/PressureDrop.jl
Created by: @jnoynaert
Version: v1.0.1
Commit: 192a995bb03891a41e18f08cc0456f11f3dfc051
Reviewed by: @jnoynaert
Reference: https://github.com/jnoynaert/PressureDrop.jl/issues/2

Is skipping 1.0.0 intentional?

Yes; 1.0.0 ended up being an internal-use release. If it helps with registry hygiene I can squash them together to avoid skipping, though.

It's up to you which you prefer.

This way is fine and will retain the actual history. Thanks!


[hal] elide lifetime parameter

Fixes #issue
PR checklist:

[x] make succeeds (on *nix)
[ ] make reftests succeeds
[x] tested examples with the following backends:

[x] metal

[x] rustfmt run on changed code

Can we do the same for submit_nosemaphores?

No. Tried and failed when I remove the 'a param. Compiler requires an explicit lifetime for submit_nosemaphores.

Are you seeing a compiler/lint warning about the lifetime you remove here? We might otherwise just leave it for consistency: it's not harming.

Removing lifetime of present_without_semaphores  causes no compile error or warning. It just work with Rust 1.35.0.

Ok. If there is no warnings with the current code, I'd slightly prefer to keep the lifetime consistent between present and submit functions, so for it to say as it is.

Ok.


change saga-spring port to made same as readme.md

change saga-spring port to made same as readme.md

Coverage decreased (-0.09%) to 95.4% when pulling 15e740ff3948da5542c11a2ebe050f9cacd5ab81 on eumji025:master into 2cd8ec43284e5c37a54bd1412f170add1b02001b on apache:master.


How to input Chinese in a input widget

I want to know how to input Chinese in PixelStreaming

Hi @Shooter-oss, the issue is already opened here and we are tracking it.
We've unfortunately not been able to prioritize it, but it is on our radar.
Thank you for your patience with this.


API version bump to 1.4

Upcoming hub version bumps up the API version to 1.4. No info on any other changes so once it goes live other things may break as well.

Version bump is trivial, once the update is live need to test what actually breaks before committing this and pushing out a new version of the library.

Looks like after the version bump no further issues are immediately, so this is good to push.


app-catalog: Filter installed apps by namespace and text search

As in the workloads views, I would like to have the same filters for namespace and text search in the apps/installed view.
Preselecting the namespace like when switching between workloads views would be the icing on the cake. ;-)

This would be huge for me as well... my cluster has many namespaces each with many apps deployed via Helm (over 2000 at the moment)...

Maybe this PR fixes this? https://github.com/headlamp-k8s/plugins/pull/110
cc/ @ashu8912


Use Span (and Pipe API) for network code and packet handling

C# offers a new way to handle pieces of arrays: C# - All About Span: Exploring a New .NET Mainstay. I think it could help to improve the performance in the network code and would result in a lower amount of heap allocations in this project. This means, the garbage collector would have less work to do.
Currently, we make good use of byte arrays, e.g. for every received network packet at least one byte array is created. If a packet was encrypted, e.g. by SimpleModulus, even more byte arrays are created.

I tested it a bit and created a branch for it: https://github.com/MUnique/OpenMU/tree/span

The new Pipe API could be amazing for this projection, too :)
More info about the new API can be found here: https://blog.marcgravell.com/2018/07/pipe-dreams-part-1.html

The new Pipe API could be amazing for this project, too :)
More info about the new API can be found here: https://blog.marcgravell.com/2018/07/pipe-dreams-part-1.html

I would have the following suggestions to improve the network performance:

To avoid GC to collect objects, all objects should be initialised on startup and reused. So that no new-keywords are necessary during the network read / write operations and references are kept.
A big buffer array (to pin it on the large object heap [https://stackoverflow.com/questions/8951836/why-large-object-heap-and-why-do-we-care]) should be created and potions of it should be assigned to the SocketAsyncEventArgs via SetBuffer()-Method. This prevents compacting of the buffer thus improves performance.
To avoid copying memory / object instantiation we should use Span
This will be only possible by limiting the total number of connections from the beginning. As the necessary resources would be calculated on startup. (Ex. BufferSize = MaxConnections * BufferSize, Number of instantiated SocketAsyncEventArgsat the beginning, etc. will all be based on that number)
Additionally investigate if the c# Pipeline is better than a big buffer. (https://github.com/dotnet/corefx/issues/31953) As there might be some concerns with implementation.

The following changes should be applied to https://github.com/MUnique/OpenMU/blob/master/src/Network/Listener.cs and https://github.com/MUnique/OpenMU/blob/master/src/Network/Connection.cs as they look like the Server / Client core parts of the application if I'm not mistaken.
I would propose the following process of change:

Start by creating an Interface for Listener and Connection classes, so the implementation can be swapped.
Create a test to measure the performance. (By load testing and measuring times, as well as external monitoring of CPU and Memory)
Try to create an Implementation with the above suggested changes.
Compare results and draw conclusions where we can improve further.

This would not yet introduce Span to the whole project, but on a smaller level that we then might be able to test with some test cases sending and modifying alot of data (what the server usually would do) After a successful result we can discuss on exposing/returning the Span parts or other steps that can help to read/write the data more easy / performant.
If any of these steps are not reasonable or another workflow should be taken, I'm happy to discuss on that so we can find a way to improve the performance together.

Well, I would propose to make it as simple as possible in a first step. For the sake of simplicity we should not case about managing our own memory ;-)
That means:

Using System.Buffers.MemoryPool in the PlayerView implementation instead of the currently used byte arrays. One thing to note: C3/C4 packets require slightly more space when they are encrypted. So it might be needed to use a memory pool during the encryption (SimpleModulus) too.
System.IO.Pipelines and Span in the Connection class and encryption stuff. Luckily for us, we can expect that decrypted packets are not bigger than the encrypted ones.

I have already started doing some transformation of the Connection class to use PipeReader and PipeWriter (interesting article: https://blogs.msdn.microsoft.com/dotnet/2018/07/09/system-io-pipelines-high-performance-io-in-net/). However, I'll need to update the span-branch to the state of the current master branch first. I did a migration to .net standard/core on master which is not in span yet. After that, I'll try to integrate my changes of the Connection class. I'm quite busy at the moment, so this could take some days.
When that works, we could write some benchmark tests and see how much changed in terms of speed and heap allocations. If required, we can then optimize further.

To make use of pipelines with sockets there is this project (with nuget package): https://github.com/mgravell/Pipelines.Sockets.Unofficial

Added a branch/PR with my current state.

I added a benchmark, and I have to say... I'm utterly impressed. I'm searching for errors in my methodology, but I can't find them. If you can, please tell me ;-)
You can find the code of the benchmark here.
The results:

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.17134.345 (1803/April2018Update/Redstone4)
Intel Core i5-3570 CPU 3.40GHz (Ivy Bridge), 1 CPU, 4 logical and 4 physical cores
.NET Core SDK=2.1.400
  [Host]     : .NET Core 2.0.9 (CoreCLR 4.6.26614.01, CoreFX 4.6.26614.01), 64bit RyuJIT
  Job-VXAUGJ : .NET Core 2.0.9 (CoreCLR 4.6.26614.01, CoreFX 4.6.26614.01), 64bit RyuJIT

Runtime=Core  InvocationCount=10  UnrollFactor=1

Method
Mean
Error
StdDev
Gen 0
Allocated

PipelinedSimpleModulusEncryptor
3.493 us
0.2589 us
0.7132 us
-
4.85 KB

SimpleModulusEncryptor
687,805.169 us
6,397.1918 us
5,670.9434 us
15500.0000
47656.55 KB

If I read (and calculate) this correctly, the pipelined version is almost 200000 times faster while allocating no additional memory during encryption. If it would allocate just a one byte per packet, we would see over 100 KB allocated ;-)

Merged pipe branch into master :)


Terminal font completely different than in the editor

As requested in #35991, creating a new issue.

VSCode Version: 1.18.0
OS Version: (K)Ubuntu 17.10

Steps to Reproduce:

Set editor.fontFamily to "Verdana".
Open terminal
Look at each letter using half the screen.

Reproduces without extensions: Yes

Duplicate https://github.com/Microsoft/vscode/issues/35681


⚠ WSINGRESOS - RECIBOPAGO has degraded performance

In 9a1ebe7, WSINGRESOS - RECIBOPAGO (https://srvshyweb.yucatan.gob.mx/cgi-bin/wspd_cgi.sh/WService=wsingresos/internet/servicios/wsRecibosPagosWeb.r) experienced degraded performance:

HTTP code: 200
Response time: 2168 ms

Resolved: WSINGRESOS - RECIBOPAGO performance has improved in 3956afa after 1 hour, 2 minutes.


Implement ReCom

Depends on #2 -- a spanning tree abstraction / lib

Now that I think about this more:

Why reimplement it, if it doesn't impose arbitrary filters (a la ALARM) and is stable?
Reverse vs. regular -- I don't have a prayer of understand this newer paper!

Using MGGG's GerryChain implementation


Setting the period in insights

Feature Description
I'd like to be able to change the period in the statistics. Having a start and end date, for example, would be a real plus for this page.
Screenshots

Yep, I've also mentioned that when reviewing that page.
From my side, this is definitely the intended behavior, plus depending on implementation, it even allows for reproducible URLs.


Tell us it isn't available yet right away

I don't know who all writes the documentation for Microsoft, but please understand that we don't all have infinite time to read every detail of your documentation. Move the fact that the SDK is in development clear to the top please. This will tell everyone that yes you are working on it, but it isn't available yet. Also, you could pretty easily change the title of this article to reflect that as well so we know that the option will come, but not available without even reading it. Please consider our time important, we spend a lot of money on you.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 2790f804-e26d-d830-cfd3-f856fb2f4d45
Version Independent ID: 0049bc46-c7c6-fd43-b9ca-2b8e76c3ef55
Content: Windows Virtual Desktop Linux Support - Azure
Content Source: articles/virtual-desktop/linux-overview.md
Service: virtual-desktop
GitHub Login: @Heidilohr
Microsoft Alias: helohr

@ChrisCochrun Thanks for the Feedback.
Agree that it makes more sense to have a Note in the top of the doc that the SDK is under development.
I have updated the doc with a note and the changes will go live soon.


Grafana Charges reports incorrect start time resulting in wrong kWh added / used

Describe the bug:
Charges overview in Grafana uses incorrect start time resulting in wrong kWh added / used. I have for my company Tesla both TeslaMate and TeslaFi running in order to have a back-up in case one system has issues, and TeslaFi correctly reports start time as well as kWh added / used.
Expected behavior:
Reporting of the real start time of charging in Grafana’s Charges overview, and using the real start time to calculate (the correct) kWh added / used during that session.
How to reproduce it (as minimally and precisely as possible):
In the example below, 'Grafana > TeslaMate > Charges' is showing a starting time of 2021-08-04 22:53:49 at 17% SoC.
When going into the 'Charge Details' of this charge, you will initially see the same starting time and starting SoC. But you can change the time range to an earlier one. In the example attached I changed start time to 2021-08-04 22:00:00. Then out of the TeslaMate database, the starting SoC was 8% instead of 17%, and start time in the database is 22:10:42 instead of 22:53:49. So data and logging of data started earlier than reported in 'Charges' under Grafana.
Steps to reproduce the behavior:
See ‘How to reproduce it’ .
Relevant entries from the logs
See below relevant part of the TeslaMate log
…
teslamate2_1  | 2021-08-04 20:07:04.336 car_id=1 [info] Received stream data: :inactive
teslamate2_1  | 2021-08-04 20:07:05.507 [error] Vehicle Error: Vehicle is offline
teslamate2_1  | 2021-08-04 20:07:12.197 car_id=1 [info] Start / :offline
teslamate2_1  | 2021-08-04 20:07:12.206 car_id=1 [info] Disconnecting ...
teslamate2_1  | 2021-08-04 20:07:54.064 car_id=1 [info] Start / :online
teslamate2_1  | 2021-08-04 20:07:54.081 car_id=1 [info] Connecting ...
teslamate2_1  | 2021-08-04 20:07:54.084 car_id=1 [info] Driving / Start
teslamate2_1  | 2021-08-04 20:09:18.441 [info] GET https://nominatim.openstreetmap.org/reverse -> 200 (158.828 ms)
teslamate2_1  | 2021-08-04 20:09:18.917 [info] GET https://nominatim.openstreetmap.org/reverse -> 200 (474.122 ms)
teslamate2_1  | 2021-08-04 20:09:18.934 car_id=1 [info] Driving / Ended / 0 km – 1 min
teslamate2_1  | 2021-08-04 20:09:18.934 car_id=1 [info] Start / :online
teslamate2_1  | 2021-08-04 20:10:32.137 car_id=1 [info] Charging detected: -2 kW
teslamate2_1  | 2021-08-04 20:10:32.138 car_id=1 [info] Charging detected: -1 kW
teslamate2_1  | 2021-08-04 20:10:32.138 car_id=1 [info] Fetch already in progress ...
teslamate2_1  | 2021-08-04 20:10:33.192 car_id=1 [info] Charging detected: -3 kW
teslamate2_1  | 2021-08-04 20:10:33.192 car_id=1 [info] Fetch already in progress ...
teslamate2_1  | 2021-08-04 20:10:34.528 [info] GET https://nominatim.openstreetmap.org/reverse -> 200 (100.289 ms)
teslamate2_1  | 2021-08-04 20:10:34.554 car_id=1 [info] Charging / SOC: 8% / Thuis
teslamate2_1  | 2021-08-04 20:10:34.554 car_id=1 [info] Disconnecting ...
teslamate2_1  | 2021-08-04 20:11:53.395 [info] GET /
teslamate2_1  | 2021-08-04 20:11:53.403 [info] Sent 200 in 7ms
teslamate2_1  | 2021-08-04 20:11:54.784 [info] GET /
teslamate2_1  | 2021-08-04 20:11:54.793 [info] Sent 200 in 8ms
teslamate2_1  | 2021-08-04 20:24:50.700 [info] GET https://owner-api.teslamotors.com/api/1/vehicles/221606540911/vehicle_data -> 408 (29874.367 ms)
teslamate2_1  | 2021-08-04 20:24:50.916 car_id=1 [warn] Discarded incomplete fetch result
teslamate2_1  | 2021-08-04 20:51:05.551 [info] GET /
teslamate2_1  | 2021-08-04 20:51:05.559 [info] Sent 200 in 8ms
teslamate2_1  | 2021-08-04 20:51:08.912 [info] GET /
teslamate2_1  | 2021-08-04 20:51:08.921 [info] Sent 200 in 8ms
teslamate2_1  | 2021-08-04 20:53:13.077 [info] SIGTERM received - shutting down
teslamate2_1  |
teslamate2_1  | waiting for postgres at database2:5432
teslamate2_1  | 2021-08-04 20:53:38.662 [info] Migrations already up
teslamate2_1  | 2021-08-04 20:53:45.178 [info] System Info: Erlang/OTP 24 (emu)
teslamate2_1  | 2021-08-04 20:53:45.178 [info] Version: 1.23.7
teslamate2_1  | 2021-08-04 20:53:45.705 [info] POST https://auth.tesla.com/oauth2/v3/token -> 200 (408.664 ms)
teslamate2_1  | 2021-08-04 20:53:47.681 [info] POST https://owner-api.teslamotors.com/oauth/token -> 200 (1970.318 ms)
teslamate2_1  | 2021-08-04 20:53:47.681 [info] Refreshed api tokens
teslamate2_1  | 2021-08-04 20:53:47.702 [info] Scheduling token refresh in 5 wk 5 d
teslamate2_1  | 2021-08-04 20:53:47.714 [info] Running TeslaMateWeb.Endpoint with cowboy 2.9.0 at :::4000 (http)
teslamate2_1  | 2021-08-04 20:53:47.725 [info] Access TeslaMateWeb.Endpoint at http://localhost
teslamate2_1  | 2021-08-04 20:53:48.951 [info] Starting logger for 'Nova'
teslamate2_1  | 2021-08-04 20:53:48.997 [info] MQTT connection has been established
teslamate2_1  | 2021-08-04 20:53:56.460 car_id=1 [info] Start / :online
teslamate2_1  | 2021-08-04 20:53:56.495 car_id=1 [info] Connecting ...
teslamate2_1  | 2021-08-04 20:53:56.712 [info] GET https://nominatim.openstreetmap.org/reverse -> 200 (209.745 ms)
teslamate2_1  | 2021-08-04 20:53:56.750 car_id=1 [info] Charging / SOC: 17% / Thuis
teslamate2_1  | 2021-08-04 20:53:56.751 car_id=1 [info] Disconnecting ...
teslamate2_1  | 2021-08-04 20:54:45.261 [info] GET /
teslamate2_1  | 2021-08-04 20:54:45.300 [info] Sent 200 in 38ms
teslamate2_1  | 2021-08-04 20:57:13.859 [info] GET /charge-cost/18
teslamate2_1  | 2021-08-04 20:57:13.879 [info] Sent 200 in 19ms
teslamate2_1  | 2021-08-04 21:03:58.942 [info] GET /
teslamate2_1  | 2021-08-04 21:03:58.950 [info] Sent 200 in 8ms
teslamate2_1  | 2021-08-04 22:17:38.542 [warn] GET https://owner-api.teslamotors.com/api/1/vehicles/221606540911/vehicle_data -> 504 (10183.681 ms)
teslamate2_1  | 2021-08-04 22:17:38.542 car_id=1 [error] Error / :timeout
teslamate2_1  | 2021-08-04 22:24:45.298 [warn] GET https://owner-api.teslamotors.com/api/1/vehicles/221606540911/vehicle_data -> 504 (10228.541 ms)
teslamate2_1  | 2021-08-04 22:24:45.298 car_id=1 [error] Error / :timeout
teslamate2_1  | 2021-08-05 00:08:21.909 [info] GET https://owner-api.teslamotors.com/api/1/vehicles/221606540911/vehicle_data -> 408 (10559.282 ms)
teslamate2_1  | 2021-08-05 00:08:21.910 [warn] TeslaApi.Error / %{"error" => "https://mothership-api.vn.teslamotors.com:443/vehicles/196220303194 => operation_timedout with 10s timeout", "error_description" => "", "response" => nil}
teslamate2_1  | 2021-08-05 00:08:21.910 car_id=1 [error] Error / :unknown
teslamate2_1  | 2021-08-05 00:51:36.017 car_id=1 [info] Phase correction: 2 -> 3
teslamate2_1  | 2021-08-05 00:51:36.030 car_id=1 [info] Derived efficiency factor: 136.7 Wh/km (10x confirmed)
teslamate2_1  | 2021-08-05 00:51:36.031 car_id=1 [info] Charging / Complete / 40.89 kWh – 238 min
teslamate2_1  | 2021-08-05 00:51:36.034 car_id=1 [info] Start / :online
teslamate2_1  | 2021-08-05 00:51:36.058 car_id=1 [info] Connecting ...
teslamate2_1  | 2021-08-05 00:54:41.646 car_id=1 [info] Suspending logging
teslamate2_1  | 2021-08-05 01:04:23.273 car_id=1 [info] Fetching vehicle state ...
teslamate2_1  | 2021-08-05 01:04:23.866 car_id=1 [info] Start / :asleep
teslamate2_1  | 2021-08-05 01:04:23.878 car_id=1 [info] Disconnecting ...
teslamate2_1  | 2021-08-05 07:33:46.412 car_id=1 [info] Start / :online
teslamate2_1  | 2021-08-05 07:33:46.439 car_id=1 [info] Connecting ...
teslamate2_1  | 2021-08-05 07:36:51.846 car_id=1 [info] Suspending logging
teslamate2_1  | 2021-08-05 07:38:00.171 [info] GET /
teslamate2_1  | 2021-08-05 07:38:00.180 [info] Sent 200 in 8ms
teslamate2_1  | 2021-08-05 07:38:03.046 [info] GET /
teslamate2_1  | 2021-08-05 07:38:03.056 [info] Sent 200 in 9ms
teslamate2_1  | 2021-08-05 07:49:12.338 car_id=1 [info] Fetching vehicle state ...
teslamate2_1  | 2021-08-05 07:49:12.937 car_id=1 [info] Start / :asleep
teslamate2_1  | 2021-08-05 07:49:12.949 car_id=1 [info] Disconnecting ...
…
Screenshots:

TeslaMate Charges print screen

TeslaMate Charge Details without changing start time

TeslaMate Charge Details with changing start time
Data:
charges.csv
charging_processes.csv
Environment:

TeslaMate version: 1.23.7
Type of installation: Docker
OS TeslaMate is installed on: Raspberry Pi 4B with all updates installed
User OS & Browser: Mac OS X 10.15.7, Firefox 90.0.2 (all latest versions)
Others:

The problem is that you restarted TeslaMate at 22:53 (which is shown as 20:53 UTC in the log). That left an unclosed charging session in the database (id 20 in charging_processes.csv), which is not visible from the Charges dashboard. When Teslamate started again, it made a new charging session (id 21) for the rest of the time.

I did not restart TeslaMate. I am 100% sure of that.
I see that the Tesla token got updated at that time, so could it not be that TeslaMate updated the vehicle token while it was still charging? Should maybe token update only be initiated when vehicle is asleep or attempts to go asleep?
Another root cause could have been that my internet connection went down for a minute when I remember back, but this is a situation it should be able to deal with.

Something certainly did restart it:
teslamate2_1 | 2021-08-04 20:53:13.077 [info] SIGTERM received - shutting down
teslamate2_1 |
teslamate2_1 | waiting for postgres at database2:5432
teslamate2_1 | 2021-08-04 20:53:38.662 [info] Migrations already up
teslamate2_1 | 2021-08-04 20:53:45.178 [info] System Info: Erlang/OTP 24 (emu)

Possibly apt did that, depending on what was upgraded. You could check the logs for the other docker containers to see if they all restarted.

When checking /var/log/dpkg.log I believe we can safely assume that 'sudo apt update'  and ' sudo apt full-upgrade' are for blame for interrupting TeslaMate. It matched the time-frame in the TeslaMate log (see log extract below). So this bug can be closed, as root cause is found.
Now, how to solve. Is there any documentation which helps me tap into the database to correct manually data, or is there another way on how to solve?
 I'll close this then.
There is documentation on how to fix drives/charges that were interrupted: https://docs.teslamate.org/docs/maintenance/manually_fixing_data
Manually adding data is not covered yet. Database management tools like TablePlus or pgadmin are helpful to manually add data.


Switch to EntityManagerInterface

Just to get rid off one of our PHPStan errors :)

thanks ! @chr-hertel


updated field to using in list view

added .buttonStyle(.plain) TogglableSecureField for prevent unexpected behavior in the list view

Looks good to me, thanks!


backgorund image

great app but there is one issue which is when you add an image to the background its not fit do you know how to fix that?
in the view controller
backgroundsettingvc "function"
self.board.backgroundColor = UIColor(patternImage: Backgroundimage)
you have to implement something called
CGContextSetPatternPhase
but i dont know how can you help me?

Hope this will help you: http://stackoverflow.com/questions/10629403/ios-colorwithpatternimage-stretch-image-full-screen


About the classification label

Hi, authors, thank you for your contribution! I have a question after reading the codes:
If the labels of classification tasks are encoded to more than 1 token, we just calibrate the first token and calculate acc in code. Right?
Thank you!

sry, I understand now.


 I have extended a line which extends Glass fiber to each house.
Now, I'd like to cluster the points to assign a distributor to each cluster. The maximum of house points per cluster should be 20.
I calculated an adjacency matrix based on the Glass fiber line for the data set which has 61 points, here as an example. I do the clustering by this library and the pre-computed adjacency matrix. However, I sometimes see a wrong clustering, which is observable in the picture.
It is my code:
"am" is the adjacency matrix of distances
db = KMeansConstrained(n_clusters = 4,size_max=20, random_state=0)
result = db.fit_predict(am)
In the picture, the black line is the glass fiber line which is the base of calculation and the colorful points are the my clustered points by the algorithm. The green and yellow clusters are not in the best state, as you see.
I have sometimes the same issue with other datasets, as well.

I appreciate any help to improve the result.
Versions:

Python: 3.9
Operating system: Windows
k-means-constrained: 0.7.2
numpy: 1.23.2
scipy: 1.9.1
ortools: 9.4.1874
joblib: 1.1.0
cython (if installed): is not installed

Best regards,
Mostafa

You need to provide a minimum, reproducing example otherwise it’s not possible to debug this issue. Closing ticket until more information is provided.


specify python version for pip install

To avoid error caused by multiple versions of python and pip

Pull request information

Status: ? (ready to merge / in development)
Kind of changes: ? (bug fix / new feature / documentation...)
Related PR or issue: ? (optional)

Description

? (general description)

Pull Request Test Coverage Report for Build 5302

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 89.901%

Totals

Change from base Build 5270:
0.0%

Covered Lines:
9000

Relevant Lines:
10011

💛  - Coveralls

@eugenevinitsky @kanaadp @kjang96 @Yasharzf I can confirm that this is okay for linux. can someone make sure it's okay on mac as well?

Confirmed.