Adds Windows NetFx and NetCore specific build configs

Background
We need to run the NetCore builds inside a container in TeamCity, but windows was building both NetFx and NetCore in the same step, this allows us to run them separately.
Results

Fixes https://github.com/OctopusDeploy/Issues/issues/... (optional public issue)
Fixes https://github.com/OctopusDeploy/ResearchAndDevelopment/issues/... (optional private issue)
See How we use GitHub Issues (including this flowchart
Before

After

How to review this PR

Quality :heavy_check_mark:

Pre-requisites

[ ] I have read How we use GitHub Issues for help deciding when and where it's appropriate to make an issue.
[ ] I have considered informing or consulting the right people, according to the ownership map.
[ ] I have considered appropriate testing for my change.

This PR is not relevant anymore


fix: jwt verification

Description:
Fix jwt verification
Checklist:

[ ] My PR follows the contribution guidelines of this project
[ ] My PR is free of third-party dependencies that don't comply with the Allowlist
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have checked the PR title to follow the conventional commit specification

Pull Request Test Coverage Report for Build<PHONE_NUMBER>7
Details

4 of 47 (8.51%)  changed or added relevant lines in 3 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage increased (+1.7%) to 43.459%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

EdgeAgentSDK/Pollux/Sources/Models/JWT/JWTRevocationCheck.swift
1
2
50.0%

EdgeAgentSDK/Pollux/Sources/PolluxImpl+CredentialVerification.swift
3
4
75.0%

EdgeAgentSDK/Pollux/Sources/Models/JWT/JWTRevocationStatus.swift
0
41
0.0%

Files with Coverage Reduction
New Missed Lines
%

EdgeAgentSDK/Pollux/Sources/PolluxImpl+CredentialVerification.swift
2
76.4%

Totals

Change from base Build<PHONE_NUMBER>:
1.7%

Covered Lines:
5089

Relevant Lines:
11710

💛  - Coveralls


add clip_size option to data prep

not quite sure what this referred to -- window size? -- so closing


chore(deps): update to support eslint v7

BREAKING CHANGE: Update eslint to v7
BREAKING CHANGE: Update peer dependencies by re-running install command in README

:tada: This PR is included in version 9.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Webmock/Net:HTTP (via faraday) opens a connection when disallowing connections.

      WebMock.disable_net_connect!(:allow_localhost => false)
      # ....
      stub_request(:get, "http://localhost:1234")
      # ....

will result in the following:
Faraday::ConnectionFailed:
Connection refused - connect(2) for "localhost" port 1234

it opens the connection even though it shouldn't be opening any outgoing connections (because we're mocking).
is this intended behaviour?

Can you please provide a failing code example?

require 'spec_helper'
require 'webmock/rspec'
require 'pry'
describe Failure do
before :all do
WebMock.disable_net_connect!(:allow_localhost => false)
end
describe "failing example" do
it "should not fail" do
url = "http://localhost:9080"
stub_request(:delete, url)
  conn = Faraday.new(:url => url) do |faraday|
    faraday.request  :json                    # json params
    faraday.response :logger                  # log requests to STDOUT
    faraday.adapter  Faraday.default_adapter  # make requests with Net::HTTP
  end

  conn.delete do |req|
    req.body = {}
  end
end

end

ok so is that related to delete request?

no. happens on every type I've tried. Get, Post, Delete.
require 'spec_helper'
require 'webmock/rspec'
require 'pry'

describe Failure do
  before :all do
    WebMock.disable_net_connect!(:allow_localhost => false)
  end

  describe "failing example" do
    let(:url) do
      "http://localhost:9080"
    end

    let(:conn) do
      conn = Faraday.new(:url => url) do |faraday|
        faraday.request  :json                    # json params
        faraday.response :logger                  # log requests to STDOUT
        faraday.adapter  Faraday.default_adapter  # make requests with Net::HTTP
      end
    end

    it "should not fail on delete" do
      stub_request(:delete, url)

      conn.delete do |req|
        req.body = {}
      end
    end

    it "should not fail on post" do
      stub_request(:post, url)

      conn.post do |req|
        req.body = {}
      end
    end

    it "should not fail on get" do
      stub_request(:get, url)

      conn.get do |req|
        req.body = {}
      end
    end
  end
end

It's probably because the connection is opened before any request is made.
WebMock can only intercept http requests by their pattern.
You open a new connection at conn = Faraday.new(:url => url), and at that point WebMock doesn't know what requests are going to follow.

order of the Faraday.new, my actual code that exposed it calls it after the mocks were set up.

sure, but the stubs are on http requests, not on https connections.
request is stubbed by url pattern. connection only uses host, protocol and port.
There can be multiple requests done using same connection. Some of them stubbed, some of them not.
I'm surprised it's happening because WebMock by default prevents Net::HTTP from doing any connections. It's possible that Faraday creates a new connection in a different way.

possibly, I'll investigate how faraday makes the connection, and let you know.

I just hit this problem to, because the Oauth2 client gem uses Faraday. Any updates?

Another write in for hitting the problem, also with the Oauth2 gem.

Can someone provide a failing code snippet to reproduce the issue?

https://github.com/bblimke/webmock/issues/622 fixed it for me. Use WebMock.enable!.

Could the fact that faraday-net_http uses Net::HTTPGenericRequest under the hood have something to do with it?
https://github.com/lostisland/faraday-net_http/blob/11953160f75dd488133a74857c2b07d41be8995d/lib/faraday/adapter/net_http.rb#L80-L93
It looks like WebMock isn't redefining that particular constant:
https://github.com/bblimke/webmock/blob/1b23927c10971bbdfdc01d7e433c26f527debd0f/lib/webmock/http_lib_adapters/net_http.rb#L16-L21
I'm not terribly familiar with the innards of Net::HTTP so I might not know what I'm talking about here.


some facebook profile pictures broken

e.g. amber's: https://www.brid.gy/facebook/31600719
looks like the problem is that some FacebookPage.picture values have URLs with username instead of user id, e.g. http://graph.facebook.com/caseorganic/picture?type=large , which don't work. easiest fix is just to update all of those values in the datastore.
granary currently always returns facebook profile picture URLs with user ids, so this was probably just a temporary growing pain at some point.

i've updated the FacebookPage.picture values in the datastore. i pulled them with remote_api, extracted just the username ones, and wrote them back with their ids instead.
~/google_appengine/remote_api_shell.py brid-gy
...
import facebook
for f in facebook.FacebookPage.query():
  print f.key.id(), f.picture
^D

# copy output to ~/fb_pictures.txt

sort -k 2 ~/fb_pictures.txt

# extract just URLs with usernames

~/google_appengine/remote_api_shell.py brid-gy
...
import facebook

@ndb.transactional
def fix(id):
  f = facebook.FacebookPage.get_by_id(id)
  f.picture = 'https://graph.facebook.com/v2.2/%s/picture?type=large' % id
  f.put()

with open('Users/ryan/fb_usernames.txt') as f:
  for line in f:
    id = line.split(' ')[0]
    fix(id)
    print id

i'm attaching the list of updated user ids here.
user_ids.txt
there was also one oddity:<PHONE_NUMBER>43524's profile picture was http://pbs.twimg.com/profile_images/3046920697/8a2a4524535a1e60fbaf0ae3f0058fb9.jpeg . weird. i've updated that one too.


feat(diagnostics): update autogenerated schema and description

Updates the schema generated by the diagnostics app to use the current Schema and Table syntax.
Also updates the comment above the schema to be more descriptive.
Below is an example of a before/after.
Before:
// This displays the inferred schema currently used by the diagnostics app.
// This is based on downloaded data, rather than the source database.
// If a table is empty, it will not display here.
// Tables and columns are only added here. Nothing is removed until the database is cleared.
new Schema([
  new Table({
    name: 'profiles',
    columns: [
      new Column({ name: 'household_id', type: ColumnType.TEXT })
    ]
  }),
  new Table({
    name: 'grocery_items',
    columns: [
      new Column({ name: 'household_id', type: ColumnType.TEXT }),
      new Column({ name: 'created_at', type: ColumnType.TEXT }),
      new Column({ name: 'name', type: ColumnType.TEXT }),
      new Column({ name: 'comment', type: ColumnType.TEXT })
    ]
  }),
  new Table({
    name: 'groups',
    columns: [
      new Column({ name: 'household_id', type: ColumnType.TEXT }),
      new Column({ name: 'created_at', type: ColumnType.TEXT }),
      new Column({ name: 'name', type: ColumnType.TEXT }),
      new Column({ name: 'index', type: ColumnType.INTEGER }),
      new Column({ name: 'is_aisle', type: ColumnType.INTEGER })
    ]
  }),
  new Table({
    name: 'groups_grocery_items',
    columns: [
      new Column({ name: 'household_id', type: ColumnType.TEXT }),
      new Column({ name: 'group_id', type: ColumnType.TEXT }),
      new Column({ name: 'grocery_item_id', type: ColumnType.TEXT })
    ]
  })
])

After:
/**
 * This is the inferred schema of the data received by the diagnostics app.
 * Because this schema is generated on-the-fly based on the data received by the app, it can
 * be incomplete and should NOT be relied upon as a source of truth for your app schema.
 * If a table is empty, it will not be shown here.
 * Tables and columns are only added here. Nothing is removed until the database is cleared.
 */
export const profiles = new Table(
  {
    // id column (text) is automatically included
    household_id: column.text
  },
  { indexes: {} }
);

export const grocery_items = new Table(
  {
    // id column (text) is automatically included
    household_id: column.text,
    created_at: column.text,
    name: column.text,
    comment: column.text
  },
  { indexes: {} }
);

export const groups = new Table(
  {
    // id column (text) is automatically included
    household_id: column.text,
    created_at: column.text,
    name: column.text,
    index: column.integer,
    is_aisle: column.integer
  },
  { indexes: {} }
);

export const groups_grocery_items = new Table(
  {
    // id column (text) is automatically included
    household_id: column.text,
    group_id: column.text,
    grocery_item_id: column.text
  },
  { indexes: {} }
);

export const AppSchema = new Schema({
  profiles,
  grocery_items,
  groups,
  groups_grocery_items
});

export type Database = (typeof AppSchema)['types'];

LGTM,
We will want to adapt the generators from https://github.com/powersync-ja/powersync-service/blob/d8e8c4d16e649ca45ae1a95980b66f7b453835c3/packages/sync-rules/src/SchemaGenerator.ts#L5 to allow for the case of not using sync rules to generate schema, as is the case in here. This will avoid us duplicating the schema generation in future and would allow us to import the generators from @powersync/service-sync-rules.


Unknown variables in framework

https://github.com/TomographicImaging/CIL/blob/master/Wrappers/Python/cil/framework/framework.py#L2568
https://github.com/TomographicImaging/CIL/blob/master/Wrappers/Python/cil/framework/framework.py#L1828

I think the links are now out of date @epapoutsellis


[Todo] Probably want to enforce Node version

This makes it easier to not worry about rmSync stuff.

We just need to add a min version tag. 15 for example should be fine. In theory we can do down to 14.14.0 (see https://nodejs.org/api/fs.html#fs_fs_rm_path_options_callback) but we can probably just keep it to 15 so it’s simpler.

We just need to add a min version tag. 15 for example should be fine. In theory we can do down to 14.14.0 (see https://nodejs.org/api/fs.html#fs_fs_rm_path_options_callback) but we can probably just keep it to 15 so it’s simpler.


docs: Add example for Google Tag Manager's consent mode

The script in this example will result in a fine-grained consent experience for the user.
The following services can be toggled separately:

ad_storage
ad_user_data
ad_personalization
analytics_storage
functionality_storage
personalization_storage
security_storage

Preview: https://stackblitz.com/edit/stackblitz-starters-pybz8y?file=script.js
Closes #669
Related to #110

When testing with Google Tag Assistant I noticed two issues with this example:

I believe we need the brackets below when setting default consent to denied:

// Set default consent to 'denied'
gtag('consent', 'default', {
    [SERVICE_AD_STORAGE]: 'denied',
    [SERVICE_AD_USER_DATA]: 'denied',
    [SERVICE_AD_PERSONALIZATION]: 'denied',
    [SERVICE_ANALYTICS_STORAGE]: 'denied',
    [SERVICE_FUNCTIONALITY_STORAGE]: 'denied',
    [SERVICE_PERSONALIZATION_STORAGE]: 'denied',
});

I got the error "A tag read consent state before a default was set". Do you think the snippet above has to be executed before the following pre-existing code I had in the header?

gtag('js', new Date());
gtag('config', '{{ site.google_analytics }}');

@orkun1675 thx, you are totally right, fixed it.
And point 2 makes sense, try to set defaults before any other dataLayer update. I do this and don't get any errors like you have (also updated the docs about it)

This is working for me now by the way. We are using this on a live website.


How can HL modules use Titanium APIs like classic modules have?

I'm not very familiar with how classic modules work, but I think we have a set of APIs to hook into things right?
The HL modules will need these as well. But frankly, as I pointed out in #3 HL modules should be nothing more then packaged CommonJS files, which means these utilities would be available anywhere in your app via Ti.* namespace objects.

Part of this epic -> https://jira.appcelerator.org/browse/TIMOB-20417

My two cents: standard stuff from titanium should be accessed through explicit requires, not through magic namespaces.
// from
Ti.UI.createView({ });
// to
import { View } from 'titanium/ui/common';
// or
import View from 'titanium/ui/common/view';

Indeed @yuchi and from what I heard from @hansemannn that will be the case.

Hey, we will move from namespacing to require-statements.

Really?! Awesome!!!

Update on this: We will, but not in the 5.4.0 timeframe (more likely for 6.0.0). Anyway, closing this Github issue with reference to the JIRA ticket above. Thanks guys!


Background service?

Thanks for this very helpful app! I´m trying to make an app that logs the times it detects specific BT networks over the course of several days, so I need it to work in the background as well. Any idea how/if it would be possible to modify this app to have this functionality?

Sorry, I have no experiences with that. But I would be very interested in any results of your research on that!


[RFC] Refs in v4

refs in v3

refs are not included in the API documentation on material-ui.com
type declarations forbid ref e.g. <Button ref={handleRef} /> but allow innerRef for every component

The latter is wrong in some cases. E.g. The inner components of Paper or Backdrop are function components and can therefore not hold refs. It will trigger a runtime warning.

withStyles from @material-ui/core does not forward refs from ref but innerRef.

named props for refs can be problematic if multiple HOCs are used. E.g. Using styled-components to style material-ui components will prevent innerRef from being passed to the inner material-ui component.

Proposal
explicitly document ref
Couple of alternatives:

We could start by collecting what components allow this at the moment (basically every component wrapped in withStyles). This will break in v4 if #13676 is accepted
Disallow ref on all of our components. If the DOM node is required RootRef should be used.

The type declarations follow from the documentation.
ref in v4
With #13676 and the replacement of @material-ui/core/styles with @material-ui/styles as our internal styling solution <Button ref /> would return the ref to the inner component by default. In it's current state some components would loose the ability to hold ref (e.g. AppBar).
There is an argument to be made that ref inherently cause all sorts of problems in general. However I would argue that all of our components return some html element. Interfacing our components so that they only return a ref to an abstract HTMLElement should therefore cause no problems in the future. It doesn't bind us to any specific implementation (other than a html element). This allows e.g. layout computations if necessary or imperative focus which is what we use internally.
I'm in favor of enabling refs on all of our existing components. It allows for a simpler API where one doesn't have to look at the API docs for every single component and check if it allows ref (and then be annoyed because we didn't have your use case in mind). (Although TS users can be lazy and don't have to check API docs).
Summary

Add explicit documentation
Fix typescript declarations
One of

Disallow ref on all components i.e. use ref at your own risk. Prefer RootRef if you need the DOM node

RootRef will trigger findDOMNode deprecation warnings

Don't change ref behavior for inner components. ref behavior on actual component will change though with #13676.

users are stuck with findDOMNode

Allow ref on all components.  RootRef should be already deprecated since it's warned in strict mode. When fragment refs will come we can't know.

Allow ref on all components. This is the best solution from a DX perspective. I think that we should aim for it, as long as, the UX implications are acceptable. For instance, does it requires us to increase the bundle size? Does is slow the components down? I don't have the answer to these questions. I only know that using forwardRef on a class component is a "nightmare" but we can build tools around the problem. What if we move all our components to functions leveraging hooks?

That would awesome. Will force users to upgrade react though.
We could also use hooks to get rid from cloneElement which failed for me a few times when I need to wrap MenuItem.

For instance, does it requires us to increase the bundle size?

Yes, because we have to use React.forwardRef. However the increased bundle size should be very small. gzip might result in a O(1) increase instead of O(n) where n is the number of components affected.

Does is slow the components down?

The more interesting question would be "by how much" since any added behavior slows it down. I remember a discussion about this concerning forwardRef in react-redux and the gist was basically that while it was slower it wasn't because of forwardRef itself but because any additional component is slower. However this is a micro-benchmark fallacy. Sure you slow down one specific code path down by e.g. 100% but if that codepath only takes up .001% of your hole app then the micro benchmark is not that interesting.

I only know that using forwardRef on a class component is a "nightmare" but we can build tools around the problem.

I had the same issue originally but I'm comfortable with it now. The fact that ref is not treated like other props is what makes this hard. There is an initial learning curve where you have to switch from "a component has props" and "a component has props and a ref which is not visible in the syntax". I don't think this pattern is that complicated: Declare a forwardRef on the class component, use forwardRef((props, ref) => <ClassComponent forwardRef={ref} {...props} />).

What if we move all our components to functions leveraging hooks?

What do hooks solve WRT to ref? Function components would result in "no ref allowed".

@eps1lon
export const menu = React.forwardRef((props, ref) => {
  const [open, setOpen] = React.useState(false);
  return (

 setOpen(v => !v)}>{open ? 'open' : 'closed'}

  );
});

@eps1lon
export const Menu = React.forwardRef((props, ref) => {
  const [open, setOpen] = React.useState(false);
  return (

 setOpen(v => !v)}>{open ? 'open' : 'closed'}

  );
});

What do hooks solve here with regards to ref?

You don't need additional component wrapper like with classes.

You don't need additional component wrapper like with classes.

The usage of state in your example is arbitrary. You picked one feature of classes this.setState and ported this to hooks. However, you did non show me how hooks solved any issue related to ref.
Try not to think about any specific implementation detail here. The original point

What if we move all our components to functions leveraging hooks?

did not adress the concern of this issue. We already have function components that do not use hooks. What could you achieve with hooks in function components that solves any issue with refs?
For example using <AppBar innerRef /> will result in a runtime warning because the inner component is a function component. In v3 you could however do <AppBar ref /> because the inner component is wrapped in withStyles which wraps the inner component in a class component. If we forward ref in withStyles then users don't have the possibility to get the DOM node without using RootRef. That is what I wanted to adress: Do we forward ref on the inner component too or do we not allow ref on our components (or do we revert ref forwarding in withStyles and force users to use findDOMNode).

My point was that hooks works inside of forwardRef function. So performance case you mentioned before is solved.
I believe you won't use withStyles with hooks. You already have better tools. So infinite components wrappers will gone. This should also increase performance.
You will not end up with runtime warning. That's the point of forwardRef, isn't it?

My point was that hooks works inside of forwardRef function. So performance case you mentioned before is solved.

Probably if we actually port class components to function components. My statement was only made in the context of "given a component: what perf implication has forwardRef" which especially applies to already existing function components. But I think you implied that we wouldn't wrap those in withStyles but rather use the hooks implementation so the perf might even out. Again: Probably but let's discuss one issue at a time. We didn't officially determine if we want to bump the peer dependency on react to 16.8.

I believe you won't use withStyles with hooks. You already have better tools. So infinite components wrappers will gone. This should also increase performance.

Maybe. I haven't seen any benchmarks comparing class components with function components + hooks though.

You will not end up with runtime warning. That's the point of forwardRef, isn't it?

Exactly. This is the discussion we have: use forwardRef or not.

I have looked into the performance implication, TL;DR it's not significative. I have tried the following:

function NakedButton(props) {
  return <button type="button" {...props} />;
}

class HocButton extends React.Component {
  state = {};

  render() {
    return <NakedButton {...this.props} />;
  }
}

const ForwardRef = React.forwardRef((props, ref) => <button ref={ref} type="button" {...props} />);

suite
  .add('NakedButton', () => {
    ReactDOMServer.renderToString(<NakedButton>Material-UI</NakedButton>);
  })
  .add('ForwardRef', () => {
    ReactDOMServer.renderToString(<ForwardRef>Material-UI</ForwardRef>);
  })
  .add('HocButton', () => {
    ReactDOMServer.renderToString(<HocButton>Material-UI</HocButton>);
  })
  .on('cycle', event => {
    console.log(String(event.target));
  })
  .run();

It outpus the following:

NakedButton x 738,357 ops/sec ±3.19% (187 runs sampled)
ForwardRef x 674,264 ops/sec ±1.22% (188 runs sampled)
HocButton x 423,407 ops/sec ±2.22% (181 runs sampled)

I have looked at how styled-components handles the problem. They don't: https://github.com/styled-components/styled-components/issues/2154.

const StyledButton = styled(Button)`
  background: linear-gradient(45deg, #fe6b8b 30%, #ff8e53 90%);
  border-radius: 3px;
  border: 0;
  color: white;
  height: 48px;
  padding: 0 30px;
  box-shadow: 0 3px 5px 2px rgba(255, 105, 135, 0.3);
`;

It outputs:

I have looked at how styled-components handles the problem. They don't: styled-components/styled-components#2154.

That's not how I read the log. The ref points to the material-ui Button. This is what you expect from a component that forwards ref. They do forward refs.

I have looked into the performance implication. It's not significant.

Good to know.

Do we have any concern left to address?

Do we have any concern left to address?

I don't think so. My primary concern was to consolidate all the concerns (and gather my own thoughts). If we implement this then #13394 should be resolved completely. Just looks better on paper to be "strict mode compliant".

💥


stdenv/linux/default.nix: add gcc rebuild during bootstrap

Description of changes
A bit of background how set of changes came to be with more words: https://trofi.github.io/posts/275-nixpkgs-bootstrap-deep-dive.html
before the change our rebuild chain was roughly the followin:

bootstrapTools install
binutils built by bootstrapTools
glibc built by bootstrapTools
gcc built by bootstrapTools

As a result glibc contained code generated by bootstrapTools's
gcc. And (what worse) copied libgcc_s.so.1 as is from it.
Such libgcc_s.so.1 was not compatible with nixpkgs gcc at
least on aarch64 where newer libgcc is expected to have new symbols.
As a result linking failed as:
ld: /build/test.o: in function `foo(int)':
test.cpp:(.text+0x2c): undefined reference to `__aarch64_ldadd4_acq_rel'
collect2: error: ld returned 1 exit status

The change rejigs rebuild sequence as:

bootstrapTools install
binutils built by bootstrapTools
gcc built by bootstrapTools
glibc built by gcc
gcc built by gcc

As a result glibc gets built by fresher gcc and embeds a copy
of libgcc_s.so.1.
To make it work as is I had to use a --sysroot= to evade
accidental pull of system headers embedded by gcc.
While at it decreased rebuild counts for bash and xz. These are
simple tools we can use as is built by bootstrapTools until we
build final stage.
List of packages with more than one build before the change:
$ nix-store --query --graph $(nix-instantiate -A stdenv) |
    grep -P " -> " | awk '{print $3}' | sort -u | sed 's/"[0-9a-z]\{32\}-/"/g' |
    sort | uniq -c | sort -n | awk '$1 > 1'

  2 "autoconf-2.71.drv"
  2 "automake-1.16.5.drv"
  2 "bootstrap-stage1-stdenv-linux.drv"
  2 "bootstrap-stage2-stdenv-linux.drv"
  2 "bootstrap-stage4-stdenv-linux.drv"
  2 "bzip2-1.0.8.drv"
  2 "file-5.43.drv"
  2 "help2man-1.49.2.drv"
  2 "hook.drv"
  2 "libtool-2.4.7.drv"
  2 "patchelf-0.15.0.drv"
  2 "perl5.36.0-gettext-1.07.drv"
  3 "binutils-2.39.drv"
  3 "binutils-wrapper-2.39.drv"
  3 "expand-response-params.drv"
  3 "gettext-0.21.drv"
  3 "libxcrypt-4.4.33.drv"
  3 "perl-5.36.0.drv"
  3 "texinfo-6.8.drv"
  3 "zlib-1.2.13.drv"
  4 "bash-5.1-p16.drv"
  4 "xz-5.2.9.drv"

List of packages with more than one build after the change:
$ nix-store --query --graph $(nix-instantiate -A stdenv) |
    grep -P " -> " | awk '{print $3}' | sort -u | sed 's/"[0-9a-z]\{32\}-/"/g' |
    sort | uniq -c | sort -n | awk '$1 > 1'

  2 "autoconf-2.71.drv"
  2 "automake-1.16.5.drv"
  2 "bash-5.2-p15.drv"
  2 "bootstrap-stage1-stdenv-linux.drv"
  2 "bootstrap-stage2-stdenv-linux.drv"
  2 "bootstrap-stage3-stdenv-linux.drv"
  2 "bootstrap-stage5-stdenv-linux.drv"
  2 "bzip2-1.0.8.drv"
  2 "file-5.43.drv"
  2 "gcc-11.3.0.drv"
  2 "gmp-with-cxx-6.2.1.drv"
  2 "help2man-1.49.2.drv"
  2 "hook.drv"
  2 "libtool-2.4.7.drv"
  2 "mpfr-4.1.1.drv"
  2 "patchelf-0.15.0.drv"
  2 "perl5.36.0-gettext-1.07.drv"
  2 "texinfo-6.8.drv"
  2 "which-2.21.drv"
  2 "xz-5.4.0.drv"
  3 "perl-5.36.0.drv"
  4 "binutils-2.39.drv"
  4 "binutils-wrapper-2.39.drv"
  4 "expand-response-params.drv"
  4 "gettext-0.21.drv"
  4 "libxcrypt-4.4.33.drv"
  4 "zlib-1.2.13.drv"

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Sorry, last minute "fix" slipped in and there is a one onwanted reference to bootstrap tools that fails the stdenv build. The fix is one-liner and will not rebuild too much if you already tried the branch:
--- a/pkgs/stdenv/linux/default.nix
+++ b/pkgs/stdenv/linux/default.nix
@@ -459,7 +459,8 @@ in
       # other purposes (binutils and top-level pkgs) too.
       inherit (prevStage)
           bash gettext gnum4 bison
-          perl texinfo which xz
+          perl texinfo which
+          # xz - rebuild to pull in rebuilt version into final stdenv
           zlib linuxHeaders libidn2 libunistring;

       ${localSystem.libc} = getLibc prevStage;

I'll push it in a few minutes along with comment updates.

I'll push it in a few minutes along with comment updates.

Pushed the update. Was able to build stdenv on x86_64 and a bunch of c++ packages on top of it.
Should be ready to test.

Currently attempting to bootstrap my entire server from this branch, will see if it builds.

I'm rebuilding my entire aarch64 test VM with a pretty standard Xfce + Firefox on this branch. Then I will try again with GCC 11.

Aaaand the server config is built, with one unrelated failure (https://github.com/NixOS/nixpkgs/pull/209118). Time for the desktop, and time for me to go get some sleep.

The desktop build ended up running out of disk space, but it got almost all the way there. I think this should really be good to go. Has anyone tested aarch64 with GCC 11 yet?

Does this still not violate the stance that none of the files in bootstrap or generated by bootstrap tools are not used directly?

Got another snag: vte (uses clangStdenv) failed for inability to dins libstdc++ librray. It's a side-effect of --sysroot= hack. Added a safe workaround for it to provide cross-looking gcc.lib paths with:
--- a/pkgs/development/compilers/gcc/11/default.nix
+++ b/pkgs/development/compilers/gcc/11/default.nix
@@ -221,6 +221,7 @@ stdenv.mkDerivation ({
   };

   targetConfig = if targetPlatform != hostPlatform then targetPlatform.config else null;
+  targetPlatformConfig = targetPlatform.config;

   buildFlags = optional
     (targetPlatform == hostPlatform && hostPlatform == buildPlatform)
diff --git a/pkgs/development/compilers/gcc/builder.sh b/pkgs/development/compilers/gcc/builder.sh
index 113bd83ea53..71a997a7df9 100644
--- a/pkgs/development/compilers/gcc/builder.sh
+++ b/pkgs/development/compilers/gcc/builder.sh
@@ -203,6 +203,17 @@ preInstall() {
         ln -s lib "$out/${targetConfig}/lib32"
         ln -s lib "${!outputLib}/${targetConfig}/lib32"
     fi
+
+    # cc-wrappers uses --sysroot=/nix/dir/does/not/exist as a way to
+    # drop default sysheaders search path. Unfortunately that switches
+    # clang++ into searching libraries in gcc in cross-compiler paths:
+    #   from ${!outputLib}/lib (native)
+    #   to ${!outputLib}/${targetPlatformConfig}/lib
+    # We create the symlink to make both native and cross paths
+    # available even if the toolchain is not the cross-compiler.
+    if [ ! -e ${!outputLib}/${targetPlatformConfig} ] ; then
+        ln -s . ${!outputLib}/${targetPlatformConfig}
+    fi
 }

Applied to all gcc versions. It fixes vte and other clang-based builds. Did not seem to break pkgsLLVM or cross-builds either.

Another snag: /nix/dir/does/not/exist trips purity check of darwin compiler wrapper. Switched to /nix/store for now.

My system rebuild got stopped overnight by the vte issue. I have not tried GCC 11 yet but plan to.

Firefox does not build, but the whole rest of the system does build and (appear to) work with GCC 9.
Firefox build excerpt:
/nix/store/va8d5ymiwr61bsx3p9rmdirg5r17gv52-wasm32-unknown-wasi-clang-wrapper-12.0.1/bin/wasm32-unknown-wasi-c++ -Qunused-arguments -std=gnu++17 --sysroot=/nix/store/fpflf31ij31lb05m0hwjzqjh5ypyz3yc-wasi-sysroot -o woff2_common.wasm -c -I/build/firefox-108.0.1/security/rlbox -I/build/firefox-108.0.1/mozobj/security/rlbox -I/build/firefox-108.0.1/third_party/wasm2c/wasm2c -I/build/firefox-108.0.1/extensions/spellcheck/hunspell/glue -I/build/firefox-108.0.1/extensions/spellcheck/hunspell/src -I/build/firefox-108.0.1/gfx/graphite2/src -I/build/firefox-108.0.1/media/libogg -I/build/firefox-108.0.1/parser/expat/lib -I/build/firefox-108.0.1/modules/woff2/include -I/build/firefox-108.0.1/mozobj/dist/include -I/nix/store/x6f22yq0wybxhsq7ginmb054cgqylv4z-nspr-4.35-dev/include -I/nix/store/x6f22yq0wybxhsq7ginmb054cgqylv4z-nspr-4.35-dev/include -I/nix/store/l06kidribbnrwyh8slz7a6ysbf0nhd3s-nss-3.86-dev/include/nss -I/build/firefox-108.0.1/mozobj/dist/include/nss -I/nix/store/bxxq3qnkms0y3q18pd2xxiz1lkymd27v-libpng-apng-1.6.39-dev/include/libpng16 -I/nix/store/wb3zs29jznhqd355jnhp2wh69961jqcx-zlib-1.2.13-dev/include -I/nix/store/3ipg6q8gkyrm3xz3k808rf4gnz32v8sc-pixman-0.42.2/include/pixman-1 -g -Os -fno-exceptions -fno-strict-aliasing -DNDEBUG=1 -DTRIMMED=1 -DMOZILLA_CLIENT -D_WASI_EMULATED_PROCESS_CLOCKS -DMOZ_IN_WASM_SANDBOX '-DPACKAGE_VERSION="moz"' '-DPACKAGE_BUGREPORT="http://bugzilla.mozilla.org/"' -DGRAPHITE2_STATIC -DGRAPHITE2_NFILEFACE -DGRAPHITE2_NTRACING -DGRAPHITE2_NSEGCACHE -DHAVE_EXPAT_CONFIG_H -MD -MP -MF .deps/woff2_common.wasm.pp   /build/firefox-108.0.1/modules/woff2/src/woff2_common.cc
security/rlbox/woff2_dec.wasm
/nix/store/va8d5ymiwr61bsx3p9rmdirg5r17gv52-wasm32-unknown-wasi-clang-wrapper-12.0.1/bin/wasm32-unknown-wasi-c++ -Qunused-arguments -std=gnu++17 --sysroot=/nix/store/fpflf31ij31lb05m0hwjzqjh5ypyz3yc-wasi-sysroot -o woff2_dec.wasm -c -I/build/firefox-108.0.1/security/rlbox -I/build/firefox-108.0.1/mozobj/security/rlbox -I/build/firefox-108.0.1/third_party/wasm2c/wasm2c -I/build/firefox-108.0.1/extensions/spellcheck/hunspell/glue -I/build/firefox-108.0.1/extensions/spellcheck/hunspell/src -I/build/firefox-108.0.1/gfx/graphite2/src -I/build/firefox-108.0.1/media/libogg -I/build/firefox-108.0.1/parser/expat/lib -I/build/firefox-108.0.1/modules/woff2/include -I/build/firefox-108.0.1/mozobj/dist/include -I/nix/store/x6f22yq0wybxhsq7ginmb054cgqylv4z-nspr-4.35-dev/include -I/nix/store/x6f22yq0wybxhsq7ginmb054cgqylv4z-nspr-4.35-dev/include -I/nix/store/l06kidribbnrwyh8slz7a6ysbf0nhd3s-nss-3.86-dev/include/nss -I/build/firefox-108.0.1/mozobj/dist/include/nss -I/nix/store/bxxq3qnkms0y3q18pd2xxiz1lkymd27v-libpng-apng-1.6.39-dev/include/libpng16 -I/nix/store/wb3zs29jznhqd355jnhp2wh69961jqcx-zlib-1.2.13-dev/include -I/nix/store/3ipg6q8gkyrm3xz3k808rf4gnz32v8sc-pixman-0.42.2/include/pixman-1 -g -Os -fno-exceptions -fno-strict-aliasing -DNDEBUG=1 -DTRIMMED=1 -DMOZILLA_CLIENT -D_WASI_EMULATED_PROCESS_CLOCKS -DMOZ_IN_WASM_SANDBOX '-DPACKAGE_VERSION="moz"' '-DPACKAGE_BUGREPORT="http://bugzilla.mozilla.org/"' -DGRAPHITE2_STATIC -DGRAPHITE2_NFILEFACE -DGRAPHITE2_NTRACING -DGRAPHITE2_NSEGCACHE -DHAVE_EXPAT_CONFIG_H -MD -MP -MF .deps/woff2_dec.wasm.pp   /build/firefox-108.0.1/modules/woff2/src/woff2_dec.cc
security/rlbox/woff2_out.wasm
/nix/store/va8d5ymiwr61bsx3p9rmdirg5r17gv52-wasm32-unknown-wasi-clang-wrapper-12.0.1/bin/wasm32-unknown-wasi-c++ -Qunused-arguments -std=gnu++17 --sysroot=/nix/store/fpflf31ij31lb05m0hwjzqjh5ypyz3yc-wasi-sysroot -o woff2_out.wasm -c -I/build/firefox-108.0.1/security/rlbox -I/build/firefox-108.0.1/mozobj/security/rlbox -I/build/firefox-108.0.1/third_party/wasm2c/wasm2c -I/build/firefox-108.0.1/extensions/spellcheck/hunspell/glue -I/build/firefox-108.0.1/extensions/spellcheck/hunspell/src -I/build/firefox-108.0.1/gfx/graphite2/src -I/build/firefox-108.0.1/media/libogg -I/build/firefox-108.0.1/parser/expat/lib -I/build/firefox-108.0.1/modules/woff2/include -I/build/firefox-108.0.1/mozobj/dist/include -I/nix/store/x6f22yq0wybxhsq7ginmb054cgqylv4z-nspr-4.35-dev/include -I/nix/store/x6f22yq0wybxhsq7ginmb054cgqylv4z-nspr-4.35-dev/include -I/nix/store/l06kidribbnrwyh8slz7a6ysbf0nhd3s-nss-3.86-dev/include/nss -I/build/firefox-108.0.1/mozobj/dist/include/nss -I/nix/store/bxxq3qnkms0y3q18pd2xxiz1lkymd27v-libpng-apng-1.6.39-dev/include/libpng16 -I/nix/store/wb3zs29jznhqd355jnhp2wh69961jqcx-zlib-1.2.13-dev/include -I/nix/store/3ipg6q8gkyrm3xz3k808rf4gnz32v8sc-pixman-0.42.2/include/pixman-1 -g -Os -fno-exceptions -fno-strict-aliasing -DNDEBUG=1 -DTRIMMED=1 -DMOZILLA_CLIENT -D_WASI_EMULATED_PROCESS_CLOCKS -DMOZ_IN_WASM_SANDBOX '-DPACKAGE_VERSION="moz"' '-DPACKAGE_BUGREPORT="http://bugzilla.mozilla.org/"' -DGRAPHITE2_STATIC -DGRAPHITE2_NFILEFACE -DGRAPHITE2_NTRACING -DGRAPHITE2_NSEGCACHE -DHAVE_EXPAT_CONFIG_H -MD -MP -MF .deps/woff2_out.wasm.pp   /build/firefox-108.0.1/modules/woff2/src/woff2_out.cc
security/rlbox/rlbox.wasm
rm -f rlbox.wasm
/nix/store/va8d5ymiwr61bsx3p9rmdirg5r17gv52-wasm32-unknown-wasi-clang-wrapper-12.0.1/bin/wasm32-unknown-wasi-c++ -Qunused-arguments -std=gnu++17 --sysroot=/nix/store/fpflf31ij31lb05m0hwjzqjh5ypyz3yc-wasi-sysroot -o rlbox.wasm -Wl,--export-all -Wl,--stack-first -Wl,-z,stack-size=262144 -Wl,--no-entry -Wl,--growable-table ogg_alloc.wasm ogg_bitwise.wasm ogg_framing.wasm xmlparse.wasm xmlrole.wasm xmltok.wasm wasm2c_sandbox_wrapper.wasm mozHunspellRLBoxSandbox.wasm affentry.wasm affixmgr.wasm csutil.wasm hashmgr.wasm hunspell.wasm phonet.wasm replist.wasm suggestmgr.wasm GraphiteExtra.wasm CmapCache.wasm Code.wasm Collider.wasm Decompressor.wasm Face.wasm FeatureMap.wasm FileFace.wasm Font.wasm GlyphCache.wasm GlyphFace.wasm Intervals.wasm Justifier.wasm NameTable.wasm Pass.wasm Position.wasm Segment.wasm Silf.wasm Slot.wasm Sparse.wasm TtfUtil.wasm UtfCodec.wasm call_machine.wasm gr_char_info.wasm gr_face.wasm gr_features.wasm gr_font.wasm gr_logging.wasm gr_segment.wasm gr_slot.wasm json.wasm RLBoxWOFF2Sandbox.wasm table_tags.wasm variable_length.wasm woff2_common.wasm woff2_dec.wasm woff2_out.wasm -lwasi-emulated-process-clocks
wasm-ld: error: unable to find library -lc++abi
wasm-ld: error: unable to find library -lc++
wasm-ld: error: unable to find library -lc++abi
clang-12: error: linker command failed with exit code 1 (use -v to see invocation)

Firefox build log:
ff_fail_log.txt.gz
Now to try with GCC 11, then to do stdenv build time comparisons.

Firefox does not build, but the whole rest of the system does build and (appear to) work with GCC 9.
Firefox build excerpt:

Reproduced the same failure on x86_64-linux. Probably another --sysroot= casualty. Looking at it to figure why.

Confirmed that --sysroot=... hack is too late in cc flags: it clobbers wasi --sysroot=/. testing the following locally:
--- a/pkgs/build-support/cc-wrapper/default.nix
+++ b/pkgs/build-support/cc-wrapper/default.nix
@@ -335,13 +335,15 @@ stdenv.mkDerivation {
     # We use --sysroot=/nix/store/does/not/exist to drop embedded default
     # path to glibc headers gcc was built against. Without it -idirafter
     # only appends to the list and outdated glibc headers end up being
-    # used.
+    # used. 'cc-cflags-before' is used to allow user's --sysroot= option
+    # to override our default.
     + optionalString (libc != null) (''
       touch "$out/nix-support/libc-cflags"
       touch "$out/nix-support/libc-ldflags"
       echo "-B${libc_lib}${libc.libdir or "/lib/"}" >> $out/nix-support/libc-crt1-cflags
     '' + optionalString (!(cc.langD or false)) ''
-      echo "--sysroot=/nix/store/does/not/exist -idirafter ${libc_dev}${libc.incdir or "/include"}" >> $out/nix-support/libc-cflags
+      echo "--sysroot=/nix/store/does/not/exist" >> $out/nix-support/cc-cflags-before
+      echo "-idirafter ${libc_dev}${libc.incdir or "/include"}" >> $out/nix-support/libc-cflags
     '' + optionalString (isGNU && (!(cc.langD or false))) ''
       for dir in "${cc}"/lib/gcc/*/*/include-fixed; do
         echo '-idirafter' ''${dir} >> $out/nix-support/libc-cflags

Will update the branch once confirm it fixes firefox for me. Will be another world rebuild unfortunately.

Rebuilt my aarch64-linux system with GCC 11 by just removing the exception from all-packages.nix and everything built properly and seems to basically work. I did not try Firefox again. No linker errors or mysterious runtime failures. Seems this does in fact fix the issue.

I would like it if theoretically bootstrapTools wouldn't need any further updates except for maybe adding new platforms. I also very dislike anything from bootstrap leaking into the final closure (like libgcc_s). So I find the idea of this very important. How exactly it should be done implementation-wise I don't really mind.

I would like it if theoretically bootstrapTools wouldn't need any further updates except for maybe adding new platforms.

Is it practical though? Even today we are using gcc-8 (went out of upstream support last year) to build latest and greatest binutils, gcc and friends. What do we do if gcc-8 starts generating something strange on fresher glibc? Say, for fancy new __int128 type. backport fixes and features back to gcc-8 source release? Do you even want to debug years old compiler just to get something fresh and working? I would say having reasonably fresh (1-2 years old) bootstrapTools is an inevitable requirement unless we want to build a chain of compilers within a few releases from one another to get to the fresh state auto of the very old bootstrapTools (and then manage these compiler chains on a per-architecture basis).

Ideally we'd probably do what Guix is doing, which is bootstrap from a very simple handwritten compiler to a chain of GCCs to the most recent one, but that's definitely not something we can just pull out of a hat right now.

On my 8 core aarch64-linux VM running 2 jobs in parallel, building the entire stdenv from scratch with no substitutions takes about 40% longer wallclock time with the latest edition of this PR (6e67eab4) as compared to the staging commit it's based on (4acd5a97). This is not lovely, but doesn't sound immediately unreasonable to me. That is ~55 and ~40 minutes respectively.
I don't know how this ratio might change when looking at an arbitrary change which requires a stdenv rebuild.
# for each commit above, with system running on nixos-22.11 branch
# get rid of stdenv
nix-collect-garbage
# download sources
nix-build -A pkgs.stdenv --dry-run --option substituters '' 2>&1 | grep '\.tar' | xargs nix-build
# build stdenv
time nix-build -A pkgs.stdenv --option substituters '' -j2

No functional change from previous iteration. Factored out --sysroot= hack into a separate commit and also proposed as https://github.com/NixOS/nixpkgs/pull/210004

A bit of background how set of changes came to be with more words if you are into that kind of thing: https://trofi.github.io/posts/275-nixpkgs-bootstrap-deep-dive.html

Restores excessive rebuilds of xz, texinfo and friends to make main commit leaner. Will reintroduce it separately this evening.

Restores excessive rebuilds of xz, texinfo and friends to make main commit leaner. Will reintroduce it separately this evening.

Proposed 2 of 3 perl builds removal separately in https://github.com/NixOS/nixpkgs/pull/210118

Proposed one texinfo removal from binutils as https://github.com/NixOS/nixpkgs/pull/210132

Filed upstream bug for poor build parallelism in gettext: https://savannah.gnu.org/bugs/index.php?63641

No functional change. No need to retest.
Rebased against current staging. This left us with a single gcc stage addition change (if we ignore https://github.com/NixOS/nixpkgs/pull/210741)
While at it added missing minor rename:
--- a/pkgs/stdenv/linux/default.nix
+++ b/pkgs/stdenv/linux/default.nix
@@ -415,7 +415,7 @@ in
       gcc-unwrapped =
         let makeStaticLibrariesAndMark = pkg:
               lib.makeOverridable (pkg.override { stdenv = self.makeStaticLibraries self.stdenv; })
-                .overrideAttrs (a: { pname = "${a.pname}-stage3"; });
+                .overrideAttrs (a: { pname = "${a.pname}-stage4"; });
         in super.gcc-unwrapped.override {
         # Link GCC statically against GMP etc.  This makes sense because
         # these builds of the libraries are only used by GCC, so it
@@ -466,7 +466,7 @@ in
       # force gmp to rebuild so we have the option of dynamically linking
       # libgmp without creating a reference path from:
       #   stage5.gcc -> stage4.coreutils -> stage3.glibc -> bootstrap
-      gmp = lib.makeOverridable (super.gmp.override { stdenv = self.stdenv; }).overrideAttrs (a: { pname = "${a.pname}-stage4"; });
+      gmp = lib.makeOverridable (super.gmp.override { stdenv = self.stdenv; }).overrideAttrs (a: { pname = "${a.pname}-stage5"; });

       # To allow users' overrides inhibit dependencies too heavy for
       # bootstrap, like guile: https://github.com/NixOS/nixpkgs/issues/181188

@trofi I guess this PR is now superseded by #210741?

@trofi I guess this PR is now superseded by #210741?

No, this PR includes that one and adds a gcc rebuild stage step.

Rebased past gettext conflict.

@ofborg build icu

@trofi should we just close this as superseded maybe?

Sure. I did not yet get the feedback from @Ericson2314 and I think it's a prerequisite. If the PR languished for a month and problem persisted for as long as I used nixpkgs it can wait for another year.
You will need another form of the  https://github.com/NixOS/nixpkgs/issues/208412
Not the first time I (and others) threw away most of the effort around the bootstrap bootstrap improvements away. I can keep the patch locally indefinitely. I already have about 200 of them :)

All I meant was that https://github.com/NixOS/nixpkgs/pull/209870 achieves the same thing and seems to be a better approach. Are there issues with that change that don't occur with this one?

Good question. I don't know. It's too hard for me to reason about it as a whole. I usually diff before/after state to see the difference. It's very hard to do there.

I usually diff before/after state to see the difference. It's very hard to do there.

I initially tried factoring it into multiple PRs, but ended up getting "i don't see the point of this" feedback for the parts that were not useful separately from external-bootstrap (example).  So to avoid time being wasted on those unproductive discussions I put it all under a single PR.


[WIP] Custom Social Icons POC

What kind of changes does this PR include?

Changes to Starlight code

Description

Add prettier config
POC for a custom "socials" type to allow a user to set a custom icon for the socials part of the header.

I'd still have some refinements to the types and the way the SocialIcon component uses those types, but wanted to at least get a POC up to see if this is something you all would like to add in before I spent time on it ;)

Closing this one out in favour of the exciting component customisation system coming soon ™ ;)


fix: explicitly request that git log does not use a pager

Summary
Resolves #2161
Short checklist

[x] Did you run bundle exec rubocop -a to fix linter issues?
[x] If relevant, did you create a test?
[x] Did you confirm that the RSpec tests pass?

Other Information
I wanted to write a feature test for this but couldn't come up with a way to have git use a pager in a manner that didn't cut across the actual tests themselves - I plan to have a second crack at this later this week, but don't think that should block landing the fix.

Looks great!
@digitalmoksha can you confirm this branch works for you?

@mattbrictson yep, it's working for me!
Thanks @G-Rath 🚀

Merged! I'll release 3.19.1 with the fix later today or tomorrow.

Released: https://github.com/capistrano/capistrano/releases/tag/v3.19.1


add / remove users to a room

What?
You can add and remove users by their id to a room
Why?
I'll cover all functions from the official docs in this sdk.

Thanks @mludi !
I'll get this merged now and push a new release later today 👍

Hey @hamchapman nice, thank you 👍


Reversions for new theme switcher approach

Taking another approach that relies on CSS Custom Properties and keeping the atomic classes, which will keep the code easier to maintain and require less change.

Remove theme-based partial folders
Add back mono color variables

See this article for reference.


[performance] 200B win opportunity, clsx/lite

Summary 💡
There seems to be an opportunity to shave bundle size with https://github.com/lukeed/clsx/releases/tag/v2.1.0
The full version looks bigger by about this amount<EMAIL_ADDRESS>For the source, we already have https://www.npmjs.com/package/babel-plugin-optimize-clsx that rewrites everything, so maybe the solution could be executed there.
Examples 🌈
No response
Motivation 🔦
No response

Hey y'all 👋🏼 I have opened a pull request to the babel-plugin-optimize-clsx package. The tests were all passing which is a good sign, let's see what happens on that pull request - if so, hopefully we can merge this and update it on the MUI side 🚀
https://github.com/merceyz/babel-plugin-optimize-clsx/pull/23


[Bug]: My inner workflow faces a timeout error

Bug Description
Despite specifying a generous timeout limit for all my workflows, my inner workflow of my nested workflow always times out within 10 seconds once deployed on LlamaDeploy.

This error did not exist on Jupyter Notebooks
When I pinged the inner workflow directly using LlamaDeployClient I get no timeout error

Which means that the timeout error only happens when I ping the outer workflow which then routes to the inner workflow.
Version
0.11.17
Steps to Reproduce
Deployment of core
from llama_deploy import (
    deploy_core,
    ControlPlaneConfig,
)
from llama_deploy.message_queues.simple import (
    SimpleMessageQueueConfig
)
control_plane_config = ControlPlaneConfig()
message_queue_config = SimpleMessageQueueConfig()

if __name__ == "__main__":
    import asyncio

    asyncio.run(
        deploy_core(
            control_plane_config = control_plane_config,
            message_queue_config=message_queue_config,
            disable_control_plane=False,
            disable_message_queue=False,
        )
    )

Deployment of workflow
from llama_index.core.workflow import Workflow, StartEvent, StopEvent, step
import asyncio
from llama_deploy import (
    WorkflowServiceConfig,
    ControlPlaneConfig,
    deploy_workflow,
)

class InnerWorkflow(Workflow):
    @step()
    async def run_step(self, ev: StartEvent) -> StopEvent:
        result = ... #some long step here
        return StopEvent(result=str(result))

class OuterWorkflow(Workflow):
    @step()
    async def run_step(
        self, ev: StartEvent, inner: InnerWorkflow
    ) -> StopEvent:
        arg1 = ev.get("arg1")
        result = await inner.run(arg1=arg1)

        return StopEvent(result=str(result))

inner = InnerWorkflow(timeout=1200.0,verbose=True)
outer = OuterWorkflow(timeout=6000.0, verbose = True)
outer.add_workflows(inner=inner)

async def main():
    inner_task = asyncio.create_task(
        deploy_workflow(
            inner,
            WorkflowServiceConfig(
                host="<IP_ADDRESS>", port=8003, service_name="inner"
            ),
            ControlPlaneConfig(),
        )
    )

    outer_task = asyncio.create_task(
        deploy_workflow(
            outer,
            WorkflowServiceConfig(
                host="<IP_ADDRESS>", port=8002, service_name="outer"
            ),
            ControlPlaneConfig(),
        )
    )

    await asyncio.gather(inner_task, outer_task)

if name == "main":
    import asyncio
    asyncio.run(main())

Calling of workflow
from llama_deploy import AsyncLlamaDeployClient, ControlPlaneConfig
control_plane_config = ControlPlaneConfig()
client = AsyncLlamaDeployClient(control_plane_config)
session = await client.create_session()
result = await session.run(
   "outer"
   query = ...
)

Relevant Logs/Tracbacks
Traceback (most recent call last):
  File "/app/tlim2/anaconda3/envs/llamaindex/lib/python3.12/site-packages/llama_index/core/instrumentation/dispatcher.py", line 281, in handle_future_result
    result = future.result()
             ^^^^^^^^^^^^^^^
  File "/app/tlim2/anaconda3/envs/llamaindex/lib/python3.12/site-packages/llama_index/core/workflow/workflow.py", line 380, in _run_workflow
    raise WorkflowTimeoutError(msg)
llama_index.core.workflow.errors.WorkflowTimeoutError: Operation timed out after 10.0 seconds

Hey @tituslhy thanks for reporting the issue, I'll try to reproduce and will update the issue with my findings.

@tituslhy your code isn't working for me, the workflow seems to run but there's the error ValueError: No consumer for 'outer' has been registered.  buried in the logs.
So I added an await asyncio.sleep(3) in-between the two tasks spawning the inner and outer workflows like this:
async def main():
    inner_task = asyncio.create_task(
        deploy_workflow(
            inner,
            WorkflowServiceConfig(
                host="<IP_ADDRESS>", port=8003, service_name="inner"
            ),
            ControlPlaneConfig(),
        )
    )

    await asyncio.sleep(3)

    outer_task = asyncio.create_task(
        deploy_workflow(
            outer,
            WorkflowServiceConfig(
                host="<IP_ADDRESS>", port=8002, service_name="outer"
            ),
            ControlPlaneConfig(),
        )
    )

    await asyncio.gather(inner_task, outer_task)

and this seems to work for me, can you try adding the sleep and let me know?

Hmm it's actually the same example as the nested workflows example here: https://docs.llamaindex.ai/en/logan-llama_deploy_docs/module_guides/workflow/deployment/
Except that the run_step in my inner workflow has a placeholder ("...") but it's a long-running step (60s). Pasting my code here again for convenience:
from llama_index.core.workflow import Workflow, StartEvent, StopEvent, step
import asyncio
from llama_deploy import (
    WorkflowServiceConfig,
    ControlPlaneConfig,
    deploy_workflow,
)

class InnerWorkflow(Workflow):
    @step()
    async def run_step(self, ev: StartEvent) -> StopEvent:
        **result = ... #some long step here**
        return StopEvent(result=str(result))

class OuterWorkflow(Workflow):
    @step()
    async def run_step(
        self, ev: StartEvent, inner: InnerWorkflow
    ) -> StopEvent:
        arg1 = ev.get("arg1")
        result = await inner.run(arg1=arg1)

        return StopEvent(result=str(result))

inner = InnerWorkflow(timeout=1200.0,verbose=True)
outer = OuterWorkflow(timeout=6000.0, verbose = True)
outer.add_workflows(inner=inner)

The problem I get is that my inner workflow always times out after 10s (which is the default) though I specified a timeout of 1200s. So I think the adding of the sleep won't fix this because I didn't face the error you faced. Mine ran perfectly except for that timeout problem.

Thanks @masci! I notice that the fix is in the Llama-Deploy repository instead of the LlamaIndex repository. Why though? And sorry for reporting this issue in the wrong repo


Validate and confirm item quantity for large donations

Resolves #1090
Description
This PR adds a confirmation message to verify unusually large donations.
This hardcodes the "large donation" limit (100,000). This also adds model validations for invalid values for 4-byte signed integers (in ruby, > 2e31 or < -2e31-1). I added those "magic numbers" as named constants in the line_items and inventory_items.
I haven't used CoffeeScript (and barely use front-end JS), so my JS validation might not be following best practices.
Type of change

Bug fix (add model validation)
New feature  (add confirm dialog)
Code style (delete unused asset files)

How Has This Been Tested?

New model tests (for validations)
New feature test (for confirm dialog)
Manual verification:

Log in
New donation
Fill out required form values
Verify functionality:

Add small item quantity -> submits successfully
Add unusually large item quantity -> confirm dialog appears, submits successfully
Add several items, including a large quantity ->  confirm dialog appears, submits successfully

Screenshots

@duaimei This is the PR -- Could you check that the barcode feature you added still works with this? I did a merge conflict resolve and I just want to be sure that I didn't break anything since JS is involved.
Thanks!


Raise error for no dataset name

What changes are proposed in this pull request?
Raise an error for attempts to load a Dataset without providing a name to prevent any further operations that may lead to upserting a dataset document with no name.

Also, in the unfortunate scenario where a user winds up with a dataset named None, it would be nice for fo.load_dataset(None) to work.

I've seen you suggest this before but don't understand why, because this already raises an informative error:
import fiftyone as fo

dataset = fo.load_dataset(None)
# ValueError: Dataset 'None' not found

fo.Dataset(None, _create=False)
# ValueError: Dataset 'None' not found

Since all datasets need names, Dataset with name =n None is in an error state. We would not/should not allow the dataset to continue to "work"

If a None-named dataset does happen somehow, this PR would prevent the user from loading it. I think it is better to let them load it so they can fix the problem with dataset.name = "valid-name".


Hash utilities like md5sum only available in busybox

Describe the bug
Currently, to get utils like md5sum, sha256sum, one installs the busybox package. However, it shadows other programs, e.g. less.
(I hope the claim above is true. I searched bin/md5sum with nix-locate, and there's only a handful of matches.)
Steps To Reproduce
Steps to reproduce the behavior:

In an overlay, define busybox = prev.lowPrio busybox
List busybox in either environment.packages or home.packages
Run less. Now it becomes colorless. Note that I don't list less explicitly

Expected behavior
I just want the hash utilities part of the busybox. Can we have a dedicated package for that, something named like hashutils?
Additional context
For now I just define my own hashutils in an overlay:
final: prev: {
  hashutils = prev.symlinkJoin {
    name = "hashutils";
    paths = map (prog: prev.writeScriptBin prog "${prev.busybox}/bin/${prog} $@") [
      "md5sum"
      "sha256sum"
      "sha512sum"
    ];
  };
}

But I imagine these hash utils are used frequently, and I feel install busybox whole sale can be a bad idea, because it shadows some other tools and users may be surprised.

I must've messed up something. The sha256sum is already available to me.


mktemp: avoid directories with @

This is a workaround for some build tools that have trouble building if the current path contains a @
This was originally proposed https://github.com/Homebrew/brew/pull/9007 in the context of versioned gcc formulas, but I fixed the bug upstream and we decided we didn't want to change it in brew.
Now it appears that this is also an issue with ninja (https://github.com/ninja-build/ninja/issues/1606) and it doesn't look like they're actively working on it. (They use the @ character for another special meaning). This is currently blocking the build of qt@5: https://github.com/Homebrew/homebrew-core/pull/67536
Given that… I suggest we include this workaround in brew. Other that the slight effect of surprise, I don't really see a downside.

Review period skipped due to critical label.


Switch to Kotlin embeddable compiler v1.3.0

Resolves #4.
Updates all Kotlin dependencies to v1.3.0 (stable release) and switches out the kotlin-compiler dependency for kotlin-compiler-embeddable.
All unit tests continue to pass (or skip) as before, and using my fork as an included build I verify that I can use this library alongside other libraries like kotlintest.

Thanks, I will look in to this (sorry, I have been busy on other things).

I am merging this and will test it.


[LiveComponent] Fix doc typo

Made from phone while reading this readme
Did not check fully if other types are wrong

Good catch, thanks @94noni.


tinyxml-2: 4.0.1 -> 6.0.0

Motivation for this change
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[x] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Fits CONTRIBUTING.md.

@grahamcofborg build tinyxml2 lgogdownloader

Failure on x86_64-darwin (full log)
Partial log (click to expand)
Package ‘lgogdownloader-3.2’ in /Users/graham/nix-borg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/pkgs/games/lgogdownloader/default.nix:21 is not supported on ‘x86_64-darwin’, refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

Success on x86_64-linux (full log)
Partial log (click to expand)
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/p4gxi0cycfmksyd2fic4a1nbd073idkj-lgogdownloader-3.2
shrinking /nix/store/p4gxi0cycfmksyd2fic4a1nbd073idkj-lgogdownloader-3.2/bin/lgogdownloader
gzipping man pages under /nix/store/p4gxi0cycfmksyd2fic4a1nbd073idkj-lgogdownloader-3.2/share/man/
strip is /nix/store/5qj61lcvzlap87rf6blvf8p577d482bv-binutils-2.28.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/p4gxi0cycfmksyd2fic4a1nbd073idkj-lgogdownloader-3.2/bin
patching script interpreter paths in /nix/store/p4gxi0cycfmksyd2fic4a1nbd073idkj-lgogdownloader-3.2
checking for references to /tmp/nix-build-lgogdownloader-3.2.drv-0 in /nix/store/p4gxi0cycfmksyd2fic4a1nbd073idkj-lgogdownloader-3.2...
/nix/store/d4xfwapkln8z5d4gpwwajrh4hvrq4p00-tinyxml-2.6.2
/nix/store/p4gxi0cycfmksyd2fic4a1nbd073idkj-lgogdownloader-3.2

Success on aarch64-linux (full log)
Partial log (click to expand)
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/bbipr3mi0zlqnhpnxx6jdy7mwc0xsfjx-lgogdownloader-3.2
shrinking /nix/store/bbipr3mi0zlqnhpnxx6jdy7mwc0xsfjx-lgogdownloader-3.2/bin/lgogdownloader
gzipping man pages under /nix/store/bbipr3mi0zlqnhpnxx6jdy7mwc0xsfjx-lgogdownloader-3.2/share/man/
strip is /nix/store/xmpjypwjmp2qi1chs5kr0hacnh161ls4-binutils-2.28.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/bbipr3mi0zlqnhpnxx6jdy7mwc0xsfjx-lgogdownloader-3.2/bin
patching script interpreter paths in /nix/store/bbipr3mi0zlqnhpnxx6jdy7mwc0xsfjx-lgogdownloader-3.2
checking for references to /build in /nix/store/bbipr3mi0zlqnhpnxx6jdy7mwc0xsfjx-lgogdownloader-3.2...
/nix/store/fii6q20536crggiqrdz919893bdnis2p-tinyxml-2.6.2
/nix/store/bbipr3mi0zlqnhpnxx6jdy7mwc0xsfjx-lgogdownloader-3.2

Bump and ping @dezgeg and @joachifm as they are the two previous contributors to this package.


How long does it take to train networks?

Thank you for sharing codes!
How long does it take to train networks for each stage?
I also want to know your environment.
Thank you in advance.

You can find the environmental setting in the requirements.txt. For the optical flow network, it usually takes ~36h in a single Titan Xp card and the second and third stages need ~24h for each. For nyu dataset it takes longer time thanks to much more training data.

@thuzhaowang Thank you for your reply!


regList.wsf(0, 2) (null): 0x80041002

Electron app error
[2023-06-05 06:29:27.649] [error]                             Unhandled Exception Error: Command failed: c:\windows\system32\cscript.exe //Nologo C:\Users\PC\AppData\Local\Programs\My App\resources\vbs\regList.wsf A HKLM\SOFTWARE\WOW6432Node\Microsoft\Windows\CurrentVersion\Uninstall\{key}_is1
C:\Users\PC\AppData\Local\Programs\My App\resources\vbs\regList.wsf(0, 2) (null): 0x80041002

    at ChildProcess.exithandler (node:child_process:407:12)
    at ChildProcess.emit (node:events:527:28)
    at ChildProcess.emit (node:domain:475:12)
    at maybeClose (node:internal/child_process:1092:16)
    at Socket.<anonymous> (node:internal/child_process:451:11)
    at Socket.emit (node:events:527:28)
    at Socket.emit (node:domain:475:12)
    at Pipe.<anonymous> (node:net:709:12)

When i try run this via shell
C:\Users\PC>c:\windows\system32\cscript.exe "C:\Users\PC\AppData\Local\Programs\My App\resources\vbs\regList.wsf" A HKLM\SOFTWARE\WOW6432Node\Microsoft\Windows\CurrentVersion\Uninstall\{key}_is1
Microsoft (R) Windows Script Host versión 5.812
Copyright (C) Microsoft Corporation. Reservados todos los derechos.

C:\Users\PC\AppData\Local\Programs\My App\resources\vbs\regList.wsf(0, 2) (null): 0x80041002

Environment:
Windows  10.0.19045 x64

is your shell running in elevation? (as administrator)

My error message is:
The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

When i exec regList.wsf manually.
Is there any service must start to run wsf scripts?

most likely it's Winmgmt service: https://learn.microsoft.com/en-us/windows/win32/wmisdk/winmgmt


Fastboot is undefined with latest version 1.4.0.

see https://github.com/simplabs/ember-simple-auth/blob/master/CONTRIBUTING.md#issues


No tasks found (gulp file autodetect no longer works)

VSCode Version: 1.9
OS Version: Windows 10

Steps to Reproduce:

Update to the latest version of VS code (v1.9)
Try to list tasks (specifically those in gulpfile.js) and see "No tasks found".

Same problem here.  My project has only gulp tasks, I don't have a task.json.  This was working before I clicked update.  Argh!

I added a tasks.json and pointed it at my gulp task, that worked.
{
	"version": "0.1.0",
	"tasks": [
		{
			"taskName": "startup",
			"command" : "gulp",
			"args": ["startup"],
			"isShellCommand": true
		}
	]
}

So looks like auto-discovery issue vs flat out broken.  Should the location of the gulpfile matter?  Mine is in my project root folder.

Sorry for the breakage. I tried to reproduce this using a simple example but I couldn't. The example is as follows:

gulp installed locally in a workspace using npm install gulp
tasks.json

{
	"version": "0.1.0",
	"command": "gulp",
	"isShellCommand": true,
	"args": ["--no-color"],
	"showOutput": "always"
}

gulpfile.js

var gulp = require('gulp');

gulp.task('compile', function() {
	console.log('Compile...');
});

gulp.task('build', function() {
	console.log('Build...');
});

This gives:

Can someone share the project so that I can investigate more. I changed code in the area so there is a likelihood that I broke something.

Remove the tasks.json. Per my prevous post the issue is with auto-discovery of gulp tasks, not running them through a project.json

In previous versions, I would be able to run Tasks: Run Task without a tasks.json file, and it would correctly list my gulp tasks. It seems that in 1.9, you need to have a tasks.json file in order for auto-discovery to work. @tkoestler try running Tasks: Configure Task Runner to generate the tasks.json and removing the "tasks" section -- it should still automatically list the tasks defined in your gulpfile.js

@vukers Did what you suggested. It works and I like it better than what I came up with.  Thanks.

@tkoestler thanks. It reproduces without a tasks.json.

I debugged it and the breakage is actually caused by a change in the configuration service. It now returns a empty object literal if a configuration doesn't exist instead of undefined.
@sandy081 moving to you as discussed. Let me know when you have a fix and I will verify.

@sandy081  Since a work around exists this might not fully qualify for a recovery build. Lets discuss in the next standup depending on the risk of the fix. I could add a work around fix on my end as well.

@dbaeumer Fixed it in configuration service to return undefined values for a non-existent configuration.
Fix is less risky. But added more tests to make sure that behavior is not changed and covered most corner cases.

Oops.. Did not see your last comment. If we decide not to fix this for recovery I will revert the commits.

I reviewed the fix. Looks good to me.


added members events endpoint

Description of change:
Added support to events
See: https://mailchimp.com/developer/reference/lists/list-members/list-member-events/
Description of implementation:
Risks & Mitigation:

This looks 💯 to me. The build is failing on lint/style errors that I had introduced previously. I will merge over the linter shortly, and make a minor release so that the updated version can be pulled through composer.


How to export custom d.ts files?

Does anyone know how I can export the d.ts file in build with the Material UI theme adaptations so that I don't need to duplicate these statements across every consumer in my library?

Duplicate of #113 . There is currently no feature for this, but you can use rollup-plugin-copy or something similar to put this file into your dist folder.


Quick Fix for Forestry Stable <IP_ADDRESS>

Combs be shown (without a crash) in the new stable version from Forestry.

I mean Forestry Stable <IP_ADDRESS>.


add default prefix of "appscope" when outputting prometheus metrics

Add a default value for statsdprefix in the helm chart generated by the scope k8s command.  The current helm chart just omits this field from the metric->format.  Here's a relevant snippet from running scope k8s
apiVersion: v1
kind: ConfigMap
metadata:
  name: scope
  namespace: default
data:
  scope.yml: |
    metric:
      enable: true
      format:
        type: ndjson
        verbosity: 4
      watch:

I've updated the cli to provide a default prefix of "appscope" specifically when metricformat is prometheus e.g., scope k8s -c server:port --metricformat prometheus.  We are not providing a default prefix for statsd nor ndjson formats. This value could be changed manually before the output is applied to a cluster, or overridden by setting SCOPE_STATSD_PREFIX where the scoped processes are running.
With the prefix, the prometheus metric output looks like this example:
# TYPE appscope_proc_start counter
appscope_proc_start{proc="sleep",pid="2840392",gid="1000",groupname="ubuntu",uid="1000",username="ubuntu",host="ip-10-8-102-86",unit="process"} 1
# TYPE appscope_proc_cpu counter
appscope_proc_cpu{proc="sleep",pid="2840392",host="ip-10-8-102-86",unit="microsecond"} 67897

One more note.  I'm choosing not to rename SCOPE_STATSD_PREFIX to SCOPE_METRIC_PREFIX at this time because it would be a breaking change for our users.


eventsource@v3: looking for testers

eventsource v3
TL;DR
I want to modernize this module, making it both easier to maintain and more cross-platform friendly.
You can help test the next version of this module by installing eventsource@next from npm, and reading the migration guide to see how to update your code.
Any issues you find can be reported on this issue. The code for the new version is available in the v3 branch.
Background
This package is soon to be 13 years old. When Aslak and Einar created the first version of this module, node.js wasn't even at 0.10 yet. A lot has changed since those days:

Node.js, browsers and other Javascript environments have aligned more closely. We now have the fetch API available in all modern browsers, Node.js, Deno, Bun etc. A recent addition is the "web streams" API - also available in the same environments.
The module system discussion has changed. While CommonJS is still used, more and more tooling and modules are moving towards ES Modules.
TypeScript has become the defacto standard for writing type-safe JavaScript. This module was written in the days before TypeScript was a thing, and currently relies on an external @types module for TypeScript support.
With the advent of AI/LLMs, EventSource/Server-Sent Events are becoming more relevant - many providers use it as the protocol to stream responses to clients.

Proposal
Use ES Modules (ESM), with CommonJS fallback
We'll build the module using @sanity/pkg-utils (because we use it where I work, and I am familiar with it), and use conditional exports to provide both ESM and CommonJS variants. Will also switch to named exports, since it has a better cross-module system compatibility.
Split the parser and the client
I've made a parser module for EventSource that is minimal, well tested and works in "any" JavaScript environment. Splitting the concerns makes sense from a testing and maintenance perspective.
Use TypeScript for type safety
I find it easier to maintain and think about code when it's typed. Not having to keep the @types/eventsource module in sync with the main module is a big plus.
Use fetch, ReadableStream, EventTarget
Allows us to reuse the same code across Node.js, browsers and other environments that support the fetch API. The polyfill for browsers becomes very slim, and there are few behavioral differences between the environments.
Remove (most) request-based extensions
In v2, there are some extensions to the WhatWG spec - allowing users to pass custom headers, adding support for proxies, specifying HTTPS options etc. I propose that all of these are removed, and instead left up to the user to implement: they are trivially implemented by passing a custom fetch function. This creates a single extension point, making the module easier to maintain.
Drop support for Node.js < 18
In order to use fetch, web streams and other modern APIs, we need to drop support for Node.js versions < 18. This is a tradeoff I'm willing to make, as Node 18 is going out of LTS in April 2024 and older versions are already out of LTS. Users who are on older versions can still rely on v2 of this module.
Repo: Use conventional commits, automate releases
In order to make the module easier to maintain, I propose we switch to using conventional commits, and automate releases using GitHub Actions. This will make it easier to maintain the module, and make it easier for contributors to understand the release process. Automatic generation of changelogs is a nice bonus.
Repo: Move to main branch
I propose we move the default branch from master to main. This better aligns with the new default branch naming in GitHub.
Disclaimer
While these changes may seem drastic, they are necessary for me to continue supporting this module long term. It also falls within the bounds of what Aslak said when he passed on the maintainership of the module to me:

Be conservative - if you allow non-standard extensions to the W3C API, only do so if it cannot be done easily with a wrapper around the library.

This is exactly what I am proposing, by encapsulating all/most non-standard extensions in a single extension point (the fetch property).

Dependency free

While not 100% true, it only has a single dependency - eventsource-parser, which has zero dependencies of its own.

Keep it working in browsers

This is probably the biggest change - in a positive way. Instead of having to rely on a webpacked file that sets some browser globals, we can now use the module directly in the browser, and with no node.js-polyfills bundled.

Keep it open

I will continue to welcome contributions, but also want to ensure that we keep an eye on security. 19+ million downloads per month means we have a responsibility to our users. By making releases built and signed by GitHub actions, the risk of malicious code being published by someone accidently gaining access to an npm account is reduced.
Preliminary test results
I've tried to exercise the new module by running a test suite that connects to a Server-Sent Events endpoint which emits events every ~10 seconds, sends comments as "keep-alives" (technically not needed given the 10s event interval) and occasionally closes the connection due to the load balancer in front of the endpoint terminating long-lived connections. The tests are run on Node.js, bun and in browsers. In server environments, I am testing the old EventSource@2 implementation vs the new one, and in browsers I am running the new implementation vs the browser native implementation and vs the yaffle polyfill. I am not running it against the browser-compiled variant of EventSource@2 since I consider it almost "an accident" that it works in browsers, given the amont of polyfills for node internals it uses.
This is all running on my laptop on a wifi connection which tends to be online, but sometimes goes into slumber or drops the wifi or similar. The benefit here is that I get to test and compare both the "optimal" case as well as the "difficult" cases (wifi disconnect, computer goes into sleep mode) and see that things recover as expected.
The results are promising:

The old implementation actually seems to have problems with stalling at times, in Node.js. It simply stops receiving events. The new implementation doesn't stall - it seems to be much better at recovering.
The browser implementation works flawlessly and has a 1:1 match with the browser native API.

One thing I did discover with the browser variant of the new module is that the Last-Event-ID header is not a CORS safelisted header, and thus reconnections cross-origin causes a preflight request to check if allowed to send the header. It seems odd that the native EventSource is allowed to send this header but not a fetch request, and I'm now working on rectifying this in the WhatWG fetch spec. Until then, we have two options:

Document the CORS behavior (my preference) and tell people how to respond to an OPTIONS request
Change browsers to use a query string parameter for the last event ID. This is what the yaffle polyfill does (lastEventId=<id>), as well as the amvtek polyfill (evs_last_event_id=<id>). Since this isn't standardized, I am not super eager to implement this - it could also be done manually by passing a custom fetch implementation which replaces the last-event-id header with a query string parameter of choice.

I use eventsource3, and the program will exit inexplicably after running for a few hours without any alarm (I open and close a different eventsource every minute). With eventsource2, there is no such problem.

I use eventsource3, and the program will exit inexplicably after running for a few hours without any alarm (I open and close a different eventsource every minute). With eventsource2, there is no such problem.

Under which environment is this? Node.js? Which version? I'm not familiar with the language above, but could you add a catch handler to your promise that awaits the espromise that logs any errors?

i think
await wget url, join(JPG, fname)
throw a error in onmessage
event2 will log the error
3 not log the error

i think await wget url, join(JPG, fname) throw a error in onmessage
event2 will log the error 3 not log the error

I am unable to reproduce. I've tried recreating something that follows your code here: https://github.com/rexxars/es-repro-329
You can switch between EventSource@v2 and EventSource@next by swapping the EventSource variable. When running this, I get (in both v2 and v3):
><EMAIL_ADDRESS>start
> node repro.js

/Users/espenh/webdev/es-repro/repro.js:64
    setTimeout(reject, 100, new Error('wget error'))
                            ^

Error: wget error
    at /Users/espenh/webdev/es-repro/repro.js:64:29
    at new Promise (<anonymous>)
    at wget (/Users/espenh/webdev/es-repro/repro.js:63:10)
    at es.onmessage (/Users/espenh/webdev/es-repro/repro.js:24:13)
    at /Users/espenh/webdev/es-repro/node_modules/eventsource-v3/dist/index.cjs:55:119
    at dispatchEvent (/Users/espenh/webdev/es-repro/node_modules/eventsource-parser/dist/index.cjs:69:24)
    at parseLine (/Users/espenh/webdev/es-repro/node_modules/eventsource-parser/dist/index.cjs:22:7)
    at Object.feed (/Users/espenh/webdev/es-repro/node_modules/eventsource-parser/dist/index.cjs:17:7)
    at /Users/espenh/webdev/es-repro/node_modules/eventsource-v3/dist/index.cjs:44:46
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Are you able to modify my test repo code to recreate the case you are seeing?

v3 is now released, so I am closing this


Warnings

Using SkyFloatingLabelTextFieldWithIcon , changed the icon type value to 1 and set image icon but still on every text field getting warning
WARNING - Did set iconImage when the iconType is set to IconType.font. The image will not be displayed.

Hey @invinciible,
I'm not sure if Xcode maintains order when setting User Defined Runtime Attributes, if it does moving iconTypeValue before iconImage should remove the warning. If it doesn't I think there might be no way around this


Enable multislice in launch script

Adds node_count in config to launch jobs with multislice training.
Refactored the launch script to use queued-resources API.
mention #629

Some UX design discussions: what behavior do we want our launch script to have?

Before: with retry>0, the launch script will retry even if the job finishes without error
After: the launch script will exit (no retry) if the job finishes without error

Before: (not using queued-resources API, not sure about the actual behavior)
After: If autodelete is True, and the job finishes with no error, the launch script will tear down the TPU automatically (so that it doesn't eat up the TPU quota). Do we want this to happen when the launch script finishes all the retries as well? e.g. tear down the TPU after the job errors and retries 100 times.

If the user manually kills the script (e.g. by ctrl-c), the TPU will still stay in the queued-resources list, taking up some quota even after suspended. Is there something we can do here?

To answer your questions:

new behavior is better
Use the existing argument autodelete is better. Existing behavior is kinda weird imho
I think the current behavior is the right behavior, especially with TRC TPUs. Maybe we should add a cleanup script or something but I think it's fine.

(waiting for a preemption event to see if my script retries correctly and push a final commit addressing @dlwh 's comments before I merge)

will just deleting the TPU (not the resource) simulate preemption well enough?

@blahBlahhhJ is this ready to merge?


Fix grammatical errors and typo in code

Fixed:
Changed "Commands must" to "Commands must be" for grammatical correctness.
Corrected "import { geSSLHubRpcClient" to "import { getSSLHubRpcClient" to fix the typo.
Why it's useful:
These changes fix grammatical errors and a typo, ensuring the code is syntactically correct and easier to read.

PR-Codex overview
This PR focuses on correcting a typo in the documentation and updating a function name in the code.
Detailed summary

In apps/hubble/www/docs/docs/cli.md, the phrase "Commands must invoked" was corrected to "Commands must be invoked".
In packages/hub-nodejs/docs/Client.md, the function name geSSLHubRpcClient was updated to getSSLHubRpcClient.

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

Hi team! Could someone please review and merge my pull request? Let me know if any changes are needed. Thanks!
@cryptobenkei @mboyle @vrypan


run.vbs gives error on task scheduler run

i installed python, entered my spotify api keys, setup the playlist image and data,  made the task with highest privileges, but when i run the task it gives this error:
script: C:\Spotify-Playlist-Doctor\resources\scripts\run.vbs
line: 6
char: 1
error: the system cannot find the file specified
code: 80070002
source: (null)
i believe i installed everything correctly, i ran the pip command specified from the main directory (C:\Spotify-Playlist-Doctor for me)
is there somehting that i missed or didn't do correctly? i know that the task points to the correct path and i didn't modify anything other than the files pointed by the readme (playlists, images and config)
this is the only good project i found for fighting the playlist bot report issue and i really want to get it working

Hey @Naxy21,
For the error you receive my bet is that something is misconfigured within your Task Scheduler.
Make sure to set the task up like so:
Add arguments: C:\python-spotify-playlist-doctor\resources\scripts\run.vbs
Start in: C:\python-spotify-playlist-doctor\resources\scripts\

Let me know whether it works out!

What if you run Main.py directly?
On Tue, Jun 13, 2023, 20:38 Hazel_888 @.***> wrote:

it does work now, but when the task runs scheduled (5 minutes after
clicking run), nothing happens, and nothing shows in output.log
—
Reply to this email directly, view it on GitHub
https://github.com/drikkk/Spotify-Playlist-Doctor/issues/1#issuecomment-1589749870,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADLP22DWXFBQMNHSYYWODMTXLCQQVANCNFSM6AAAAAAZCMKS2E
.
You are receiving this because you commented.Message ID:
@.***>


The Island Private School Of Limassol

@slds1 First stage of verification passed, official domain for The Island Private School Of Limassol is isl.cy as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision.

@slds1 Thank you for your patience while waiting for a review and processing your request. Pull request merged.
Please start requesting the licenses in about an hour, to let the changes to propagate through our system.


Migration and seed for groups

Description
I have created the database table and I did it with Knex. Also, add a seed file for development to add a couple of initial groups.
I had a problem to understand this first but now it's clear.
Fixes # (22)
How to test?
Checklist

[ X ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ X ] This PR is ready to be merged and not breaking any other functionality

Closing this issue #22 for the new issue we are working on right now #65. (create groups table)


Remove sbt 1.0 docs, or print warning that it's a pre-released docs

Debian/Ubuntu users are confused because the installation instruction points them to experimental stuff.

Do you mean "or" instead of "for"?
I'm happy to remove it, it'll need a proper revising for 1.0, it's just confusing/wrong atm.

Maybe rename to 1.0-beta/ and tell Google to not index?

Personally I would remove it. We can use read the sources for now until we've reviewed it.

When the beta comes out though I would like a place to document new features, changes, list of plugins etc.

Yeah, when we're ready for the beta we should definitely bring it back, with accurate information.

sbt 1.0 is out.

I'm getting this error " unable to locate sbt package"

@Masood-Iqlect please open a new issue on sbt/sbt following the instruction here - https://www.scala-sbt.org/community.html#how-can-I-help


fix(router): ensure routes have distinct names

This is in reference to #366.
Router will throw an error if a route with the same name in addRoute

Should the check be a part of function validateRouteConfig()?
Should the comparison be case insensitive? Different casing is not tested.

@CasiOo

Should the check be a part of function validateRouteConfig()?

That is a good point regarding design. I can make that change if necessary.

Should the comparison be case insensitive? Different casing is not tested.

Interesting point. @EisenbergEffect can you chime in?

If possible, it's probably better to locate that inside the existing validation method. As far as case sensitivity goes, the recognizer's name lookup is currently case-sensitive.

@EisenbergEffect Thank you. I have updated the code per your comment.
In validateRouteConfig(), I also needed to pass Router's routes since it was not contained in the validateRouteConfig's context.

Thank you!


Permission sets aren't provisioning in 3.1.7

I notice in 3.1.7 that none of my permission sets are getting provisioned into the account using SSO Extensions. These are valid permission sets, which do not throw an error in any of the Lambda processors. The permissionSetTopicProcessor gives log messages that indicate a status of "SUCCESS" on the waiter that requests provisioning, but the provisioning operation doesn't happen.
When I go into the permission set and manually click the "Update" button to provision it, it works. And all the permissions I asked for in the permission set JSON are reflected in the live permission set.

Hey @allquixotic , please help me understand the correct flow here:

you had a permission set created through the solution interface, and then also provisioned this to some accounts as well through the solution interface
Now, you are triggering an update of the permission set through the solution interface.
The permission set chanages (for ex, managed policies, permission set attributes) etc are propagated to the permission set object
However, you are noticing that the reprovisioning of the permission set into the accounts it's provisioned already is not happening, and you needed to do this manually from the console?

Leela,
That is correct.

Hi @allquixotic ,
After some debugging, I now understood the root cause of the issue. I am still trying to figure out a clean solution to handle this, and will update you when done. For context, here's where the issue is

When a permisison set is udpated, not all the changes are syncrhonous, specifically the change handling for both AWS and customer managed policies are asynchronous and the permissionSetTopicProcessor does not currently have control/visibility into when the asynchronous processes are completed.
Additionally, the code syncrhonously triggers a re-provisioning operation as well.  This is referred in this section of the code
When this provision call is triggered, it could either be when all the permission set changes are fully complete and therefore the most up-to-date copy is being reprovisioned (correct scenario), or it could be when the changes are still ongoing, and the reprovision call is triggered on a stale copy of the permission set (buggy scenario).
I need to figure out a way to make the code wait until the asyncrhonous calls are complete, so that a reprovisioning call is always done on a valid PS copy instead of a stale one.


the bias of visible layer

Hi, you did awesome job!
But I'm a little confused about the bias of visible layer which is returned by W, Bv, Bh = rbm.get_weights(), sometime I often see bias of hidden layer, what is the bias of input/visible layer? thx!
best wishes!
SunnyMarkLiu

RBM is a bidirectional network. So, at first we compute hidden representation of input data as follows:
hidden_data = s1(input_data @ W + hidden_bias)

After that, we should reconstruct input data from this hidden representation with the same W matrix, but obviously with different bias (because of different dimensionalities):
data_reconstructed = s2(hidden_data @ W.T + visible_bias)

And now we can compute all updates.
See also original paper:
https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf
Equations (7) and (8). Visible and hidden biases are denoted as a and b respectively.

Note: when RBM's weights is used for deep network initialization, visible bias should be ignored.

Thank you very much!


fix @ledger/react-native-hid for android tiramisu

https://github.com/LedgerHQ/ledger-live/issues/7786

✅ Checklist

[x] npx changeset was attached.
[x] Covered by automatic tests. Bugfix must bring non-regression
[x] Impact of the changes:  the changes impact only the lib @ledger/react-native-hid

📝 Description
Android sdk 34 give the following error when use transporter from the lib @ledger/react-native-hid:
One of RECEIVER_EXPORTED or RECEIVER_NOT_EXPORTED should be specified when a receiver isn't being registered exclusively for system broadcasts.
To fix this bug is needed call getReactApplicationContext() with the Context.RECEIVER_NOT_EXPORTED attribute.
implementation that throw the error: https://github.com/nicolas-meilan/RungeCoin/blob/master/src/web3/wallet/wallet.ts#L210

❓ Context

JIRA or GitHub link:   ISSUE 7786

🧐 Checklist for the PR Reviewers

The code aligns with the requirements described in the linked JIRA or GitHub issue.
The PR description clearly documents the changes made and explains any technical trade-offs or design decisions.
There are no undocumented trade-offs, technical debt, or maintainability issues.
The PR has been tested thoroughly, and any potential edge cases have been considered and handled.
Any new dependencies have been justified and documented.
Performance considerations have been taken into account. (changes have been profiled or benchmarked if necessary)

Hello @nicolas-meilan, first of all, thank you for your contribution!
Can you resync your fork, please? We had an issue with our CI that we had to fix.
Thank you!

Thanks, looks about the same as the fix I had on our fork, which is working in prod. Once this is resolved we'll move back to the official package.

Hello @nicolas-meilan, thank again for your continuous sync of your branch.
Since I do not have an Android Tiramisu to test, it will be @mbertin-ledger that will take over and do the test.
Thanks again for your patience 🙏

@nicolas-meilan you can stop merging develop into your branch for now. As soon as it's reviewed, we will need one last merge/rebase and you should be good to go

@valpinkman & @mbertin-ledger should we reopen the PR ?


Disabled metric collection when GENERAL_MICROMETER_ENABLED is false

Prior to this change user specified MeterRegistry's were not added when GENERAL_MICROMETER_ENABLED was false. JVM metrics could still be collected, but never reported to a destination.
Closes #4989

Full IT build passed.


Missing feature to terminate processing

I just evaluated this library for use in our product. But sadly, the missing termination feature from within the poll lambda is a show stopper.  If I can only fail a record over and over again, there must be a way to stop processing gracefully altogether.
Is this feature coming soon?
And why is the poll lambda relying on exception handling instead of a proper return value? Using exception handling for flow control is very outdated and should not be the norm anymore.
Thanks!

Hi @Zordid - can you please elaborate? I do not fully follow - what do you mean by termination feature?
Afaik you can close parallel consumer instance from within poll call - either signal main app thread that it should close the Parallel Consumer instance or just call the close from within poll directly on a different thread - something like this:
        parallelConsumer.poll(recordContexts -> {
            if (shouldSignalClose()) {
                CompletableFuture.runAsync(() -> parallelConsumer.closeDontDrainFirst(Duration.ofSeconds(10)));
            }
            processRecord(recordContexts);
        });

Normally though application would not close Parallel Consumer due to failing record - if the issue that leads to record processing to fail - is bad data in the record - then Dead Letter Queue approach could be used or that record just skipped and processing moved to next record - depending on requirements, but the general goal is to not block processing.
If the processing is failing due to external system being unavailable - then that could be monitored through health monitoring of dependent systems and ParallelConsumer can be paused / closed on unavailability of external system (similar to circuit breaker pattern) and resumed / restarted when health of external system is back to good / green - that, in my opinion, would be triggered outside of the Poll lambda though.
Using exceptions to indicate a failure is normal enough in applications to my knowledge - could you elaborate ?

All I mean is, it is a cleaner approach to give the lambda a way to indicate process result by a return value than to use exceptions. General rule of thumb for me, because exceptions ar least make it very hard to understand the contract by looking at the signature of a function.
If the lambda that processes the record would need to return a plain enum with ok, retry or stop it would be much easier to understand.
Well, the reason stop is a thing we need is quite simple. The product we build does not know anything about its context, it is all driven by configuration of a customer.
And one of the possible reactions for errors is to shut down the application - the host environment would then restart it all and there is a chance the problem is solved.
If it were just a pure Kafka app, I'd agree, but we run completely different things connecting to the very unpleasant SAP world in this app...
Cheers and thanks for one trick to kill the processing from inside!

This is not the right attitude, specially for free software

Personally, in general - i have a view that "it depends" - how application / library is written, what expectations are of its users etc.
I find that Java functional interfaces are a bit lacking when it comes to handling branching logic - pattern matching, exception flows etc - i've seen and used that more in Scala, or using something like Vert.x with future handler composition.
Specifically with Parallel Consumer - its code base is not written in functional style, and it does need to handle unhandled exceptions from user code with some default behaviour, and i would argue it would make sense to tie specific behaviour to exception types. That is widely used and familiar pattern to many Java developers.
In addition - User Function already has a return value - for poll and produce flow. Of course it can be wrapped into Result object with status of execution and any data returned, but it just makes it more complex.
So - while i agree that there are cases when it is better and more natural to use return object with different statuses for success / failure modes - i do not think that is a good fit for Parallel Consumer code base.

Keeping it open - as i believe it would be beneficial to implement - bandwidth permitting.
Related enhancement that was closed as stale - but may have useful work in it already done / partially done - #291


Create SECURITY.md to document security policy, reporting, etc

https://medium.com/@Trewaters/security-readme-files-security-md-cb34803afcff

https://github.com/Innablr/revolver/pull/320