:bar_chart: move Google mobility trends from covid repo to ETL

Proof of concept for migrating COVID datasets from covid-19-data to ETL.
The dataset was first migrated to get all metadata using
ENV=.env.prod backport-migrate --dataset-id 5085 --namespace covid --short-name google_mobility_trends --version latest

and then modified to use etag://... of the CSV file in covid repo.
ETL is still missing some features implemented in GrapherBaseUpdater

Slack notifications
Incrementing charts versions to trigger rebake
enqueue_deploy that enqueues deploy

I'm not sure how charts work at the moment. I thought that all of them get rebaked irrespective of version, but I might be wrong. As for deploy, we could have a single CRON job instead that would be triggering deploys daily or every few hours.
If this dataset works, I can start migrating other datasets.

@Marigold, thanks for doing this.
The code looks good! Some comments/questions:

Currently, the data flow looks like covid-19-data → Grapher → backport → ETL → Grapher? I guess I am missing something. Is the idea to move the scraping/getting data part from covid-19-data to snapshot?
Side comment, Google stopped updating this dataset during (I believe) October 2022.
Slack notifications are especially useful for vaccinations/hospitalizations pipelines. For the remaining ones these are good to have, but it is fine.
On the chart version increment, I honestly don't know what the impact of this is. In general, I am not sure if we even make any use of the chart versions. AFAIK the chart will display the updated version irrespective of the version being bumped up or not.
enqueue_deploy: Are you talking about running something like etl covid once a day? I think that'd be a nice idea.

Currently, the data flow looks like covid-19-data → Grapher → backport → ETL → Grapher?

It's actually covid-19-data -> .csv -> ETL -> Grapher. The backport-migrate is not backporting periodically, I just used it to generate ETL steps (with metadata) that I updated.

Side comment, Google stopped updating this dataset during (I believe) October 2022.

Yeah, I know. It was just a proof of concept.

Slack notifications are especially useful for vaccinations/hospitalizations pipelines. For the remaining ones these are good to have, but it is fine.

Would it be enough to send notifications after .csv is generated?

and 5.

Good!
If you think it's viable approach I'll remove the .csv -> Grapher part from covid repo and run it from ETL. If it works I can start with other covid datasets.

Right, all makes sense!
Re: removing .csv → Grapher, it is currently not being executed on the covid repo:
https://github.com/owid/covid-19-data/blob/c5c053b41c65e3661592f5d50821968430ad3ff9/scripts/scripts/grapherupdate.sh#L61
But for the other processes, I think we could start with those listed in:
https://github.com/owid/covid-19-data/blob/c5c053b41c65e3661592f5d50821968430ad3ff9/scripts/scripts/grapherupdate.sh#L56-L65

cases/deaths
decoupling
hospitalisations
oxcgrt
uk-nations
sweden


Added more eggs

Added 2 eggs for myself

please make review

Thank you for you innsats, please look at #6
regards,
Anna


[improvement-#14995] get namespace in real time

Purpose of the pull request

plz close this issues : #14995

Brief change log
Verify this pull request

This pull request is code cleanup without any test coverage.

it has overlapping parts with a pr #14977 ,so i close this issues cc @EricGao888 @songjianet


[Bug] link formatting in Readme.md

Description

In the README.md every link that is present in the repository is  inline-block except sources.md. Put sources.md in inline-block like history.md as shown in the screenshot.
https://github.com/creativecommons/quantifying/blob/8c444b52f560b034ecf464dafdea34ca520afc91/README.md?plain=1#L84
Screenshots

Resolution

[x] I would be interested in resolving this bug.

there you go

This should be closed now


Better kivy settings: flag setting that need kivy reload

See title

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/99143-better-kivy-settings-flag-setting-that-need-kivy-reload?utm_campaign=plugin&utm_content=tracker%2F42681&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F42681&utm_medium=issues&utm_source=github).

This issue has been around since 2011, @tito could you explain more about this? I don't seem to understand the issue. It would be better to open a new issue with more explanation.


Undefined Method for <ActiveRecord::Associations::CollectionProxy []>

I am seeing this issue on Rails <IP_ADDRESS>, Administrate 0.13.0 and the current master branch of NestedHasMany.
My configuration is very simple:
class LabDashboard < Administrate::BaseDashboard
  ATTRIBUTE_TYPES = {
    payor_lab_settings: Field::NestedHasMany.with_options(skip: :lab)
  }

  ...

  FORM_ATTRIBUTES = [
    :payor_lab_settings
  ].freeze
end

class PayorLabSettingDashboard < Administrate::BaseDashboard
  ATTRIBUTE_TYPES = {
    payor: Field::BelongsToSearch.with_options(class_name: 'Payor'),
  }.freeze

  ...

  FORM_ATTRIBUTES = [
    :payor
  ].freeze
end

Here is the relevant section of the stack trace:
PayorLabSetting::ActiveRecord_Associations_CollectionProxy#method_missing
activerecord (<IP_ADDRESS>) lib/active_record/relation/delegation.rb, line 108
Administrate::Page::Form#get_attribute_value
administrate (0.13.0) lib/administrate/page/base.rb, line 35
Administrate::Page::Form#attribute_field
administrate (0.13.0) lib/administrate/page/base.rb, line 29
block in Administrate::Page::Form#attributes
administrate (0.13.0) lib/administrate/page/form.rb, line 15
Administrate::Page::Form#attributes
administrate (0.13.0) lib/administrate/page/form.rb, line 14
Administrate::Field::NestedHasMany#nested_fields
() bundle/ruby/2.5.0/bundler/gems/administrate-field-nested_has_many-0145a35e4497/lib/administrate/field/nested_has_many.rb, line 23

The exact same code with Field::HasMany works correctly. All dashboards are working individually.

Further digging, it looks like this is likely due to a change in Administrate between 0.12.0 and 0.13.0:
In 0.13.0 the relevant code: https://github.com/thoughtbot/administrate/blob/v0.13.0/lib/administrate/page/base.rb#L35
Vs 0.12.0: https://github.com/thoughtbot/administrate/blob/v0.12.0/lib/administrate/page/base.rb#L36
Administrate core stopped rescuing NameError and returning nil. My guess is if I revert back to 0.12.0 in Administrate,  the code here was always technically broken, because of an attempt to initialize the Form with a collection, rather than an instance of the associated model, but because of Administrate's rescue, it wasn't blowing up, and instead returned a set of Fields with nil values.

@nickcharlton I have opened PR #34 which addresses this problem. It also captures the same changes made in #31 and #32 in order to get tests passing against Administrate 0.13.0.

Confirming that the issue is solved with the PR.


Add peerpodconfig-ctrl and peerpod-ctrl into operator

We've created peerpodconfig-ctrl and peerpod-ctrl to help manage VM resources we create for peer pods. Once we've done the work in https://github.com/confidential-containers/cloud-api-adaptor/issues/871 (and pre-reqs) to move to the coco operator and away from our forks, we should consider adding the deployment of these controllers into the cluster.

This should include webhook as well.. Updating the issue title accordingly

@caavery is working on it and hence changing the status to in-progress

@caavery do you think this will converge for release 0.7.0 Feature freeze on 12th of July (GA on the 19th) or should we move this to release 0.8.0?

It should be moved to 0.8.0 as it will not be ready for 0.7.0.

The concern I have with adding components from CAA to operator is introducing cyclical dependency.
CAA depends on operator to install all the necessary components. And installing stuff that CAA owns from the parent introduces cyclical dependency.
Also this will make the release harder. As in we first cut a release of operator and use that tag in CAA. But by making this change we don't know which release to cut when?

@surajssd any suggestions to avoid this dep?
Should we move the controllers to a separate repo (eg addon-controllers) since the controllers import CAA module? So release sequence may look like

Create CAA release
Create peerpod/peerpodconfig release
Create operator release

cc @stevenhorsman

So, I think that makes sense, if the peer-pod ctrl is added to the operator.
There is also a code cyclic dependency we have at the moment is that peerpod-ctrl depends on cloud-api-adaptor, but cloud-api-adaptor also depends on peerpod-ctrl, so I think we might want to think about that as well before doing any separation?

@surajssd any suggestions to avoid this dep? Should we move the controllers to a separate repo (eg addon-controllers) since the controllers import CAA module?

We can move the code out yes.

So release sequence may look like

Create CAA release
Create peerpod/peerpodconfig release
Create operator release

Isn't the order upside down, with operator release being the first to go out?

I've created a draft PR at https://github.com/confidential-containers/operator/pull/249

Is there a corresponding issue for operator?

@bpradipt @jensfr
This issue has been fixed in https://github.com/confidential-containers/operator/pull/249

Pradipta merged this this under https://github.com/confidential-containers/operator/pull/349


Fix call to nil stat

Fixes #10242

Hehe

Drone having issues pushing/pulling it seems.

@cpuguy83 but do you know why that mkdir part was there?

Actually, I'm looking at this, it should need the MkdirAll, but not the other stuff.
It seems binds are being created here: https://github.com/docker/docker/blob/47e3da848ffbe88d0188ae6cfc09d6e1668bc293/daemon/start.go#L54-L66
Which is why we weren't getting the panic.
Removing this block of code is giving some other weird issues atm.

Can you rebase or something to make drone happy?

This fixes the panic only, but I will need to look at the binds setup in daemon/start.go for another PR.

What is wrong with binds?

Nothing is "wrong" other than having the bind paths created in "setupHostConfig" was masking the issue with the stat here, and it really should be created by the volumes system.

Ok, SGTM

LGTM

LGTM


Set difference, set intersection support?

Is there existing/planned support for set difference or set intersection operators?
e.g. in table "Likes" with columns "Person" and "Pizza":

To find people who like both mushroom and onion pizza
SELECT person FROM likes WHERE pizza="mushrooms"
INTERSECT
SELECT person FROM likes WHERE pizza="onions"

To find people who like onions but not ham on their pizzas
SELECT person FROM likes WHERE pizza="onions"
MINUS
SELECT person FROM likes WHERE pizza="ham"

I am new to SQL, so maybe there is a similarly simple way to do this, but set operations make these clear.

I think this could be implemented in pypika pretty easily, since we have code already for things like unions that produce similar SQL. Would you like to try and raise a PR?

@twheys I have raised a PR regarding the above issue. Could you please take a look

Wow, thank you @DimpleK26 ! I'm pretty inexperienced with the workflow on here. Now that your changes have been merged, do I go ahead and click "Close Issue"?

@roger-rojas Yes . If you are happy with intersect ,minus and except operations, kindly go ahead :)


feat(array): product

[!TIP]
The owner of this PR can publish a preview release by commenting /publish in this PR. Afterwards, anyone can try it out by running pnpm add radashi@pr<PR_NUMBER>.

Summary

Add a Cartesian product function for arrays, called product.
NOTE: Unfortunately, this only supports a single type. Mixed types are not fully supported, for example, _.product([[1,2],["hello","world"]]) will throw a type error. Fully mixed types are tricky to achieve in TypeScript for Cartesian Products, because of how things get "mixed up". In the above example, we know the result is composed of arrays like [1, "world"] with type [number, string]. I haven't found a good way to solve it. See discussion here: https://gist.github.com/ssippe/1f92625532eef28be6974f898efb23ef
Related issue, if any:

For any code change,

[x] Related documentation has been updated, if needed
[x] Related tests have been added or updated, if needed
[x] Related benchmarks have been added or updated, if needed

Does this PR introduce a breaking change?

No

Bundle impact

Status
File
Size [^1337]

A
src/array/product.ts
202

[^1337]: Function size includes the import dependencies of the function.

@yamcodes thanks for the PR. some considerations.

I really don't understand why you have to pass [...any[]] instead of using the spread operator "..."

export function product<T>(...arrays: T[]): T[][] {

about types, I think I found an implementation that solves the type problem

export function product<T extends any[]>(...arrays:  [...T]): Array<{[K in keyof T]: T[K][number]}>
export function product<T extends any[]>(...arrays: T): T[][] {
  let out: T[][] = [[]]
  for (const array of arrays) {
    const result: T[][] = []
    for (const currentArray of out) {
      for (const item of array) {
        const currentArrayCopy = currentArray.slice()
        currentArrayCopy.push(item)
        result.push(currentArrayCopy)
      }
    }
    out = result
  }
  return out
}

and since it is now possible to have better types, I recommend creating a test for the types, a suggestion:
import * as _ from 'radashi'

describe('product types', () => {
  test('with an empty array', () => {
    const result = _.product([])
    expectTypeOf(result).toEqualTypeOf<[never][]>()
  })
  test('with two arrays of the same type', () => {
    const result = _.product(['red', 'blue'], ['fast', 'slow'])
    const [[v1, v2]] = result
    expectTypeOf(result).toEqualTypeOf<[string, string][]>()
    expectTypeOf(v1).toEqualTypeOf<string>()
    expectTypeOf(v2).toEqualTypeOf<string>()
  })
  test("with two arrays of different types", ()=> {
    const result = _.product(['red', 'blue'], [1, 2])
    const [[v1, v2]] = result
    expectTypeOf(result).toEqualTypeOf<[string, number][]>()
    expectTypeOf(v1).toEqualTypeOf<string>()
    expectTypeOf(v2).toEqualTypeOf<number>()
  })
  test("with three arrays of different types", ()=> {
    const result = _.product(['red', 'blue'], [1, 2], [true, false])
    const [[v1, v2, v3]] = result
    expectTypeOf(result).toEqualTypeOf<[string, number, boolean][]>()
    expectTypeOf(v1).toEqualTypeOf<string>()
    expectTypeOf(v2).toEqualTypeOf<number>()
    expectTypeOf(v3).toEqualTypeOf<boolean>()
  })
  test("with constant arrays of different types", ()=> {
    const result = _.product(['red', 'blue'] as const, [1, 2] as const)
    const [[v1, v2]] = result
    expectTypeOf(result).toEqualTypeOf<["red" | "blue", 1 | 2][]>()
    expectTypeOf(v1).toEqualTypeOf<"red" | "blue">()
    expectTypeOf(v2).toEqualTypeOf<1 | 2>()
  })
  test("with a mix of constant and non-constant types", ()=> {
    const result = _.product(['red', 'blue'], [1, 2] as const)
    const [[v1, v2]] = result
    expectTypeOf(result).toEqualTypeOf<[string, 1 | 2][]>()
    expectTypeOf(v1).toEqualTypeOf<string>()
    expectTypeOf(v2).toEqualTypeOf<1 | 2>()
  })
})

Thanks for the PR! I think naming the function cartesianProduct would make its purpose clearer to anyone reading the code. The term "product" can be a bit ambiguous since it could also refer to element-wise multiplication or dot products. Using cartesianProduct would help avoid any confusion and match the common terminology for this operation. What do you think?

@MarlonPassos-git
(1) I also prefer supplying the arrays as parameters, so we should go with your solution. I'll add that I followed the example set by unzip, so it'd be a good idea to update that as well. @aleclarson
(2) Oh. My. God. I'm so blown away!!! The usage of:
(1) Spreading generic tuple parameters ([...T])
(2) Overloads to simplify the implementation (instead of type assertions)
(3) Mapped types on tuples
Is pure genius. I've learned something new today because I had no idea you could spread generic types. Thank you, Marlon!
@aleclarson Thanks! It's a pleasure to join your community and finally contribute!
I also prefer cartesianProduct, since it eliminates a cognitive overhead, so I'm glad you brought it up. I just got the impression that the library prefers shorter and "cuter" names rather than explicit ones. I'll go ahead and rename.

Guys, I've updated the PR to reflect your great suggestions. Let me know what you guys think of the feature as a whole!

Suggestions were applied and I've also enforced that T extends any[][] to enforce the input parameters to all be arrays.

Hey! There's a new requirement for PRs that introduce new features. Without this requirement met, we won't be able to merge this. Note that this PR can still be included in a beta prerelease before this requirement is fulfilled.

[ ] Edit the next-minor.md file

⚠ Note: You need to run git rebase main before this file will appear locally.

I've improved the type signature, but otherwise all is good 👍
Thanks for the contribution, @yamcodes! Hope to see more from you in the future :)

Wonderful, thank you and happy to help, looking forward to contribute again :D


Reproducible training and evaluation

Currently the list is created from all lstmf files by using sort -R which creates a randomly sorted list. That solution gives results which cannot be reproduced which is unwanted for several reasons.
Using a pseudo random sort with a known seed which can optionally be set by the user would fix this.

https://github.com/stweil/ocrd-train/commit/32b63c5c28173d7f7ac2cf23da95714e23045190 implements a possible solution.

From @wrznr (see #72):

There is a way to pass seeds to sort -R:
seq 1 100 | sort -R --random-source=<(openssl enc -aes-256-ctr -pass pass:"42" -nosalt </dev/zero 2>/dev/null)
where 42 is the seed. This may make the additional script superfluous.

No need for openssl as far as I see. This works, too:
seq 1 100 | sort -R --random-source=<(echo 42)

Even better. Although, python may be faster, I'd like to stick with coreutils.

It only makes a difference of about 30 s on a large data set with 300000 lines, so sticking to coreutils is reasonable. Done in PR #76.

There remains a problem:
The code currently shuffles the result lines from find. Those lines are unsorted. So they must be sorted before they are shuffled to get a reproducible result.

Pull request #77 addresses that additional problem.

This works, too

The simple code without openssl works on MacOS (where I had run that test), but fails on Linux. I added a commit to PR #77 to fix that.

The lists for training and evaluation are now reproducible. First tests with the latest code show that the training results are nevertheless not reproducible: a comparison of the same training running on one machine with latest Tesseract . This needs more examination. I therefore updated the subject of this issue to cover that.
My current hypothesis is that the different results might be caused by differences in the calculation of the dot product. To test this, the same training must be run with identical Tesseract versions on different machines with similar hardware (AVX support). Maybe Tesseract must also be restricted to a single thread.

Very, very interesting.

You can try with --sequential_training to check if that helps.

The rule which builds the unicharset from the box files also uses find. As that command does not sort its findings, the entries in the resulting unicharset have more or less a random order.

With sorted unicharset and disabled OpenMP, the training process seems to be reproducible at least on the same machine. I still have to see whether enabled OpenMP would also give reproducible results.
I tried the same training also on a different host. While it worked nice on the Intel server with a CER of 14.47 % after 10000 iterations, it did not on a 64 bit ARM system where the CER did not drop below 100 %. That's really strange and needs more examination


win: optimize udp receive performance

https://github.com/libuv/libuv/issues/3704
Do at most 32 nonblocking udp receive in a row.

Should this change go to v1.x ?

@bnoordhuis It would be appreciated that if you guys could move this PR forward.

@bnoordhuis @vtjnash Would you mind helping land this PR?

I was about to merge it but I see you're targeting the master branch. If you want to see it in the next release, can you target the v1.x branch instead?

Merged in #3807.


Could you add the license file?

Your article was helpful, thank you.

Glad you found it helpful! Added license here :)


Add CAS1 Placement Matching Outcomes Report

Relates to https://dsdmoj.atlassian.net/browse/APS-472

I do have big and fundamental reservations about how we are providing metrics, CAS-wide. I think we need to have a x-team chat on this topic and consider different approaches and architectures which may be more maintainable, robust and easy to comprehend. But that's another discussion, for the long-term...

Yes, i'm inclined to agree, especially with the move to only provide 'raw' data and let those ingesting it do the heavy lifting


ROX-24331: Fix problem with comma in report form mailing lists

Description
Problem
Thanks to Brad for careful testing to notice that you cannot delete trailing comma in Distribution list text input box.
Analysis
Because mailingLists property has string[] type in payload and response, the usual single-source-of-truth principle of React implies on change:

Call splitAndTrimMailingListsString to convert to split and trim string value from TextInput element.
Call mailingLists.join(', ') to convert back to string.

Here is the unintended result:

As soon as someone types a trailing comma, splitAndTrimMailingListsString appends an empty string to the array, and join with ', ' appends a space to the string.
A backspace deletes the space, but leaves the comma.
The split-join round trip replaces the space.

Solution
A hypothetical solution would replace TextInput element with array of TextInput elements plus explicit interactions to add or delete items from mailingLists array.
Because a redesign is out of scope, second source of truth is an alternative solution.
Checklist

[ ] Investigated and inspected CI test results
[x] ~Unit test and regression tests added~

Testing Performed
Prerequisite for yarn start in ui folder:
export ROX_COMPLIANCE_REPORTING=true

yarn lint in ui
yarn build in ui and then compute branch - master for the following

wc build/static/js/*.js
main.js 0 = 4635821 - 4635821
total 220 = 11712125 - 11711905
ls -al build/static/js/*.js | wc
files 0 = 172 - 172

yarn start in ui

Add temporary console.log statements:

in splitAndTrimMailingListsString function of NotifierConfigurationForm component.
in updateMailingListsString function of NotifierConfigurationForm component.
in NotifierMailingLists component.

Manual testing of Vulnerability Reporting
Prerequisite: create at least one collection for vulnerability reports.
Make sure that both sources of truth are consistent and correct for one delivery destination.

Visit /main/vulnerabilities/reports and click Create report.

Click Add delivery destination and then select a notifier.

See notifier.labelDefault in TextInput element value.

Type comma.

See comma instead of comma-space in TextInput element value.
See absence of empty string item in mailingLists formik value.

Press delete key.

See absence of comma in TextInput element value.

Type comma, double space, another email address, and then double space.

See string exactly as typed in TextInput element value.
See 2 correctly trimmed items in mailingLists formik value.

Click Next.

See 2 correctly trimmed and joined items in Distribution lists column of table.

Click Create.

See expected mailingLists property value in payload and response of POST /v2/reports/configurations request.

Click link in reports table to open report details page.

See expected mailingLists property value in response of GET /v2/reports/configurations/id request.
See 2 correctly trimmed and joined items in Distribution lists column of table.

Manual testing of Compliance Reporting
Make sure that both sources of truth are consistent and correct for multiple delivery destinations.

Visit /main/compliance/schedules and click Create scan schedule.

Click Add delivery destination and then select a notifier.

See notifier.labelDefault in TextInput element value.

Edit Email template to add custom subject.

Click Add delivery destination and then create a notifier.

See notifier.labelDefault in TextInput element value.

Edit the original string, and then add a second string.

See 2 correctly trimmed and joined items in mailingLists formik value.

Type multiple consecutive commas.

See only 2 correctly trimmed and joined items in mailingLists formik value.

Edit Email template to add custom body.

Click Add delivery destination and then create a notifier.

See notifier.labelDefault in TextInput element value.

Edit the original string.

See expected item in mailingLists formik value.

Click Next.

See correctly trimmed and joined items in Distribution lists column of table.

Click Back and then delete the first delivery destination.

See correct values of Distribution list and Email template for the original second and third delivery destinations.

Click Next and then click Create.

See expected mailingLists property value in payload but unexpected notifiers: [] in response of POST /v2/compliance/scan/configurations request.

Click link in reports table to open report details page.

See unexpected notifiers: [] property value in response of GET /v2/compliance/scan/configurations request/id request.

Thank you for careful testing again, Brad.

I wonder if not allowing spaces in the input would have alleviated some of that paranoia..?

For comparison to entering an address when composing an e-mail message in Gmail (although only approximate, because this feature cannot autocomplete based on customer contacts list) the input box allows multiple spaces.

It seems like swimming against the current to use comma as separator in this context.
From viewpoint of component reuse, in hindsight it seems like a better investment of effort:

Omit simulated approximate preview of e-mail contents.
Invest the extra effort on this interaction to render each split item as a deletable chip as soon as user types a space (an a light imitation of every e-mail system for the past 25 years).


Add property to load the config.json on runtime

@pmuir I've force pushed the commit to keep a cleaner history, wdyt about this new implementation?

Looks good.


MANGO-459. Feature: updated workflows.

Codecov Report

Merging #194 (2dcbeea) into develop (7a83a5c) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop     #194   +/-   ##
========================================
  Coverage    59.83%   59.83%
========================================
  Files          202      202
  Lines         2963     2963
  Branches       110      110
========================================
  Hits          1773     1773
  Misses        1169     1169
  Partials        21       21

Flag
Coverage Δ

web
59.83% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7a83a5c...2dcbeea. Read the comment docs.


Radial progress bug iOS

Seems like radial regress is broken

Thanks for reporting.
Which iOS version is that?

It's Chrome on 14.8.1


Hide table footer based on view type

Hi, I need to show/hide table footer based on view type. I've tried with this code and it doesn't working:
<ngx-gantt-table>
    <ngx-gantt-column name="标题" width="180px">
      <ng-template #cell let-item="item"> {{ item.title }} </ng-template>
    </ngx-gantt-column>
      <ng-template #tableFooter [ngIf]="gantt?.viewType == 'day'">

test

      </ng-template>
  </ngx-gantt-table>

v 15.2

Hi, I need to show/hide table footer based on view type. I've tried with this code and it doesn't working:
<ngx-gantt-table>
    <ngx-gantt-column name="标题" width="180px">
      <ng-template #cell let-item="item"> {{ item.title }} </ng-template>
    </ngx-gantt-column>
      <ng-template #tableFooter [ngIf]="gantt?.viewType == 'day'">

test

      </ng-template>
  </ngx-gantt-table>

v 15.2

Angular templates are not slots, so they do not support dynamic conditional if. Let me give you an idea to add different view classes when switching views, and then hide them through the class


Long compilation time for complex narrwoing conditions.

Reported originally by @robwormald
Repro attached.
app.ngfactory.txt

@ahejlsberg has a fix for this.


Overview: Delegation

Task description
For the Sirius release we are going to add the option to delegate an amount to an open baker pool, via the Concordium Mobile Wallet.
The new delegation feature will consist of a handful of new flows and a status page. Each flow and the status page will have their own issues, and this issue can be seen as an overview of them all.
In Figma, the delegation flows will be laid out in an order, to demonstrate how the states of the delegation can change, and how that will influence the different flows and the status page. Some screens will be semi-transparent, indicating that they are redundant to what have been seen earlier, but they have been kept in the flows to give some contextual information.   If you need access to the Figma flows, please reach out to @jens-concordium.
Sub-tasks

[x] The delegation status page
[x] Registering a delegation
[x] Updating a current delegation
[x] Stopping a delegation

Notes:

From a technical perspective, there is just a single transaction related to delegation, namely the ConfigureDelegation transaction. When first registering a delegation, all parameters must be submitted. Subsequently, when making updates, the parameters are optional. Lastly, submitting a delegation amount of 0, equals stopping a delegation.
Making an update transaction that decreases the delegation amount will cause a cool down period for the amount. While the cool down is still pending, no changes can be made to the amount. This also means the delegation cannot be stopped, as that in practice is changing of the amount to 0. This will be reflected on the delegation status page
All other changes to a delegation will be locked at the last epoch up to a pay day. Should the transaction be made during the last epoch up to a pay day, the changes will be locked from last epoch up to the next pay day.

Verified.
Versions

Software Version: 3.0.0 (40)
OS:14.8.1
Browser
Mobile device: iPhone SE


WIP Solid Earth tide correction

This PR introduces the Solid Earth tide (SET) correction in the series of LUTs to give as an input to the geocode SLC core module. SET are computed using pySolid which has been introduced as a new dependency in COMPASS
TO DO:

Validate that computed SET are correct. From preliminary testing, I am able to correctly get SET in ENU. However, when converting ENU SET to the radar grid of rdr2geo, the result of scipy.interpolate.RegularGridInterpolator is zero
Double-check that the correct convention and conversion between azimuth angle and orbit azimuth angle has been applied
SET in LOS and azimuth need to be interpolated to the same grid of the other corrections
Convert SET along azimuth from meters to seconds
SET contributions need to be added to the total LUT in range and azimuth direction
Write the SET contribution (and the other correction layers) inside the output product

Posting Yunjun's preliminary PR review shared on Slack:
Hi Virginia, I am testing your solid Earth tides branch on COMPASS. After configuring the yaml file, s1_cslc.py went through without error, which is very nice.
What’s the output of this step? It seems that it’s computed but not written into files. I would suggest the following to help your own development/validation and communication with others:
a. embed a debug_mode code inside compass, to help testing and future diagnose. I have a lot of this in mintpy (https://github.com/insarlab/MintPy/blob/0b0a8eb6c6e4ac1945398e378bc881bbfd099b0e/src/mintpy/plate_motion.py#L113-L130). We could discuss this in the CSLC tag-up, or you may bring this up in the ADT meeting (I won’t attend ADT meetings until after AGU due to a time conflict with a class).
OR b. have a jupyter notebook uploaded to our notebook repo.
I actually prefer option a, as it’s easier.
Attached screenshot is the output range and azimuth lut value. Both are not reasonable yet, because:

the output unit of pysolid is meter, which need to be converted into our lut unit, which are pixel or second?
the en2az() is not right yet, check here (https://github.com/insarlab/MintPy/blob/main/src/mintpy/utils/utils0.py#L621) please. The LOS azimuth angle and orbit azimuth angle are different, see the definitions here (https://github.com/insarlab/MintPy/blob/main/src/mintpy/utils/utils0.py#L531). Because of this easy confusion (to us and to the future users for sure), I suggest we start to document all the geometry and sign conventions in the github wiki (https://github.com/opera-adt/COMPASS/wiki), starting from this one.
After fixing the two, I think we may be good to go for the PR.

Below are two more minor suggestions:

In the scratch dir for the solid_earth_tides directory, the heading.rdr file is actually the LOS azimuth angle, right? If so, I would suggest to use azimuth_angle.rdr instead, as HEADING angle is technically different from LOS azimuth angle.
I would suggest to use incidence_angle.rdr for the filename as well, to be sure of clarity.

Answering Yunjun's comments:

Debugging statements are actually a good idea but since we are running a bit out of time with the Beta delivery, I would prefer focusing on the theoretical part of the SET computation and whether this is done right. We can always submit an issue and get to it when we have time
You are correct, we need to convert the SET azimuth contribution from meters to seconds before summing it up to the overall azimuth LUT which is the input to the geocode SLC core module.
You are right, I think in the comments I explicitly indicated the convention used for the various declination of azimuth angles. I introduced this function https://github.com/vbrancat/COMPASS/blob/312fd507a5c4627e0c20e1bb33deca0c88ff90d0/src/compass/utils/lut.py#L253  to do the conversion from what we call "heading angle" and to the definition of azimuth angle needed to project the SET contribution from enu to LOS and azimuth directions. I would appreciate if you can double-check
It would be not correct to change the name of heading.rdr into azimuth_angle.rdr as its definition differs from that of the community. I reject this comment
Changed the name of incidence.rdr in incidence_angle.rdr

Closing because branch is stale and takes too much time to resolve conflicts. Will open a new PR.


proposal: garbage collect goroutines blocked forever

Hi everyone!
As of today, Go's runtime does not garbage collect blocked goroutines. The most cited reason being, that goroutines blocked forever are usually a bug and that collecting them would hide this bug. I would like to show a few examples, where garbage collected goroutines would really be appreciated and lead to a much safer and less buggy code.
What does it mean?
package main

import "fmt"

func numbers() <-chan int {
	ch := make(chan int)
	go func() {
		for i := 0; ; i++ {
			ch <- i
		}
	}()
	return ch
}

func main() {
	for i := 0; i < 1000; i++ {
		for num := range numbers() {
			if num >= 1000 {
				break
			}
			fmt.Println(num)
		}
	}
}

This code has a memory leak in current Go. The function numbers returns a channel that generates an infinite sequence of natural numbers. We get this sequence 1000 times in the main function, print the first 1000 numbers of each sequence and quit. The numbers function spawns a goroutine that feeds the channel with numbers. However, once we print the first 1000 numbers produced by the goroutine, the goroutine stays in the memory, blocked forever. If the first for-loop iterated forever, we would run out of memory very quickly.
How to collect goroutines?
I'd suggest the following algorithm:

All non-blocked goroutines are marked active.
All channels not reachable by active goroutines are marked inactive.

Edit: Based on the discussion, I add one detail to the implementation. A goroutine marked dead would be collected in a way that's identical to calling runtime.Goexit inside that goroutine (https://golang.org/pkg/runtime/#Goexit).
Edit 2: Based on the further discussion, runtime.Goexit behavior is debatable and maybe not right.
What are the benefits?
Once such goroutine garbage collection is enabled, we can solve large variety of new problems.

Infinite generator functions. Generator is a function that returns a channel that sends a stream of values. Heavily used in functional languages, they are currently hardly possible in Go (we'd have to send them a stopping signal).
Finite generator functions. We can do them in Go, however, we have to drain the channel if we want to avoid a memory leak.
Manager goroutines. A nice way to construct a concurrent-safe object in Go is to, instead of guarding it by a mutex, create a manager goroutine, that ranges over a channel that sends commands to this object. Manager goroutine then executes this commands as they come in. This is hard to do in Go, because if the object goes out of scope, manager goroutine stays in the memory forever.

All of the described problems are solvable. Manual memory management is also solvable. And this is just like that. Inconvenient, error-prone and preventing us from doing some advanced stuff, that would be really useful.
I'd like to point out, that the whole talk "Advanced Go Concurrency Patterns" would be pointless, if Go collected blocked goroutines.
What do you guys think?

This makes the garbage collector more expensive.
Also, how do you "mark dead" a goroutine? A goroutine can't be killed by another goroutine.  Does the channel operation panic?
This would require a lot of design & implementation work for marginal benefit.

I'm not sure about the details, but it seems logical that the deferred statements would run and then the goroutine would be silently killed. It would not be killed by another goroutine, it would be killed by the garbage collector.
I personally don't think the benefits are marginal. It would make a lot of the obvious code work correctly.
If you watch Rob Pike's "Go Concurrency Patterns" talk, you see that his code examples would not be correct, if the goroutines would not be garbage collected like this. Also, in one point in the talk, being asked by some guy "What happens to that goroutine", he responds something like, "Don't worry, it'll be collected".

I'm not sure about the details,

The proposal process is really about the details, though.

but it seems logical that the deferred statements would run and then the goroutine would be silently killed.

What about locks it's currently holding? Would deferred statements be run?
What about the cases where it actually is a bug to have the leak and the goroutine shouldn't be silently swept under the rug?
Maybe an alternate proposal would be that the sender goroutine should have to explicitly select on the receiver going away:
func numbers() <-chan int {
	ch := make(chan int)
	go func() {
		for i := 0; ; i++ {
			select {
                        case ch <- i:
                        unreachable:  // language change, though.
                                // choose: panic, return silently, etc
                        }
		}
	}()
	return ch
}

But really, now that we have context in the standard library, the answer should be to start the generator goroutine with a context.

I think that deferred calls should be run. If one of the deferred calls is a mutex unlock, than it'll be run. If the blocked goroutine holds a mutex, but no deferred call unlock that mutex, then it's a bug and although the goroutine will be collected, the mutex won't be unlocked which would definitely show the bug, just as good as a goroutine memory leak.
On the Context: of course, it's possible to use here, however, you have to manually cancel it. It's just like manual memory management.

Okay, so your proposal is that a channel send or receive operation that can be dynamically proven to never complete turns into a call to runtime.Goexit (https://golang.org/pkg/runtime/#Goexit)?

Exactly, yes.

I don't agree with calling Goexit.  If a goroutine is blocked forever, we can just garbage collect it without changing any semantics (and maybe sample it or something, for the bug case).  It would never have run again, so there isn't an observable difference except for memory footprint.  Why would it call Goexit, run defers, etc.?  That seems like an unexpected behavior while waiting on a channel.  And it feels like finalizers - we can only provide nebulous guarantees about whether we can detect it and how quickly we do so.
Holding a lock while doing a channel operation should be heavily frowned upon :(

@randall77 You've got a valid point. This would need to be discussed.

How would goroutines waiting on Tickers or Timers work?
What about channels in channels?

@fraenkel Tickers and Timers theoretically always have a running goroutine behind them, counting the time, which eventually sends a signal. So, they're always active.
Channels in channels don't make any difference, if I'm not missing anything. If a channel is not reachable by any active goroutine, then no values will ever be sent or received on it.

If this is implemented (and I think it indeed would be useful), deferred functions should not run. Essentially this should not change the observable behavior of any program except for memory usage.

In fact, it might be a good idea to panic if the garbage collector wants to collect a goroutine but finds that it has pending defers (or more conservatively, is holding a lock), since that's always a bug.

I think this could work, but it would require folding goroutines into the marking phase of the GC.
Instead of treating all goroutines as roots, start with only active goroutines. (Active here means not blocked on a channel.  Goroutines in syscalls are considered active for this.)  Any time we mark a channel, mark all the goroutines waiting on that channel as active, and schedule their stacks for scanning.  At the end of mark, any goroutines not marked active can be collected.  As a bonus, we never need to scan their stack (and we can't, because then we would find a reference to the channel they are waiting on).
There are lots of tricky details:

Channels have direction - if an active goroutine has a send-only reference to a channel, it could never wake up a goroutine receive-waiting on that channel.  The GC has no way to tell different channel references apart (send/receive/bidirectional is only a compile-time thing).
The GC needs to know that channels (or possibly something the channel references, like the goroutine struct) are special objects.  It needs that information to know that when marking the channel, it needs to do extra work it doesn't need to do for normal objects.  We'd need to allocate channels on segregated pages or something.  or maybe marks similar to how we do finalizers?
Goroutines could be selecting on several channels at once.  Any one of those could cause it to be active.

On a more meta level, this proposal would encourage using forever-blocking goroutines during the normal course of business.  Right now they are considered a bug.  The proposal suggests this is just a defense against memory leaks.  But this is a slippery slope people are surely going to drive a mac truck down.  Lazy-evaluating Haskell interpreter, anyone?  Go isn't really a good fit for that, as a goroutine is a really heavy future implementation.  But people will try anyway, and get frustrated with the performance.

If one happens to write incorrect code which creates deadlocks or goroutines which are "dead" for some reason, one cannot easily determine what went wrong since the evidence is now magically removed.

@fraenkel It seems to me that deadlocks could easily be handled as a special case.

@randall77 I already use goroutines heavily as a future implementation that requires manual memory management. The whole point of goroutines being extremely lightweight and scalable is that doing this is not "really heavy". If goroutines are just used where you would use threads in Java, etc, then it has lost some of its utility.

(for example, I often use goroutines to express a sequence of numbers, such as an allocator for object IDs, to avoid intertwining the logic for incrementing counters, etc with the main loop where the business logic happens)

@randall77, you could imagine doing exactly what you propose in the GC, but instead of doing anything else to kill the goroutine, just ~sync.Once a warning about a leaked goroutine to stderr. The runtime doesn't complain to stderr often, but this seems justified enough.

This would also allow some of the more common erlang patterns to be useful in go.

It's possible to detect when a goroutine is blocked for more than some time by examining the stack (proof-of-concept https://github.com/egonelbre/antifreeze/blob/master/monitor.go). As such, one of the approaches could be that you can specify a time limit for how long a goroutine can be blocked.
With regards to the example, this is faster, shorter and doesn't require an additional goroutine:
type Numbers struct{ next int64 }
func (n *Numbers) Next() int { return int(atomic.AddInt64(&n.next, 1)) }

func main() {
	for i := 0; i < 10; i++ {
		var numbers Numbers
		for {
			num := numbers.Next()
			if num >= 1000 {
				break
			}
			fmt.Println(num)
		}
	}
}

@egonelbre Using a timeout would be horrible, as it will either be too long or too short for some purposes. Especially accidentally timing out a goroutine which is still reachable would cause weird unpredictable bugs.
Integrating goroutine-collecting with the GC would make sure that all the collecting is completely "unobservable" without dumping stack traces. And as @DocMerlin mentions, it does significantly increase the expressivity of Go compared to languages like Erlang with somewhat similar concepts of communicating lightweight processes.

I was missing this for a long time too and still don't quite understand, why memory is considered a resource that a programmer should obviously not need to care about, while goroutines are (cf. "What about the cases where it actually is a bug to have the leak and the goroutine shouldn't be silently swept under the rug?" - I could just as easily ask "what about cases where a leaked pointer is actually a bug and the GC is sweeping that under the rug?").
That being said, I believe a) this proposal so far to be too hand-wavy. I think there are a lot of questions to be answered about the details of what goroutines can and can't be collected. And as this feature is only useful, once it's codified in the spec (otherwise a program can't rely on the collection, thus any actual use of it would still be incorrect), these questions would need good, complete but also concise answers.
And b) that indeed, context seems like a good enough solution to at least all cases I care about. In 99% of code, I need to plumb through a context anyhow and rely on it being cancelled upstack.
So, it would be cool, if this could happen, but the edge-cases should definitely be thought hard about.

@Merovius Fully agree. This proposal is not meant to be something complete, I was rather going for to see whether other people were missing this too.
I think it would be useful if some more people from the Go team stated their opinion on this (such as @griesemer, @robpike, @adg, etc. not mentioning @bradfitz, who already contributed).
If the proposal turns out to be reasonable and acceptable, then we might start thinking about the details in depth.

The @golang/proposal-review group meets on Mondays, but generally doesn't comment on proposals until they've been open with community input for a week.

@bradfitz Aaa, thanks, I didn't know that. Does that mean that this proposal will be discussed on the Monday in two weeks?

It depends on backlog.

@Merovius context is similar to manually managing memory using free in my mind, especially in code that does not use the context package pervasively, or even for which a workflow involving context makes no sense (say, an in-memory data structure package)

@Merovius It seems like there's a very simple way of defining "unreachable":

A goroutine is unreachable if it is waiting on a resource that is unreachable (in the GC sense) from any other goroutine stack.

@bunsim: not quite.

A goroutine is unreachable if it is waiting on a resource that is unreachable (in the GC sense) from any other goroutine stack.

... from any other reachable goroutine's stack.
Which of course is a recursive definition.  GC marking solves exactly this problem for objects.

Which is why doing this from the GC makes sense
On Sat, Mar 25, 2017 at 15:14 Keith Randall<EMAIL_ADDRESS>wrote:

@bunsim https://github.com/bunsim: not quite.
A goroutine is unreachable if it is waiting on a resource that is
unreachable (in the GC sense) from any other goroutine stack.
... from any other reachable goroutine's stack.
Which of course is a recursive definition. GC marking solves exactly this
problem for objects.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/19702#issuecomment-289236550, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABLfWwFj-0ZeGnWWmHRbgAllZwlgCdY3ks5rpXWJgaJpZM4MoqnK
.

@bunsim I believe comparing cancelling contexts with using free is a false equivalency. The closest equivalency to memory-management, I believe, would be having a pool-allocator, passing that down-stack and free it as a whole once the request is finished, precisely eliminating the need to call free.
In general, your http/rpc/whatever server should call cancel for you at some point, eliminating the need for you to do it. Now, that doesn't help with code that isn't request oriented, but that's why I emphasized the "I" in my post; personally, I just can't think of any code I would write that a) would care about a goroutine-leak because it's long-running and b) isn't some sort of server with context-scoped work.
In response to your definition of unreachable, waving your hands more vigorously isn't really progress. There are selects to think about, read-only channels, write-only channels, sending channels over channels, changing the values of channel variables, interfaces… I'm not saying, that there isn't a good precise definition to be given that covers all these cases. But I also wouldn't be surprised if solving this problem precisely would turn out to be equivalent to solving the halting problem. It's just very hard to tell, as long as the level of discussion is this high-level.
It is still worthwhile to think about even partial heuristics that can be employed (it would probably be the first step anyway) without codifying it in the spec, just as a clutch to contain the effects of bugs, but as I said, any such heuristic couldn't be relied upon. So as for correctness you can't assume them anyway, making them much less interesting as a feature for the average go-user, I think. At least for the next couple of releases, I, at least, am going to contain my excitement here.

@Merovius There are other languages that support this feature, and "collecting infinitely blocking threads" is something that is already an established concept. For example, Racket does exactly this, with a well-defined specification:
A thread that has not terminated can be garbage collected (see Garbage Collection) if it is unreachable and suspended or if it is unreachable and blocked on only unreachable events through functions such as semaphore-wait, semaphore-wait/enable-break, channel-put, channel-get, sync, sync/enable-break, or thread-wait. Beware, however, of a limitation on place-channel blocking; see the caveat in Places.

Racket has an even more complex system of select-like constructs ("events") than Go's channel select --- everything from events to file descriptors to channels could be put in a select-equivalent --- but yet that paragraph fully describes how the GC collects unreachable threads.
It is certainly not equivalent to solving the halting problem, and is similar in difficulty to memory GC.
This feature would be pretty useless and indeed "bug-hiding" if it were just used to fix up broken programs that leak goroutines. It is certainly possible to implement this as a strict guarantee with strict definitions of unreachability, and in fact that would be the only way I would support this proposal.

Will this proposal keep Go 1 compatibility, for example:
func numbers(p *T) <-chan int {
	ch := make(chan int)
	go func() {
		for i := 0; ; i++ {
			ch <- i
		}

		runtime.KeepAlive(p)
	}()
	return ch
}

if the generator goroutine is collected, then p will be too, which may be not the expected.

We don't know whether the chan will be read in later. If code only know that the chan should be closed or not. So exit groutine explicitly or using time-out is better, I think.
select {
case ch <- i:
case <-time.After(time.Second):
	close(ch)
	return // exit goroutine
}

often, I want to block a goroutine for ever deliberately.
This proposal will make this impossible.

@golang101 This argument doesn't seem very reasonable because:

A blocked unreachable goroutine will not stop the main goroutine from exiting.
A blocked unreachable goroutine does nothing and will never do anything. Collecting it makes no difference except for memory consumption.

So blocked-for-ever main goroutine is not a blocked unreachable goroutine?
Not unreasonable.
There should at least one more condition to support this proposal: there is no other heap value references after the blocking point. The example in https://github.com/golang/go/issues/19702#issuecomment-289259906 obviously doesn't satisfy this condition.

@golang101 I also cannot easily see a situation where your code with runtime.KeepAlive would result in different output with goroutine-collecting on, and goroutine-collecting off.
Can you construct a simple program that uses runtime.KeepAlive in a way so that the program outputs one thing when goroutines are collected, and another thing when goroutines are not collected? runtime.KeepAlive is weird since essentially the only time where it ever affects observable behavior is in a small handful of cases related to syscalls and finalizers.

This proposal does cause finalizers to run that would never have otherwise.

True, but when do finalizers run is already something that's poorly defined and implementation specific. I wouldn't imagine any "real code" breaking if goroutines are collected; indeed right now infinitely blocking goroutines are a memory leak so they are unlikely to occur in non-buggy code.

Unfortunately those finalizers will have access to otherwise blocked
channels.
ch := make(chan int)
runtime.SetFinalizer(obj, func () {
doSomethingWithChannel(ch)
})
On Sun, Mar 26, 2017 at 12:09 PM, Keith Randall<EMAIL_ADDRESS>wrote:

This proposal does cause finalizers to run that would never have otherwise.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/19702#issuecomment-289293924, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA7Wn5Uh-SAjbW7x6pM2uwZZfY8nZ3-7ks5rpo23gaJpZM4MoqnK
.

@randall77 I'd actually say that if you have an object, that is kept from being finalized by a goroutine that's blocked forever, I'd say that that's always a bug and a memory leak and an unintended behaviour. Correct me if I'm wrong.

@RLH : objects with finalizers are GC roots, so if a finalizer can get to a channel, anyone waiting on that channel would not get GCd.  So a goroutine could still block forever without getting GCd if it holds a reference to an object whose finalizer can reference the channel you're forever blocked on.
@faiface: probably.  Just another data point to consider.  We're skirting the edge of the Go1 compatibility promise.

I just pointed an example of going through an infinite loop in the groutine. My opinion is explicitly exit groutine (it self). Because the chan may be read in later.
func main() {
	for i := 0; i < 1000; i++ {
		n := numbers() {
		for num := range n {
			if num >= 1000 {
				break
			}
			fmt.Println(num)
		}
		if rand.Int() % 2 == 0 {
			doSomething(n)
		}
	}
}

finalizer seems good for me but it may be leak.

@mattn I didn't quite get your point here... If the channel can be used later, than it's not unreachable/inactive, so the goroutine sending values on it wouldn't be collected. A channel is unreachable/inactive only in the case, when it's not reachable by following references/pointers from an active goroutine.

@faiface

If the channel can be used later, than it's not unreachable/inactive

I just mean it's harder to detect whether the chan will be unreachable/inactive or not.  (, without finalizer)

Indeed, if goroutines holding references to finalizers will not be collected even if they are blocked indefinitely, it would be very surprising behavior for users, since finalizers might even be set by code in other packages. Goroutine collection would not be a good idea if programmers have to guess whether or not the code they wrote leaks goroutines.
Finalizers are also often used to call free on C-managed memory, in which case running them does make sense when collecting goroutines. I propose that we simply say that the existence of finalizers do not matter, and when goroutines are collected they may be run even if they wouldn't be run otherwise. This slightly breaks compatibility, but literally every change to the GC slightly breaks compatibility w.r.t. when do finalizers run. The introduction of runtime.KeepAlive already broke compatibility to a greater degree, IMO.

@bunsim As the current guarantee re finalizers basically are "they might not run, even if an object is unreachable", I disagree with your characterization. The guarantee allows running fewer or more finalizers of objects that are unreachable. It does not, however, allow running finalizers for objects that are (logically) reachable.
Anyway, I also don't think it's a particularly important breakage. Finalizers are broken anyway; and it seems we were fine adding just such a breakage recently (and introducing runtime.KeepAlive as a poor patch for that).

Exactly. runtime.KeepAlive already breaks the connection between "logical" reachability and finalizers running, so adding goroutine collection will almost certainly not break anybody's correctly written program.

@Merovius

"they might not run, even if an object is unreachable"

Here, it is some different.
If the blocked goroutine doesn't exit, the finalizers will absolutely not run.
But if it exits, the finalizers may run.

Maybe the rare case is not worth to keeping compatibility for it.
I have another simple example. Should the two new created gorotuines be collected or not?
package main

func f(c1, c2 chan int) {
	<-c1
	c2 <-1
}

func main() {
	c1, c2 := make(chan int), make(chan int)
	go f(c1, c2)
	go f(c2, c1)

	// ...
}

Looks the answer is yes.
The two new created gorotuines will not be collected just for there is a global variables holds the channels?
Then how about this:
package main

var cs []chan int

func f(c chan int) {
	<- c
}

func main() {
	c1, c2 := make(chan int), make(chan int)
	go f(c1)
	go f(c2)

	cs = []chan int{c1, c2}

	// ...
}

I see many CLs on the stdlib to reduce the amount of garbage generated, continual effort on the GC to improve its performance and characteristics, and work on the compiler to improve its escape analysis. To collect blocked-forever goroutines would be encouraging more garbage to be created as the balance of when to use a goroutine shifts. Does it give any benefit that can't be obtained with a little bit of extra coding, e.g. select on a closing channel, or Context?

In situations where you are programming in an actor-style with large amounts of communicating goroutines, each acting as an object, then using closing channels or Context becomes very cumbersome, similar to manually managing memory. Sometimes it is nice to be able to program in a style centered entirely around goroutines communicating with channels.
I do agree that in most existing Go code there is no need, but I think that's precisely because it is difficult to write in such a style currently, so it is avoided in favor of things like explicit mutexes or abusing Context in cases where "request scope" makes no sense. Go is supposed to be about programming in a CSP style, not in a C-like "call free/unlock on everything" style (which admittedly Go makes a bit easier due to defer), and collecting garbage goroutines will make it a lot easier.
The improvement to the GC, IMO, instead should be a reason to worry less about garbage in Go.  I've already started to remove all of my usages of tricks like object pooling (which really often circumvents memory safety) unless profiling shows significant performance decrease.

 The answer is that it would be too hard to program in that world, in fear that your goroutine might be killed at any moment. Each goroutine shares state with other goroutines.
What happens to the locks the goroutine holds? The invariants they protect may have been temporarily violated (that's what locks are for) and not yet reestablished. Releasing the lock will break the invariants permanently; not releasing the lock will deadlock the program.
What happens to wait groups waiting for that goroutine? If there's a deferred wg.Done, should it run? If the waitgroup just wants to know the goroutine is no longer exiting, maybe that's OK. But if the waitgroup has a semantic meaning like "all the work I kicked off is done", then it's probably not OK to report that the goroutine is done when in fact it's not really done, just killed.
What happens to the other goroutines that goroutine is expected to communicate with? Maybe the goroutine was about to send a result on a channel. The killer can possibly send the result on the killed goroutine's behalf, but how can the killer know whether the goroutine completed its own send or not before being killed?
For all these reasons and more, Go requires that if you want a goroutine to stop executing, you communicate that fact to the goroutine and let it stop gracefully.
Note that even in Unix, where processes don't share memory, you still end up with problems like in a pipeline p1 | p2 when p2 is killed and p1 keeps trying to write to it. At least in that case the write system call has a way to report errors (in contrast to operations like channel sends or mutex acquisitions), but all the complexity around SIGPIPE exists because too many programs still didn't handle errors correctly in that limited context.
The proposal in this issue amounts to "have the GC kill permanently blocked goroutines". But that raises all the same questions, and it's a mistake for all the same reasons.
More fundamentally, the GC's job is to provide the illusion of infinite memory by reclaiming and reusing memory in ways that do not break that illusion, so that the program behaves exactly as if it had infinite memory and never reused a byte. For all the reasons above, killing a goroutine would very likely break that illusion. If defers are run, or locks are released, now the program behaves differently. If the stack is reclaimed and that happens to cause finalizers to run, now the program behaves differently.
Perhaps worst of all, collecting blocked goroutines would mean that when your whole program deadlocks, there are no goroutines left! So instead of a very helpful snapshot of how all the goroutines got stuck, you get a print reporting a deadlock and no additional information. Deadlocks are today the best possible concurrency bug: when they happen, the program stops and sits there waiting for you to inspect it. Contrast that with race conditions, where the eventual crash happens maybe billions of instructions later and you have to find some way to reconstruct what might have gone wrong. If the GC discards information about deadlocked goroutines, even for partial deadlocks, this fantastic property of deadlocks - that they are easy to debug because everything is sitting right there waiting for you - goes out the window.
Debuggability is the same reason we don't do tail call optimization: when something goes wrong we want to have the whole stack that identifies how we got to the code in question. The useful information discarded by tail call optimization is nothing compared to the useful information discarded by GC reclaiming  blocked goroutines.
On top of all these problems, it's actually very difficult in most cases to reliably identify goroutines that are blocked forever. So this optimization would very likely not fire often, making it a rare event.
I just don't see collecting permanently blocked goroutines happening in any form. There is a vanishingly small intersection between the set of situations where you even identify the opportunity reliably and the set of situations where discarding the goroutines is safe and doesn't harm debugging.
The GC could address both of these problems - correctness and debuggability - by reclaiming goroutines but being careful to keep around any state required so that it looks like they're still there: don't run defers, record all stack pointers to keep those objects and any finalizers reachable (now or in the future) from those objects live, record a text stack trace to show in the eventual program crash, and so on. But this is really just compression, not collection, since some information must still be preserved. Effort spent compressing leaked memory is probably wasted: better to make it easier for programmers to find and fix leaks instead.

@rsc,

Effort spent compressing leaked memory is probably wasted: better to make it easier for programmers to find and fix leaks instead

Thoughts on my comment to Keith above?
https://github.com/golang/go/issues/19702#issuecomment-289178306

@bradfitz, I don't think this is worthwhile. The detection rate is so low that it won't be worth the complexity. Leaked goroutines don't even matter until there are a lot of them; people who care can watch runtime.NumGoroutines() against a limit they choose, and that will be much better at detecting. Or just take a look at /debug/pprof once in a while.

@rsc I don't think this should be about "killing" goroutines. Such a change will never change the observable behavior of any program except for memory usage. It is not about killing "infinitely blocked" goroutines, but rather collecting "unreachable" goroutines. A goroutine that's infinitely blocked but has pointer on its stack to a waitgroup that somebody else is waiting on is obviously not unreachable.
Nothing will change from the perspective of programmers using existing Go idioms, except perhaps debugging, which can easily be solved by adding a flag to GODEBUG's GC optiosn that prints something when goroutines are collected. The only things goroutine collecting would add are new idioms / design patterns.
Of course, it's fine if the larger Go community feel that idioms such as using background goroutines to synchronize an object don't belong in Go, but my point was that this proposal isn't about magically fixing memory leaks in existing programs, but rather to enable programs that haven't been written yet to be written.

@bunsim I understand your motivation was for new patterns. But I don't believe you can split the hairs well enough here to collect only the goroutines that aren't stuck due to bugs. And you shouldn't have to set a GODEBUG flag (that breaks the idioms you want to enable!) after the fact to get useful information about your deadlocked program.

@rsc When you or anybody else have dealt with goroutines stuck forever, how often was it a bug that wouldn't be solved by garbage collecting that goroutine? If it's often the case, could you give any example?

The GODEBUG flag wouldn't break the idioms, it would just print "# of goroutines collected this cycle" as it already does for bytes etc. And you already can totally disable the GC using debugging options anyway.
Nobody is suggesting some weird heuristic to collect only goroutines that aren't stuck due to bugs. It's perfectly okay to collect ones that are stuck due to bugs, it only loses us some debugging info we can recover by simply disabling the GC. And a large amount of existing "buggy code" will simply be the correct way of doing things in the future.
The whole argument sounds suspiciously like "we shouldn't use a GC because it hides bugs from Valgrind".

I want to stress @rsc 's point that if this is to change the way that people write Go code, then it is essential that it be clear when goroutines will be collected, and that goroutines will be reliably collected in that state.  That seems to me to be difficult.  It's a lot harder to tell when a goroutine is blocked than it is to tell when memory is not referenced.  The canonical case here is a goroutine blocked reading from or writing to a channel, but the goroutine itself is holding a reference to the channel; for example, it may be an argument to the function that blocked, and therefore be on the goroutine's stack.  We need to reliably detect not that the channel is unreferenced, but that the only references to the channel are from memory that is only visible to the blocked goroutine.  That seems hard.

@ianlancetaylor That doesn't seem especially harder than GC in general, though?

Weak pointers are easier: you have a pointer type that the GC explicitly ignores (except that it gets updated when an object gets moved, for systems with a moving GC, and it gets cleared when an object is freed.)
What we need for this is something different: given a goroutine G1 blocked on a channel, we need to run a GC from all roots except G1, and see whether we marked the channel.  That is straightforward but too inefficient to actually use.  I don't know an efficient way to implement this.  Perhaps there is one.

@ianlancetaylor What if we run the GC only from the roots of running/runnable goroutines, and then collect all goroutines blocked on channels we didn't mark? This should collect everything in one go, though it's might be hard to integrate it into Go's concurrent GC.

There are many times when a goroutine will be blocked on a channel in a data structure that is accessible to other goroutines if they follow the right chain of pointers in memory, but none of them will. That case and many other related ones fundamentally fool any GC-based algorithm, so that in many cases goroutines will not be collected, and the conditions will be very unclear, depending potentially on optimizations in the compiler that move data between stack and heap, and maybe other optimizations as well. The result will be that it's fairly unpredictable when a blocked goroutine would be collected vs not.
As Ian said, "it is essential that it be clear when goroutines will be collected, and that goroutines will be reliably collected in that state."

@rsc There are many times when a data structure is under a chain of pointers that's accessible from a goroutine if it follows the right chain of pointers in memory, but it never will. So the conditions at which memory will be collected is not clear and GC is "fooled", so garbage collection is fundamentally undecidable. I'm sure you could even construct a scenario where freeing an object at the "right" moment means solving the halting problem. Better use free()!
I've programmed extensively in Racket, a language where unreachable blocked threads are collected, and it is definitely not "fairly unpredictable" when a blocked thread will be collected. There are several common design patterns that involve blocked threads being collected, and in each case it's very clear why it's being collected. In Go's case, we wouldn't be removing any code to shut goroutines down at all in most existing code; we would simply have new idioms in which whether goroutines are collected or not is very obvious.

Just as an example, this proposal would allow users not to call Stop() on time.Ticker. Calling stop on objects that don't manage external resources should be the GC's job.

The issue of calling Stop on a time.Ticker is actually a different problem.  Implementing this suggestion would not solve it.  The problem there is that a time.Ticker has an entry in the runtime timer table, and there is nothing that removes that entry even if the time.Ticker is garbage collected.

@ianlancetaylor Ah, I always thought the reason was that time.Ticker was backed by a sleeping goroutine in a loop.


Can't compile custom rustfmt.

nightly works now so I've been using that.

I have a patch to change the Readme


Coin type changes

This pull request comes as a result of actually using the library and finding couple of things rather cumbersome to use, in particular the Coin type. There is no easy way to work with its amount due to the lack of any getters on the Decimal and the type itself only providing addition-related capabilities. I've decided to take a possibly controversial approach of changing the amount field to an u128. I've done it for a couple of reasons:

even cosmos-sdk's Coin is represented with a big.Int. (Dec is used for DecCoin). Now, the counter argument is that big.Int is capable of representing values up to 2^256 - 1 which is a significantly more than what u128 provides. Plus it can also represent negative values. However, do negative coins make any sense and would be useful on their own? Also, u128 still has higher range than the current Decimal with internally used u64.
I've taken some inspiration from CosmWasm's coin type. In their case, however, they have an additional thin Uint128 wrapper around the u128 primitive mostly to help with the serialization for their clients, which is not applicable in this case.
the crate is still in the pre 1.0 stage thus, even though it's a breaking change, I'd argue it's not as disruptive as if it would have been done later.

What's your opinion on it? I would be more than happy to discuss it and introduce any additional changes. Perhaps rather than going with primitive u128 a different wrapper with properties equivalent to Go's big.Int would be more appropriate?

I agree the current Decimal newtype is unhelpful in its current state.
The only usage of Decimal is amount, so if you're migrating that field away from it you can delete the Decimal newtype entirely.
We can consider this for the next round of breaking changes.

I agree the current Decimal newtype is unhelpful in its current state.
The only usage of Decimal is amount, so if you're migrating that field away from it you can delete the Decimal newtype entirely.
We can consider this for the next round of breaking changes.

Thanks! This will go out in the next breaking release (v0.8)


Last version of EAS CLI breaks Android build.

Build/Submit details page URL
https://expo.dev/accounts/elenitaex5/projects/proapp/builds/b79c6596-0670-400d-8fc2-9f7fcdc5e409
Summary
I have been creating my builds (both Android and iOS) with EAS perfectly right till version<EMAIL_ADDRESS>and I thing 3.14.X too.
But since I upgrade to versions 3.16.X and 3.17.0 my app is crashing. EAS BUILD and submission seems to be ok, but the app crashed since the beginning.
Problem comes because new versions of eas-cli is editing AndroidManifest.xml changing as it said in this issue comment
https://github.com/expo/expo/issues/21238#issuecomment-1636277775
This is the  error shown when app crashes:
java.lang.AssertionError: The embedded manifest is invalid or could not be read. Make sure you have configured expo-updates correctly in android/app/build.gradle. null
	at expo.modules.updates.manifest.EmbeddedManifest.get(EmbeddedManifest.kt:34)
	at expo.modules.updates.launcher.NoDatabaseLauncher.<init>(NoDatabaseLauncher.kt:70)
	at expo.modules.updates.launcher.NoDatabaseLauncher.<init>(NoDatabaseLauncher.kt:23)
	at expo.modules.updates.UpdatesController.start(UpdatesController.kt:238)
	at expo.modules.updates.UpdatesController$Companion.initialize(UpdatesController.kt:506)
	at expo.modules.updates.UpdatesPackage$createReactNativeHostHandlers$handler$1.onWillCreateReactInstanceManager(UpdatesPackage.kt:46)
	at expo.modules.ReactNativeHostWrapperBase.createReactInstanceManager(ReactNativeHostWrapperBase.kt:27)
	at com.facebook.react.ReactNativeHost.getReactInstanceManager(ReactNativeHost.java:41)
	at com.abiglobalhealth.proapp.MainApplication.onCreate(MainApplication.java:66)
	at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1278)
	at android.app.ActivityThread.handleBindApplication(ActivityThread.java:7092)
	at android.app.ActivityThread.-$$Nest$mhandleBindApplication(Unknown Source:0)
	at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2215)
	at android.os.Handler.dispatchMessage(Handler.java:106)
	at android.os.Looper.loopOnce(Looper.java:210)
	at android.os.Looper.loop(Looper.java:299)
	at android.app.ActivityThread.main(ActivityThread.java:8261)
	at java.lang.reflect.Method.invoke(Native Method)
	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:559)
	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:954)

Managed or bare?
Bare
Environment
expo-env-info 1.0.5 environment info:
    System:
      OS: macOS 12.6.3
      Shell: 5.8.1 - /bin/zsh
    Binaries:
      Node: 16.19.0 - /opt/homebrew/opt/node@16/bin/node
      Yarn: 1.22.19 - /opt/homebrew/bin/yarn
      npm: 8.19.3 - /opt/homebrew/opt/node@16/bin/npm
      Watchman: 2<IP_ADDRESS> - /opt/homebrew/bin/watchman
    Managers:
      CocoaPods: 1.11.3 - /opt/homebrew/bin/pod
    SDKs:
      iOS SDK:
        Platforms: DriverKit 22.2, iOS 16.2, macOS 13.1, tvOS 16.1, watchOS 9.1
    IDEs:
      Android Studio: 2021.2 AI-212.5712.43.2112.8609683
      Xcode: 14.2/14C18 - /usr/bin/xcodebuild
    npmPackages:
      @expo/webpack-config: ^18.0.4 => 18.1.2
      expo: ~48.0.18 => 48.0.20
      react: 18.2.0 => 18.2.0
      react-dom: 18.2.0 => 18.2.0
      react-native: 0.71.8 => 0.71.8
      react-native-web: ~0.18.11 => 0.18.12
    npmGlobalPackages:
      eas-cli: 3.17.0
      expo-cli: 6.3.10
    Expo Workflow: bare
    ```

✔ Validating global prerequisites versions passed
✔ Checking for incompatible packages passed
✔ Checking for conflicting global packages in project passed
✔ Verifying prebuild support package versions are compatible passed
✔ Checking dependency versions for compatibility with the installed Expo SDK passed
✔ Validating Expo Config passed
✔ Checking package.json for common issues passed

Didn't find any issues with the project!

### Error output

_No response_

### Reproducible demo or steps to reproduce from a blank project

Upgrading a wellformed project from eas-cli 3.13.X or 3.14.X to 3.16.X or 3.17.X

Try this...
https://github.com/expo/expo/issues/23599#issuecomment-1641991609

Same here

+1

+1

I fixed the issue with adding expo.updates.url (https://exp.host/[organization]/[project]) in app.json

I fixed the issue with adding expo.updates.url (https://exp.host/[organization]/[project]) in app.json

I'll try this out.
Thanks for letting us know!!!
We managed to push the app out using was cli v 1.1.1.

I fixed the issue with adding expo.updates.url (https://exp.host/[organization]/[project]) in app.json

I have to confirm this is working.
Currently on Expo SDK 46.

I fixed the issue with adding expo.updates.url (https://exp.host/[organization]/[project]) in app.json

please post here , link is not working


Kamal 2 Upgrade fails with a host thats only running accessories

If a host does not have the image the tagging fails, I feel like it should be done only on primary host. The log seems self-explanatory.
  INFO [e693dc5b] Running docker tag registry.gitlab.com/image:latest registry.gitlab.com/image:latest on HOST1
  INFO [e693dc5b] Running docker tag registry.gitlab.com/image:latest registry.gitlab.com/image:latest on HOST2
...
  INFO [e693dc5b] Finished in 0.090 seconds with exit status 0 (successful).
  INFO [2e229f51] Finished in 0.108 seconds with exit status 0 (successful).
Releasing the deploy lock...
  ERROR (SSHKit::Command::Failed): Exception while executing on host HOST2: docker exit status: 1
docker stdout: Nothing written
docker stderr: Error response from daemon: No such image: registry.gitlab.com/image:latest

I did not.


Why do you use the px4 autopilot/Tools/simulation/gz/models/x500 The imu update_rate of /model.sdf is 250hz. In the description, you are advised to set the IMU update_rate to 400hz

Why do you use the px4 autopilot/Tools/simulation/gz/models/x500
The imu update_rate of /model.sdf is 250hz. In the description, you are advised to set the IMU update_rate to 400hz

I keep encountering this warning after using make px4_sitl gz_x500
WARN [rl_tools_policy] Policy frequency deviation: 250.00Hz (target 400.00Hz)


Problem configuring access log using a log config file

Thanks for adding access logging to Granian.  I have been trying to configure the access log format using a json config file, but have been unable to do so successfully.  I have been using a config file to change the default log format for a while.  After studying the changes to the log.py file I updated my log_config.json file, but receive a logging error whenever I load a web page.  I've attached my json config file.
To keep things simple I only have the addr atom in my config file, but I would add others once I get this working.  The error displayed is "KeyError: 'addr'.  A little farther down  "ValueError: Formatting field not found in record: 'addr'"  is shown.  At the end of the traceback message the following two lines are displayed:
`Message: '[%(time)s] %(addr)s - "%(method)s %(path)s %(protocol)s" %(status)d %(dt_ms).3f'
Arguments: {'addr': '<IP_ADDRESS>', 'time': '2024-05-29 09:35:05 -0500', 'dt_ms': 48.99859428405762, 'status': 200, 'path': '/Folder/MakePage', 'query_string': b'', 'method': 'GET', 'scheme': 'http', 'protocol': 'HTTP/1.1'}`
The arguments dictionary has the addr key, so I'm confused by the error message.  I've probably got something misconfigured in my config file, but I've worked on this for a couple of hours, and haven't been able to solve the problem.  If the entire traceback message is needed, I can supply it.  Any suggestions for what I need to change/fix are appreciated.  Thanks.
log_config.json

You should use --access-log-fmt to select which atoms to apply. So given your configuration, probably your log_config.json should look like this for access:
{
    "formatters": {
        "access": {
            "fmt": "%(asctime)s.%(msecs)03d] %(message)s",
        }
    }
}

and then you should pass this to granian: --access-log-fmt '%(addr)s'

Thanks for the response.  I probably won't have a chance to try out your suggestion until tomorrow or Friday.

I was able to get the access log formatted the way I wanted based off of your suggestion.  Thanks for the help.  I'll close the issue if I have the permissions necessary.


Add version: JetBrains.Kotlin.Compiler version 1.3.50-eap-5

Pull request has been created with Komac v1.9.1 :rocket:
Microsoft Reviewers: codeflow:open?pullrequest=https://github.com/microsoft/winget-pkgs/pull/116249&drop=dogfoodAlpha

@wingetbot waivers Add Validation-Executable-Error


Register zarrs differently to other file references

The root of the zarr format is a directory. This is stored on FIRE as a zip archive usually with the suffix .zarr.zip. We need to create a workflow to deal with this format.

Need to copy from FIRE, unzip, then copy to S3
Annotate the fileref with type 'zarr' or 'zarr_archive'?
When summarising filetypes just get the size of the archive (do we need both compressed and uncompressed sizes)?

 Register zarrs differently to other file references

There's a prototype of this in https://github.com/BioImage-Archive/bia-integrator/blob/main/tools/scripts/assign_zipped_zarr_from_filerefs.py, it uses "zipped_zarr" as the type, but we don't have to stick to that.


main player graphics shrink

My player text / graphics are tiny, If I minimize the window, and then maximize, it is full scale.  However, when I click on anything, it goes back to being small.

Getting the same issue, but the player gets larger whenever I switch to mini player. Then the app refuses to go back to normal zoom levels, and keeps getting larger

This reliably happens to me on linux. Minimize, then maximize, then navigate anywhere and the font size and images scale down.

Duplicate https://github.com/MarshallOfSound/Google-Play-Music-Desktop-Player-UNOFFICIAL-/issues/2269


Not sure how to use FHIR

[Enter feedback here]
From this documentation I have 0 idea on how to use this feature. Instead, I go to the Language Studio page https://language.cognitive.azure.com/tryout/healthAnalysis and toggle the FHIR toggle on the side. I don't see any different output in the window. The python example here https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/textanalytics/azure-ai-textanalytics/samples/sample_analyze_healthcare_action.py says it showcases the FHIR example but there is no such showcase. The other two examples do not have FHIR in them.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: b3fd449a-0bb6-793f-afc8-465273c4c898
Version Independent ID: 6a81ba92-1ef0-1dff-5bf8-84e14b24db99
Content: How to call Text Analytics for health - Azure AI services
Content Source: articles/ai-services/language-service/text-analytics-for-health/how-to/call-api.md
Service: azure-ai-language
GitHub Login: @jboback
Microsoft Alias: jboback

@QiMata
Thanks for your feedback! We will investigate and update as appropriate.

@QiMata
Thanks for your feedback! I've assigned this issue to the author who will investigate and update as appropriate.

Thanks for your dedication to our documentation. Unfortunately, at this time we have been unable to review your issue in a timely manner and we sincerely apologize for the delayed response. The requested updates have not been made since the creation of this issue, and the timeline for resolution may vary based on resourcing, so we've created an internal work item to incorporate your suggestions. We are closing this issue for now, but feel free to comment here as necessary. #please-close


Is it possible to trigger fzf completion with a key on blesh ?

Hello there!
I currently have fzf completion and bindings working perfectly with blesh, but I was wondering if I could do sort of an hybrid of both, basically instead of using the ** to trigger fzf completion would it be possible to use a custom binding to trigger this kind of completion ?
I've seen this issue https://github.com/junegunn/fzf/issues/1804 where he says to create a custom function for each command which I could do, but I'm wondering if there's a way to just use the existing completion.bash from them.
Thanks in advance

to trigger this kind of completion ?

I'm not sure if I understand the request correctly, but doesn't the example provided in https://github.com/junegunn/fzf/issues/1804#issuecomment-574476419 work?
The completion by ** is implemented as merely parts of shell functions here and here, but there is basically no way to execute only a part of the specified shell function. Then, one needs to anyway extract the relevant part to a new shell function and call it from the keybinding. That seems essentially equivalent to the solution provided in https://github.com/junegunn/fzf/issues/1804#issuecomment-574476419.
I'm not sure if the fzf completion really needs to be called through the programmable completion, but if one needs to force it without implementing a new function or modifying completiion.bash, one may do it in a brute force way by replacing all the completion settings temporarily:
function ble/widget/fzf-complete {
  local completion_save FZF_COMPLETION_TRIGGER=
  ble/util/assign completion_save 'complete -p' # save the current completion settings
  complete -r
  complete -F _fzf_path_completion -D # e.g. if you want to trigger the ** completion for the path completions
  ble/widget/complete; local ext=$?
  complete -r
  builtin eval -- "$completion_save" # restore the original completion settings
  return "$ext"
}
ble-bind -f 'C-x C-i' fzf-complete

To be honest the only advantage I'm looking for is merely aesthetic since the solution proposed by junegunn hides current prompt and replaces it with the fzf program, however the ** completion places the fzf program underneath the line and still displays the current prompt so I would just prefer that solution.
Your snipped works perfectly. From my understanding it temporarily disables the default completion method and replaces it with the fzf one and then restores the previous completions methods. Is there any drawbacks to this approach ? It seems like it still fallsback to default completion for some commands unless they are the ones defined on completion.bash of fzf, but I dont't understand why since the completions should have been cleared...
Either way I guess this should allow me to write a wrapper to achieve what I need!
Thanks once again :)

since the solution proposed by junegunn hides current prompt and replaces it with the fzf program,

Ah, OK. You are right. That is the behavior of Bash for bind -x bindings. Bash hides the last line of the prompt and the command line before calling the shell command registered to bind -x and restores it after the shell command is completed. For the compatibility with Bash, ble.sh also behaves in the same way. ble.sh behaved in a different way in the past, but there was a request to make the behavior of bind -x consistent with Bash.

Is there any drawbacks to this approach?

I worried about the case where the execution of the widget is canceled in the middle and the completion settings are left in the temporary state, but I think it won't happen practically because as far as I remember, I haven't experienced any cancellation of the widget execution in the middle for about nine years since I started to use ble.sh as a daily driver.

It seems like it still fallsback to default completion for some commands unless they are the ones defined on completion.bash of fzf, but I dont't understand why since the completions should have been cleared...

Ah, OK. I forgot that there is another path for ble.sh to find the completion settings. Even if no completion settings are set (in particular complete -p -D is not set), if a shell function _comp_load or _completion_loader exists, ble.sh tries to dynamically load a completion setting using the function. To eliminate also this path, one needs to mask the shell function _comp_load.
In addition, it would probably be better to mask also the completion settings that the fzf completion internally saves.
Here's an updated version:
function ble/widget/fzf-complete {
  # Mask completion settings cached by fzf
  local -a fzf_orig_completions
  fzf_orig_completions=("${!_fzf_orig_completion_@}")
  local "${fzf_orig_completions[@]/%/=}"

  # Replace the programmable-completion setting
  local completion_save
  ble/util/assign completion_save 'complete -p'
  complete -r
  complete -F _fzf_path_completion -D

  # Disable completion auto-loader
  ble/function#push _comp_load 'return 1'

  # Trigger fzf-completion without '**'
  local FZF_COMPLETION_TRIGGER=

  ble/widget/complete; local ext=$?

  # restore the settings
  ble/function#pop _comp_load
  complete -r
  builtin eval -- "$completion_save"
  return "$ext"
}

In the latest push, I added the above ble/widget/fzf-complete (with some changes) in ble-import integration/fzf-completion, so you can now just bind it using ble-bind -f KEYSPECS fzf-complete.

That's awesome! Much appreciated! I'll start using it this way
Thanks once again for your work, ble.sh is simply amazing!

Thank you!


Add common logging function

We should provide a module-level logging function which can be used to enable/disable all logging and to provide common structure to logging output.

Swapped to the enhancement label because this is more of a utility than an actual feature.