[workflow] added notification if scheduled build fails

📌 Checklist before creating the PR

[x] I have created an issue for this PR for traceability
[x] The title follows the standard format: [doc/gemini/tensor/...]: A concise description
[x] I have added relevant tags if possible for us to better distinguish different PRs

🚨 Issue number

Link this PR to your issue with words like fixed to automatically close the linked issue upon merge
e.g. fixed #1234, closed #1234, resolved #1234

Fixed #2575 .
📝 What does this PR do?

Summarize your work here.
if you have any plots/diagrams/screenshots/tables, please attach them here.

This PR added a script to send message to lark if the daily build and test fails.
Better merge this after #2558
💥 Checklist before requesting a review

[x] I have linked my PR to an issue (instruction)
[x] My issue clearly describes the problem/feature/proposal, with diagrams/charts/table/code if possible
[x] I have performed a self-review of my code
[x] I have added thorough tests.
[x] I have added docstrings for all the functions/methods I implemented

⭐ Do you enjoy contributing to Colossal-AI?

[x] 🌝 Yes, I do.
[ ] 🌚 No, I don't.

Tell us more if you don't enjoy contributing to Colossal-AI.

Closed 2580 as well.


Bug: Limit to subdomains, not work

PHP Version
8.0
CodeIgniter4 Version
4.1.9 develop
CodeIgniter4 Installation Method
Manual (zip or tar.gz)
Which operating systems have you tested for this bug?
Windows
Which server did you use?
apache
Database
No response
What happened?

https://codeigniter4.github.io/CodeIgniter4/incoming/routing.html#limit-to-subdomains
You can restrict it to any subdomain by setting the value to an asterisk, (*). If you are viewing from a URL that does not have any subdomain present, this will not be matched:

// Limit to any sub-domain
$routes->add('from', 'to', ['subdomain' => '*']); <-not work
Steps to Reproduce
Change in file: https://github.com/codeigniter4/CodeIgniter4/blob/develop/system/Router/RouteCollection.php#L1354
change old: unset($host[count($host)]);
to new: unset($host[count($host)-1]);
Expected Output
bug fix
Anything else?
No response

or array_pop($host)

@szajens Thank you!
This must be a bug.

I sent a PR #5961.


Remove unused files, add configuration guide for Issuer Kit services.

Added configuration guide for Issuer Kit services.
The files that have been removed are unused.

Yes, the way the demo mode works - as an example - is by building the docker images for each one of the services and starting them with overrides for one (or more) of the configuration files.
I think the docs highlight where the overrides need to be placed, I could maybe reference the docker-compose explicitly, but was trying to avoid doing so in order to not have outdated links pointing at the wrong lines in the file (it's standard docker patterns we are talking about here).


gluesql, litestream, checkpoint + wal, s3

I am aware gluesql can use sled.
Is there a way to have gluesql write to "checkpoint + write ahead log"
The goal is to have something like litestream, where changes are streamed to s3

Could readysets wal connector be useful for this
https://github.com/readysettech/readyset/blob/main/replicators/src/postgres_connector/connector.rs


Cannot update deliver on sierra

New Issue Checklist

[x] Updated fastlane to the latest version
[x] I have read the Contribution Guidelines

Issue Description
Cannot update / install
$ sudo gem install deliver
ERROR:  While executing gem ... (Errno::EPERM)
    Operation not permitted - /usr/bin/fastlane-credentials

$ sudo gem update
Password:
Updating installed gems
Updating activesupport
ERROR:  Error installing activesupport:
	activesupport requires Ruby version >= 2.2.2.
Updating credentials_manager
ERROR:  While executing gem ... (Errno::EPERM)
    Operation not permitted - /usr/bin/fastlane-credentials

Environment
-bash: fastlane: command not found
deliver 1.13.3
Do you use bundler to execute fastlane (i.e. bundle exec fastlane)?
no
Do you use a Ruby environment manager (e.g. chruby, rbenv, rvm)?
no

Hi @rogomantik What happens if you run fastlane run update_fastlane
Let me know if that fixes things for you. Are you using RVM? I'd suggest that since it lets you manage your rubies without using sudo. Thanks!

I don't have fastlane installed, only deliver

$ fastlane run update_fastlane
-bash: fastlane: command not found

If I dont use sudo i get

$ gem install deliver
Fetching: credentials_manager-0.16.2.gem (100%)
ERROR:  While executing gem ... (Gem::FilePermissionError)
You don't have write permissions for the /Library/Ruby/Gems/2.0.0 directory.

I cannot update gem too

$ sudo gem update --system
Updating rubygems-update
ERROR:  While executing gem ... (Errno::EPERM)
Operation not permitted - /usr/bin/update_rubygems
$ gem update --system
Updating rubygems-update
Fetching: rubygems-update-2.6.7.gem (100%)
ERROR:  While executing gem ... (Gem::FilePermissionError)
You don't have write permissions for the /Library/Ruby/Gems/2.0.0 directory.

I think is something related to the sierra update

Hi @rogomantik Can you run ruby -v? It looks like you need to update your ruby version. You might need to install rvm to do this. Thanks!

$  ruby -v
ruby 2.0.0p648 (2015-12-16 revision 53162) [universal.x86_64-darwin16]

Tried to upload data to itunes connect (the screenshots are correctly sized)

Error finding relevant GitHub issues: undefined method `map' for nil:NilClass
[!] The request could not be completed because:
In order to save your changes, all iPad Pro screenshots must be a .jpeg, .jpg, or .png file that is 2732x2048 or 2048x2732 pixels, at least 72 DPI, and in the RGB color space. In order to save your changes, all iPad Pro screenshots must be a .jpeg, .jpg, or .png file that is 2732x2048 or 2048x2732 pixels, at least 72 DPI, and in the RGB color space. In order to save your changes, all iPad Pro screenshots must be a .jpeg, .jpg, or .png file that is 2732x2048 or 2048x2732 pixels, at least 72 DPI, and in the RGB color space. In order to save your changes, all Screenshots for 5.5-Inchs must be a .jpeg, .jpg, or .png file that is 2208x1242 or 1242x2208 pixels, at least 72 DPI, and in the RGB color space. In order to save your changes, all Screenshots for 5.5-Inchs must be a .jpeg, .jpg, or .png file that is 2208x1242 or 1242x2208 pixels, at least 72 DPI, and in the RGB color space. In order to save your changes, all Screenshots for 5.5-Inchs must be a .jpeg, .jpg, or .png file that is 2208x1242 or 1242x2208 pixels, at least 72 DPI, and in the RGB color space. In order to save your changes, all Screenshots for 5.5-Inchs must be a .jpeg, .jpg, or .png file that is 2208x1242 or 1242x2208 pixels, at least 72 DPI, and in the RGB color space. In order to save your changes, all iPad Pro screenshots must be a .jpeg, .jpg, or .png file that is 2732x2048 or 2048x2732 pixels, at least 72 DPI, and in the RGB color space. In order to save your changes, all Screenshots for 5.5-Inchs must be a .jpeg, .jpg, or .png file that is 2208x1242 or 1242x2208 pixels, at least 72 DPI, and in the RGB color space.
#######################################################################
deliver 1.14.2 is available.
It is recommended to use the latest version.
Update using 'sudo gem update deliver'.
#######################################################################
1.14.2 Improvements

Show information that we just skip screenshots if we have too many (#6446)
Update team selection and documentation across all fastlane actions (#6444)
Updated spaceship dependency for improved team selection

1.14.1 Hot fix for saving images

Show language next to screenshot download (#6424)

Fix upload of image assets to iTunes Connect (#6423)

1.14.0 Add support for Stickers app category

Fix Sticker category by adding correct Apps prefix (#6257)
Update internal dependencies (#6104)

Hi @rogomantik Thanks for following up. Can you try installing ruby 2.3+? That should clear this up.

@rogomantik You're using an outdated version of deliver, as shown in your output

deliver 1.14.2 is available.

Please update to the latest release


[Feature Request][Gluon] Add text processing utilities for Gluon like Pytorch/Text

Similar to vision module, MXNet Gluon should have utilities to ease NLP experiments by providing modules like BucketingIterator, Popular dataset loaders etc.
Similar module for PyTorch: https://github.com/pytorch/text

Thanks for the request. It's on my radar.

Thanks Sheng.

#8763

Check out https://github.com/dmlc/gluon-nlp


Lmql playground crashes with: [WinError 2] The system cannot find the specified file

Hi all,
i have the following system:
Win11; python 3.11.6 or pyhton 3.12.0
Fresh ennvironment by python-venv
i try to run lmql playground but got stuck with the following:
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in run_code
File "C:\Users\I009200.pyenv\pyenv-win\versions\3.12.0\Scripts\lmql.exe_main.py", line 7, in
File "C:\Users\I009200.pyenv\pyenv-win\versions\3.12.0\Lib\site-packages\lmql\cli.py", line 260, in main
command0
File "C:\Users\I009200.pyenv\pyenv-win\versions\3.12.0\Lib\site-packages\lmql\cli.py", line 127, in cmd_playground
if subprocess.call(["yarn", "--version"]) != 0:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\I009200.pyenv\pyenv-win\versions\3.12.0\Lib\subprocess.py", line 389, in call
with Popen(*popenargs, **kwargs) as p:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\I009200.pyenv\pyenv-win\versions\3.12.0\Lib\subprocess.py", line 1026, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\I009200.pyenv\pyenv-win\versions\3.12.0\Lib\subprocess.py", line 1538, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] The system cannot find the specified file
Can you help me?
Thanks in advance

Hi there :) Have you installed yarn and node in your command line environment, and are those command readily available there? Otherwise, can you maybe try running with WSL as an alternative?

Hi :)
I have both installed. Does that mean linux is the prefered environment?

@ChristophRaab - did you figure this out?  I'm hitting the same issue

Got this working - had to modify the cli.py file - anywhere subprocess was being used with the yarn command, I needed to add the argument shell=True

This solution worked for me as well, cheers @AndrewTHEManeri!


Button Dark Mode

Initial buttons landed in PR #4602. However, they don't have any concept of dark mode. Once Theming is landed #4491 I need to loop back and upgrade Buttons to account for dark mode modifications.

Resolved by #4655


Readme: add deprecation notice

As we talked about in many issues (recently this), jQuery.Turbolinks will no longer be updated.
I've added links to "alternatives" (if you can call it that):

http://ricostacruz.com/rsjs/ - A document with instructions on how to write Turbolinks-compatible JS, and
https://www.npmjs.com/package/onmount - a 1kb helper library to help with writing Turbolinks-compatible JS behaviors.

Lovely, thank you a lot!


Reimplement useMachine to always start service immediately

⚠⚠⚠Right now this is just an experiment I'd like to gather feedback on ⚠⚠⚠
There is this mantra repeated by React team that subscriptions should be done in a commit phase (inside an effect) - mainly because only then we get also an "access" to a cleanup mechanism. And given the React model - this all is true.
Here, however, we don't deal with an external source - but rather we create our own and we own it. So it seems to me that rules are a little bit different for such a scenario. We could even move subscribing to the moment right after calling .start() (in render phase). We don't actually need a cleanup phase because we should be able to rely upon GC doing its job when our component unmounts (or doesn't ever get committed) - the source (service) should be shared only in its React subtree, so if this gets removed from memory then it should just free all references to created service and it should get removed from as well. useState could be also seen as such "internal" source and a component is "subscribed" to it right from the start after all.

@davidkpiano I've removed the possibility to swap a machine as discussed. The code has changed quite a bit from a prior version so this is ready for a re-review now. Do you have in mind any particular tests I should add for this?

@davidkpiano WDYT about this PR as a whole? should I go for it? or do you maybe have some concerns about the used approach?

This LGTM


Doc update for v1.3.7

Updated manifest and deployment versions

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 65.628%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
2316

Relevant Lines:
3529

💛  - Coveralls


Removing model import - new SDK change

Removed import model "from azure.cognitiveservices.vision.customvision.prediction.prediction_endpoint import models". The new SDK change doesn't require this line and having this will break the code.

@cbtham : Thanks for your contribution! The author, @areddish, has been notified to review your proposed change.

@cbtham  Thanks! yes this line should be removed as the naming has changed.

#sign-off


JavadocType fix

remove this:
https://github.com/eo-cqrs/eo-kafka/blob/26e54d848913812f8365a832b5c4c84d71fc5df6/checkstyle-suppressions.xml#L33
and fix all warnings until the build is successful
part of #231
@h1alexbel set milestone please

I would like to do this

@NikiTuz18 sure, go ahead

@l3r8yJ or @h1alexbel, you can advise me about this todo's in "FkConsumer.java" and "FkProducer.java":
`/**

@todo #49:45m/DEV Fake Consumer implementation
/ /*
@todo #10:45m/DEV Fake Producer implementation
*/`

They prevent maven install. Can I remove this todo's tags?

There is one more point. During the build, an error occurs, in my opinion, unrelated to the documentation:
"[ERROR] src\main\java\io\github\eocqrs\kafka\parameters\KfFlexibleEnvelope.java:[46,23] (design) VisibilityModifier: Variable 'settings' must be private and have accessor m
ethods.
"

@l3r8yJ or @h1alexbel, you can advise me about this todo's in "FkConsumer.java" and "FkProducer.java":
/**
 * @todo #49:45m/DEV Fake Consumer implementation
 */
/**
 * @todo #10:45m/DEV Fake Producer implementation
 */

[ERROR] src\main\java\io\github\eocqrs\kafka\fake\FkConsumer.java:[42,4] (javadoc) JavadocType: Unknown tag 'todo'.
[ERROR] src\main\java\io\github\eocqrs\kafka\fake\FkProducer.java:[38,4] (javadoc) JavadocType: Unknown tag 'todo'._

They prevent maven install. Can I remove this todo's tags?

@NikiTuz18 you just need to remove the first character * and it should pass, i.e:
/*
 * @todo #10:45m/DEV Fake Producer implementation
 */

There is one more point. During the build, an error occurs, in my opinion, unrelated to the documentation: [ERROR] src\main\java\io\github\eocqrs\kafka\parameters\KfFlexibleEnvelope.java:[46,23] (design) VisibilityModifier: Variable 'settings' must be private and have accessor methods.

@NikiTuz18 simply fix the existing suppression for this rule in checkstyle-suppressions.xml

@l3r8yJ or @h1alexbel, you can advise me about this todo's in "FkConsumer.java" and "FkProducer.java":
/**
 * @todo #49:45m/DEV Fake Consumer implementation
 */
/**
 * @todo #10:45m/DEV Fake Producer implementation
 */

[ERROR] src\main\java\io\github\eocqrs\kafka\fake\FkConsumer.java:[42,4] (javadoc) JavadocType: Unknown tag 'todo'.
[ERROR] src\main\java\io\github\eocqrs\kafka\fake\FkProducer.java:[38,4] (javadoc) JavadocType: Unknown tag 'todo'._

They prevent maven install. Can I remove this todo's tags?

@NikiTuz18 you just need to remove the first character * and it should pass, i.e:
/*
 * @todo #10:45m/DEV Fake Producer implementation
 */

@l3r8yJ If do this that give a another error:
[ERROR] src\main\java\io\github\eocqrs\kafka\fake\FkConsumer.java:[43] (javadoc) JavadocType: Missing a Javadoc comment.
[ERROR] src\main\java\io\github\eocqrs\kafka\fake\FkProducer.java:[39] (javadoc) JavadocType: Missing a Javadoc comment.

Done, it's fixed

check out this

There is one more point. During the build, an error occurs, in my opinion, unrelated to the documentation: [ERROR] src\main\java\io\github\eocqrs\kafka\parameters\KfFlexibleEnvelope.java:[46,23] (design) VisibilityModifier: Variable 'settings' must be private and have accessor methods.

@NikiTuz18 simply fix the existing suppression for this rule in checkstyle-suppressions.xml

Done, it's fixed

@NikiTuz18, great!
please submit pull request

@NikiTuz18, great! please submit pull request

That's not all, i need to fix it:

@l3r8yJ or @h1alexbel, you can advise me about this todo's in "FkConsumer.java" and "FkProducer.java":
/**
 * @todo #49:45m/DEV Fake Consumer implementation
 */
/**
 * @todo #10:45m/DEV Fake Producer implementation
 */

[ERROR] src\main\java\io\github\eocqrs\kafka\fake\FkConsumer.java:[42,4] (javadoc) JavadocType: Unknown tag 'todo'.
[ERROR] src\main\java\io\github\eocqrs\kafka\fake\FkProducer.java:[38,4] (javadoc) JavadocType: Unknown tag 'todo'._

They prevent maven install. Can I remove this todo's tags?

@NikiTuz18 you just need to remove the first character * and it should pass, i.e:
/*
 * @todo #10:45m/DEV Fake Producer implementation
 */

@l3r8yJ If do this that give a another error: [ERROR] src\main\java\io\github\eocqrs\kafka\fake\FkConsumer.java:[43] (javadoc) JavadocType: Missing a Javadoc comment. [ERROR] src\main\java\io\github\eocqrs\kafka\fake\FkProducer.java:[39] (javadoc) JavadocType: Missing a Javadoc comment.

@NikiTuz18, great! please submit pull request

That's not all, i need to fix it:

@l3r8yJ or @h1alexbel, you can advise me about this todo's in "FkConsumer.java" and "FkProducer.java":
/**
 * @todo #49:45m/DEV Fake Consumer implementation
 */
/**
 * @todo #10:45m/DEV Fake Producer implementation
 */

[ERROR] src\main\java\io\github\eocqrs\kafka\fake\FkConsumer.java:[42,4] (javadoc) JavadocType: Unknown tag 'todo'.
[ERROR] src\main\java\io\github\eocqrs\kafka\fake\FkProducer.java:[38,4] (javadoc) JavadocType: Unknown tag 'todo'._

They prevent maven install. Can I remove this todo's tags?

@NikiTuz18 you just need to remove the first character * and it should pass, i.e:
/*
 * @todo #10:45m/DEV Fake Producer implementation
 */

@l3r8yJ If do this that give a another error: [ERROR] src\main\java\io\github\eocqrs\kafka\fake\FkConsumer.java:[43] (javadoc) JavadocType: Missing a Javadoc comment. [ERROR] src\main\java\io\github\eocqrs\kafka\fake\FkProducer.java:[39] (javadoc) JavadocType: Missing a Javadoc comment.

I can remove comment's with todo

@NikiTuz18 don't remove it, we need those tickets to be in code for now


package name is not PEP8

"Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability. Python packages should also have short, all-lowercase names, although the use of underscores is discouraged."

https://www.python.org/dev/peps/pep-0008/#package-and-module-names

It is possible my pytest doesn't work due to this irregularity in the python package name.
output of my python test log:
ERROR collecting tests/test_exclusions.py __________________
ImportError while importing test module 'C:\Users\ali_m\AnacondaProjects\PhD\Semiology-Visualisation-Tool\tests\test_exclusions.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests\test_exclusions.py:2: in <module>
    from mega_analysis.semiology import mega_analysis_df
E   ModuleNotFoundError: No module named 'mega_analysis'

I propose

rename repo to "SemiologyVisualisationTool"

we use the command

$ git remote set-url origin new_url
https://help.github.com/en/github/administering-a-repository/renaming-a-repository

agree?

The Python package name is mega_analysis, it has nothing to do with the name of the repo. It is a valid name, like face_recognition. I suggest you try pytest on a toy repo.


(operate) feat: Add column for how much xdai needed in staking contracts page

Proposed changes

Added a "Minimum Operating Balance Required" column
updated a few styles.

Fixes

Types of changes
What types of changes does your code introduce?
Put an x in the boxes that apply

[ ] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Just be mindful of CoinGecko rate limits as this list grows longer, can probably cache some requests if it becomes a problem, or reuse values


"catchall" enum?

Wondering if there is an easy way to implement an enum mapper that designates one value as a "catchall" rather than throwing an error if an enum is not within the known set.
For example if I am building a Flutter app and today the app and the server are both sync'ed, we have an enum like:
@MappableEnum()
enum PreferredContactMethod {
  PhoneCall, Email, SMS
}

and next month the developers implement WhatsApp and WeChat. People who have an updated app will be able to receive these enum values from the server, but older clients will throw an error when the server sends PreferredContactMethod.WhatsApp and that is not a known value. Getting your entire userbase to upgrade their app can take years.
Instead, what if we anticipated the issue and built the initial enum like:
@MappableEnum()
enum PreferredContactMethod {
 @MappableCatchall() Unknown,
 PhoneCall,
 Email,
 SMS
}

and any value returned from the server that wasn't a known value would be routed to Unknown. This is preferred over throwing an error, the client can display a 'please update' message or default to a different choice or cleanly handle some other way.
Thank you.
EDIT: just found out this exists in json_mappable with the JsonKey.unknownEnumValue annotation

Use `@MappableEnum(defaultValue: PreferredContactMethod.Unknown)


Fixed bad indentation for code example in README file

Fixed typo in code example of Store Revisions section

Repo too old. No one responding.


Emit signal groups in GTKW files

Fixes #764.
Things to look at in the review:

Group names for views and interfaces. The current code uses "view" and "interface". Maybe the name should reference a layout/signature in some way?
Treating single-field view as in https://github.com/amaranth-lang/amaranth/issues/764#issuecomment-2014220740 is not implemented. It could be implemented by adding a special case.
I've added tests that check if the code doesn't raise an exception, the resulting gtkw files are not checked. I'm not sure how to best check them; I could compare them with a reference, but this would freeze the signal naming. Is signal naming for views and interfaces specified?
Had to add another change related to the recent private names commit. Now that there are three separate recursive functions in this PR, maybe the recursion pattern should be abstracted away?

@tilk Could you rebase the PR please? We've finally landed MemoryData, a proper replacement for MemoryIdentity that also improves tracing experience, but this caused a conflict here.

@tilk, the rework of the simulator necessary for the RFC 36 has introduced some merge conflicts here--could you fix them please?

Could you squash the PR please? The commit message could be something like
sim: group signal traces according to their function.

Thanks!


[fix](inverted index) fix the issue where need_read_data does not han…

…dle non-pushdown conditions
https://github.com/apache/doris/pull/36292
Issue Number: close #xxx

run buildall


Q5: git annex for machine state

"Q5 / Future Thinks + No Budget"-office here again. :wink:
I stumbled over https://git-annex.branchable.com/ and it looks like a suitable idea on how to manage user state.
cc/ @cole-h since his readme prominently exposes the challenge.

Though gitlab decided to drop support a while ago in favor of git lfs, details: https://gitlab.com/gitlab-org/gitlab/-/issues/1648

Though gitlab decided to drop support a while ago in favor of git lfs, details: https://gitlab.com/gitlab-org/gitlab/-/issues/1648

@blaggacao What do you mean by “user state” exactly? Only immutable state provided by “the system”?
Things to consider:

You only need git-annex for large files, so not password files etc.
It ties you completely to git, which hinders progress (look e.g. at pijul)
Git-annex only has gains over plain (possibly encrypted) git-lfs if you want to have a partial state in some repos and configure the spread of the data over the repositories.
At least you'd have to handle unlocking+locking/commiting of files as by default everything is immutable and referenced by symlinks.
For mutable files/non-symlink mode git-annex uses gits smudge/clean filters which the whole file is in principle piped through (though git-annex uses some tricks there). This is inefficient if you don't really need these filters.
Yet another really complex component of this system. Nix + git-annex + dhall + systemd for a basic system looks a bit complex to be suitable for a “community consensus on how to manage a NixOS system”, which should also be understandable by beginners (#137).

For mutable state: What about dysnomia?

Those are very thoughtful points, thank you!
I'm especially intrigued by pijul which seems to be a modern alternative to sqlite's fossil. I will keep an eye on it.

What do you mean by “user state” exactly? Only immutable state provided by “the system”?

Not sure.
In principle, git lfs does the job of putting blobs under the same workflow as system state, too. I have no understanding if pijul/fossil have a similar feature.
A vcs based workflow might serve two goals, in this context:

snapshotting
syncing (between hosts)

Dysnomia sounds interestingly convoluted which might or might not constitute a quality attribute (which I'm far from able to judge). I hope the docs get more accesible as the project evolves.
Maybe we actually need syncing and snapshotting only as a useful afterthought for some classes of state?


TitleWidget: Improve compatibility of ShortTitles.

This should fix some issues people have with short titles and Firefox, and possibly other programs too.
This should improve our existing functionality. See https://github.com/rickbutton/workspacer/pull/118 for discussion.

PRs haven’t been merged in months and Rick haven’t been answering emails either.
Right now it seems a fork is needed to keep workspacer going forwards.
I’m discussing this in a issue here: https://github.com/rickbutton/workspacer/issues/211. Feel free to let me know your thoughts.


RTE: enable per-node-group tuning

Introduce per-node-group config (NodeGroupConfig) object, in order to enable fine-tuning of RTE.
This is needed to enable cluster-admins to explore the proper tradeoff between cluster load and scheduling precision.
We will also need to have some slack to explore the best default to set, and we need some scalability knobs for the scale benchmarks (due "soon").
The new Config object means we need to deprecate the DisablePodsFingerprinting bool we added in the 4.11 cycle.
This also plays nice with the ongoing deprecation of the bool flags in the k8s APIs.

/hold
missing controller tests. We don't need e2e tests, but controller tests we definitely need.

/hold cancel
tests added

/retest

Thanks for addressing all the review comments.
/lgtm
/approve

/cherry-pick release-4.11

/cherry-pick release-4.11

/cherry-pick release-4.11

/cherry-pick release-4.11


Create Create sl.json

Slovenian translations.

While translating into Slovene, I noticed what I believe are two typos in the .en.json file. Could you please review the following lines:

Line 311: It seems to be identical to line 307, but perhaps it should read, "Will be played when an exercise is closed."
Line 588: Is the period here intentional? ". Exercise for @stagelabel""

@miropuhek Thanks for being so observant!
The first one was in fact a copy&paste error where I forgot to replace "opened" with "closed".
The second one is in fact correct. It will be concatenated to the stage title that is fetched from the exercise. But I can see how this can be confusing. I'll change this.


reflect: fatal error: reflect mismatch when reflect.MakeFunc is used as a goroutine

Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?
go version go1.10.3 linux/amd64
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GORACE=""
GOROOT="/usr/lib/google-golang"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/google-golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="clang"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build622183756=/tmp/go-build -gno-record-gcc-switches"

What did you do?
I have a function consume() made by reflect.MakeFunc(). When I run it as a goroutine go consume(...), if it coincides with a GC cycle, the program dies with the following error:
runtime: confused by reflect.makeFuncStub
fatal error: reflect mismatch
...
runtime stack:
...
goroutine 166 [runnable (scan)]:
reflect.makeFuncStub(0xc42001e0c0, 0xc420014140, 0x0, 0x0, 0x0, 0x0, 0xc42013e800, 0x0, 0x0, 0x0, ...)
        /usr/lib/google-golang/src/reflect/asm_amd64.s:12
created by main.runConsume
        /usr/local/google/home/liulk/tmp/test-go/reflectmismatch.go:52 +0x7e

Each time the fatal error happens, the goroutine with runnable (scan) status is always in reflect.makeFuncStub(). However, if I wrap the goroutine under a func() closure like go func() { consume(...) }() then the problem goes away.
Here is a playground link showing that this issue is reproducible in the playground: https://play.golang.org/p/vDyOEMZoAWf
There is a previous issue also mentioning reflect mismatch, but I think that's a different issue that is already fixed: https://github.com/golang/go/issues/18635
What did you expect to see?
The program should run until completion.
What did you see instead?
The program crashed with "fatal error: reflect mismatch"

/cc @ianlancetaylor @aclements

Works in Go 1.3, fails in every version since.  Works with gccgo.  Easy to work around.  Setting milestone to 1.12.

Kindly paging also @randall77

The easy, shallow fix for this is probably to teach the compiler to do the “wrap in a closure” truck itself. It already does that for a print and some other builtins; this probably just requires adding a special case for (say) any calls in package reflect. Might be a decent starter issue for someone working on the compiler.

It looks like the problem is scanning a goroutine where the top level function is reflect.makeFuncStub, but the goroutine hasn't started yet. There's no frame for reflect.makeFuncStub that the runtime can grab the arg map from. We'd have to grab that map from the ctxt register (or, more precisely, the g.gobuf.ctxt value), like we do with defers.
Here's a simpler repro:
package main

import (
	"reflect"
	"runtime"
)

const N = 100

func main() {
	c := make(chan bool, N)
	for i := 0; i < N; i++ {
		f := reflect.MakeFunc(reflect.TypeOf(((func(*int))(nil))),
			func(args []reflect.Value) []reflect.Value {
				c <- true
				return nil
			}).Interface().(func(*int))
		go f(nil)
	}
	runtime.GC()
	for i := 0; i < N; i++ {
		<-c
	}
}

run with
GOMAXPROCS=1 go run issue25897.go

@josharian I don't think the wrapper strategy will work. At the point of the go statement, the thing we're running is just a closure - we don't know where the implementation comes from.  Or are you saying that we should wrap just makeFuncStub in another wrapper function?


Error starting userland proxy: bind: address already in use - TASK [Ensure internet-monitoring environment is running.]

After resolving other issues (thus after having run the playbook a few times), the docker containers have been created and re-running the playbook causes this issue :
TASK [Ensure internet-monitoring environment is running.] *******************************************************************************************************************************************************
fatal: [<IP_ADDRESS>]: FAILED! => {"changed": false, "errors": ["ERROR: for internet-monitoring_speedtest_1  Cannot start service speedtest: driver failed programming external connectivity on endpoint internet-monitoring_speedtest_1 (d8dcfa01510ba4d07f5e5438a9dcc9426fd233fee10549ffe4e4b5b9db716662): Error starting userland proxy: listen tcp <IP_ADDRESS>:9798: bind: address already in use", "ERROR: for internet-monitoring_nodeexp_1  Cannot start service nodeexp: driver failed programming external connectivity on endpoint internet-monitoring_nodeexp_1 (8f9ce9324d12c44b75eed9c566c4fd3749979f6383f6e3379f3e31e88c96a3b2): Error starting userland proxy: listen tcp <IP_ADDRESS>:9100: bind: address already in use"], "module_stderr": "Starting internet-monitoring_speedtest_1 ...
Starting internet-monitoring_ping_1      ...
Starting internet-monitoring_nodeexp_1   ...
Host is already in use by another container
Starting internet-monitoring_speedtest_1 ... error

ERROR: for internet-monitoring_speedtest_1  Cannot start service speedtest: driver failed programming external connectivity on endpoint internet-monitoring_speedtest_1 (d8dcfa01510ba4d07f5e5438a9dcc9426fd233fee10549ffe4e4b5b9db716662): Error starting userland proxy: listen tcp <IP_ADDRESS>:9798: bind: address already in use

Host is already in use by another container
Starting internet-monitoring_nodeexp_1   ... error

ERROR: for internet-monitoring_nodeexp_1  Cannot start service nodeexp: driver failed programming external connectivity on endpoint internet-monitoring_nodeexp_1 (8f9ce9324d12c44b75eed9c566c4fd3749979f6383f6e3379f3e31e88c96a3b2): Error starting userland proxy: listen tcp <IP_ADDRESS>:9100: bind: address already in use\nHost is already in use by another container
Starting internet-monitoring_ping_1      ... error
", "module_stdout": "", "msg": "Error starting project Encountered errors while bringing up the project."}

I've tried to remove the containers/purging the system, but when I do this the system has to re-download all images etc., which causes timeout errors and halts the playbook.

@fred-corp - You can also try just killing any active Docker containers that cause that; run docker ps to find running containers and then docker rm -f [container id here] to remove.

Running docker rm -f [container id here] did not work; it would delete the defined container, but the error would persist. I solved it by starting from zero (deleting this VM and creating a new one). Then :

install docker
make sure $USER & root are members of docker group
install the latest version of ansible
disable systemd-resolved (Just an Ubuntu thing to do)
Running the ansible-galaxy and ansible-playbook commands

A bit of a shame I couldn't resolve it without having to reinstall ubuntu...


updated C# server_reflection.md

The server reflection example in C# is old (as service registration happen in in startup.cs and build in program.cs)
https://github.com/grpc/grpc/blob/master/doc/csharp/server_reflection.md
I have a working example which had server reflection implementation in a another way (can be read as "better" way). this implementation in easy to understand and inline with current .net core structure.
https://github.com/rupeshtech/k8s-grpc-dotntecore/blob/master/SampleGrpcService/Services/ReflectionImplementation.cs
https://github.com/rupeshtech/k8s-grpc-dotntecore/blob/master/SampleGrpcService/Startup.cs
release notes:
Added  another implementation of server reflection for language csharp (c#)
language: C# (chsarp)
Version: .net core 3.0
@nicolasnoble

@nicolasnoble , CLA is signed..

gonna create another pull request with Language label


Doing a pip install in root folder of spesmilo/electrumx will install previous upstream 1.14.0

Inside spesmilo/electrumx (latest git master branch) if you do a pip3 -m pip install * will install ex 1.14.0 (previous upstream release) instead of spesmilo/electrumx 1.16.0 -- is this expected?

pip3 -m pip install *

Uhm... do you mean python3 -m pip install . ?
Try running
python3 -m pip uninstall electrumx
python3 -m pip uninstall e-x
python3 -m pip install .

Maybe show some terminal output.

Yes, sorry,  I meant python3 -m pip install *
I have run python3 -m pip uninstall electrumx and python3 setup.py install and it installed the correct package. But python3 -m pip install * will not install the correct one.


The signature is not written red although it's invalid when the sender/receiver is changed

This is a valid example-transaction at /blockchain:

When I edit the amount (in $), the signature gets invalid => signature is written red

But when I edit the sender or the reciever instead, the signature should also be invalid and be written red in my opinion (as it's invalid). That's not the case:

Yeah, thanks for bringing this up. There is a known bug in the message signing library used that will take some work to fix. I agree, it causes confusion in the way things are working right now. I'll eventually fix this although my time is short.

Thanks for this. I've fixed it in PR https://github.com/anders94/public-private-key-demo/pull/6 which I'll shortly be deploying at https://anders.com/blockchain/public-private-keys/ as well.


Unable to run 'nanobox dev'.

Hey there! I have some trouble trying to setup nanobox on my project. I've already looked to another similar issues but nothing of that helps.
I've downloaded nanobox with and without Vagrant and VirtualBox.
I've cloned my repo from a remote server.
Execute 'nanobox dev', on my root project folder. Everything seems to install correctly, I type my password and for some reason I'm having the same error.
This my terminal output:

➜  site-frontend git:(master) nanobox dev
+> Creating /Users/Gabo/.nanobox directory
+> Installing nanobox/boot2docker...
   [***************************************   99%] 192k/s (0:00:01 remaining))
+> Creating a nanobox
+> Nanobox may require admin privileges to modify your /etc/hosts and /etc/exports files.
Password:

   - Importing nanobox base image....
   - Booting virtual machine........
   - Configuring virtual network.........................A vagrant error occured (See /Users/Gabo/.nanobox/apps/site-frontend/vagrant.log for details)
A vagrant error occured (See /Users/Gabo/.nanobox/apps/site-frontend/vagrant.log for details)
A vagrant error occured (See /Users/Gabo/.nanobox/apps/site-frontend/vagrant.log for details)
A vagrant error occured (See /Users/Gabo/.nanobox/apps/site-frontend/vagrant.log for details)
A vagrant error occured (See /Users/Gabo/.nanobox/apps/site-frontend/vagrant.log for details)
A vagrant error occured (See /Users/Gabo/.nanobox/apps/site-frontend/vagrant.log for details)
A vagrant error occured (See /Users/Gabo/.nanobox/apps/site-frontend/vagrant.log for details)
A vagrant error occured (See /Users/Gabo/.nanobox/apps/site-frontend/vagrant.log for details)
A vagrant error occured (See /Users/Gabo/.nanobox/apps/site-frontend/vagrant.log for details)
A vagrant error occured (See /Users/Gabo/.nanobox/apps/site-frontend/vagrant.log for details)
A vagrant error occured (See /Users/Gabo/.nanobox/apps/site-frontend/vagrant.log for details)
A vagrant error occured (See /Users/Gabo/.nanobox/apps/site-frontend/vagrant.log for details)
A vagrant error occured (See /Users/Gabo/.nanobox/apps/site-frontend/vagrant.log for details)

A fatal Vagrant error occurred (See /Users/Gabo/.nanobox/apps/site-frontend/vagrant.log for details). Exiting...%

Then, of course, I go the log file and this is what it looks like:
2016-04-12 10:45:06 INFO  Bringing machine 'site-frontend' up with 'virtualbox' provider...
2016-04-12 10:45:06 INFO  ==> site-frontend: Importing base box 'nanobox/boot2docker'...
2016-04-12 10:45:07 INFO  2016-04-12 10:45:08 INFO  Progress: 10%!(NOVERB)
2016-04-12 10:45:08 INFO  Progress: 20%!(NOVERB)
2016-04-12 10:45:09 INFO  Progress: 30%!(NOVERB)
2016-04-12 10:45:10 INFO  Progress: 40%!(NOVERB)
2016-04-12 10:45:10 INFO  Progress: 90%!(NOVERB)
2016-04-12 10:45:10 INFO  ==> site-frontend: Matching MAC address for NAT networking...
2016-04-12 10:45:10 INFO  ==> site-frontend: Checking if box 'nanobox/boot2docker' is up to date...
2016-04-12 10:45:12 INFO  ==> site-frontend: Setting the name of the VM: site-frontend
2016-04-12 10:45:13 INFO  ==> site-frontend: Clearing any previously set network interfaces...
2016-04-12 10:45:13 INFO  ==> site-frontend: Preparing network interfaces based on configuration...
2016-04-12 10:45:13 INFO  site-frontend: Adapter 1: nat
2016-04-12 10:45:13 INFO  site-frontend: Adapter 2: hostonly
2016-04-12 10:45:13 INFO  ==> site-frontend: Forwarding ports...
2016-04-12 10:45:13 INFO  site-frontend: 22 (guest) => 12047 (host) (adapter 1)
2016-04-12 10:45:13 INFO  ==> site-frontend: Running 'pre-boot' VM customizations...
2016-04-12 10:45:13 INFO  ==> site-frontend: Booting VM...
2016-04-12 10:45:14 INFO  ==> site-frontend: Waiting for machine to boot. This may take a few minutes...
2016-04-12 10:45:14 INFO  site-frontend: SSH address: <IP_ADDRESS>:12047
2016-04-12 10:45:14 INFO  site-frontend: SSH username: docker
2016-04-12 10:45:14 INFO  site-frontend: SSH auth method: password
2016-04-12 10:45:36 INFO  site-frontend:
2016-04-12 10:45:36 INFO  site-frontend: Inserting generated public key within guest...
2016-04-12 10:45:36 INFO  site-frontend: Removing insecure key from the guest if it's present...
2016-04-12 10:45:36 INFO  site-frontend: Key inserted! Disconnecting and reconnecting using new SSH key...
2016-04-12 10:45:37 INFO  ==> site-frontend: Machine booted and ready!
2016-04-12 10:45:37 INFO  ==> site-frontend: Checking for guest additions in VM...
2016-04-12 10:45:37 INFO  ==> site-frontend: Configuring and enabling network interfaces...
2016-04-12 10:45:37 INFO  ==> site-frontend: Exporting NFS shared folders...
2016-04-12 10:45:37 INFO  ==> site-frontend: Preparing to edit /etc/exports. Administrator privileges will be required...
2016-04-12 10:45:38 INFO  The nfsd service does not appear to be running.
2016-04-12 10:45:38 INFO  Starting the nfsd service
2016-04-12 10:45:38 INFO  ==> site-frontend: Mounting NFS shared folders...
2016-04-12 11:07:13 ERROR The following SSH command responded with a non-zero exit status.
2016-04-12 11:07:13 ERROR Vagrant assumes that this means the command failed!
2016-04-12 11:07:13 ERROR 2016-04-12 11:07:13 ERROR mount -o 'nfsvers=3,proto=tcp' <IP_ADDRESS>:'/Users/Gabo/Projects/site-frontend' /vagrant/code/site-frontend
2016-04-12 11:07:13 ERROR 2016-04-12 11:07:13 ERROR Stdout from the command:
2016-04-12 11:07:13 ERROR 2016-04-12 11:07:13 ERROR 2016-04-12 11:07:13 ERROR 2016-04-12 11:07:13 ERROR Stderr from the command:
2016-04-12 11:07:13 ERROR 2016-04-12 11:07:13 ERROR mount: RPC: Authentication error; why = Client credential too weak
2016-04-12 11:07:13 ERROR mount: mounting <IP_ADDRESS>:/Users/Gabo/Projects/site-frontend on /vagrant/code/site-frontend failed: Bad file descriptor
2016-04-12 11:07:13 FATAL [commands/create] vagrant.Up() failed - exit status 1
2016-04-12 11:07:13 *LOG* Closing log now

I've tried on two different clean install machines.
I've already tried by downloading Vagrant and VirtualBox using standalone installers.
I've Mac OS X El Capitan 10.11.4 in both machines.
Considering that it is a Vagrant error, I've tried to setup a Vagrant Box successfully using other image. So, I don't think Vagrant it's not installed correctly.

Any advice on that matter?
Thanks in advance.

It looks like the issue is specifically trying to mount the shared volumes via NFS. Nanobox can be configured to use VirtualBox's native driver instead.
@sdomino Can you please provide a reference within the .nanofile where this can be accomplished?

@tylerflint @gabovanlugo I had a similar problem today, in this case was due to a conflict in the nfs /etc/exports file. Try running the following to check if there is an issue:
sudo nfsd checkexports

Just a heads up, there has been some progress on this. Nanobox has been completely re-written from the ground up, and we just released the v1 into beta this week. Nanobox no longer depends on Vagrant, and so this is likely not an issue anymore. We would love to get your feedback on the new version. The new guides are still a WIP but should be usable within the next week. If you're interested, we'd love to talk with you on irc #nanobox or we can bring you into our slack channel as well.
Additionally, Nanobox and Nanobox Cloud have been merged into a unified process. If you would be interested in early access to the private beta please let me know and I can get you an access token.
Just reach out to<EMAIL_ADDRESS>and we can get you going.
I really appreciate both of your help on this. I'll go ahead and close this ticket and hope to hear from you directly.


refactor: use ExternalAddrToStr

Summary by CodeRabbit

Refactor

Updated address formatting across various modules to use a new standardized method for improved consistency.

Tests

Revised test cases to align with the updated address formatting methodology.

Documentation

Enhanced error messages to include specific unrecognized chainName for better clarity in troubleshooting.

mg


feat: [M3-7272] - Disable Public IP Address for VPC only Linode

Description 📝
Visually disable Public IP addresses for VPC-only Linodes and add tooltip explanation.
A VPC-only Linode is a Linode that has at least one config interface with primary set to true and purpose vpc and no ipv4.nat_1_1 value.
Changes  🔄

Disable the Public IP Address column in the Linodes landing table
Disable the Public IP Addresses and Access sections in the Linode's details page
Disable the Public IPv4 row in the Linode's details -> Network tab -> IP Addresses table
Refactored components into their own file and added unit testing
Created custom hook useVPCConfigInterface to share logic

Preview 📷

Linode landing before
Linode landing after

Linode details before
Linode details after

Linode Network IP Address table before
Linode Network IP Address table after

How to test 🧪
Prerequisites

Ensure your account has vpc customer tags
Have a VPC-only Linode

You can either create a Linode and add a VPC or edit a Linode's config so that the primary interface is VPC. (make sure the assign a public IPv4 address checkbox is unchecked)

Verification steps
For the VPC-only Linode

The Public IP Address column in the Linodes landing table should be disabled and there should be a tooltip
The Public IP Addresses and Access sections in the Linode's details page should be disabled and there should be a tooltip
The Public IPv4 row in the Linode's details -> Network tab -> IP Addresses table should be disabled and there should be a tooltip

As an Author I have considered 🤔
Check all that apply

[x] 👀 Doing a self review
[x] ❔ Our contribution guidelines
[x] 🤏 Splitting feature into small PRs
[x] ➕ Adding a changeset
[x] 🧪 Providing/Improving test coverage
[x] 🔐 Removing all sensitive information from the code and PR description
[x] 🚩 Using a feature flag to protect the release
[x] 👣 Providing comprehensive reproduction steps
[x] 📑 Providing or updating our documentation
[x] 🕛 Scheduling a pair reviewing session
[x] 📱 Providing mobile support
[x] ♿  Providing accessibility support

Update: Just edited debian-us-lax-001 to have the following network interfaces:

However, it still has a disabled public IP address:

Is this expected behavior?

@coliu-akamai Could it have been bc the "Assign a public IPv4 address" was checked? 🤔


[SUPPORT] parquet bloom filters not supported by hudi

hudi 0.12.1
spark 3.2.1
hudi has its own bloom implementation used mainly for fast lookup on the hudi key. Up to 0.11 hudi allow to store them in the metadata table, and also add bloom on several columns. So far those bloom are not use at read time.
Spark leverage parquet bloom filters at write time. They are then used at read time to skip files. Those bloom can improve queries a lot. But so far, hudi don't support them :
Basic parquet table => bloom works as expected: the accumulator is not used when filtered values are outside df values, but incremented when values matches
spark.sql("set parquet.filter.bloom.enabled=true")
spark.sql("set parquet.filter.columnindex.enabled=false")
spark.sql("set parquet.filter.stats.enabled=false")

import org.apache.spark.util.{AccumulatorContext, AccumulatorV2, Utils}
import org.apache.spark.sql.Row
{spark.sql("select '0' as a, '1' as b, '2' c, 4 as d union select '1', '2', '3', 4").write.mode("overwrite").option("parquet.bloom.filter.enabled#b", "true").parquet("/tmp/bloom")}

class NumRowGroupsAcc extends AccumulatorV2[Integer, Integer] {
  private var _sum = 0
  override def isZero: Boolean = _sum == 0
  override def copy(): AccumulatorV2[Integer, Integer] = {
    val acc = new NumRowGroupsAcc()
    acc._sum = _sum
    acc
  }
  override def reset(): Unit = _sum = 0
  override def add(v: Integer): Unit = _sum += v
  override def merge(other: AccumulatorV2[Integer, Integer]): Unit = other match {
    case a: NumRowGroupsAcc => _sum += a._sum
    case _ => throw new UnsupportedOperationException(
      s"Cannot merge ${this.getClass.getName} with ${other.getClass.getName}")
  }
  override def value: Integer = _sum
}
val accu = new NumRowGroupsAcc
sc.register(accu)
spark.read.format("parquet").load("/tmp/bloom").filter("b = '3'").foreachPartition((it: Iterator[Row]) => it.foreach(_ => accu.add(0)))
println(accu)
spark.read.format("parquet").load("/tmp/bloom").filter("b = '2'").foreachPartition((it: Iterator[Row]) => it.foreach(_ => accu.add(0)))
println(accu)

/**
NumRowGroupsAcc(id: 5611, name: None, value: 0)
NumRowGroupsAcc(id: 5611, name: None, value: 1)
/**

Hudi parquet table => bloom works NOT as expected: the accumulator is incremented when match and no match
import org.apache.spark.util.{AccumulatorContext, AccumulatorV2, Utils}
import org.apache.spark.sql.Row
spark.sql("set parquet.filter.bloom.enabled=true")
spark.sql("set parquet.filter.columnindex.enabled=false")
spark.sql("set parquet.filter.stats.enabled=false")
import org.apache.spark.util.{AccumulatorContext, AccumulatorV2, Utils}

val basePath = "/tmp/bloom_hudi/"
val hiveOptions = Map[String, String](
  "hoodie.table.name"-> "hudi_test",
"hoodie.datasource.write.recordkey.field"-> "a",
"hoodie.datasource.write.partitionpath.field"-> "d",
"hoodie.datasource.write.precombine.field"-> "c",
"hoodie.datasource.write.table.name"-> "hudi_test",
"hoodie.datasource.write.operation" -> "bulk_insert",
"parquet.bloom.filter.enabled#b" -> "true"
)
spark.sql("select '0' as a, '1' as b, '2' c, 4 as d union select '1', '2', '3', 4").write.format("hudi").
options(hiveOptions).
mode("overwrite").
save(basePath)

class NumRowGroupsAcc extends AccumulatorV2[Integer, Integer] {
  private var _sum = 0
  override def isZero: Boolean = _sum == 0
  override def copy(): AccumulatorV2[Integer, Integer] = {
    val acc = new NumRowGroupsAcc()
    acc._sum = _sum
    acc
  }
  override def reset(): Unit = _sum = 0
  override def add(v: Integer): Unit = _sum += v
  override def merge(other: AccumulatorV2[Integer, Integer]): Unit = other match {
    case a: NumRowGroupsAcc => _sum += a._sum
    case _ => throw new UnsupportedOperationException(
      s"Cannot merge ${this.getClass.getName} with ${other.getClass.getName}")
  }
  override def value: Integer = _sum
}
val accu = new NumRowGroupsAcc
sc.register(accu)
spark.read.format("hudi").load(basePath).filter("b = '3'").foreachPartition((it: Iterator[Row]) => it.foreach(_ => accu.add(0)))
println(accu)
spark.read.format("hudi").load(basePath).filter("b = '2'").foreachPartition((it: Iterator[Row]) => it.foreach(_ => accu.add(0)))
println(accu)
/**
accu: NumRowGroupsAcc = NumRowGroupsAcc(id: 635, name: None, value: 1)
accu: NumRowGroupsAcc = NumRowGroupsAcc(id: 635, name: None, value: 2)

/*

hi @nsivabalan bloom at read time is an useful feature for read performance. thought ?

Hudi is able to benefit from parquet files written with blooms. (tested by replacing the hudi parquet files with the vanilla spark's one, and it hudi datasource triggers the bloom).
Digging the source code, I guess the reason blooms are not taken in consideration is in the hudi's parquetWriter wrapper. It then calls [the parquetWriter](https://github.com/apache/parquet-mr/blob/cac8f7cf55b390c2ac5ef5d14a6aa72597b99284/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetWriter.java#L231-L236 public constructor) which has very limited parquet feature support. There is a more complete constructor but sadly it's access is limited to package.
Accessing to package constructor can be done by changing the HoodieBaseParquetWriter package to org.apache.parquet.hadoop, but also the ParquetWriter has to be present in the same jar (common package cannot be spread over multiple jars).
A better option would be parquet provides more suitable constructors. Or I am missing something ?

@danny0405 thought ?

which has very limited parquet feature support. There is a more complete constructor but sadly it's access is limited to package.

2 questions:

What is exactly the feature you want for parquet writers? The BloomFilter written by Hudi parquet writer wrapper does not meet your request?
How spark writes their BloomFilters? How they solved the package/accessability issue as you mentioned.

Yeah, please elaborate more about 1, and for 2, can Hudi also take the similiar manner for writers?

hudi should support parquet vanilla bloom filters, because this is a standard optimization method supported by every query engines using parquet 1.12 and above. Moreover hudi does not provide such optimization method. Hudi blooms are not used for select queries. Hudi blooms are only useful for update operations. Providing vanilla parquet bloom support to hudi would allow an other set of optimization (such z-order, parquet stats) for almost free.
as for 2, providing bloom filters to hudi is just passing the configurations to parquet. But after analysis, it likely needs a small refactoring of the way we configure parquet writers. (by stopping using deprecated ParquetWriter constructor and use the builder method which support all parquet configurations)

more details:

parquet bloom doc
spark bloom config

by stopping using deprecated ParquetWriter constructor and use the builder method which support all parquet configurations

I'm okay with that.

that's a good point. I don't know, I found that code in the spark tests. The point is it does increment !

https://github.com/apache/hudi/pull/10278 I am working on the FileGroup Reader for Hudi 1.0 and that test was failing but if I change it to accu.add(1) then it works. So that's why I'm asking. I don't want to break parquet bloom filters.

Got it. Can you share your draft ?

Okay will double check thanks for reaching

@parisni update on this. I am working on getting vectorized reader enabled in more scenarios with the new fg reader and now that test works again


dolt: 0.15.0 -> 0.15.2

Semi-automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/liquidata-inc/dolt/releases.
meta.description for dolt is: "Relational database with version control and CLI a-la Git."
meta.homepage for dolt is: "https://github.com/liquidata-inc/dolt"
Updates performed:

Golang update

Release on GitHub
Compare changes on GitHub

Checks done (click to expand)

built on NixOS

The tests defined in passthru.tests, if any, passed

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 0.15.2 with grep in /nix/store/nwmn9g21mf9kpijvb068hx9fbc1sj07b-dolt-0.15.2

directory tree listing: https://gist.github.com/58e90c15ef8adcbe75bf6a26f4ffd4a6

du listing: https://gist.github.com/d4b6a96ca64739c28c1e69903a59e979

Rebuild report (if merged into master) (click to expand)

4 total rebuild path(s)
1 package rebuild(s)
1 x86_64-linux rebuild(s)
1 i686-linux rebuild(s)
1 x86_64-darwin rebuild(s)
1 aarch64-linux rebuild(s)
First fifty rebuilds by attrpath
dolt

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/nwmn9g21mf9kpijvb068hx9fbc1sj07b-dolt-0.15.2 \
  --option binary-caches 'https://cache.nixos.org/ https://r-ryantm.cachix.org/' \
  --option trusted-public-keys '
  r-ryantm.cachix.org-1:gkUbLkouDAyvBdpBX0JOdIiD2/DP1ldF3Z3Y6Gqcc4c=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(r-ryantm's Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A dolt https://github.com/r-ryantm/nixpkgs/archive/fcc9e59971377f044e83bbd8386c798bfcfb5514.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/nwmn9g21mf9kpijvb068hx9fbc1sj07b-dolt-0.15.2
ls -la /nix/store/nwmn9g21mf9kpijvb068hx9fbc1sj07b-dolt-0.15.2/bin

cc @danbst @kalbasit for testing.

Result of nixpkgs-review pr 83959 1

1 package built:

dolt


[Feature Request]  Implement graceful shutdown

Tell us about your feature request
I'd like to propose the implementation of graceful shutdown functionality in the project. This feature would ensure that the application gracefully handles shutdown signals, allowing it to complete ongoing tasks and close connections cleanly before exiting.
Would be happy to work on implementing this if it is something that is wanted.
Disclaimer

[X] I agree

Tell us about your feature request
I'd like to propose the implementation of graceful shutdown functionality in the project. This feature would ensure that the application gracefully handles shutdown signals, allowing it to complete ongoing tasks and close connections cleanly before exiting. Would be happy to work on implementing this if it is something that is wanted.
Disclaimer

[x]  I agree

That would be great because graceful shutdown is important when dealing with SQL. For example, with MySQL, if the connection is not closed properly, it might look like a deadlock (keep opening), and potentially many open connections could remain open.

@Adembc I will assign you the issue. Thanks for opening the ticket.

Yo Yo yo, Will this be implemented ? Because I have a better way to implement graceful shutdown, and it's only possible in Go, not in any other language.
This A better async method written in go
example:
// Start initiates the Fiber server on the specified address.
// It logs server startup and any errors encountered during Listen.
func (s *FiberServer) Start(addr string) {
	go func() {
		Logger.LogInfof("Starting server on %s", addr)
		if err := s.app.Listen(addr); err != nil {
			Logger.LogErrorf("Server listen failed: %v", err)
		}
	}()
}

// Shutdown attempts to gracefully shut down the Fiber server without interrupting any active connections.
func (s *FiberServer) Shutdown() error {
	return s.app.Shutdown()
}

// CleanupDB disconnects from the database if a connection exists.
// It logs the outcome of the operation.
func (s *FiberServer) CleanupDB() error {
	if s.db != nil {
		if err := s.db.Close(); err != nil {
			Logger.LogErrorf("Failed to close database connection: %v", err)
			return err
		}
		Logger.LogInfo("Database connection successfully closed.")
	}
	return nil
}

// StartServer sets up and starts the server while also preparing for a graceful shutdown.
// It accepts a server interface, address, path for monitoring, and a timeout for shutdown.
func StartServer(server Server, addr, monitorPath string, shutdownTimeout time.Duration) {
	startServerAsync(server, addr, monitorPath)
	waitForShutdownSignal(shutdownTimeout, server)
}

// startServerAsync launches the server in a separate goroutine.
// This allows the server to start listening for incoming requests without blocking the main thread.
func startServerAsync(server Server, addr, monitorPath string) {
	server.Start(addr, monitorPath)
}

// waitForShutdownSignal listens for OS interrupt or SIGTERM signals to initiate a graceful server shutdown.
// The server attempts to shut down within the specified timeout, logging the process and any errors.
func waitForShutdownSignal(shutdownTimeout time.Duration, server Server) {
	quit := make(chan os.Signal, 1)
	signal.Notify(quit, os.Interrupt, syscall.SIGTERM)

	sig := <-quit
	Logger.LogInfof("Received shutdown signal: %v", sig)

	ctx, cancel := context.WithTimeout(context.Background(), shutdownTimeout)
	defer cancel()

	shutdownTimer := time.NewTimer(shutdownTimeout)
	defer shutdownTimer.Stop()

	go func() {
		if err := server.Shutdown(); err != nil {
			Logger.LogErrorf("Error during server shutdown: %v", err)
		}

		if err := server.CleanupDB(); err != nil {
			Logger.LogErrorf("Database cleanup failed: %v", err)
		}

		cancel()
	}()

	select {
	case <-ctx.Done():
		Logger.LogInfo("Server shutdown completed within timeout.")
	case <-shutdownTimer.C:
		Logger.LogWarning("Server shutdown exceeded the timeout.")
	}
}

Another Method works well on fiber frameworks
// Shutdown gracefully shuts down the Fiber server using the provided context.
func (s *FiberServer) Shutdown(ctx context.Context) error {
	return s.app.ShutdownWithContext(ctx)
}

// waitForShutdownSignal listens for OS interrupt or SIGTERM signals to initiate a graceful server shutdown.
// The server attempts to shut down within the specified timeout, logging the process and any errors.
func waitForShutdownSignal(shutdownTimeout time.Duration, server Server) {
	quit := make(chan os.Signal, 1)
	signal.Notify(quit, os.Interrupt, syscall.SIGTERM)

	sig := <-quit
	Logger.LogInfof("Received shutdown signal: %v", sig)

	ctx, cancel := context.WithTimeout(context.Background(), shutdownTimeout)
	defer cancel()

	go func() {
		if err := server.Shutdown(); err != nil {
			Logger.LogErrorf("Error during server shutdown: %v", err)
		}

		if err := server.CleanupDB(); err != nil {
			Logger.LogErrorf("Database cleanup failed: %v", err)
		}

		cancel()
	}()

	select {
	case <-ctx.Done():
		Logger.LogInfo("Server shutdown completed within timeout.")
	case <-time.After(shutdownTimeout):
		Logger.LogWarning("Server shutdown exceeded the timeout.")
		cancel()
	}
}

I would like to volunteer to work on this if it hasn't been started yet.

@Adembc, what is your status on resolving this ticket? @Spankie is interested, and I can assign it to him if you don’t have time currently.


callback iteration

more payment types
we currently only trigger callback for lightning receive and intraledger receive.
we should add onchain for receive, and also trigger event for the send flow
more information in the event
currently the event only contain the hash of the event. we don't have information such as amount, currency and the other element that we would have in a Transaction
make a clear definition of the event
svix define an Event Type Schema. this would help people integrate with the event if we define a schema for event type
make example using asymmetric key for the signature
the event is signed. it's possible to create asymmetric and symmetric key. I think asymmetric is best for our use case because even thought this a bit computing intense, it make the set up easier as there is no need to share a secret between the callback server and the recipient. (note that svix recommend the symmetric option, but I think it's just because of the compute optimization on their end, this matter more for use case when a lot more event could be trigger than a use case like us for payment)

accountId received over the webhook may be wrong (cc @siddhart1o1 )


Wrong browser family returned for Edge

Version: 2.0
For users using MS Edge based on Chromium engine, instead of the Edge browser family, we get Chrome. Currently only confirmed that for Edge 86; we're not sure how far back the issue goes.
Example:
>>> import user_agents
>>> ua = user_agents.parse('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36 Edg/86.0.622.58')
>>> ua.browser
Browser(family='Chrome', version=(86, 0, 4240), version_string='86.0.4240')

You can check the user agent string:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36 Edg/86.0.622.58

on https://developers.whatismybrowser.com/useragents/parse/ where it is parsed properly as Edge 86.

Turns out, the issue is fixed in the 2.2.0 release.


rework evidence page flow

Description

[ ] WHY - What is the problem this feature would answer?

Replace this text with a description here. Think of it as the 1-paragraph blog post you would write once the feature is released on the Open Targets platform

[ ] HOW are we going to solve the problem?

Describe technical design of the solution.
Paste here google doc link or link to markdown file, if already written somewhere else

[ ] WHAT would this look like?

Screenshots and mock-ups are great

Progress Tracker
Get to a prototype at branch.targetvalidation.org:

Write code + tests + docs. Isolate the work in feature branches.
Use as many PRs as you need.
Write tests in the same or different PRs, as is convenient for you.
As each PR is merged, add a comment to this issue referencing the PRs.

[ ] Feature tag Code should have a feature tag to be disabled by default.
[ ] Minimal functional testing
[ ] Minimal docs
[ ] if on webapp, test a deployment on netlify

Push to staging

[ ] merged code and pushed to staging.targetvalidation.org
PULL request should be reviewed by 1-2 people at least
[ ] at least 1 week internal testing?
[ ] received some feedback/input from users (eg. pharma scientist)

Released

[ ] in the release notes?
[ ] documentation updated?
[ ] blog post?
[ ] users that requested it originally informed?

Make sure you can measure impact

As dev team we want to know if a new feature are succesful.
To do that we need to define metrics to assess the impact of each new feature we introduce to the platform

[ ] metric identified
[ ] identify the one metric that matters (OMTM)

Issue closed due to technical and design constraints, but will be considered during a re-write and re-design of the Platform


formatting of PP reference when threat inherited from Base-PP

In PP-Modules we currently have the ability to bring a threat in from a Base-PP for the purpose of mapping it to an objective in the Module. See for example section 4.3 of (https://commoncriteria.github.io/email-client/master/email-client.html). However, there appears to be no way to put a space in the referenced PP name.
The actual code uses a tag  where 'base' must be the same as the 'id' parameter of a . However, it then just puts the actual id value as-is in brackets. Since the id value can't contain spaces, this creates inaccurate references. So for example, instead of saying "from [App PP]" which is the desired behavior, it must say "from [AppPP]" because there's no way to override the 'id' value (that can't contain a space).
More a nuisance cosmetic issue than an actual problem but worth flagging.

I think this is fixed. Now it produces "App PP" rather than "AppPP."

confirmed fixed


SubscribeToShard support

This one is a challenge. It requires Http2 server-push functionality and the documentation on this is very limiting. I believe Http4s has support for server-push but perhaps not in the way that SubscribeToShard needs to be supported.
https://docs.aws.amazon.com/kinesis/latest/APIReference/API_SubscribeToShard.html
https://aws.amazon.com/blogs/aws/kds-enhanced-fanout/
Without this support, users will have to use the PollingConsumer in their tests.

Curious if there's been any movement on this.  Off the top of my head, it wouldn't seem that using a polling consumer in testing and a subscription consumer in production results in a valid test.  But this is my teams first Kinesis project,  I need to go back and study the differences.


Add Unit Tests for Eraser

Fixes #22

looks like there's a conflict

@AldairCoronel looks like some conflicts now


can we have one for Sendgrid

@pooran - Have you tried CustomeReceiver sample for this.
Documentation for SendGrid WebHooks is avaialble here: Hooks

This issue was copied to aspnet/AspNetWebHooks#19

This issue was moved to aspnet/AspLabs#52


[admin] merge to live

Docs Build status updates of commit 48976f3:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/structure-of-a-presentationml-document.md
:bulb:Suggestion

Details

docs/structure-of-a-presentationml-document.md

Line 33, Column 78: [Suggestion-docs-link-absolute] Absolute link 'https://docs.microsoft.com/dotnet/api/documentformat.openxml.presentation.presentation?view=openxml-2.8.1' will be broken in isolated environments. Replace with a relative link.
Line 33, Column 78: [Suggestion-preserve-view-not-set] Did you mean to link to a specific version? It's usually better not to specify a version, so the link always goes to the canonical version. If you really need to link to a specific version, add &preserve-view=true to the URL: https://docs.microsoft.com/dotnet/api/documentformat.openxml.presentation.presentation?view=openxml-2.8.1. Otherwise, remove the view parameter.
Line 34, Column 68: [Suggestion-docs-link-absolute] Absolute link 'https://docs.microsoft.com/dotnet/api/documentformat.openxml.presentation.presentationproperties?view=openxml-2.8.1' will be broken in isolated environments. Replace with a relative link.
Line 34, Column 68: [Suggestion-preserve-view-not-set] Did you mean to link to a specific version? It's usually better not to specify a version, so the link always goes to the canonical version. If you really need to link to a specific version, add &preserve-view=true to the URL: https://docs.microsoft.com/dotnet/api/documentformat.openxml.presentation.presentationproperties?view=openxml-2.8.1. Otherwise, remove the view parameter.
Line 35, Column 79: [Suggestion-docs-link-absolute] Absolute link 'https://docs.microsoft.com/dotnet/api/documentformat.openxml.presentation.slidemaster?view=openxml-2.8.1' will be broken in isolated environments. Replace with a relative link.
Line 35, Column 79: [Suggestion-preserve-view-not-set] Did you mean to link to a specific version? It's usually better not to specify a version, so the link always goes to the canonical version. If you really need to link to a specific version, add &preserve-view=true to the URL: https://docs.microsoft.com/dotnet/api/documentformat.openxml.presentation.slidemaster?view=openxml-2.8.1. Otherwise, remove the view parameter.
Line 36, Column 79: [Suggestion-docs-link-absolute] Absolute link 'https://docs.microsoft.com/dotnet/api/documentformat.openxml.presentation.slidelayout?view=openxml-2.8.1' will be broken in isolated environments. Replace with a relative link.
Line 36, Column 79: [Suggestion-preserve-view-not-set] Did you mean to link to a specific version? It's usually better not to specify a version, so the link always goes to the canonical version. If you really need to link to a specific version, add &preserve-view=true to the URL: https://docs.microsoft.com/dotnet/api/documentformat.openxml.presentation.slidelayout?view=openxml-2.8.1. Otherwise, remove the view parameter.
Line 37, Column 87: [Suggestion-docs-link-absolute] Absolute link 'https://docs.microsoft.com/dotnet/api/documentformat.openxml.drawing.theme?view=openxml-2.8.1' will be broken in isolated environments. Replace with a relative link.
Line 37, Column 87: [Suggestion-preserve-view-not-set] Did you mean to link to a specific version? It's usually better not to specify a version, so the link always goes to the canonical version. If you really need to link to a specific version, add &preserve-view=true to the URL: https://docs.microsoft.com/dotnet/api/documentformat.openxml.drawing.theme?view=openxml-2.8.1. Otherwise, remove the view parameter.
Line 38, Column 85: [Suggestion-docs-link-absolute] Absolute link 'https://docs.microsoft.com/dotnet/api/documentformat.openxml.presentation.slide?view=openxml-2.8.1' will be broken in isolated environments. Replace with a relative link.
Line 38, Column 85: [Suggestion-preserve-view-not-set] Did you mean to link to a specific version? It's usually better not to specify a version, so the link always goes to the canonical version. If you really need to link to a specific version, add &preserve-view=true to the URL: https://docs.microsoft.com/dotnet/api/documentformat.openxml.presentation.slide?view=openxml-2.8.1. Otherwise, remove the view parameter.
Line 39, Column 79: [Suggestion-docs-link-absolute] Absolute link 'https://docs.microsoft.com/dotnet/api/documentformat.openxml.presentation.notesmaster?view=openxml-2.8.1' will be broken in isolated environments. Replace with a relative link.
Line 39, Column 79: [Suggestion-preserve-view-not-set] Did you mean to link to a specific version? It's usually better not to specify a version, so the link always goes to the canonical version. If you really need to link to a specific version, add &preserve-view=true to the URL: https://docs.microsoft.com/dotnet/api/documentformat.openxml.presentation.notesmaster?view=openxml-2.8.1. Otherwise, remove the view parameter.
Line 40, Column 80: [Suggestion-docs-link-absolute] Absolute link 'https://docs.microsoft.com/dotnet/api/documentformat.openxml.presentation.notesslide?view=openxml-2.8.1' will be broken in isolated environments. Replace with a relative link.
Line 40, Column 80: [Suggestion-preserve-view-not-set] Did you mean to link to a specific version? It's usually better not to specify a version, so the link always goes to the canonical version. If you really need to link to a specific version, add &preserve-view=true to the URL: https://docs.microsoft.com/dotnet/api/documentformat.openxml.presentation.notesslide?view=openxml-2.8.1. Otherwise, remove the view parameter.
Line 41, Column 77: [Suggestion-docs-link-absolute] Absolute link 'https://docs.microsoft.com/dotnet/api/documentformat.openxml.presentation.handoutmaster?redirectedfrom=MSDN&view=openxml-2.8.1' will be broken in isolated environments. Replace with a relative link.
Line 41, Column 77: [Suggestion-preserve-view-not-set] Did you mean to link to a specific version? It's usually better not to specify a version, so the link always goes to the canonical version. If you really need to link to a specific version, add &preserve-view=true to the URL: https://docs.microsoft.com/dotnet/api/documentformat.openxml.presentation.handoutmaster?redirectedfrom=MSDN&view=openxml-2.8.1. Otherwise, remove the view parameter.
Line 42, Column 79: [Suggestion-docs-link-absolute] Absolute link 'https://docs.microsoft.com/dotnet/api/documentformat.openxml.presentation.commentlist?redirectedfrom=MSDN&view=openxml-2.8.1' will be broken in isolated environments. Replace with a relative link.
Line 42, Column 79: [Suggestion-preserve-view-not-set] Did you mean to link to a specific version? It's usually better not to specify a version, so the link always goes to the canonical version. If you really need to link to a specific version, add &preserve-view=true to the URL: https://docs.microsoft.com/dotnet/api/documentformat.openxml.presentation.commentlist?redirectedfrom=MSDN&view=openxml-2.8.1. Otherwise, remove the view parameter.
Line 43, Column 73: [Suggestion-docs-link-absolute] Absolute link 'https://docs.microsoft.com/dotnet/api/documentformat.openxml.presentation.commentauthorlist?redirectedfrom=MSDN&view=openxml-2.8.1' will be broken in isolated environments. Replace with a relative link.
Line 43, Column 73: [Suggestion-preserve-view-not-set] Did you mean to link to a specific version? It's usually better not to specify a version, so the link always goes to the canonical version. If you really need to link to a specific version, add &preserve-view=true to the URL: https://docs.microsoft.com/dotnet/api/documentformat.openxml.presentation.commentauthorlist?redirectedfrom=MSDN&view=openxml-2.8.1. Otherwise, remove the view parameter.

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


eslint complains about "Do not use 'new' for side effects"

ESLint doesn't like "new"
I initialize FlowRouterMeta:
new FlowRouterMeta(FlowRouter); new FlowRouterTitle(FlowRouter);
Then eslint complains about it:

error  Do not use 'new' for side effects        no-new

Is there an other way to do this?
Thank you!

Hello @floek ,
Don't worry about it, just a warning. If it's too annoying fo you simply assign it to some var:
const frPlugins = {
  meta: new FlowRouterMeta(FlowRouter),
  title: new FlowRouterTitle(FlowRouter)
};

It may warn you what frPlugins defined, but not used :)


doc(README) : add prerequisites

:grey_question: About
While trying to play with ollama-python on a brand new env, I found out that following the current steps of the README do not take in account that ollama must be up and running (ie. that it's not ollama-python role).
:dart: Objective
The purpose of this PR is to make the Getting started even easier to people can just copy/paste and enjoy :smile:

This would close #29.

Worth mentioning that there are a few ways to do this. For example, you might also add something like:
## Local Ollama Setup
You need to have a local ollama server running to be able to continue. To do this:

- Download: https://ollama.com/
- Run an LLM: https://ollama.com/library
    - Example: `ollama run llama2`
    - Example: `ollama run llama2:70b`

:pray: @connor-makowski

Big thanks @adriens

Dear maintainers,
this is a very useful documentation update. Consider merging it. 😉

:partying_face:

Tweet

Referencing this to: https://github.com/ollama/ollama-python/pull/155 as an extension for adding clarity to the getting started process.


CIR1F model sigma value range seems can not be larger than 0.1?

hi, I have tested the CIR1F model, seems the sigma value range can not be larger than 0.1, if set to 0.3, the error occur.
cir1f = xen.CIR1F('cir1f', r0=0.02, alpha=0.1, longterm=0.042, sigma=0.03)
Is that correct?
Thanks.

hi, cir model has a condition that avoids the possibility of negative interest rates.
that is correct condition.
can you check following wiki-link?

restriction calc
2ab >= sigma^2
2 * 0.1 * 0.042 < 0.1^2
0.0084 < 0.01 is not valid for inequality

thanks.

hi, cir model has a condition that avoids the possibility of negative interest rates.
that is correct condition.
can you check following wiki-link?
restriction calc
2ab >= sigma^2
2 * 0.1 * 0.042 < 0.1^2
0.0084 < 0.01 is not valid for inequality
thanks.

oh, my fault, many thanks!


DEQ - Spills app rebuild

deq-spills repository
Benefit

Rebuild using new ArcGIS Maps SDK for JS, which is replacing ArcGIS API for JS 3.x
Integrate with new DEQ spills SalesForce app

Acceptance Criteria

Spills app rebuilt according to specs and requirements of DEQ
Mapping app is integrated into DEQ Spills SalesForce

DEQ Wishlist

Incorporate a couple Enviro layers:

Public Water System Facilities

Calculate distance to buffered point
Return some attribute

Ground Water Protection Zone

Return some attributed of the spatial data if there’s an intersection with a GWPZ

Ability to see the highway mile markers better on the base map, and a wish for railroad mile markers (though the railroads are usually pretty good about giving us coordinates instead of their milepost)

This was completed last week. Barry got back to me this morning and verified that production is working.

We worked with Barry last week to get the current (3.x) app pushed to production and the production salesforce instance pointed at it. I'm not sure if this issue should be closed or if it should stay open to track the rebuild for esri js 4.x, etc.

We worked with Barry last week to get the current (3.x) app pushed to production and the production salesforce instance pointed at it. I'm not sure if this issue should be closed or if it should stay open to track the rebuild for esri js 4.x, etc.

Let's close this, as I have a Q4 task to sort out a plan for either a 4x rebuild, or a more enterprise SF map integration that could replace the existing map widget


Connector Developer Portal cannot open?

When open https://outlook.office.com/connectors/Home/Login#/configurations, it always show error:
:-(
Something went wrong.
Mailbox address missing in URL
What happens?
Help!
Rukai.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 0b92bda5-e2b6-843f-efdb-3cd1fda4d77a
Version Independent ID: 1119b12b-8650-f99a-4298-cc3f93f420f3
Content: Connectors developer dashboard - Outlook Developer
Content Source: docs/actionable-messages/connectors-dev-dashboard.md
Product: outlook
Technology: o365-connectors
GitHub Login: @jasonjoh
Microsoft Alias: jasonjoh

I'll report this to the appropriate team.

I also received this error, unable to access the connector develop portal configurations page. The page initially flashes up, then the below error displays (in both latest Chrome and Edge):

There was an issue with the service but that has been resolved. I'm now able to get to the dashboard without the error on multiple accounts. Can you try again and let us know if you're still unable to log in?

Hi Jason, yes - can confirm that this is now resolved. Thanks.

@jasonjoh Yesterday it was working but today I'm getting the same error. I've tried to log in using Edge, Edge Dev, Chrome and Firefox (also in incognito/private mode) and always get the same error.

Same here, doesn't work.

It happens to me to, I've been working and having this problem since a week ago. Today even when i can enter to my dashboard, my connector still don't load.

Same here.

was there a fix?

This is not working for me too. Same error as reported.


[admin] Publish

Docs Build status updates of commit 5aeed48:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/overview/office-add-ins-community-call.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


[Feature request] An option to clear the output console.

I think an option to clear the output is useful.

And a toggle in the settings for doing it automatically when a scene is played.

Both of these would be really nice to have. To have the options directly on the console and to not have to go into the editor settings would be nice too.

Yes please! Anooyed me many, many times.
Workaround: use the -d option when starting Godot from a terminal. You'll have all the output there and can reset the terminal to clear the old output (at least on Ubuntu I can).

Here is my proposal for a button on top right of output console to clear it out. Clearing currently need a click on this button for now, but I'll work on a way to make this automatic on Running the scene. What do you prefer ?

a checkbox "Clear console on Run" above the console (Unity3d-like)
an option in the parameters

Tell me what you think.

The first is the more complete, but with the second checked  the first could also not be visible, thus both. :)

Quoting reduz :
"seems fine to me, you can add the clear option on start in the editor settings and use the EDITOR_DEF() macro to read it at the right place"
Thus, here is the planning :

[x] Get a "Clear" button working
[ ] Put a CheckButton "Clear on Play" working, clearing the output at each run (Play/F5, Debug/F6)
[ ] Create and show an option in Godot parameters ("Always clear output on Play ? Yes/No"
[ ] Add a condition to display the CheckButton above the output only if the preceding parameter is set to "No"

I suggest closing this Feature Request ;)

I'd like to request that this be re-opened. I'd like a command like console.clear() to clear the console (NOT just the OUTPUT window, but rather the CONSOLE window.)
Also, is there a project option to clear the console whenever a project is run? I'd like to see that too.

I suggest a new feature request instead, on godotengine/godot or godotengine/godot-proposals (is that right yet @akien-mga ?). What you're asking is a different feature, even if related.
However I suspect a won't do in this case since I don't think Godot can control clearing on the standard output - only writing, like any other software.
Concerning the option to clear the console whenever a project is run, a small search in Editor settings would have told you it is already there.

A feature request was opened: #28632
@StraToN We don't advertise godot-proposals just yet, it's still in the design phase.


Add the remaining HTML quiz questions

There are only two questions in the HTML quiz.
Add the remaining questions/answers that have been posted on Instagram/Twitter
HTML Quiz fixtures
Twitter HTML questions

Hello...I'm interested in working with this issue.

Hello...I'm interested in working with this issue.
Hi @Swetha-kodam your help is most welcome.  I will assign the ticket over to you.


Theme color does not apply to current page icon while switching between themes

After changing the colour theme, the Home Icon in curved navigation bar does not respond to the theme change.
However it adapts the new theme if we change the screens (from curved navigation bar only)

This is not specific to the home screen but any screen from which we change the setting

ye solve ho gya na @kshitijalwadhi ?

No sir, its there on one device am testing and absent on the other

Fixed by changes in upstream of library/flutter.


feat: LDAP User management

Is your feature request related to a problem? Please describe.
Users should be managed via LDAP, similar to Portainer
Describe the solution you'd like
LDAP user management implemented
Describe alternatives you've considered

OAUTH
Microsoft Active directory

I believe LDAP is the best starting point
Additional context
One key feature of ollama-webui is its ability to have users, but without LDAP this doesn't have same network level utility.

Yes agree, this would be a use case for us as well.

Related: https://github.com/ollama-webui/ollama-webui/issues/483

I can help test this.

This would be great for us as well - an auth via Header, set by an reverse proxy, would be a great alternative. Then we could add different auth mechanisms (OpenID, LDAP, Kerberos etc) without implementing each of them in webui itself...

This would be great for us as well - an auth via Header, set by an reverse proxy, would be a great alternative. Then we could add different auth mechanisms (OpenID, LDAP, Kerberos etc) without implementing each of them in webui itself...

Excellent news, this has been implemented:

#1347

How can I connect Open-WebUI with Microsoft Active directory? I'm looking for an excellent solution. A fixed module in Open-WebUI for Active Directory (LDAP) would be a dream

Me too! +100 from me ;-)

+1

+1

Would be awesome to have this feature.

+1

I solved my case with Microsoft authentication by this
https://docs.openwebui.com/tutorial/sso#microsoft

I solved my case with Microsoft authentication by this
https://docs.openwebui.com/tutorial/sso#microsoft

does not help with local active directory / ldap server sadly, but might be useful for some.

My implementation: https://github.com/WilsonZiweiWang/open-webui/commit/bfe556010bed6e6e4a0a63a9ef1e27fe20e3ca17
It worked in my case with my LDAP server. Feel free to play with it and leave a comment. Thanks.

Bravo @WilsonZiweiWang for your LDAP implementation. @tjbck could @WilsonZiweiWang's code be integrated to this project?

Feel free to make a PR!

@WilsonZiweiWang with some refac that might be workable for more servers. Not working on my AD server, but honestly I'm fairly sure it's either a layer8 problem, or we just need more environment variables to tune the (too many) options for LDAP binding, user search and field mappings.

WilsonZiweiWang with some refac that might be workable for more servers. Not working on my AD server, but honestly I'm fairly sure it's either a layer8 problem, or we just need more environment variables to tune the (too many) options for LDAP binding, user search and field mappings.

My implementation: WilsonZiweiWang@f4e4874 It worked in my case with my LDAP server. Feel free to play with it and leave a comment. Thanks.

I wasn't quite able able to get this working out the box, but adding
LDAP_USER_DN = f'cn={cn},{LDAP_USERS_DN}'
inside if uid == form_data.user:

I got this to work with @Peter-De-Ath's modification on a test LDAP server, but still unable to get working on my (probably misconfigured by my predecessor) AD domain.
Some notes I'd add:

This won't interact well with WEBUI_LOGIN=false, the email/username and password fields are hidden
Could we not use the email field for login? We already get it from the LDAP server to create the account
Would prefer if LDAP is enabled that a button be shown in similar style to the OIDC login buttons, hidden otherwise
May want to be able to configure which field is used for the username

hi all, thanks for testing my code and leaving comments, I have created the PR for this issue: https://github.com/open-webui/open-webui/pull/5056

My implementation: WilsonZiweiWang@f4e4874 It worked in my case with my LDAP server. Feel free to play with it and leave a comment. Thanks.

I am getting the following error when I install your fork over the official one:
OperationalError: (sqlite3.OperationalError) no such table: config
[SQL: SELECT config.id AS config_id, config.data AS config_data, config.version AS config_version, config.created_at AS
config_created_at, config.updated_at AS config_updated_at
FROM config ORDER BY config.id DESC
 LIMIT ? OFFSET ?]
[parameters: (1, 0)]

hi all, thanks for testing my code and leaving comments, I have created the PR for this issue: #5056

I tried to install from your fork following the instructions on Ollama's documentation for git installation, but I do not see any options to connect to a LDAP server in Admin Panel > Connections. Am I supposed to install it another way?

LDAP support merged to dev. More testing wanted from the community here!

Hello! We managed to make it work with LDAP. Thanks a lot for adding this feature.
Some small issues I ran into:

Auths.insert_new_auth only gets mail/hash/cn parameters. Other auth methods also accept a role (default to pending otherwise no matter how DEFAULT_USER_ROLE  is defined. It's also not possible to add profile picture (although it could be difficult with ldap, at least in my experience with our LDAP, ldapsearch returns a path to a temp image file, not a real URL).
The LDAP_SEARCH_FILTERS is always messy to get right. I didn't understand that it already did f&(LDAP_ATTRIBUTE_FOR_USERNAME={login})({LDAP_SEARCH_FILTERS}).
In any case testing with dev was very easy, amazing work all around.

I also got LDAP working with a simple ldap filter to a specific group, TLS and no issues authenticating.  The only issue I see is the default Role is Pending instead of User as mentioned above.

Are the LDAP variables set in the .env file?

Are the LDAP variables set in the .env file?

You can set initial values to them in the .env file or edit them with the interface

works great. thank you.

thanks for the implementation :) I also tested it, works like a charm.

would it be possible to share a sample of a working configuration?

would it be possible to share a sample of a working configuration?

You will need to know how your active directory is configured.

I know, but still not able to integrate and the binding error appearing. If there is a guide or tutorial for setting up AD integration would be helpful to everyone.

I know, but still not able to integrate and the binding error appearing. If there is a guide or tutorial for setting up AD integration would be helpful to everyone.

The configurations can be very different from case to case, I think first it is best that you let us know what errors you are having. You could create an issue.

Thank you for the hard work! I have the LDAP auth running but wondering how am i able to map different Ldap-Groups to OpenWebUI's Permission-Groups. Is this to customized so that i have to fork or do you see the same use case?

Thank you for the hard work! I have the LDAP auth running but wondering how am i able to map different Ldap-Groups to OpenWebUI's Permission-Groups. Is this to customized so that i have to fork or do you see the same use case?

Why not just create a single group and nest groups under that one?

Thank you for the hard work! I have the LDAP auth running but wondering how am i able to map different Ldap-Groups to OpenWebUI's Permission-Groups. Is this to customized so that i have to fork or do you see the same use case?

Why not just create a single group and nest groups under that one?

I already have a meta-group in ldap, so that the login works for both groups. But i want ldap-group-a to have openwebui-permission-group-a (can only access model llamaX) and ldap-group-b to have openwebui-permission-group-b (can also access model gpt4 + working area).

Out of interest, did anyone get sAMAccountName working for LDAP_ATTRIBUTE_FOR_USERNAME?  Thanks.
Trying filter: (sAMAccountName=%s)


Cloud-build

[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 06f7abf1-fbaf-82dc-16a7-0e977675f624
Version Independent ID: 111a77e0-d565-355f-3ee0-425f61912e75
Content: Azure enterprise scaffold is now the Cloud Adoption Framework for Azure - Cloud Adoption Framework
Content Source: docs/resources/azure-scaffold.md
Service: cloud-adoption-framework
Sub-service: general
GitHub Login: @rdendtler
Microsoft Alias: martinek

@16CentAstrology - can you please elaborate on what feedback you're providing?


Tests globally mock moment

Description
Mock moment to moment-timezone, so tests pass on CI.

Pull Request Test Coverage Report for Build 325

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 33.655%

Totals

Change from base Build 319:
0.0%

Covered Lines:
157

Relevant Lines:
460

💛  - Coveralls


feat(cdk-experimental/menu): add context menu trigger directive

Add a directive which opens an attached context menu when a user right clicks within the
triggering element. The context menu trigger also considers nested context menu triggers within
an element opting to open the lowest level non-disabled context menu.

This pr implements the basic context menu logic. A follow on PR will implement keyboard triggering

@teflonwaffles @crisbeto @jelbourn feedback should be addressed


hi ayushdabra I'm working on my project similar to your's.

Hi, thank you for your project posted on public.
I just want to know how ImageDataGenerator() works, and It'd be real helpful if you show your image's directory structure. Can you do it for faster look-up if you don't mind?
it would be like:
data/
    train/
        dogs/
            dog001.jpg
            dog002.jpg
            ...
        cats/
            cat001.jpg
            cat002.jpg
            ...
    validation/
        dogs/
            dog001.jpg
            dog002.jpg
            ...
        cats/
            cat001.jpg
            cat002.jpg
            ...

+) why need to_categorical() when you've already assigned class_mode = 'categorical'?
I did this and found out I had to, because when I checked labels by train_generator.labels,
It showed the same label as when class_mode = 'sparse', but I don't know why this happened..

All problem solved thank you for sharing your project again


Research: Integrate Lifecycle Controller with ArgoCD Resource Hooks

Goal
Research how we can integrate the LifeCycle Controller with ArgoCD and document the changes we would need.
Technical Details
ArgoCD offers hooks that are triggered before and after a deployment.
Acceptance Criteria

[ ] Document which components of the LifeCycle controller are still required (e.g., Event Controller) and which are not needed anymore (e.g., Keptn Scheduler)
[ ] Document how the pre- and post-Sync hooks should look like
[ ] Document any found limitation

DoD

[ ] Document how much of the LifeCycle Controller can be built on top of ArgoCD as part of this ticket

Dependencies

https://github.com/keptn-sandbox/lifecycle-controller/issues/15

Resources

https://argo-cd.readthedocs.io/en/stable/user-guide/resource_hooks/

Maybe it's something where we could actually use the CloudEvents /CDEvents interface

@thisthat @thschue can we close this?


baltimore tooltips WIP

I rule and have completed some work on Case Manager that's ready for review!
Added tooltip config to display baltimore config if FUND == 'BAF'.
Not sure what the actual bmore fund is, but assuming BAF.
This pull request makes the following changes:

Adds baltimore tooltip config

It relates to the following issue #s:

Fixes #1119
Bumps #Y

@tingaloo since this is something that might have the propensity to shift a little bit I was thinking about switching this to a config. Probably the steps there are add something in the config model of accepted configs, and then futz around with the names, but you're like halfway there by identifying the right methods. Any interest in taking a whack?

Not really, I thought this was an easy fix. And I have no clue how config models work.

Resolved via #1215


[HUDI-4810] Fixing Hudi bundles requiring log4j2 on the classpath

Change Logs
In https://github.com/apache/hudi/pull/6170, we've rebased Hudi to instead mostly rely on Log4j2 bridge and implementations (in tests).
However we actually missed the fact that log4j-1.2-api isn't actually an API module (as advertised) but rather a fully-fledged implementation bringing in requirement to provide Log4j2 impl jar on the classpath.
Impact
Risk level: Medium
TBD Manual bundles compatibility verification.
Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

CI is green:

https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=11262&view=results


auto adds the discovery token

WIP because while this is called it doesn't actual add the token like the code would suggest @JamesSmartCell please advise

@JamesSmartCell ready to go

related? https://github.com/AlphaWallet/alpha-wallet-android/issues/870