what is mx.arraycollection similar to jewel.arraylistview? or another class?

Hi, since this is not an issue but a question about why we did something in a particular way, I think you should ask for it in:

stack overflow here: https://stackoverflow.com/questions/tagged/apache-royale
In in mailing lists

so others can benefit from the responses.
Thanks! :)


Translations update from Hosted Weblate

Translations update from Hosted Weblate for pappl/PAPPL.
Current translation status:

A short comment about this: I tried to fix the worst translation errors (the automatic translation was somehow funny, but sometimes very misleading). This certainly needs some more finetuning later - ideally by someone from the print industry, since I do not know all of the special keywords and paper formats, etc


.installPlugin(createjs.Sound) crashes chrome browser

So I think, preload.installPlugin(createjs.Sound) crashes chrome browser with multiple refreshes.
Below is the code that may matter of how Im using it. It seems to work sometimes with ease sometimes, then other times it crashes - with no console.log errors

Also, Overall Progress of the queue seem to iterate up to 1.2 (120%) for some reason, so to have it end at 100% I have to use the number "83.3334". - ref function handleOverallProgress()

Possibility:  Maybe I am not doing something correctly, maybe someone can help me tweak/rewrite.
Note: I would like it to work with loadQueue() as I have it. Though I've read sounds can also be handled using registerSounds(), Im also using preloadJS to load folders, and multiple types of file extensions as well.
Let me know. Thanks much. :)
var manifestFolder;
var manifest;
//===============================================================
function pload_mainSounds(){
manifestFolder="s/";
manifest = [{ id: "msgIntroLoop", src: "msgIntroLoop_5305-13513.mp3" },
{ id: "rIntro", src: "rIntro_13513-30464.mp3" },
{ id: "landnIntro", src: "landnIntro_27227-80581.mp3" },
{ id: "bkgLoop", src: "bkgLoop_79146-106907.mp3" },
{ id: "msgIntro", src: "msgIntro_0-5305.mp3" }
];
loadFiles();
}
//===============================================================
function loadFiles() {
if (preload != null) { preload.close(); }
createjs.Sound.alternateExtensions = ["mp3"];
preload = new createjs.LoadQueue(true, manifestFolder);
createjs.Sound.registerSounds(manifest, manifestFolder);
createjs.Sound.registerPlugins([createjs.WebAudioPlugin, createjs.HTMLAudioPlugin]);
preload.installPlugin(createjs.Sound); //see question #1 above
//crashes chrome browser for some reason with multiple refreshes.
preload.loadManifest(manifest);preload.on("fileload", handleFileProgress);preload.on("progress", handleOverallProgress);preload.on("fileprogress", handleFileProgress);preload.on("error", handleFileError);preload.setMaxConnections(2);
preload.on("complete", ploadComplete);
}
//===============================================================
function handleOverallProgress(event) {
console.log((preload.progress*83.3334|0) + " OverallProgress" ); //see question #2 above
}
//===============================================================
function ploadComplete() {
createjs.Sound.play("msgIntroLoop", interrupt="any", delay=0, offset=0, loop=-1, volume=1)
}

There are a couple of things going on here.
First, there seems to be an issue with the latest release of Chrome, with more details available on Stack Overflow.
Second, your code seems to be mixing SoundJS internal loading and PreloadJS loading in a confusing way.  You are meant to only use one or the other.  The Preloading Tutorial can help explain things more clearly.
Hope that helps.


[BUG] Responsive Design Issue of the Contact Page

Description
The contact page while checking the responsiveness the UI of the page is not showing good and it can be improved.
Screenshots

The name box end getting hidden by the picture and the UI of the page is not showing good, I can fix this error.
@swaraj-das Please assign me this task. Please add gssoc-ext and hacktober-accepted labels.
Any additional information?
No response
What browser are you seeing the problem on?
No response
Checklist

[X] I have checked the existing issues.
[X] I have read the Code Of Conduct
[X] The changes don't break the code
[X] Please attach the screenshots and video of the development work once it is done

@swaraj-das Can you please assign me the level

@swaraj-das I have fixed the bug of the contact page please check it and merge it. Please assign level to my PR

@swaraj-das Please merge the PR


Provide Prometheus metrics

Provide a /metrics endpoints that can be used to monitor the application to Prometheus.
Some metrics that can be monitored:

How many users that joined to the group (per group name) -- either just joined, finished, or didn't finished the captcha
How many users that finished the captcha (per group name)
How many users that didn't finish the captcha (per group name)
How many users are donig the captcha right now (per group name)
How many commands are the bot sending to Telegram (per group name) -- commands means everything from banning a user, deleting a message, and sending a message.

Bang ini kalau mau dapetin nama grupnya harus manggil method atau field apa ya?

Cek ke dokumentasi Telebotnya aja.
Sent from Proton Mail for iOS
On Sun, Oct 9, 2022 at 11:58, Iqbal H @.***> wrote:

Bang ini kalau mau dapetin nama grupnya harus manggil method atau field apa ya?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

Ini metrik per nama grupnya diregister setelah botnya ditambahkan ke group gitu ya bang


chore(main): release docs-site 1.11.6

:robot: I have created a release beep boop
1.11.6 (2024-07-19)
Documentation

docs-site: address docs and scripts friction points (#17815) (c74968b)
docs-site: update deploy-a-proverset.mdx (#17813) (dd09223)

This PR was generated with Release Please. See documentation.

:robot: Created releases:

docs-site-v1.11.6
:sunflower:


Feature/return constraint symbol

Updated the Model.add_constraint() method to return the constraint symbol, which can be inspected by the user to determine constraint feasibility. This will allow users to more easily track which constraints are violated if a model state is infeasible compared to using the Model.iter_constraints() method. Update summary:

Add return type to Model.add_constraint() method in model.pyi.
Return value parameter in Model.add_constraint() method in model.pyx.
Update unit tests for the Model.add_constraint() method in tests/test_model.py.
Add release note.

No explicit updates to the docs, as the new documentation for the Model.add_constraint() method should be built automatically by sphinx with the updated method docstring. However, I can add a blurb about checking constraint feasibility to docs/intro.rst if the addition would add value.

One interesting thing I noticed is the following behaviour:
...
>>> i = model.integer()
>>> c = model.constant(5)
>>> c_sym = model.add_constraint(i <= c)
>>> c_iter, = model.iter_constraints()
>>> id(c_sym) == id(c_iter)
False
>>> c_sym.__repr__()
<dwave.optimization.symbols.LessEqual at 0x0123...>
>>> c_iter.__repr__()
<dwave.optimization.symbols.LessEqual at 0x0123...>
>>> c_sym.__repr__() == c_iter.__repr__()
True
>>> c_sym.equals(c_iter)
True

I think this is because iter_constraints() is constructing a new symbol from the pointer in the constraints vector, so there are actually two distinct Python objects pointing to the same constraint. The equals method shows that they are still pointing to the same graph node, but don't know if the differing ids will cause any issue.

Yes, there is a many-to-1 relationship between the Python Symbol objects, and the underlying node. You can do
symbol1.id() == symbol2.id()

to test whether they are using the same node. This is part of the design, for better or worse.


Reorganise d3/d3 issues

(This is going to be self-referential).
Can we help reorganise the open issues in https://github.com/d3/d3/issues ? they should follow the new d3 way of separating topics.
Browsing through the list of currently 87 open issues, I see several types:

General questions that are more or less well answered, but stay opened for some reason (for example, because they contain a nice idea to contemplate for a future development; or maybe just because no one pushed the "close" button). I would recommend to close them, possibly writing down the idea in an idea list.

Issues that have been solved with v4 — and can be closed. Often this means, when there is a test case, adapting the d3v3 test case to v4.

Issues that haven't been solved with v4, but would need to be transferred to the relevant d3-xxxx repo. We could close them and create a reference issue in d3-xxxx.

Issues that belong to d3 “core” and are not solved. Usually these are tough nuts to crack and they may benefit from a little more explanations or test cases. They should be kept open, and could be tagged “core”; they also include "meta" issues such as this very issue.

Issues that belong to d3 “project” and are not solved. Usually they are requests for new features, which should now be treated as ideas for new plugins. They could be kept and tagged “plugins”, or regrouped in some other repo, and added to the idea list.

Additionally I think it would help to add an issue template.

I’ve added an issue template. Thank you for the suggestion.
I’ve closed all the open pull requests, since none of them can be merged now that the code has been moved to different repositories.
Perhaps this my frustration speaking, but I’m partly inclined to turn off issues on this repository, since the vast majority of issues filed here are invalid: they would be better handled as questions on Stack Overflow or as issues filed in the appropriate repository. The main downside of this is that past (closed) issues would no longer be searchable. Unfortunately GitHub doesn’t allow issues to be read-only.

@Fil It would also be good if we also have labels good first bug tagged to help the newbie d3 developers, like me to start contributing.


styleguide

Which do you prefer? .eslint or .jshint single or double quotes etc etc.

I definitely prefer eslint. For my full configuration preferences, see eslint-config-kentcdodds


feat(core) - Allow for configurable exit codes

Issue #, if available:
Description of changes:

Allow for the configuration of how the exit code process is handled

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Still wonder if there's a way to sync data/CfnLintCli/config/schema.json with the CLI parameters

Still wonder if there's a way to sync data/CfnLintCli/config/schema.json with the CLI parameters

Open to ideas for that.  It would be nice for sure.


AR sync, 1.13.18, libglib

@conda-forge-admin, please rerender
^-- just opening a PR for me to work on.

Hi! This is the friendly automated conda-forge-webservice.
I just wanted to let you know that I rerendered the recipe in conda-forge/dbus-feedstock#21.


wireguard: 1.0.20200401 -> 1.0.20200413

Motivation for this change
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

The sed patch did not work for me. I found a different way.

@Ma27 yes.

Backported as 24cf10d08084f5e314bdc23a07bab3c8ed3eb583, a49a2fb8c10c7219a9bdff9801bf09bb168905c6 (only two commits since I squashed the two passthru.tests changes together and removed the wg-quick test since it's not available on 20.03).


workbox-webpack-plugin - chunks option in conflict with precaching index.html

Library Affected:
workbox-webpack-plugin
Browser & Platform:
all browsers
Issue or Feature Request Description:
Background here: we are using the webpack-html-plugin to generate index.html and workbox-webpack-plugin to inject the __precacheManifest into our service worker file.
When chunks option is not specified, InjecInjectManifestwill generate a list of all the files compiled by webpack (including all the static assets like html, images etc). However this is not really an option for us, as we have 100+ JS files. It's not wise to precache all of them upfront.
So we are using the chunks option to only precache the application shell related code. However, since the webpack-html-plugin generated html will have no associated chunks. The index.html will be filtered out from the precacheManifest
I tried to work around the issue by using the old glob way. However it's meant only for external files, not the webpack compilations.
I dig into the source code and seems no way to work around this issue. I'm pretty sure this is a common use case for large websites like us, am I missing anything here or this is a feature by     ####design?

Hello @pedi—Sorry that you're running into that. As an alternative to chunks, you could potentially make use of one of the following supported options:

excludeChunks, if it's cleaner to list a set of chunk names to avoid caching. Assets that don't match, like your index.html, should still be precached.

include and exclude, to specify one or more RegExp patterns that will match against asset names and allow you to opt-in or opt-out of precaching certain assets.

Do either of those options sound like they'd help?

Thanks @jeffposnick for quick follow up on this.
Option 1 might not be feasible for us as the exclude list will be long and hard to maintain
Option 2 can serve as a workaround for us at the moment. However the chunks still provide a lower long -term maintenance cost as more files could be emitted from the same chunk in future and their file names might not necessarily match the predefined regex.
Currently our workaround is to to compile sw.js as a separate webpack stage and use the built one as a template to feed into the InjectManifest plugin.
Hope the code snippet below can explain our approach clearly.
const precacheFileList = self.__precacheManifest;

workbox.precaching.precacheAndRoute(
  precacheFileList.concat([
    {
      url: 'index.html',
      revision: __GIT_SHA__,
    },
  ])
);

This should be addressed by the current Workbox v5.0.0 alpha.


[F] Update docker_october scaffold

@lthurston I tested this on Hockney and it seemed scaffold everything the way I'd expect. Can you please eyeball and/or give it a quick test for me?

@lthurston polite bump


database is dropped using alicloud RDS postgres instance

the database will be dropped suddenly, without any errors in casdoor logs.

found the problem is caused by alicloud terrraform provider. It sucks...


How to evaluate?

I see the evaluate script, but the test_***.py python file is missing.
Want to ask if it's open source, or provide implementation details

Thank you for your question.
Regarding the helpfulness dimension, we conduct tests on MTBench.
For the honesty and harmlessness dimensions, our test data can be found at: src/CPSFT/cpsft/test_data_harmlessness and src/CPSFT/cpsft/test_data_honesty. The test code is located at: src/CPSFT/cpsft/test_harmlessness.py and src/CPSFT/cpsft/test_honesty.py.
I hope this information is helpful to you.

It seems that the code provided for evaluating honesty only output answers from LLMs. There is no output metric. Can you provide the code for this?

To rate the responses generated by GPT-4 regarding the templates for "helpfulness" and "honesty" in the file preference_templates.py from the repository https://github.com/OpenBMB/UltraFeedback/blob/main/src/data_annotation/preference_templates.py


Add MeSH mappings for mouse proteins

[ ] Map MGI gene identifiers to UniProt
[ ] Identify other organisms available from MeSH

Doesn't UniProt provide MGI mappings though? I'm pretty sure it does.

See e.g, https://www.uniprot.org/uniprot/?query=braf&fil=organism%3A"Mus musculus (Mouse) [10090]"&columns=id%2Cdatabase(MGI)&sort=score

UniProt mapping is the big TODO - we can do lots of different automated mapping generations with the last push

I might be missing something but I think UniProt provides mappings to most relevant organism specific databases.

@bgyori yes they do! I haven't written code for this, so maybe you could help. I think you're working on improving ProtMapper to do this right now, right?


Add a README

Add a README for the main branch , should be the whole description of the assigment

@Cristiana959 updated the README


Kawaja pylint dup

Summary

Many lines are false-positive alerts.
Pylint disable-all was used instead of disable duplicate-code due to that option not effectively working.

Objective
--duplicate code argument for the pylint command was added while we did a version 0.1.0 to v 1.0.0 transition. The 0.1.0 code has been removed and the pylint cmd needs to be altered from:
pylint -f parseable -d duplicate-code --max-args=8 --indent-string '  ' *.py
to
pylint -f parseable --max-args=8 --indent-string '  ' *.py
Edits Summary
==tuna.driver_bn:58
==tuna.driver_conv:75
Status: [Resolved]
Comment: function cmd() is duplicated across files. Function were renamed to reflect the file name tags.
==tuna.find_db:44
==tuna.miopen_tables:410
==tuna.miopen_tables:483
Status:[Resolved - Workaround]
Comment: function solver() and session() are duplicated across three filesd under different class names. disableed pylint @ line 44 in find_db
==tuna.import_db:55
==tuna.load_job:109
Status: [Resolved - Workaround]
Comments: Ignored pylint @ line 109, renaming or changing code structure is unnessary and could result in compile errors.
==tuna.driver_bn:112
==tuna.driver_conv:138
Status: [Resolved]
Comments: renamed functions set_defaults with suffix reflecting the file/class name.
==tuna.fin_class:78
==tuna.worker_interface:121
Status:[Resolved - workaround]
Comments: pylint false-flags this line as duplicate code
==tuna.export_db:59
==tuna.go_fish:134
Status: [Resolved - workaround]
Comments: pylint false flag this line as duplicate code
==tuna.driver_bn:114
==tuna.driver_conv:140
Status: [Resolved-worked around]
Comments: pylint false flag this line as duplicate code
==tuna.driver_base:96
==tuna.helper:236
Status: [Resolved]
Comments: pylint false flag this line as duplicate code with deprecated function from the helper module.
==tuna.fin_builder:58
==tuna.worker_interface:346
Status: [Resolved]
Comments: pylint false flag this line as duplicate code
==tuna.go_fish:137
==tuna.import_db:59
Status: [Resolved]
Comments: pylint false flag this line as duplicate code
Errors Prior to modification:
`worker_interface.py:1: [R0801(duplicate-code), ] Similar lines in 2 files
==tuna.driver_bn:58
==tuna.driver_conv:75
    super().__init__(line)
    #allow cmd input to override driver line
    if cmd:
      self._cmd = cmd

  @property
  def cmd(self):
    """Setting 'private' variable"""
    return self._cmd

  @cmd.setter
  def cmd(self, value):
    """Checking for allowed conv values"""
worker_interface.py:1: [R0801(duplicate-code), ] Similar lines in 3 files
==tuna.find_db:44
==tuna.miopen_tables:410
==tuna.miopen_tables:483
  @declared_attr
  def solver(self):
    """solver column"""
    return Column(Integer, ForeignKey("solver.id"), nullable=False)

  @declared_attr
  def session(self):
    """session column"""
    return Column(Integer, ForeignKey("session.id"), nullable=False)

worker_interface.py:1: [R0801(duplicate-code), ] Similar lines in 2 files
==tuna.import_db:55
==tuna.load_job:109
  parser.add_argument(
      '--session_id',
      action='store',
      required=True,
      type=int,
      dest='session_id',
      help=
      'Session ID to be used as tuning tracker. Allows to correlate DB results to tuning sessio
ns'
  )
worker_interface.py:1: [R0801(duplicate-code), ] Similar lines in 2 files
==tuna.driver_bn:112
==tuna.driver_conv:138
  def set_defaults(self, defaults):
    """Set fds defaults"""
    for k, val in self.to_dict().items():
      if val is None and k in defaults.keys():
        setattr(self, k, defaults[k])

  @staticmethod
  def get_params(tok1):
    """Get full arg name"""
worker_interface.py:1: [R0801(duplicate-code), ] Similar lines in 2 files
==tuna.fin_class:78
==tuna.worker_interface:121
    self.session_id = None

    self.__dict__.update(
        (key, value) for key, value in kwargs.items() if key in allowed_keys)

    self.dbt = DBTables(session_id=self.session_id,
                        config_type=self.config_type)

worker_interface.py:1: [R0801(duplicate-code), ] Similar lines in 2 files
==tuna.export_db:59
==tuna.go_fish:134
  parser.add_argument(
      '--session_id',
      action='store',
      type=int,
      dest='session_id',
      help=
      'Session ID to be used as tuning tracker. Allows to correlate DB results to tuning sessio
ns'
  )
worker_interface.py:1: [R0801(duplicate-code), ] Similar lines in 2 files
==tuna.driver_base:96
==tuna.helper:236
  ret_id = None
  with DbSession() as session:
    try:
      tid = TensorTable(**tensor_dict)
      session.add(tid)
      session.commit()
      ret_id = tid.id
worker_interface.py:1: [R0801(duplicate-code), ] Similar lines in 2 files
==tuna.fin_builder:58
==tuna.worker_interface:346
      kernel_obj.kernel_name = kern_obj['kernel_file']
      kernel_obj.kernel_args = kern_obj['comp_options']
      kernel_obj.kernel_blob = bytes(kern_obj['blob'], 'utf-8')
      kernel_obj.kernel_hash = kern_obj['md5_sum']
      kernel_obj.uncompressed_size = kern_obj['uncompressed_size']
worker_interface.py:1: [R0801(duplicate-code), ] Similar lines in 4 files
==tuna.export_db:62
==tuna.go_fish:137
==tuna.import_db:59
==tuna.load_job:113
      type=int,
      dest='session_id',
      help=
      'Session ID to be used as tuning tracker. Allows to correlate DB results to tuning sessio
ns'
  )`

@reidkwja  please bring your branch up to date with develop, so its easier to review.

@reidkwja please bring your branch up to date with develop, so its easier to review.

@alexandraBara
Branch up-to-date now!


Set .spec.replaces to avoid removing published/released versions

Following https://docs.engineering.redhat.com/display/CFC/Best_Practices#Best_Practices-SkipRange

/override ci/prow/e2e-aws-operator
CSV file is not CI tested


Main report - Update cell type introduction

We need to update some of the text that is present in the introduction of the cell type section in the main QC report:

[x] Make sure that description of the annotations is in the same order that is used throughout the report. Submitter, SingleR, CellAssign
[x] Add in citations for SingleR and CellAssign

Closed by #523


Watch TargetGroupPolicy CRD event on controllers

What type of PR is this?
feature
Which issue does this PR fix:
Related to #69 and #304 but not fixing the issue yet.
What does this PR do / Why do we need it:

This is the first part of TargetGroupPolicy change PR, just "watching" the event of TargetGroupPolicy change. Splitting out actual model changes because it is a lot bigger.
The event handler is written a bit differently from other handlers, using controller library construct handler.EnqueueRequestsFromMapFunc. This clearly indicates the resource relationship, and enables easier unit testing IMO.
The watch is happening only for Route/ServiceExport controllers which actually involve target group operations.
Also brings back DNSEndpoint watcher which got lost during route controller merge.

If an issue # is not available please add repro steps and logs from aws-gateway-controller showing the issue:
Testing done on this change:

make e2etest passing
Added event handler unit tests as well - the controller package is now part of make test

Automation added to e2e:

Will this PR introduce any new dependencies?:

Will this break upgrades or downgrades. Has updating a running cluster been tested?:
Does this PR introduce any user-facing change?:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

80 of 138 (57.97%)  changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-4.9%) to 34.99%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

controllers/serviceexport_controller.go
0
14
0.0%

controllers/route_controller.go
0
17
0.0%

controllers/eventhandlers/targetgrouppolicy.go
80
107
74.77%

Totals

Change from base Build<PHONE_NUMBER>:
-4.9%

Covered Lines:
3535

Relevant Lines:
10103

💛  - Coveralls


Release 1.3.2 (Fixes 1.3.1)

New business features
None
New technical/security features
Make a script to run APIHUB process without the need of the launcher service, to make epi-runner image to be more easy monitored by the kubernetes. #732
None
Tasks implementing the new features
None
Bug Fixes
Overview: Changes in paths to accommodate with the read only images for build
Move all apihub components storage to external-volume #731
Leaflet message with no product code creates a memory leak #742
Close button for ACF iframe #730
Minor fixes:
Disable remove not completed/not implemented Demiurge features #691
Security Fixes
None
Configuration Changes
None


Run out of memory when forecast much longer steps

For ai-models-graphcast, it works fine when I predict only a few time steps. However, it fails with an "out of memory" error when I try to predict over a longer lead time, such as 10 days. I have 188 GB of memory for CPU or 24 GB for GPU. Is there any solution to avoid this issue? It appears that the memory used by the model is not released after completing each step.
Thanks for your reply in advance!

Yes, we are aware of that problem and are looking into it.

Try using these commands before running graphcast. Explanation is at https://jax.readthedocs.io/en/latest/gpu_memory_allocation.html
export XLA_PYTHON_CLIENT_PREALLOCATE=false
export XLA_PYTHON_CLIENT_ALLOCATOR=platform

Try using these commands before running graphcast. Explanation is at https://jax.readthedocs.io/en/latest/gpu_memory_allocation.html
export XLA_PYTHON_CLIENT_PREALLOCATE=false
export XLA_PYTHON_CLIENT_ALLOCATOR=platform

I have tried your solution, but it failed with the following message:
2023-10-26 10:49:40,924 INFO Download rate 3.8M/s
2023-10-26 10:49:40,924 INFO Download rate 3.8M/s
2023-10-26 10:49:49,065 INFO Creating forcing variables: 6 seconds.
2023-10-26 10:49:56,199 INFO Converting GRIB to xarray: 7 seconds.
2023-10-26 10:50:01,614 INFO Reindexing: 5 seconds.
2023-10-26 10:50:01,665 INFO Creating training data: 6 minutes 22 seconds.
2023-10-26 10:50:11,951 INFO Extracting input targets: 10 seconds.
2023-10-26 10:50:11,951 INFO Creating input data (total): 6 minutes 32 seconds.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1698288611.955721    7562 tfrt_cpu_pjrt_client.cc:349] TfrtCpuClient created.
2023-10-26 10:50:12,075 INFO Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm". Available platform names are: CUDA
2023-10-26 10:50:12,075 INFO Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory
2023-10-26 10:50:36.996835: E external/xla/xla/service/slow_operation_alarm.cc:65] Constant folding an instruction is taking > 1s:
%pad.151 = bf16[3114720,8]{1,0} pad(bf16[3114720,4]{1,0} %constant.369, bf16[] %constant.771), padding=0_0x0_4, metadata={op_name="jit()/jit(main)/while/body/remat/mesh2grid_gnn/_embed/mesh2grid_gnn/sequential/encoder_edges_mesh2grid_mlp/linear_0/dot_general[dimension_numbers=(((2,), (0,)), ((), ())) precision=None preferred_element_type=bfloat16]" source_file="/home/dyf/anaconda3/envs/graphcast/bin/ai-models" source_line=8}
This isn't necessarily a bug; constant-folding is inherently a trade-off between compilation time and speed at runtime. XLA has some guards that attempt to keep constant folding from taking too long, but fundamentally you'll always be able to come up with an input program that takes a long time.
If you'd like to file a bug, run with envvar XLA_FLAGS=--xla_dump_to=/tmp/foo and attach the results.
2023-10-26 10:50:39.321182: E external/xla/xla/service/slow_operation_alarm.cc:133] The operation took 3.32445162s
Constant folding an instruction is taking > 1s:
%pad.151 = bf16[3114720,8]{1,0} pad(bf16[3114720,4]{1,0} %constant.369, bf16[] %constant.771), padding=0_0x0_4, metadata={op_name="jit()/jit(main)/while/body/remat/mesh2grid_gnn/_embed/mesh2grid_gnn/sequential/encoder_edges_mesh2grid_mlp/linear_0/dot_general[dimension_numbers=(((2,), (0,)), ((), ())) precision=None preferred_element_type=bfloat16]" source_file="/home/dyf/anaconda3/envs/graphcast/bin/ai-models" source_line=8}
This isn't necessarily a bug; constant-folding is inherently a trade-off between compilation time and speed at runtime. XLA has some guards that attempt to keep constant folding from taking too long, but fundamentally you'll always be able to come up with an input program that takes a long time.
If you'd like to file a bug, run with envvar XLA_FLAGS=--xla_dump_to=/tmp/foo and attach the results.
W0000 00:00:1698288645.880896    7562 hlo_rematerialization.cc:2946] Can't reduce memory use below -17.35GiB (-18632099542 bytes) by rematerialization; only reduced to 52.66GiB<PHONE_NUMBER>6 bytes), down from 52.66GiB<PHONE_NUMBER>6 bytes) originally
2023-10-26 10:50:47.788508: E external/xla/xla/pjrt/pjrt_stream_executor_client.cc:2644] Execution of replica 0 failed: RESOURCE_EXHAUSTED: Failed to allocate request for 5.72GiB<PHONE_NUMBER>B) on device ordinal 0
BufferAssignment OOM Debugging.
 Set JAX_TRACEBACK_FILTERING=off to include these.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/dyf/anaconda3/envs/graphcast/bin/ai-models", line 8, in
sys.exit(main())
File "/home/dyf/anaconda3/envs/graphcast/lib/python3.10/site-packages/ai_models/main.py", line 285, in main
_main()
File "/home/dyf/anaconda3/envs/graphcast/lib/python3.10/site-packages/ai_models/main.py", line 258, in _main
model.run()
File "/home/dyf/anaconda3/envs/graphcast/lib/python3.10/site-packages/ai_models_graphcast/model.py", line 240, in run
output = self.model(
File "/home/dyf/anaconda3/envs/graphcast/lib/python3.10/site-packages/ai_models_graphcast/model.py", line 117, in
return lambda **kw: fn(**kw)[0]
jaxlib.xla_extension.XlaRuntimeError: RESOURCE_EXHAUSTED: Failed to allocate request for 5.72GiB<PHONE_NUMBER>B) on device ordinal 0
BufferAssignment OOM Debugging.

I'm using a Tesla V100-SXM2-32GB.

I'm using a Tesla V100-SXM2-32GB.
I have rented two RTX 4090 24GB GPUs from a cloud service, but unfortunately, they are not functioning properly. As a result, I am only able to make predictions around 3 to 5 times (forget the specific number).

Good day all,
I have run this on AWS, and was only able to do so using their 'p5' instance. Anything less, and I got the memory issues noted above.

Initially, I used a 3090 with 24GB, but the run failed immediately. It was not until I rented an A100 with 80GB that I found everything could operate normally.
Following the example command ai-models --input cds --date 20230110 --time 0000 graphcast, I successfully obtained a file named graphcast.grib.
Additionally, during the run, I observed that both memory and video memory usage were around 60GB (the Jax library directly utilizes about 3/4 of the video memory).
(ai) root@747c17acba6c:/# /opt/conda/envs/ai/bin/ai-models  --input cds --date 20230110 --time 0000 graphcast
2023-11-28 06:50:35,205 INFO Writing results to graphcast.grib.
/opt/conda/envs/ai/lib/python3.10/site-packages/ecmwflibs/__init__.py:81: UserWarning: /lib/x86_64-linux-gnu/libgobject-2.0.so.0: undefined symbol: ffi_type_uint32, version LIBFFI_BASE_7.0
  warnings.warn(str(e))
2023-11-28 06:50:35,531 INFO Model description:
GraphCast model at 0.25deg resolution, with 13 pressure levels. This model is
trained on ERA5 data from 1979 to 2017, and fine-tuned on HRES-fc0 data from
2016 to 2021 and can be causally evaluated on 2022 and later years. This model
does not take `total_precipitation_6hr` as inputs and can make predictions in an
operational setting (i.e., initialised from HRES-fc0).

2023-11-28 06:50:35,531 INFO Model license:
The model weights are licensed under the Creative Commons
Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0). You
may obtain a copy of the License at:
https://creativecommons.org/licenses/by-nc-sa/4.0/.
The weights were trained on ERA5 data, see README for attribution statement.

2023-11-28 06:50:35,531 INFO Loading params/GraphCast_operational - ERA5-HRES 1979-2021 - resolution 0.25 - pressure levels 13 - mesh 2to6 - precipitation output only.npz: 0.3 second.
2023-11-28 06:50:35,531 INFO Building model: 0.3 second.
2023-11-28 06:50:35,531 INFO Loading surface fields from CDS
2023-11-28 06:50:35,656 INFO Loading pressure fields from CDS
2023-11-28 06:50:48,418 INFO Creating forcing variables: 12 seconds.
2023-11-28 06:50:53,993 INFO Converting GRIB to xarray: 5 seconds.
2023-11-28 06:50:57,666 INFO Reindexing: 3 seconds.
2023-11-28 06:50:57,706 INFO Creating training data: 22 seconds.
2023-11-28 06:51:04,715 INFO Extracting input targets: 6 seconds.
2023-11-28 06:51:04,715 INFO Creating input data (total): 29 seconds.
2023-11-28 06:51:05,098 INFO Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm". Available platform names are: CUDA
2023-11-28 06:51:05,102 INFO Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory
2023-11-28 06:52:10.480192: E external/xla/xla/service/slow_operation_alarm.cc:65] Constant folding an instruction is taking > 1s:

  %pad.149 = bf16[3114720,8]{1,0} pad(bf16[3114720,4]{1,0} %constant.365, bf16[] %constant.768), padding=0_0x0_4, metadata={op_name="jit(<unnamed wrapped function>)/jit(main)/while/body/remat/mesh2grid_gnn/_embed/mesh2grid_gnn/sequential/encoder_edges_mesh2grid_mlp/linear_0/dot_general[dimension_numbers=(((2,), (0,)), ((), ())) precision=None preferred_element_type=bfloat16]" source_file="/opt/conda/envs/ai/bin/ai-models" source_line=8}

This isn't necessarily a bug; constant-folding is inherently a trade-off between compilation time and speed at runtime. XLA has some guards that attempt to keep constant folding from taking too long, but fundamentally you'll always be able to come up with an input program that takes a long time.

If you'd like to file a bug, run with envvar XLA_FLAGS=--xla_dump_to=/tmp/foo and attach the results.
2023-11-28 06:52:18.177603: E external/xla/xla/service/slow_operation_alarm.cc:133] The operation took 8.697529096s
Constant folding an instruction is taking > 1s:

  %pad.149 = bf16[3114720,8]{1,0} pad(bf16[3114720,4]{1,0} %constant.365, bf16[] %constant.768), padding=0_0x0_4, metadata={op_name="jit(<unnamed wrapped function>)/jit(main)/while/body/remat/mesh2grid_gnn/_embed/mesh2grid_gnn/sequential/encoder_edges_mesh2grid_mlp/linear_0/dot_general[dimension_numbers=(((2,), (0,)), ((), ())) precision=None preferred_element_type=bfloat16]" source_file="/opt/conda/envs/ai/bin/ai-models" source_line=8}

This isn't necessarily a bug; constant-folding is inherently a trade-off between compilation time and speed at runtime. XLA has some guards that attempt to keep constant folding from taking too long, but fundamentally you'll always be able to come up with an input program that takes a long time.

If you'd like to file a bug, run with envvar XLA_FLAGS=--xla_dump_to=/tmp/foo and attach the results.
2023-11-28 06:52:20.556910: E external/xla/xla/service/slow_operation_alarm.cc:65] Constant folding an instruction is taking > 2s:

  %pad.1 = bf16[1618752,8]{1,0} pad(bf16[1618745,4]{1,0} %constant.374, bf16[] %constant.687), padding=0_7x0_4, metadata={op_name="jit(<unnamed wrapped function>)/jit(main)/while/body/remat/grid2mesh_gnn/_embed/grid2mesh_gnn/sequential/encoder_edges_grid2mesh_mlp/linear_0/dot_general[dimension_numbers=(((2,), (0,)), ((), ())) precision=None preferred_element_type=bfloat16]" source_file="/opt/conda/envs/ai/bin/ai-models" source_line=8}

This isn't necessarily a bug; constant-folding is inherently a trade-off between compilation time and speed at runtime. XLA has some guards that attempt to keep constant folding from taking too long, but fundamentally you'll always be able to come up with an input program that takes a long time.

If you'd like to file a bug, run with envvar XLA_FLAGS=--xla_dump_to=/tmp/foo and attach the results.
2023-11-28 06:52:22.395348: E external/xla/xla/service/slow_operation_alarm.cc:133] The operation took 3.838535293s
Constant folding an instruction is taking > 2s:

  %pad.1 = bf16[1618752,8]{1,0} pad(bf16[1618745,4]{1,0} %constant.374, bf16[] %constant.687), padding=0_7x0_4, metadata={op_name="jit(<unnamed wrapped function>)/jit(main)/while/body/remat/grid2mesh_gnn/_embed/grid2mesh_gnn/sequential/encoder_edges_grid2mesh_mlp/linear_0/dot_general[dimension_numbers=(((2,), (0,)), ((), ())) precision=None preferred_element_type=bfloat16]" source_file="/opt/conda/envs/ai/bin/ai-models" source_line=8}

This isn't necessarily a bug; constant-folding is inherently a trade-off between compilation time and speed at runtime. XLA has some guards that attempt to keep constant folding from taking too long, but fundamentally you'll always be able to come up with an input program that takes a long time.

If you'd like to file a bug, run with envvar XLA_FLAGS=--xla_dump_to=/tmp/foo and attach the results.
2023-11-28 06:52:37,380 INFO Doing full rollout prediction in JAX: 1 minute 32 seconds.
2023-11-28 06:52:37,380 INFO Converting output xarray to GRIB and saving
2023-11-28 06:54:53,203 INFO Saving output data: 2 minutes 15 seconds.
2023-11-28 06:54:53,276 INFO Total time: 4 minutes 19 seconds.
(ai) root@747c17acba6c:/# ls
graphcast.grib  params  sspaas-fs  sspaas-tmp  stats  test.py  tf-logs
(ai) root@747c17acba6c:/# du -lh graphcast.grib
6.5G	graphcast.grib

I have the same issue above, and I am using a Tesla M40-24GB GPU. Is there any solution?

I have the same issue above, and I am using a NVIDIA 2080ti-11GB GPU. Is there any solution? Is it possible to solve this issue by
adjusting the batch_size of the input of the model,and How?

Hello all,
There is no easy solution for this, as it would require a refactor of the code. We will consider it a feature request for the future, but currently do not have any capacity to fulfil it.
Thanks,
Matthew

May I ask what is the minimum graphics card configuration and  minimum memory requirement for running the model？

May I ask what is the minimum graphics card configuration and minimum memory requirement for running the model？

From my experience, the minimum GPU memory requirement for running the model (like Pangu-wather) is 12 to 16 GiB, and for RAM when using CPU only for inference is at least 16 GiB.

May I ask what is the minimum graphics card configuration and minimum memory requirement for running the model？

From my experience, the minimum GPU memory requirement for running the model (like Pangu-wather) is 12 to 16 GiB, and for RAM when using CPU only for inference is at least 16 GiB.

How about Graphcast？

May I ask what is the minimum graphics card configuration and minimum memory requirement for running the model？

From my experience, the minimum GPU memory requirement for running the model (like Pangu-wather) is 12 to 16 GiB, and for RAM when using CPU only for inference is at least 16 GiB.

How about Graphcast？

I'm still testing Graphcast that uses GFS analysis data as the input. It used at least 18 GiB of my GPU memory.

Hello all, There is no easy solution for this, as it would require a refactor of the code. We will consider it a feature request for the future, but currently do not have any capacity to fulfil it. Thanks, Matthew

May I ask what is the minimum graphics card configuration and minimum memory requirement for running the model？I am still bothered by this issue, Is that A100 necessary?

I am trying to run the models with fewer steps and rerun the models with the output grib file as the new input file, but got errors, it seems the output grib file cannot be used as input file for those models. Any idea on why it is like this?

May be your output file only contains the NWP data in 13 pressure levels, rather than 37 pressure levels, which depends on your params files of models.
---Original---
From: @.>
Date: Thu, Apr 25, 2024 23:52 PM
To: @.>;
Cc<EMAIL_ADDRESS>Subject: Re: [ecmwf-lab/ai-models] Run out of memory when forecast much longersteps (Issue #14)
I am trying to run the models with fewer steps and rerun the models with the output grib file as the new input file, but got errors, it seems the output grib file cannot be used as input file for those models. Any idea on why it is like this?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: @.***>

I have written codes to solved this issue by myself, and you can check my forked repository.
Hope this can help you!

I have written codes to solved this issue by myself, and you can check my forked repository. Hope this can help you!

Can you provide the links to your forked repository? I couldn't find it on the main one.

I have written codes to solved this issue by myself, and you can check my forked repository. Hope this can help you!

I found your forked repository of graphcast, but after I installed it and run the ai-models the inference process got kill at first step:
(ai-models) [@localhost ~]$ ai-models --date 20240425 --time 1200 --lead-time 384 --path graphcast/2024-04-25z12:00:00.grib graphcast 2024-04-26 17:32:58,444 INFO Writing results to graphcast/2024-04-25z12:00:00.grib. 2024-04-26 17:32:58,444 INFO Loading surface fields from MARS 2024-04-26 17:32:58,638 INFO Loading pressure fields from MARS 2024-04-26 17:32:59,081 INFO Writing step 0: 0.5 second. 2024-04-26 17:32:59,512 INFO Writing step 0: 0.4 second. 2024-04-26 17:32:59,768 INFO Model description: GraphCast model at 0.25deg resolution, with 13 pressure levels. This model is trained on ERA5 data from 1979 to 2017, and fine-tuned on HRES-fc0 data from 2016 to 2021 and can be causally evaluated on 2022 and later years. This model does not take total_precipitation_6hr as inputs and can make predictions in an operational setting (i.e., initialised from HRES-fc0).
2024-04-26 17:32:59,768 INFO Model license: The model weights are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0). You may obtain a copy of the License at: https://creativecommons.org/licenses/by-nc-sa/4.0/. The weights were trained on ERA5 data, see README for attribution statement.
2024-04-26 17:32:59,768 INFO Loading params/GraphCast_operational - ERA5-HRES 1979-2021 - resolution 0.25 - pressure levels 13 - mesh 2to6 - precipitation output only.npz: 0.3 second. 2024-04-26 17:32:59,768 INFO Building model: 0.3 second. 2024-04-26 17:33:12,513 INFO Creating forcing variables: 12 seconds. 2024-04-26 17:33:15,242 INFO Converting GRIB to xarray: 2 seconds. Killed

If you have GFS analysis data, you can run demo_grib2nc_gfs.py to process it, then run demo_inference.py to inference without ai-models library.

Change the code to run the GraphCast_small model solved this problem for me:

You need to first download the low resolution asset: params/GraphCast_small - ERA5 1979-2015 - resolution 1.0 - pressure levels 13 - mesh 2to5 - precipitation input and output.npz
Modify the ai-models_graphcast/ai_models_graphcast/model.py
line 57-58: the download_files refer to the low resolution asset
line 68: change to grid = [1.0, 1,0]
Modify the ai-models_graphcast/ai_models_graphcast/input.py
line 60: change the code to .reshape(len(forcing_variables), len(dates), 181, 360)

Those changes reduce the resolution to 1.0 and thus requires much less memory

Change the code to run the GraphCast_small model solved this problem for me:

You need to first download the low resolution asset: params/GraphCast_small - ERA5 1979-2015 - resolution 1.0 - pressure levels 13 - mesh 2to5 - precipitation input and output.npz
Modify the ai_models_graphcast/ai_models_graphcast/model.py
line 57-58: the download_files refer to the low resolution asset
line 68: change to grid = [1.0, 1,0]
Modify the ai_models_graphcast/ai_models_graphcast/input.py
line 60: change the code to .reshape(len(forcing_variables), len(dates), 181, 360)

Those changes reduce the resolution to 1.0 and thus requires much less memory

Can you provide the specific operation details? For example, how to download the data?

Hello all, There is no easy solution for this, as it would require a refactor of the code. We will consider it a feature request for the future, but currently do not have any capacity to fulfil it. Thanks, Matthew

For the Pangu model, manually create the InferenceSession and destroy it after each use inside the stepper solved my problem

Change the code to run the GraphCast_small model solved this problem for me:

You need to first download the low resolution asset: params/GraphCast_small - ERA5 1979-2015 - resolution 1.0 - pressure levels 13 - mesh 2to5 - precipitation input and output.npz
Modify the ai_models_graphcast/ai_models_graphcast/model.py
line 57-58: the download_files refer to the low resolution asset
line 68: change to grid = [1.0, 1,0]
Modify the ai_models_graphcast/ai_models_graphcast/input.py
line 60: change the code to .reshape(len(forcing_variables), len(dates), 181, 360)

Those changes reduce the resolution to 1.0 and thus requires much less memory

Can you provide the specific operation details? For example, how to download the data?

In model.py you can change the code lines 57 58 to:
class GraphcastModel(Model):
download_url = "https://storage.googleapis.com/dm_graphcast/{file}"
expver = "dmgc"
# Download
download_files = [
    (
        # "params/GraphCast_operational - ERA5-HRES 1979-2021 - resolution 0.25 -"
        # " pressure levels 13 - mesh 2to6 - precipitation output only.npz"
     "params/GraphCast_small - ERA5 1979-2015 - resolution 1.0 -"
        " pressure levels 13 - mesh 2to5 - precipitation input and output.npz"
    ),
    "stats/diffs_stddev_by_level.nc",
    "stats/mean_by_level.nc",
    "stats/stddev_by_level.nc",
]

...
you can manually download the file from the download_url above.

Change the code to run the GraphCast_small model solved this problem for me:

You need to first download the low resolution asset: params/GraphCast_small - ERA5 1979-2015 - resolution 1.0 - pressure levels 13 - mesh 2to5 - precipitation input and output.npz
Modify the ai_models_graphcast/ai_models_graphcast/model.py
line 57-58: the download_files refer to the low resolution asset
line 68: change to grid = [1.0, 1,0]
Modify the ai_models_graphcast/ai_models_graphcast/input.py
line 60: change the code to .reshape(len(forcing_variables), len(dates), 181, 360)

Those changes reduce the resolution to 1.0 and thus requires much less memory

Can you provide the specific operation details? For example, how to download the data?

In model.py you can change the code lines 57 58 to: class GraphcastModel(Model): download_url = "https://storage.googleapis.com/dm_graphcast/{file}" expver = "dmgc"
# Download
download_files = [
    (
        # "params/GraphCast_operational - ERA5-HRES 1979-2021 - resolution 0.25 -"
        # " pressure levels 13 - mesh 2to6 - precipitation output only.npz"
     "params/GraphCast_small - ERA5 1979-2015 - resolution 1.0 -"
        " pressure levels 13 - mesh 2to5 - precipitation input and output.npz"
    ),
    "stats/diffs_stddev_by_level.nc",
    "stats/mean_by_level.nc",
    "stats/stddev_by_level.nc",
]

...
you can manually download the file from the download_url above.

Once you make those change how do you actually run ai-models?

So I've tried  ai-models --assets ./graphcast_assets graphcast where  ./graphcast_assets holds params/GraphCast_small - ERA5 1979-2015 - resolution 1.0 - pressure levels 13 - mesh 2to5 - precipitation input and output.npz but im running into the following: ecmwfapi.api.APIException: "ecmwf.API error 1: User ' has no access to services/mars"
Do you need access to mars?

Getting a segmentation fault 2024-08-14 21:24:09,142 INFO Loading params/GraphCast_small - ERA5 1979-2015 - resolution 1.0 - pressure levels 13 - mesh 2to5 - precipitation input and output.npz: 0.4 second.
2024-08-14 21:24:09,142 INFO Building model: 0.4 second.
2024-08-14 21:24:09,582 INFO Creating forcing variables: 0.4 second.
2024-08-14 21:24:10,085 INFO Converting GRIB to xarray: 0.5 second.
2024-08-14 21:24:10,398 INFO Reindexing: 0.3 second.
2024-08-14 21:24:10,401 INFO Creating training data: 1 second.
2024-08-14 21:24:11,068 INFO Extracting input targets: 0.6 second.
2024-08-14 21:24:11,069 INFO Creating input data (total): 1 second.
Segmentation fault

Yes, you need Mars access.
Best,
Han
On Wed, Aug 14, 2024 at 3:46 PM Sean Wang @.***> wrote:

So I've tried ai-models --assets ./graphcast_assets graphcast where
./graphcast_assets holds params/GraphCast_small - ERA5 1979-2015 -
resolution 1.0 - pressure levels 13 - mesh 2to5 - precipitation input and
output.npz but im running into the following: ecmwfapi.api.APIException:
"ecmwf.API error 1: User ' has no access to services/mars"
Do you need access to mars?
—
Reply to this email directly, view it on GitHub
https://github.com/ecmwf-lab/ai-models/issues/14#issuecomment-2289846891,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHM32CGLCV67XWEMJQXUEVTZRO6Y3AVCNFSM6AAAAAA5Y3CJVKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBZHA2DMOBZGE
.
You are receiving this because you commented.Message ID:
@.***>


Update fraud-front to connect to DynamoDB for its session storage

Proposed changes
What changed
Updated fraud front to use express-session and dynamodb-connect to store the front-end session details within an aws DynamoDB table.

Why did it change
To have persistant storage method for the front-end data that is in line with the rest of the programme and means we can avoid setting up and managing a Redis cluster as the alternative.

Issue tracking

KBV-609

Manually compared against https://github.com/alphagov/di-ipv-cri-address-front/pull/245/files


BTE 504 error in Workflow D.5 query

https://arax.ncats.io/?source=ARS&id=9b3132c8-58bb-4b4d-933d-73c520a4a7de

{
  "callback": "http://localhost:8000/ars/api/messages/d55176b4-64aa-4b04-8c57-365452979001",
  "message": {
    "query_graph": {
      "edges": {
        "e00": {
          "object": "n01",
          "subject": "n00"
        },
        "e01": {
          "object": "n02",
          "subject": "n01"
        },
        "e02": {
          "object": "n03",
          "subject": "n02"
        }
      },
      "nodes": {
        "n00": {
          "categories": [
            "biolink:ChemicalSubstance"
          ],
          "ids": [
            "DRUGBANK:DB00150"
          ]
        },
        "n01": {
          "categories": [
            "biolink:Protein"
          ]
        },
        "n02": {
          "categories": [
            "biolink:MolecularActivity"
          ]
        },
        "n03": {
          "categories": [
            "biolink:ChemicalSubstance"
          ],
          "ids": [
            "KEGG.COMPOUND:C02700"
          ]
        }
      }
    }
  },
  "validation_result": {
    "message": "",
    "size": "2 kB",
    "status": "PASS",
    "version": "1.1.1"
  }
}```

Standard three-hop explain query. BTE planned fix tracked in biothings/BioThings_Explorer_TRAPI#112

This query is now listed as an alternate for the December Demo


treeview.showIfEnabled error on version 0.7.6

I get the following on boot, the error just keeps repeating and site reports internal server error.
Rolling back to 0.7.5 resolves this.
Booting up worker for treeview
An exception was thrown as a result of invoking function showIfEnabled error: Unregistered syscall clientStore.get
Error dispatching event plug:load to treeview.showIfEnabled: Unregistered syscall clientStore.get
Error: Unregistered syscall clientStore.get
    at wa.onMessage (file:///silverbullet.js:117:27964)
    at Worker.worker.onmessage (file:///silverbullet.js:117:27483)
    at Worker.wrappedHandler (ext:deno_web/02_event.js:1401:12)
    at innerInvokeEventListeners (ext:deno_web/02_event.js:754:7)
    at invokeEventListeners (ext:deno_web/02_event.js:801:5)
    at dispatch (ext:deno_web/02_event.js:658:9)
    at Worker.dispatchEvent (ext:deno_web/02_event.js:1043:12)
    at Worker.#pollMessages (ext:runtime/11_workers.js:258:12)
    at eventLoopTick (ext:core/01_core.js:169:7)
Booting up worker for index
Error: Unregistered syscall clientStore.get
    at wa.onMessage (file:///silverbullet.js:117:27964)
    at Worker.worker.onmessage (file:///silverbullet.js:117:27483)
    at Worker.wrappedHandler (ext:deno_web/02_event.js:1401:12)
    at innerInvokeEventListeners (ext:deno_web/02_event.js:754:7)
    at invokeEventListeners (ext:deno_web/02_event.js:801:5)
    at dispatch (ext:deno_web/02_event.js:658:9)
    at Worker.dispatchEvent (ext:deno_web/02_event.js:1043:12)
    at Worker.#pollMessages (ext:runtime/11_workers.js:258:12)
    at eventLoopTick (ext:core/01_core.js:169:7)
Error: Unregistered syscall clientStore.get
    at wa.onMessage (file:///silverbullet.js:117:27964)
    at Worker.worker.onmessage (file:///silverbullet.js:117:27483)

Similar to #819, although the behaviour on my instance is very different (won't load after boot).

I'll assume this will be fixed after upgrading. If you cannot upgrade because SB doesn't boot:

Remove all files from your _plug folder in your space
Start SilverBullet again
Reinstall the treeview plug using the Plugs: Update command

Can confirm the above worked, thank you. On latest now.


chef_client_scheduled_task action :remove not working, wrong command

Cookbook version
8.1.2
Chef-client version
12.19.36
Platform Details
Windows (all)
Scenario:
Trying to remove chef_client_scheduled_task by doing the :remove action
Steps to Reproduce:
chef_client_scheduled_task 'Chef Client' do
    action :remove
end

Expected Result:
chef-client Windows task should be removed.
Actual Result:
Instead removing the task I got error:
FATAL: Chef::Exceptions::ValidationFailed: chef_client_scheduled_task[Chef Client] (development::default line 16) had an error: Chef::Exceptions::ValidationFailed: Option action must be equal to one of: nothing, create, run, change, delete, end, enable, disable!  You passed :remove.
Solution:
Change the action :remove in the file .\resources\scheduled_task.rb to action :delete
action :remove do
  windows_task 'chef-client' do
    action :delete
  end
end

This should be fixed in 8.1.4. Thanks for the report


not really destroying

tl;dr - this turned out to be me
I don't fully have a repro case for this yet, but wanted to start an issue so we can track it.
Yesterday I was working on JavaPengines. The test server I'm using just starts an http server and loads pengines. I'm not changing the timeout.
I start my Java client back up and sent a /create message and get this back
{"answer":{"data":[{"X":{"args":["taco"],"functor":"a"},"Y":3}],"event":"success","id":"03d18c99-6da2-4434-8eac-4144069f720a","more":true,"projection":["X","Y"],"time":0.000020260000000000243},"event":"create","id":"03d18c99-6da2-4434-8eac-4144069f720a","slave_limit":3}
Strange part is, that query is from yesterday!
 Notably, the pengine in question was destroyed.
So why is the server hanging onto that old query???
),answer(success('03d18c99-6da2-4434-8eac-4144069f720a',[json{'X':a(taco),'Y':3}],2.0260000000000243e-5,true))])
[Thread 13] Sending create('03d18c99-6da2-4434-8eac-4144069f720a',[slave_limit(3),answer(success('03d18c99-6da2-4434-8eac-4144069f720a',[json{'X':a(taco),'Y':3}],2.0260000000000243e-5,true))]), timout: 300
[Thread 9] Got create('03d18c99-6da2-4434-8eac-4144069f720a',[slave_limit(3),answer(success('03d18c99-6da2-4434-8eac-4144069f720a',[json{'X':a(taco),'Y':3}],2.0260000000000243e-5,true))]) from <message_queue>(0x7f634002d920)
[Thread 9] DESTROY? create('03d18c99-6da2-4434-8eac-4144069f720a',[slave_limit(3),answer(success('03d18c99-6da2-4434-8eac-4144069f720a',[json{'X':a(taco),'Y':3}],2.0260000000000243e-5,true))])
[Thread 9] Running hook: header
[Thread 9] Field: [cache_control('no-cache, no-store, must-revalidate'),pragma('no-cache'),expires('0'),content_type('application/json; charset=UTF-8')]
[Thread 9] Transfer-encoding: none
[Thread 9] Running hook: send_header
[Thread 9] Header: [connection('keep-alive'),cache_control('no-cache, no-store, must-revalidate'),pragma('no-cache'),expires('0'),content_type('application/json; charset=UTF-8')]
[Thread 9] Running hook: close
[Thread 9] [35] 200 OK (0.000 seconds; 326 bytes)
[Thread 9] Wiping posted data
[Thread httpd@9900_4] Got job requeue((0x7f6368026e80),(0x7f6368027a70),user:http_dispatch,[protocol(http),peer(ip(127,0,0,1)),pool(client('httpd@9900',user:http_dispatch,(0x7f6368026e80),(0x7f6368027a70)))])
[Thread httpd@9900_4] Waiting for keep-alive ...
[Thread httpd@9900_4] 	Timeout on keep-alive connection
[Thread httpd@9900_4] Closing connection from ip(127,0,0,1)
[Thread httpd@9900_4] Waiting for a job ...
[Thread 11] '9be79c5f-6050-4081-8477-516588fd186e': 6 = destroy => 1
[Thread 11] Reply to <message_queue>(0x7f634002abd0): destroy('9be79c5f-6050-4081-8477-516588fd186e')
[Thread 11] Sending destroy('9be79c5f-6050-4081-8477-516588fd186e'), timout: 300
[Thread 12] 'c9c19ec8-d5b9-44e2-8bcb-8208b23eada8': 6 = destroy => 1
[Thread 12] Reply to <message_queue>(0x7f634002c490): destroy('c9c19ec8-d5b9-44e2-8bcb-8208b23eada8')
[Thread 12] Sending destroy('c9c19ec8-d5b9-44e2-8bcb-8208b23eada8'), timout: 300

Sorry - this was caused by my own fumble fingeredness, as in I forced some behavior by adding debug code and didn't remove it ... oops.
My bad

I think JavaPengines is a great idea!


Use macros to generate specialized type FFI functions

Currenty in our FFI we have a number of functions that essentially to do the same thing but accepting different types. This is to make the API profile easier for consuming clients. However this makes for a lot of repetition of code and causes a lot of work to occur if Mentat's API changes in these areas.
This issue tracks creating Macros for generating type specific FFI functions to avoid duplication of work. These macros should cover the following FFI functions.

value_at_index_into_<type>
typed_value_into_<type>
query_builder_bind_<type>
entity_builder_retract_<type>
entity_builder_add_<type>
in_progress_builder_retract_<type>
in_progress_builder_add_<type>

Is this repetition that much of an issue today? My experience with macros is that their use tends towards eventually obfuscating what's going on, or at least making things a bit harder to read in the long run, especially when there's multiple moving parts/parameters. I have a feeling that regular statically typed code is in most cases preferable (but have no data to back this up).
I share that they make a lot of sense when there's extreme or simple repetition, but I didn't get that feeling when poking into the ffi code.


Fix next_key in table rows response

https://github.com/telosnetwork/antelope-rs/blob/ccd4422e8ce77a57a1f2945785d809348443d327/crates/antelope/src/api/v1/chain.rs#L266-L289
When the lower_bound is not specified in the request params, it's None, which means it defaults to the u64 of the primary key.  So we should default to treating/parsing next_key as u64.

As a workaround, we've just set the lower_bound to be TableIndexType::UINT64(0) but this should be fixed properly.


static source port

Hello,
I would  like to simulate reflection/amplification attacks but in order to do so, I would need to set a static source port. This would allow bonesi to simulate the reflection traffic. Would this be possible to add to the code?
thanks
Eric

Sure, just change line 266 of bonesi.c to a fixed number.
In general, it would be a good idea to use the min and max src port as a command line parameter also for running in parallel more efficiently. Don't have time to do so now but marking as future enhancement.


Why arabic pre-trained model is not working? where as it works well when used in transformer pipeline?

I am trying to recreate your project. I am having issues with your code when I am using certain pre-trained models as you have indicated. I downloaded them and placed them in data directory. I want to test them before I start retraining them. However, the output that comes is check if you have downloaded the model.
the output in postman is
{
"output": "Make sure you have downloaded model for ar-en translation"
}
I think the same environment works well when I use Marian directly in a separate script using pipeline function and translates well however the model is named cryptically in .cache directory.

Can you share the structure of the directory/translation models?

Closing due to inactivity, see this similar issue if you have trouble: https://github.com/kylegallatin/machine-translation-service/issues/10


Html to pdf.

How to use this library to build extinction, html to pdf.
User push in plugin button, program build pdf file and send to my server. Can this library do it?

As explained in the README, PDF.js is a PDF viewer and not a PDF creator.
Hence what you're asking isn't possible, and you may have confused this project with e.g. https://github.com/MrRio/jsPDF/.
Closing as answered.


Fix initial findings 1 2 3

Update: Readme.md spell check done
Update: Scripts now using printf blocks instead of echo
Update: Script output now identical format for tests
Update: Readme.md Bugs added / workarounds added
Update: Reorganization to provide better overview
Rename: .travis.yml now .travis.yml.j2 to be attended from galaxy
Update: update-meta-files.sh now updating more
Update: colors added
Update: LICENSE date set to 2018
Update: update-meta-files.sh now including LICENSE

Resolves: #1 #2 #3
@donvipre  Please review. After this the skeleton should be ok from my side.

changes applied :)
@donvipre maybe you can have another look.

@donvipre  fixed

@daniel-wtd : now you got on "s" missing ;)

found something else, gimme a sec

@donvipre now with more "s" in the box
@Cinux90 #6 should be addressed, too


Failed to build delete range aggregator Magic number mismatch: expected

2022-08-24T01:04:14.584219Z  WARN node{id=13 name="compactor-2"}:task{id=6083}: risingwave_storage::hummock::compactor: Failed to build delete range aggregator Magic number mismatch: expected<PHONE_NUMBER>, found: 23981349.

See https://gist.github.com/kwannoel/7f34fff04ee8f3eecdd8d10ab25bdc7f?permalink_comment_id=4608036#gistcomment-4608036 for more info.

I can successfully reproduced this issue locally and verified several things:

The encoding size and meta offset are both correct in the writer and reader side.
The content seen by the writer and the reader are indeed different.

I suspect there is something wrong in madsim s3 simulator so I turned on debug log in madsim and saw these debug logs in madsim relevant to the problematic object (id=93):
2022-08-24T01:04:09.509693Z DEBUG node{id=3 name="s3"}:task{id=4273}: madsim_aws_sdk_s3::sim::server::service: upload_part bucket="hummock001" key="hummock_001/32/93.data" upload_id="121596723" part_number=2
2022-08-24T01:04:09.511002Z DEBUG node{id=3 name="s3"}:task{id=4274}: madsim_aws_sdk_s3::sim::server::service: upload_part bucket="hummock001" key="hummock_001/32/93.data" upload_id="121596723" part_number=1
2022-08-24T01:04:09.528232Z DEBUG node{id=3 name="s3"}:task{id=4280}: madsim_aws_sdk_s3::sim::server::service: complete_multipart_upload bucket="hummock001" key="hummock_001/32/93.data" upload_id="121596723"

We can see that UploadPart for part_number=2 finishes before UploadPart for part_number=1, which is a valid behavior for S3 multipart upload since parts can be uploaded concurrently in different connections. However, after checking the madsim s3 simulator codes, the simulator seems to miss the ordering and put part2 before part1 in the object body.
Root cause: although the simulator sorts the parts by number at first and populates a selection idx, it also sorts the selection idx afterwards. This causes the selection idx ordering to be identical with the parts original ordering and thus miss to order the parts by part number.

cc @wangrunji0408

Potential fix: https://github.com/madsim-rs/madsim/pull/149

fuzzing-37.log
Similar error encountered today, but from monitored_store iter instead:
risingwave_storage::monitor::monitored_store: Failed in iter: Hummock error: Magic number mismatch: expected<PHONE_NUMBER>, found: 564084722.
  backtrace of inner error:

https://buildkite.com/risingwavelabs/generate-sqlsmith-snapshots-weekly/builds/108#0188f428-2377-4dcf-a342-456063a88aa3

fuzzing-37.log
Similar error encountered today, but from monitored_store iter instead:
risingwave_storage::monitor::monitored_store: Failed in iter: Hummock error: Magic number mismatch: expected<PHONE_NUMBER>, found: 564084722.
  backtrace of inner error:

https://buildkite.com/risingwavelabs/generate-sqlsmith-snapshots-weekly/builds/108#0188f428-2377-4dcf-a342-456063a88aa3

@kwannoel Is the issue resolved after the madsim fix?

Yup, tested this issue + fuzzing-37 locally, in https://github.com/risingwavelabs/risingwave/pull/10700.
Passes both. Thanks for your help!


Support wavefront formatted status output

By following Michael’s python script, I’ve modified foundationdb source file (mainly fdbcli.actor.cpp) so that every time we run command line
fdbcli —exec "status wavefront"

we can parse the returned json status object and turn it into wavefront supported metrics format. Also with several improvements based on the discussion we have.
(1) remove excluded from tags, and add metric like cluster.processes.4604.excluded = [0|1]. 0 represents false and 1 represents true.
(2)  add rule_id as a point tag and add metric name like cluster.processes.4604.rule.master = [0|1] (the name indicates that fdb process on port 4604 has the rule as master)
And please see attachments for command line output.
output.txt

@panghy
@jbau
@warshawd


Form shouldn't clear when you go from tx build to exploration and back

The current fix is to clear it.
What is happening under the covers is that the input fields get reset by React but the (backend) store is still there.
The solution is to make the input fields (pickers) controlled components based on the store.

Done with the redux picker upgrades


RFC: Signed Fixed-width Integer Support

Proposal
Lean currently supports unsigned fixed-width integers (UInt8, UInt16, UInt23, UInt64).
This proposal is regarding similar support for signed fixed-width integers that provide similar efficiency benefits to the UIntN types (e.g., native C representations).
Design Points for discussion

Semantics for overflow and avoiding UB.

Division by zero returning zero seems an obvious choice here like for existing UIntN types.
Signed arithmetic overflow in C/C++ is undefined behavior. To avoid this, an implementation could
i. simply behave more like Fin and BitVec in Lean4 today and provide predictable wraparound behavior on overflow (there is a slight bias for these semantics in this RFC at the moment), or
ii. choose to detect and panic on such occurrences (perhaps similar to Rust in debug mode?)

C representation

Using the expected intn_t types could make for the least surprising set of C definitions/FFI APIs
A case for reusing uintn_t types could also be made as they are already used by the compiler for some types

Where would IntN definitions ideally live in the short and long term?

a GitHub repo dedicated solely to IntN types or similar?
https://github.com/leanprover/std4?
https://github.com/leanprover/lean4?

Approaches
Approach 1: UIntN Subtypes
A UIntN is used to represent each IntN type, e.g. here is Int8:
structure Int8 where
  ofUInt8 ::
  toUInt8 : UInt8
  deriving DecidableEq, Inhabited, Hashable

Efficient Representation
This approach already provides an efficient representation in compiled code with no compiler extensions by being defined directly in terms of UInt8. I.e., all compiled definitions operating on these Int8s will use uint8_t directly. Note that this does mean that FFI code will need to manually track which generated C definitions are using uint8_t for Int8 and which are using uint8_t for some other type (e.g., UInt8, Bool, etc).
Operators
Arithmetic operations which are "sign-insensitive" can be defined directly in terms of their corresponding UInt8 operation, e.g. Int8.add:
def Int8.add (a b : Int8) : Int8 := ⟨UInt8.add a.toUInt8 b.toUInt8⟩

The remaining few sign-sensitive operations can be given the appropriate implementation in Lean (and backed with a more efficient C representation where needed), e.g., Int8.div:
@[extern "lean_int8_div"]
def Int8.div (a b : Int8) : Int8 :=
  match a.isNeg, b.isNeg with
  | true,  true => ⟨(-a).toUInt8 / (-b).toUInt8⟩
  | true,  false => -⟨(-a).toUInt8 / b.toUInt8⟩
  | false, true => -⟨a.toUInt8 / (-b).toUInt8⟩
  | false, false => ⟨a.toUInt8 / b.toUInt8⟩
instance : Div Int8 := ⟨Int8.div⟩

and the corresponding C definition lean_int8_div:
extern "C" LEAN_EXPORT uint8_t lean_int8_div(uint8_t a, uint8_t b) {
  if ((a == 0) || ((a == INT8_MIN) && (b == ((uint8_t)((int8_t)-1))))) {
    return a;
  } else {
    return ((int8_t) a) / ((int8_t) b);
  }
}

Here's an initial draft for Int8 for feedback here if we want to use this approach:

WrapUIntN.Int8 Lean definitions
WrapUIntN.Int8 C definitions (WIP)

Potential pros

requires no compiler changes required for efficient representation
requires minimal new Lean code since many Int8 ops simply use the same UInt8 op
requires minimal new C code since UInt8 already provides many appropriate C definitions that are re-used

Potential cons

without some compiler additions, generated C functions operating on IntN types will all be in terms of uintn_t types and not intn_t types. The casts between a uintn_t  and intn_t type are free and semantically correct. The book keeping to track which uintn_t means what in the FFI/C layer, however, could be painful and a common Lean4/FFI footgun.

Approach 2: Fin Subtype
A Fin is used to represent each IntN type, e.g. here is Int8:
/-- The size of type `Int8`, that is, `2^8 = 256`. -/
abbrev Int8.size : Nat := 256

/--
The type of signed 8-bit two's compliment integers.
-/
structure Int8 where
  val : Fin Int8.size

Efficient Representation
This approach would require a compiler extension like what exists for UIntN types to use the expected intn_t C type. This would obviously mean  more up front work, but the resulting compiled C/FFI code would be in terms of the expected signed integer type, which seems likely to be the least surprising choice for a user to encounter.
Operators
Arithmetic operations which are "sign-insensitive" can be defined directly in terms of their corresponding Fin operation in Lean and given the expected efficient representations in C, e.g. Int8.add:
def Int8.add (a b : Int8) : Int8 := ⟨a.val + b.val⟩

The remaining few sign-sensitive operations can be given a more complex definition when appropriate in Lean (along with a more efficient C implementation where needed), e.g., Int8.div:
@[extern "lean_int8_div"]
def Int8.div (a b : Int8) : Int8 :=
  match a.isNeg, b.isNeg with
  | true,  true => ⟨(-a).val / (-b).val⟩
  | true,  false => -⟨(-a).val / b.val⟩
  | false, true => -⟨a.val / (-b).val⟩
  | false, false => ⟨a.val / b.val⟩
instance : Div Int8 := ⟨Int8.div⟩

and the corresponding C definition lean_int8_div:
extern "C" LEAN_EXPORT int8_t lean_int8_div(int8_t a, int8_t b) {
  if ((a == 0) || ((a == INT8_MIN) && (b == -1))) {
    return a;
  } else {
    return a / b;
  }
}

Here's an initial draft for Int8 using Fin for feedback here if we want to use this approach:

WrapFin.Int8 Lean definitions
WrapUIntN.Int8 C definitions (WIP)

Potential pros

has the least surprising C definitions/FFI
defining signed fixed width integers in terms of Fin may be "more direct" or "more natural" than defining them in terms of unsigned fixed-width integers (debatable of course!)

Potential cons

requires some compiler additions to get the expected efficient C representation, likely similar in scope to what exists for UIntN

Approach 3: Int Subtypes
Each IntN type could also be defined as a subtype of Int similar to the following:
/--
`IntN n` is an integer `i` with the constraint that `-2^(n-1) ≤ i < 2^(n-1) - 1`.
-/
structure IntN (n : Nat)where
  /-- If `i : IntN n`, then `i.val : ℤ` is the described number. It can also be
  written as `i.1` or just `i` when the target type is known. -/
  val  : Int
  /-- If `i : IntN n`, then `i.2` is a proof that `-2^(n-1) ≥ i.1`. -/
  isGe : LE.le (2^(n-1)) val
  /-- If `i : Fin2C n`, then `i.3` is a proof that `i.1 < (n-1)`. -/
  isLt : LT.lt val (2^(n-1)-1)

Zulip discussion steered me away from this and towards the aforementioned UIntN approach, so less experimenting has been done with this approach.
Potential pros

generated C interfaces are in terms of the signed integer types, which could be more intuitive/less error prone when writing FFI-involved code.

Potential cons

Requires more Lean and C definitions per type than UIntN approach.
Requires compiler extension(s) to achieve a native representation.

Community Feedback
The lack of signed fixed-width integer types has come up in discussion with Lean FRO members and has been discussed some in the lean4 dev community. It was also noted as a pain point in the community when working with external code during Lean Together 2024.
Thank you to those who have helped in discussing this topic so far.
Impact
Add :+1: to issues you consider important. If others benefit from the changes in this proposal being added, please ask them to add :+1: to it.
Related Work
The following also involve supporting signed fixed-width integer types:

Std.Data.BitVec: supports similar functionality but for arbitrary bitwidths and wraps a Nat
SciLean.Data.Int64: Int64 impl using the same proposed design of wrapping an unsigned int (USize instead of than UIntN). The proposed Int8 impl is similar with a few slight differences (e.g., fewer bitwise operations, leverages a few C definitions for efficiency, etc)
Galois.Data.Bitvec: somewhat outdated but still a helpful reference, similar to Std.Data.BitVec

Are there considerations to support overflow detection?

Are there considerations to support overflow detection?

I'm not aware of specific considerations here as of yet. The tinkering so far has focused on efficiency, avoiding signed integer-related UB we may inherit from C, and just general correctness.
Does anyone know: is there any prior art in this space for the UIntN types? (If no, would such considerations equally apply to both? And, do we want such considerations to be a part of an initial addition/PR, or a part of a follow-up feature/enhancement?)

The usual semantics for unsigned types are to wrap around, without overflow. (However, add with carry/subtract with borrow/extended multiplication are often supported.) The overflow semantics for signed types are quite different; they are well supported in LLVM and as builtins in major C compilers. So support is available but might require compiler support.

The usual semantics for unsigned types are to wrap around, without overflow. (However, add with carry/subtract with borrow/extended multiplication are often supported.) The overflow semantics for signed types are quite different; they are well supported in LLVM and as builtins in major C compilers. So support is available but might require compiler support.

The proposed Int8 implementation linked in the RFC uses the corresponding unsigned operations where possible. This gives them predictable wrap around semantics and avoids any UB from signed integer overflow, e.g. (127 + 1) : Int8 = -128.
For the sign-sensitive operations, it specifically checks for and avoids the problematic cases which would trigger signed integer overflow (e.g., div and mod explicitly check the few cases where that can happen, so you get predictable albeit perhaps initially odd results like(-128 / -1) : Int8 = -128 instead of UB).
Perhaps I misunderstood what you meant by support overflow detection. The proposed design explicitly avoids signed integer overflow in the C sense and values will wrap around, but it doesn't attempt to dynamically detect specific cases at runtime in order to warn/error for the user.

The runtime detection (with catch support?) is what I'm asking about.
Just to be clear: I'm just asking whether this is currently considered, I don't think this is important for basic signed integer support.

This seems closely related to the Std.BitVec type, which implements a signless approach to bitvectors.
For example, it has Std.BitVec.udiv for unsigned division and Std.BitVec.sdiv for signed division.

Things like overflow detection and such might make more sense to implement on BitVec, and are in fact on my roadmap for them already. The proposed IntN types seem more suited for when you want to squeeze out every last bit of performance, and thus probably don't want to pay the cost of overflow detection.

This seems closely related to the Std.BitVec type

Yes, that code was helpful to reference =) I just added a Related Work section to the RFC I should have included initially  acknowledging the related implementations I'm aware of.

Things like overflow detection and such might make more sense to implement on BitVec ... the proposed IntN types seem more suited for when you want to squeeze out every last bit of performance

I am increasingly curious if Lean could reasonably help a user avoid/detect overflow while using these sorts of "primitive" UIntN and IntN types... but I think that may be a topic to dive into separately (at least based on the fact that the UIntN types today also don't feature this). Happy to hear disagreement here though =)

C/C++ make signed integer overflow undefined but efficient tools are generally available.
LLVM support: https://llvm.org/docs/LangRef.html#llvm-sadd-with-overflow-intrinsics
Clang support: https://clang.llvm.org/docs/LanguageExtensions.html#checked-arithmetic-builtins
GCC support: https://gcc.gnu.org/onlinedocs/gcc/Integer-Overflow-Builtins.html
A generic operation would have to trap overflow by checking the inputs.

This is now implemented through #5790, #5885 and #5961. Additionally UIntX was refactored to use BitVec instead in #5323 so IntX can now make use of the BitVec primitives that we already put in a lot of work over the course of this year.
To the best of our knowledge we have addressed all UB concerns by both mimicking semantics of BitVec in C and enabling the -fwrapv option in the compiler such that overflow etc. is now defined behavior.
Additional things such as proper proof APIs for both UIntX and IntX as well as things like catching overflows are future work items that are not part of this RFC afaict so I'm closing this as completed.

Awesome!! This is very useful, thank you :smiley:


Move to Poetry

In this PR

Add configuration for Poetry.
Changed links in Readme file to use Github Url instead of a relative path. ( Required for Publishing Package )

What is Poetry?
Poetry is a dependency manager as well as a publisher to Pypi. Refer their website for more info https://python-poetry.org/.
But even here, pycairo creates a problem. Well, we should need to think about it.
See https://github.com/python-poetry/poetry#installation for installation in your PC.
Also, according to https://www.python.org/dev/peps/pep-0518/ adding a pyproject.toml is necessary.
You could read https://github.com/python-poetry/poetry#what-about-pipenv.

I have an idea for fixing pycairo installation. And making it ease for the users to install using pip.
The problem of installing it comes becausepip install pycairo doesn't work because they don't compile binary for Windows. The users need C compilers to install Cairo and then install pycairo. What we can do is to create a new repo under ManimCommunity just to maintain those binarys. We can use Travis to set it up. By building it in Travis and then adding it to the repo so we don't need each time to update the link from iifc. Compile a python package for non python depency from python wiki. Doing this will make installation more easy.

I'm not sure "Move to Poetry" is the right wording for this, more like "Add support for Poetry", right? Because we're of course not going to give up on pip, we'd just allow people to use poetry if they want.

I'm not sure that we should be investing anything more into Cairo. Immediately after we finish the add-on API, we're going to be removing all dependencies on Cairo in favor of OpenGL rendering.

Immediately after we finish the add-on API, we're going to be removing all dependencies on Cairo in favor of OpenGL rendering.

More like in favor of none in specific (at least in the main repo), because the idea is that we become able to use different renderers, not just one.

Immediately after we finish the add-on API, we're going to be removing all dependencies on Cairo in favor of OpenGL rendering.

More like in favor of none in specific (at least in the main repo), because the idea is that we become able to use different renderers, not just one.

Well, yeah, but we have to include at least one renderer with Manim that's pretty much guaranteed to work. I'm not saying that it should be hard-coded, but at least come with Manim CE as a preinstalled add-on.

If we don't include one with the initial download, we are going to have many confused users.

Such an add-on would have to be on a separate repo either way though, and it can be added as a dep or as a submodule or something

Yeah, that's a good idea. I'll open a new issue about how rendering should be changed.
My point is that we don't need to touch Cairo installation because it's going to be completely replaced shortly.

I'm not sure "Move to Poetry" is the right wording for this, more like "Add support for Poetry", right? Because we're of course not going to give up on pip, we'd just allow people to use poetry if they want.

The first thing to say is Poetry and pip are actually same. What it actually is that poetry is a way to publish the package to pip! Again, Poetry is for the ease for developers and it makes no change for the users. The users will as usual use
pip install manim

for installation. So, Move to Poetry is a valid statement and we are not gonna think about publishing packages after this PR. I currently named the package as manimce until we the name manim. Also, if you could please find me necessary classifiers other than the one added already which suit this package. See https://pypi.org/classifiers/.
Also, you could see https://pycairo.readthedocs.io/en/latest/integration.html?highlight=GL #moderngl-imagesurface-as-texture which maybe useful.

Why is Travis failing for this? Idk

Travis is failing for everything at the moment. In any case, after looking over it a while I think using poetry will probably save us a lot of effort in the long run.

Also, I am creating a GitHub Action to publish packages using poetry!

@naveen521kk What's your progress on this? Tests have been fixed, so merge master into your branch and let them run.

Just leaving record here: let's work on #32 and #61 once this is closed.

Workflow for Github Action is done. Also, btw Pycairo is still a Problem! I just made https://github.com/pygobject/pycairo/pull/191 but there is no response from them. Please do say me what can be done!

@naveen521kk what is the status of this?

Waiting for https://github.com/pypa/pypi-support/issues/450 and https://github.com/pygobject/pycairo/pull/191 to be merged :(. All other things are done.

I'm marking this as draft until the above mentioned issues are merged.

Look like I need to document about using poetry somewhere. @leotrs Can you help me where to do? Also, it will change some parts of readme. This doesn't need to wait for those PR. This can directly be merged (Seems unnecessary to me). But yes, wait for me to write documentation which will change how people install manim( for Dev's).

Look like I need to document about using poetry somewhere. @leotrs Can you help me where to do?

Not sure I understood the question. If you are asking me to help you write the documentation, unfortunately I cannot because I've never used poetry. If you are asking me where to place a documentation file that you can write yourself, then the answer depends is... probably somewhere in the manim/docs/installation/ directory. Maybe a new file for_devs.rst?

Also, it will change some parts of readme.

That's fine. Most of the readme will be moved to the documentation anyway once it's up on RTD.

This doesn't need to wait for those PR. This can directly be merged (Seems unnecessary to me).

I'd suggest we only change the instructions after (or at the same time as) the workflow actually changes. No need to have instructions in the readme that don't match what actually needs to be done.

If you are asking me where to place a documentation file that you can write yourself

That's what I meant. And I am sure you will loving poetry ❤ once you start using it.

I'd suggest we only change the instructions after (or at the same time as) the workflow actually changes

Actually merging those PR it is not necessary for this to change. It is just that, Windows users must have to do some additional steps because of Pycairo and nothing else. Also, it is quite confirmed that we would be having the manim name on PyPi so it doesn't need to worry about it.

In that case I see no harm in adding a new documentation file to this (or other) PR

What are the folders we should have in distribution? I just included the folder manim here is that ok? Or anything else to be added.
@eulertour @leotrs

I believe example_scenes should perhaps be included. But I'm neutral about that. All the others seem like they can be exlucded.

@naveen521kk thank you very much for writing the documentation.
Suggestion: the file for_dev.rst is currently sitting in the docs folder but is not linked to from anywhere. Could you please add a link to it in the installation.rst page?
Question: the file mentions the fact that poetry creates a virtual environment by default. Is this always the case? Can I not use poetry to install manim AND use a standard virtual environment?

Can I not use poetry to install manim AND use a standard virtual environment

Yes you can do that easily. See configuration details https://python-poetry.org/docs/configuration/#local-configuration

Pinging @PgBiel for checking docs. :)

I will look into the changes tommorrow.

Still anything to do?

@naveen521kk thank you very much for putting this together.
My final question before merging is this: will devs have to install poetry as soon as we merge this? will they have to install/clone/update something? How to transition from pip to poetry?
If there is some sort of sudden change, I would want to make sure every dev knows about this change before we break their workflow.

I recommend that all the devs install poetry. But can continue even with their old workflow theoretically. Because pip also understand pyproject.toml and yes it will work, except that they can't do an editable install( No problem if previously done).
Nevertheless, soon all the devs should be having poetry installed so that it simplifies the development process. We should kinda announce in discord and also have a mention in readme and contributing guidelines ( I will do soon).

Ok so the steps are these:

we merge this and nothing breaks, everyone can still work. (Please confirm this is the case, we have to be very sure about this one)
we announce that the only change moving forward is that devs can no longer do an editable install, but nothing should break if they already have one (every dev surely has one)
announce that devs move to poetry quickly (we need to point them to the instructions for how to do this)
now we're all on poetry and everything is nice and good

Please confirm this is the case, we have to be very sure about this one

I have tried it. Maybe you can also. Delete setup.py and the pyproject.toml and try installing and it should work. :)

If that's the case then all we need is one more approval and to make the announcement.
It would also be helpful if somebody else could pull this branch and try the move from pip to poetry and make sure that it's smooth.

Ok so I will add some items in ReadMe contributing guidelines and also resolve conflicts by today.

Today we should merge this as I would be able to work on this later.

I think this PR is ready and can be merged.

I just tried following the instructions for installing manim via poetry, everything ran smoothly.
Apart from the one unresolved discussion (https://github.com/ManimCommunity/manim/pull/165/files#r481102250), I also think this can be merged.

One thing we should remember is to fix the pycairo thing for windows, as of this comment by maintainer they would do the whl in the next release so till then the pyproject.toml should be like this and then it can be changed.
@behackl I will change that thing.

This needs a review and then can be merged.

Yes, can this be merged? @leotrs ?

Ok this is ready for merge (sorry for the delay!)
I'm merging this in exactly 24h from the moment this post is made. @ManimCommunity/core @ManimCommunity/contributor please make sure to read the instructions (docs/source/installation/for_dev.rst ) because the next time you pull from master after this is merged, things are going to break.
If anyone has any objections, please leave a message here saying so before the 24h period is up.
NOTE: I think 24h is enough because this PR has been open for a while. Even if someone is not able to leave a message in this 24h period (perhaps they don't have wifi access for the time being), it is fine to come back here later and raise any objections. In any case, if you do have any objections, make sure to not pull from master until your questions/objections are resolved.

Actually, @behackl and I just realized that moving to poetry will kill our RTD builds, so this PR needs to include that as well. Sorry to keep delaying this y'all.
@naveen521kk could you please look into this? https://github.com/readthedocs/readthedocs.org/issues/4912

As already mentioned pip handles it now. I need to just edit the docs requirements files to use zip files(https://github.com/ManimCommunity/manim/archive/master.zip) instead of git because git dependency installs in editable mode(not possible without setup.py) while zip files are just installed and works. Which I will do soon.
I suspect this somehow broke for python 3.6 because Idk there is some issue with pip see this log from Google Collab. Using poetry is ok here. But normal pip install doesn't work out. I will try the same of windows soon.

@leotrs RTD build it successfully https://manim-naveen.readthedocs.io/en/latest/


[proposal] Bundle literals implementation

Potential implementation for Bundle literals, that doesn't use macro annotations and has passable syntax.
Example syntax:
(new MyBundle).Lit(_.a -> 42.U, _.b -> true.B)

For nested Bundles:
// specify an inner bundle with another bundle lit
(new MyOuterBundle).Lit(
  _.a -> (new MyBundle).Lit(_.a -> 42.U, _.b -> true.B)
)

// specify inner bundle fields directly
(new MyOuterBundle).Lit(
  _.a.a -> 42.U, _.a.b -> true.B,
  _.b.c -> false.B, _.b.d -> 255.U
)

and more weirdly for anonymous inner Bundles (so probably not recommended syntax?):
(new MyOuterBundle).Lit(
  b => b.b -> b.b.Lit(_.c -> false.B, _.d -> 255.U)
)

The Lit constructor takes in a list of pairs (->) from fields to values, each each element being a function from a self-type object (the clone is used internally) to a field. This gives reasonable syntax (basically you have the additional _. prefix over each field over what we could do with macro annotations) while being able to compile-time typecheck field accesses (so if you renamed fields, the typechecker would complain about trying to access an nonexistent field).
Drawbacks compared to a macro annotation approach:

the value type can't be checked at compile time, it accepts any value of type Data. With macro annotations, we can (maybe?) generate a function signature based off the field type. Note that compile-time Chisel type checks are not perfect, the dynamic typeEquivalent is much more accurate.
you need the additional magic _. prefix for each field.
_  in anonymous functions is a somewhat-flakey operator. This seems to work though. Originally, Lit has the argument signature (this.type => Data, Data)* but the _ doesn't work there.
technically you could use the _ operator in the value part (since the argument signature is (this.type => (Data, Data))*), but that will exception out inside the Lit constructor, since the Lit constructor calls the function on the newly cloned object (which obviously can't be referenced before).

But not needing a macro annotation (and requiring annotated Bundles, or the dev time to develop the annotation that we haven't had yet) is a nice property...
If we like this API, these improvements are needed:

more test cases, particularly for error cases
refactoring of the Lit constructor, while it's not terrible right not it's also not great
perhaps thoughts on supporting Vec literals, or also Bundle wire initializers (#1005) - but might be a follow-on PR that might need a deeper refactoring

Related issue: Resolves #805

Type of change: other enhancement

Impact: API addition (no impact on existing code)

Development Phase: proposal
Release Notes

This seems like a pretty reasonable proposal to me. I think getting them out there with a minimum of boilerplate required of the Bundle definer makes the _. a small price to pay. Macro's are nice but make for tough learning of internals. I'd say we should put this on the Friday agenda.

Can this work together/is this interoperable with a future @bundle annotation?

@edwardcwang potentially: this expects a varargs function of Data -> Data, and so it should have a different type signature than what we would generate with a @bundle annotation, even if it's also named Lit (though we'd want to be really careful to see if there could be potential nasty interactions).
That being said, I think if (and that's a reasonably big if) this goes through, the need for a @bundle annotation becomes much less clear. It's a lot of development effort for a small syntactic gain, especially since we already have autoclonetype. The main argument against this PR would be that it's really a bit of a hack, but Chisel has its fair share of syntactic hacks (eg .elsewhen, switch, the Module(...) wrapper).

Resolution from today's meeting was to document partial-initialization and DontCare semantics, and productionize this. The syntax seemed fine, and this is much preferable to the current way of constructing Bundle literals by defining a very low-level Lit function.
There was also discussion on naming (Lit vs lit) and invocation structure (this is very different than existing literal constructors, eg 0.U - though those may be more converters more like mySeq.toList).

This API appears to work in scastie
class BundleLitTest {
  val fieldA = 0
  val fieldB = 1

  def Lit(elems: (this.type => (Int, Int))*): this.type = {
    for (elem <- elems) {
      println(elem(this))
    }
    this
  }
}

val a = new BundleLitTest
a.Lit(_.fieldA -> 10, _.fieldB -> 11)

(0,10)
(1,11)

Tested with 2.13 and dotty
scastie.scala-lang.org

Ok, I've cleaned up the code and add test cases. Please review now.
Some things that might need more discussion:

What should isLit do for Bundle literals? It currently returns false, partially for backwards compatibility because Bundle literals behave differently than other literals. Should there be a different recommended API?
Can we have a typeEquivalent definition that the Bundle literal constructor can use to check field assignment? It works for aggregate types. but not for Bits types since the check compares width too.
Currently, partially initialized Bundle literals have uninitialized fields set to DontCare, and if assigned to another wire, any value on the wire would be overwritten with DontCare. (side note: is there a way to write a test case that a wire is actually DontCare?)
What should the recommended syntax be for initializing sub-bundles? Deep field specification (potentially breaks abstraction)? Sub-bundle specification (keeps abstraction, but syntax for anonymous inner bundle literals is weird)? Should we support both? Or just one?

Future work would include:

Vec literals, which also needs discussion on system architecture and API.

My recommendation is that isLit = elements.forall(_.isLit)--yes, technically an API change, so should consult users.  Same for eventual Vec lits.
The current DontCare scheme sounds like the right way to go.  Assigning a literal to another wire/reg should assign the entire thing--even if some fields are don't-cares.

retest this please

Not sure what's going on with the compile error, it compiles fine for me and I don't see any new style violations. Potentially a transient bug in the build system, or at least that's what the Jenkins log indicates.

We're adding some new machines to Jenkins slave network and some of the jobs impose incorrect constraints on their environment. This should be sorted out in the next day or two.

retest this please

@ducky64 Looks like this finally made it through the testing. Time to get it merged

One additional thing we can do is to require an experimental import, eg import chisel3.experimental.BundleLiteralConstructor._ as we've done with the recent testers2 experimental features. Thoughts?

Changed to:

Require import chisel3.experimental.BundleLiteralConstructor._ to access Bundle .Lit, with the Aggregate class literal constructor changed to private[chisel3] def _makeLit (it relies on protected features like bind so I can't move the entire thing into an implicit class).
Remove Data.selfBind, which was originally used to support DIY Bundle Literals. Any existing DIY Bundle Literal code will break, but that was never really supported anyways, and I really hope no one else is using the selfBind API.


Unescape dollar signs?

Hit an error publishing the microsite from v2.1.0-M10. Maybe this was it?

I think maybe this wasn't necessary after all.  Vault publishes with github4s, and probably so should this?

I think maybe this wasn't necessary after all.  Vault publishes with github4s, and probably so should this?