cos_directory_prefix is ignored in runtimes and the restricted PSS are violated

Hello, since a secure S3 storage system has to implement policies instead of giving all users dangerous admin access  i tried the following runtime
{
  "display_name": "in-cluster",
  "metadata": {
    "tags": [],
    "display_name": "in-cluster",
    "engine": "Argo",
    "auth_type": "KUBERNETES_SERVICE_ACCOUNT_TOKEN",
    "api_endpoint": "http://ml-pipeline.kubeflow.svc.cluster.local:8888",
    "public_api_endpoint": "https://istio-ingressgateway-istio-system.apps.p028.otc.mcs-paas.io/_/pipeline",
    "cos_auth_type": "USER_CREDENTIALS",
    "cos_endpoint": "http://minio-service.kubeflow.svc.cluster.local:9000",
    "cos_bucket": "mlpipeline",
    "cos_directory_prefix": "private-artifacts/julius-vonkohout",
    "cos_username": "julius-vonkohout",
    "cos_password": "....",
    "runtime_type": "KUBEFLOW_PIPELINES"
  },
  "schema_name": "kfp"
}

cos_directory_prefix is being completely ignored. Without admin access it even gets access denied at the bucket level /mlpipeline . This admin level stuff sounds very dangerous to me. Why is it trying to access anything at the root level?
Traceback (most recent call last):
  File "/home/lib64/python3.9/site-packages/elyra/pipeline/processor.py", line 481, in _verify_cos_connectivity
    CosClient(runtime_configuration)
  File "/home/lib64/python3.9/site-packages/elyra/util/cos.py", line 117, in __init__
    raise ex from ex
  File "/home/lib64/python3.9/site-packages/elyra/util/cos.py", line 101, in __init__
    if not self.client.bucket_exists(self.bucket):
  File "/home/lib64/python3.9/site-packages/minio/api.py", line 658, in bucket_exists
    self._execute("HEAD", bucket_name)
  File "/home/lib64/python3.9/site-packages/minio/api.py", line 403, in _execute
    return self._url_open(
  File "/home/lib64/python3.9/site-packages/minio/api.py", line 386, in _url_open
    raise response_error
minio.error.S3Error: S3 operation failed; code: AccessDenied, message: Access denied, resource: /mlpipeline, request_id:<PHONE_NUMBER>F71DCE, host_id: None, bucket_name: mlpipeline

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/lib64/python3.9/site-packages/tornado/web.py", line 1713, in _execute
    result = await result
  File "/home/lib64/python3.9/site-packages/elyra/pipeline/handlers.py", line 156, in post
    response = await PipelineProcessorManager.instance().process(pipeline)
  File "/home/lib64/python3.9/site-packages/elyra/pipeline/processor.py", line 165, in process
    res = await asyncio.get_event_loop().run_in_executor(None, processor.process, pipeline)
  File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/lib64/python3.9/site-packages/elyra/pipeline/kfp/processor_kfp.py", line 264, in process
    kfp_argo_compiler.Compiler().compile(pipeline_function, pipeline_path, pipeline_conf=pipeline_conf)
  File "/home/lib64/python3.9/site-packages/kfp/compiler/compiler.py", line 1175, in compile
    self._create_and_write_workflow(
  File "/home/lib64/python3.9/site-packages/kfp/compiler/compiler.py", line 1227, in _create_and_write_workflow
    workflow = self._create_workflow(pipeline_func, pipeline_name,
  File "/home/lib64/python3.9/site-packages/kfp/compiler/compiler.py", line 1005, in _create_workflow
    pipeline_func(*args_list, **kwargs_dict)
  File "/home/lib64/python3.9/site-packages/elyra/pipeline/kfp/processor_kfp.py", line 247, in <lambda>
    pipeline_function = lambda: self._cc_pipeline(  # nopep8 E731
  File "/home/lib64/python3.9/site-packages/elyra/pipeline/kfp/processor_kfp.py", line 509, in _cc_pipeline
    self._verify_cos_connectivity(runtime_configuration)
  File "/home/lib64/python3.9/site-packages/elyra/pipeline/processor.py", line 483, in _verify_cos_connectivity
    raise RuntimeError(
RuntimeError: Error connecting to cloud storage: S3 operation failed; code: AccessDenied, message: Access denied, resource: /mlpipeline, request_id:<PHONE_NUMBER>F71DCE, host_id: None, bucket_name: mlpipeline. Update runtime configuration 'in-cluster' and try again.

A proper Kubeflow installation does not need to violate company guidelines and common sense https://github.com/kubeflow/pipelines/pull/7725 All of this works properly with normal pipelines, while it fails when using python scripts.
Furthermore i noticed that Elyra base images heavily violate the official Kbernetes restricted podsecuritystandards https://kubernetes.io/docs/concepts/security/pod-security-standards/#restricted for no reason. A simple container should not require extremely dangerous root rights.
time="2022-11-04T12:56:51.784Z" level=info msg="capturing logs" argo=true
mkdir: cannot create directory ‘./jupyter-work-dir/’: Permission denied
time="2022-11-04T12:56:51.786Z" level=error msg="cannot save artifact /tmp/mlpipeline-metrics.json" argo=true error="stat /tmp/mlpipeline-metrics.json: no such file or directory"
time="2022-11-04T12:56:51.786Z" level=error msg="cannot save artifact /tmp/mlpipeline-ui-metadata.json" argo=true error="stat /tmp/mlpipeline-ui-metadata.json: no such file or directory"
Error: exit status 1

Anaconda (2021.11) with Python 3.x
Container Image continuumio/anaconda3:2021.11 was used
Expected behavior
A clear and concise description of what you expected to happen.
Deployment information
Describe what you've deployed and how:

Elyra version: 3.12
Installation source: Kubeflow Jupyterlab

Hi @juliusvonkohout
Object prefix
Thanks for reporting! It appears that version 3.12 does have an issue. I ran a quick test on 3.13 (available soon - famous last words) and this seems to work there.

S3 permissions / admin access
We'll need more time to look into this. Not much has changed in a while but I don't recall that there is a requirement for admin access. Elyra makes the following calls using the minio client library:

client.bucket_exists(bucket) - check if the user-specified bucket exists (defined in the runtime configuration)
client.make_bucket(bucket) - try to create the bucket if ^^ returns something other than success
client.client.fput_object(bucket, fq_object_name, local_file_path) - upload the files to said bucket

Runtime configuration images
[updating the comment in a few minutes with details]

Quick update on 'S3 permissions / admin access':
The following setup worked just fine in our dev environment:
$ mc admin user add localminio bert 1234123
$ mc mb localminio/rwbucket
$ mc admin policy add localminio rwbucket read_and_write.json
$ mc admin policy set localminio rwbucket user=bert

read_and_write.json
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:*"],"Resource":["arn:aws:s3:::rwbucket/*"]}]}

Does that mean that i can set "cos_directory_prefix": "private-artifacts/julius-vonkohout", in the runtime WITHOUT having to modify the pipeline manually?
Alright, i understand your approach with the images. Yes removing insecure images is the right way to go.
There will be a redesign of Kubeflows S3 storage in the near future, that enforces proper folder permissions. so you will not be able to just list the root in Kubeflow 1.7 or maybe 1.8. Also something insecure as this {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:"],"Resource":["arn:aws:s3:::rwbucket/"]}]} will not be possible. you will only have access to your own folder, not he whole bucket (usually mlpipeline). So only the users folder e.g. "s3://mlpipeline/private-artifacts/julius-vonkohout" should be liested and modified. So you should not do anything outside of cos_directory_prefix

Does that mean that i can set "cos_directory_prefix": "private-artifacts/julius-vonkohout", in the runtime WITHOUT having to modify the pipeline manually?

No, this is currently only a Elyra pipeline property. There's currently no way to define a "global" default that could be defined in the runtime configuration (where it would apply to all pipelines that are executed using that configuration) or the Elyra pipeline editor (settings apply to all pipelines).

There will be a redesign of Kubeflows S3 storage in the near future, that enforces proper folder permissions. so you will not be able to just list the root in Kubeflow 1.7 or maybe 1.8. Also something insecure as this {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:"],"Resource":["arn:aws:s3:::rwbucket/"]}]} will not be possible. you will only have access to your own folder, not he whole bucket (usually mlpipeline). So only the users folder e.g. "s3://mlpipeline/private-artifacts/julius-vonkohout" should be listed and modified. So you should not do anything outside of cos_directory_prefix

Thanks for the heads-up! I don't think Elyra explicitly queries buckets or tries to read/write objects that don't have the user-defined object prefix included.  The setup you are describing should therefore not be an issue.

Unrelated to the concerns you've raised, but since this discussion is primarily about security it's probably worth bringing up: I've just noticed that you've configured your runtime to use UID/password for object storage authentication. You should always utilize secrets in test/production deployments to prevent exposing the credentials in KFP. By default they are embedded in the DAG.

No, this is currently only a Elyra pipeline property. There's currently no way to define a "global" default that could be defined in the runtime configuration (where it would apply to all pipelines that are executed using that configuration) or the Elyra pipeline editor (settings apply to all pipelines).

That is sad to hear. For example using the python script to pipeline functionality will not work then or  it will need manual intervention as soon as only prefixes are allowed. Setting the prefix in the runtime would be a proper solution for that.

No. This should work for any type of file X (Jupyter notebook, Python/R script) that Elyra's generic component supports. All you need to do is declare the output (files) that file X produces:

After file X was executed by the Elyra bootstrapper script, the bootstrapper will try to locate the specified output files and upload them to the object storage bucket using the user-defined prefix. This pre- and post processing is transparent to file X and Kubeflow Pipelines in general. Therefore there is no need for file X to know what the object prefix is, as long as it doesn't try to upload files to object storage directly.
I should mention - because that's a common cause for confusion - that currently these outputs are a 'proprietary' mechanism that generic components use. These outputs are not the same as component outputs that a KFP component definition might declare. Conceptually, this is the generic components signature, which doesn't declare any component inputs or outputs.
name: Run a file
description: Run a Jupyter notebook or Python/R script

implementation:
  container:
    image: ...
    command: [sh, -c]
    args: ["..."]

The metadata that the component requires (such as the output files to be uploaded to object storage) is passed in using the container args.

3.13. was released https://github.com/elyra-ai/elyra/releases/tag/v3.13.0, which should fix the path prefix issue you've reported.

3.13. was released https://github.com/elyra-ai/elyra/releases/tag/v3.13.0, which should fix the path prefix issue you've reported.

But I guess it will still not allow me to set the runtime specific prefix automatically. We will need a way to set the prefix globally for all pipelines from the python and R editor. We do not need a user defined prefix but instead a runtime defined prefix.

You are correct. Adding support for a new prefix property in the runtime configuration, which would apply to all pipelines that are submitted or exported using that configuration, would be a new requirement.


URL previews have limited information

Only way around this would be to get rid of the privacy and server-fetch data which is undesirable

True, matrix.to will never be able to make the riot side of this happy with that restriction. The preview could at least be helpful though: "matrix.to - join a room" or something that describes what the site does instead of just repeating the title.

I think this is pretty important - if we want to share links out of band then they need to not look... weird.
Presumably we can inspect the URL to provide a little bit of context as to whether this is a permalink to a room, a message/event in a room, a user or a community. Perhaps something like:

Permalink to a room on matrix.org
Permalink to a message on matrix.org
Permalink to Matrix user @tom:lant.uk
Permalink to Matrix community +coolstuff:matrix.org

The problem is that URL previews generally use a meta header, and they can't be easily calculated given most parers grep the content without running javascript.

So we can't do that because the server doesn't see the actual thing that's being linked to to generate the meta header

(there isn't actually a server component to matrix.to)

Aaaaaaaand matrix.to page content is generated clientside. Boo!
Presumably we could make a server-generated matrix.to server that stuffs a more useful summary into the meta header?

By which I mean something which honours the exact same URL scheme but generates the dynamic bits serverside rather than clientside

I'm going to add open graph details to matrix.to that give a little more info about what a matrix.to link is. Unfortunately it seems there is no way to maintain privacy and provide better link previews.

Duplicate of https://github.com/matrix-org/matrix.to/issues/207


Exact match doesn't seem to work

Describe the bug
The exact search doesn't seem to be working.
Steps to reproduce (if applicable)
Steps to reproduce the behavior:
 ▲ quickwit index search --index-id wikipedia --metastore-uri file://$(pwd)/wikipedia --query 'title:apollo AND 11' | jq '.hits[].title[]'
"Apollo"
"Apollo 11"
"Apollo 8"
"Apollo program"
"Apollo 13"
"Apollo 7"
"Apollo 9"
"Apollo 1"
"Apollo 10"
"Apollo 12"
"Apollo 14"
"Apollo 15"
"Apollo 16"
"Apollo 17"
"List of Apollo astronauts"
"Apollo, Pennsylvania"
"Apollo 13 (film)"
"Apollo Lunar Module"
"Apollo Guidance Computer"
"Apollo 4"

Okay, so it seems we've found what we're looking for as a second result.
However, since the article as literally named Apollo 11 we should be able to
perform what (according to quickwit's documentation) seems to be an exact
search:
▲ quickwit index search --index-id wikipedia --metastore-uri file://$(pwd)/wikipedia --query 'title:"Apollo 11"' | jq '.hits[].title[]'

Expected behavior
The "Apollo 11" result should be showing up.
System configuration:
60f897c0f49b4a920948b2bb98ca081f5557ed22 built from source on Linux, rustc 1.56.1
Additional context

Thanks @mrusme for the bug report. Can you share your index schema? I just want to make sure that you can do phrase queries.

@fmassot thanks for the quick reply. That's the schema I'm using:
{
  "version": 0,
  "index_id": "wikipedia",
  "index_uri": "file:///home/mrus/projects/@mrusme/ulpia/wikipedia",
  "search_settings": {
    "default_search_fields": ["title", "section_texts"]
  },
  "doc_mapping": {
    "store_source": true,
    "field_mappings": [
      {
        "name": "title",
        "type": "text"
      },
      {
        "name": "section_titles",
        "type": "array<text>"
      },
      {
        "name": "section_texts",
        "type": "array<text>"
      },
      {
        "name": "interlinks",
        "type": "array<text>",
        "indexed": false,
        "stored": false
      }
    ]
  }
}

Since I couldn't find any documentation for the latest master I had to scramble together the config myself. I might make a mistake there?

sorry for the not updated documentation, we are working on it right and it will be in line with the code in 2 weeks.
With your current schema,quickwit should return an error, remove the jq part and look at the output.
Here is the schema you need to have to make phrase queries:
{
  "version": 0,
  "index_id": "wikipedia",
  "index_uri": "file:///home/mrus/projects/@mrusme/ulpia/wikipedia",
  "search_settings": {
    "default_search_fields": ["title", "section_texts"]
  },
  "doc_mapping": {
    "store_source": true,
    "field_mappings": [
      {
        "name": "title",
        "type": "text"
        "record": "position" // Record position will enable phrase query on body field.
      },
      {
        "name": "section_titles",
        "type": "array<text>"
      },
      {
        "name": "section_texts",
        "type": "array<text>"
      },
      {
        "name": "interlinks",
        "type": "array<text>",
        "indexed": false,
        "stored": false
      }
    ]
  }
}

Unfortunately it doesn't return an error (or at least it doesn't look like one):
 ▲ quickwit index search --index-id wikipedia --metastore-uri file://$(pwd)/wikipedia --query 'title:"Barack Obama"'
{
  "numHits": 0,
  "hits": [],
  "elapsedTimeMicros": 4614
}

Also, even with | jq an error should be visible, since jq only reads from stdin, not from stderr in that case.

Ok, it's a bug then, I need to check that.

Also one more thing: I applied your change to my config and performed the following command:
quickwit index ingest --index-id wikipedia --metastore-uri file://$(pwd)/wikipedia --data-dir-path ./wikipedia-data --input-path enwiki-latest-pages-articles.json --overwrite

It seems like the overwrite flag doesn't do much here though. I assume I need to rm -rf wikipedia and start from scratch, correct?

It seems like the overwrite flag doesn't do much here though. I assume I need to rm -rf wikipedia and start from scratch, correct?

Yes exactly, you need to reindex the dataset and the simplest way to do it is to create again the index with --override flag.

sorry it's --overwrite, not --override

I've just tested a query like this on the tutorial Wikipedia dataset:
➜  quickwit git:(main) ✗ cargo r --release index search --index-id wikipedia --query 'body:"Barack Obama"' | jq '.hits[].title[]'
"Speaker of the United States House of Representatives election, October 2015"
"American SAFE Act of 2015"
"Sheila Gwaltney"
"Haben Girma"
"Assistant Secretary of State for Conflict and Stabilization Operations"
"Dixie Highway in Florida"
"Catherine A. Novelli"
"Walter Naegle"
"2015 Xi Jinping visit to the United States"
"Compact of Mayors"

Here is my index config:
version: 0
index_id: wikipedia
index_uri: file:///Users/fmassot/Documents/quickwit/indexes/wikipedia

doc_mapping:
  field_mappings:
    - name: title
      type: text
      tokenizer: default
      record: position
    - name: body
      type: text
      tokenizer: default
      record: position

search_settings:
  default_search_fields: [title, body]

I have 10 hits.

@mrusme I will try your query on the complete dataset.

Yeah, just finished re-indexing and now my results for the example I gave initially look like this:
"Apollo 11"
"Apollo 11 (disambiguation)"
"Apollo 11 in popular culture"
"Apollo 11 missing tapes"
"Apollo 11 goodwill messages"
"British television Apollo 11 coverage"
"Apollo 11 (1996 film)"
"Apollo 11 lunar sample display"
"Apollo 11 Cave"
"Moonshot: The Flight Of Apollo 11"
"Apollo 11 50th Anniversary commemorative coins"
"Apollo 11 anniversaries"
"Apollo 11 (2019 film)"

It's better now and I assume for exact matches I could always pick the very first match. However, it would be nice to have an exact exact match, which literally only shows a hit for an entry that's named "Apollo 11" or no hit at all of there is none. :)

I opened a dedicated issue for the search command here #922. It will be fixed soon, thanks again for the report.

It's better now and I assume for exact matches I could always pick the very first match. However, it would be nice to have an exact exact match, which literally only shows a hit for an entry that's named "Apollo 11" or no hit at all of there is none. :)

I see yes but when I look at other search engine and how tantivy is doing queries, we have several choices to do what you want:

use a costly regex query
instead of indexing tokens, you can index the title as is and then do a query on it. But a query like "Apollo" will not match a document with "Apollo 11".
just do a top 10 query and filter out what you don't need.

I would go for the 3rd solution :)

We should add BM25 scoring for this use case. That would help.

@mrusme the search command bug has been fixed here #923
Let's close this issue and maybe add a dedicated one for the BM25 scoring?


Error - Run setup.py first

It doesnt work. I ran setup.py so much times.
PS C:\Users\<Username>\Desktop\Edu-Mail-Generator-main> py setup.py
Requirement already satisfied: requests in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (2.26.0)
Requirement already satisfied: charset-normalizer~=2.0.0 in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from requests) (2.0.7)
Requirement already satisfied: idna<4,>=2.5 in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from requests) (3.3)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from requests) (2021.10.8)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from requests) (1.26.7)
WARNING: You are using pip version 21.2.3; however, version 21.3.1 is available.
You should consider upgrading via the 'C:\Users\<Username>\AppData\Local\Programs\Python\Python310\python.exe -m pip install --upgrade pip' command.

Requirement already satisfied: clint in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (0.5.1)
Requirement already satisfied: args in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from clint) (0.1.0)
WARNING: You are using pip version 21.2.3; however, version 21.3.1 is available.
You should consider upgrading via the 'C:\Users\<Username>\AppData\Local\Programs\Python\Python310\python.exe -m pip install --upgrade pip' command.

Requirement already satisfied: faker in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (9.8.0)
Requirement already satisfied: python-dateutil>=2.4 in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from faker) (2.8.2)
Requirement already satisfied: text-unidecode==1.3 in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from faker) (1.3)
Requirement already satisfied: six>=1.5 in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from python-dateutil>=2.4->faker) (1.16.0)
WARNING: You are using pip version 21.2.3; however, version 21.3.1 is available.
You should consider upgrading via the 'C:\Users\<Username>\AppData\Local\Programs\Python\Python310\python.exe -m pip install --upgrade pip' command.

Requirement already satisfied: selenium in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (4.0.0b4)
Requirement already satisfied: urllib3[secure]~=1.26 in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from selenium) (1.26.7)
Requirement already satisfied: trio~=0.17 in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from selenium) (0.19.0)
Requirement already satisfied: trio-websocket~=0.9 in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from selenium) (0.9.2)
Requirement already satisfied: attrs>=19.2.0 in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from trio~=0.17->selenium) (21.2.0)
Requirement already satisfied: outcome in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from trio~=0.17->selenium) (1.1.0)
Requirement already satisfied: cffi>=1.14 in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from trio~=0.17->selenium) (1.15.0)
Requirement already satisfied: sortedcontainers in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from trio~=0.17->selenium) (2.4.0)
Requirement already satisfied: sniffio in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from trio~=0.17->selenium) (1.2.0)
Requirement already satisfied: async-generator>=1.9 in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from trio~=0.17->selenium) (1.10)
Requirement already satisfied: idna in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from trio~=0.17->selenium) (3.3)
Requirement already satisfied: pycparser in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from cffi>=1.14->trio~=0.17->selenium) (2.21)
Requirement already satisfied: wsproto>=0.14 in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from trio-websocket~=0.9->selenium) (1.0.0)
Requirement already satisfied: certifi in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from urllib3[secure]~=1.26->selenium) (2021.10.8)
Requirement already satisfied: cryptography>=1.3.4 in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from urllib3[secure]~=1.26->selenium) (35.0.0)
Requirement already satisfied: pyOpenSSL>=0.14 in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from urllib3[secure]~=1.26->selenium) (21.0.0)
Requirement already satisfied: six>=1.5.2 in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from pyOpenSSL>=0.14->urllib3[secure]~=1.26->selenium) (1.16.0)
Requirement already satisfied: h11<1,>=0.9.0 in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (from wsproto>=0.14->trio-websocket~=0.9->selenium) (0.12.0)
WARNING: You are using pip version 21.2.3; however, version 21.3.1 is available.
You should consider upgrading via the 'C:\Users\<Username>\AppData\Local\Programs\Python\Python310\python.exe -m pip install --upgrade pip' command.

Requirement already satisfied: colorama in c:\users\<Username>\appdata\local\programs\python\python310\lib\site-packages (0.4.4)
WARNING: You are using pip version 21.2.3; however, version 21.3.1 is available.
You should consider upgrading via the 'C:\Users\<Username>\AppData\Local\Programs\Python\Python310\python.exe -m pip install --upgrade pip' command.

Firefox
Installed verision - 94.0.1
Latest geckodriver version - v0.30.0
Downloading...
Traceback (most recent call last):
  File "C:\Users\<Username>\Desktop\Edu-Mail-Generator-main\setup.py", line 70, in <module>
    main()
  File "C:\Users\<Username>\Desktop\Edu-Mail-Generator-main\setup.py", line 29, in main
    setup_Firefox(firefox_ver)
  File "C:\Users\<Username>\Desktop\Edu-Mail-Generator-main\__dwnldDrivers\versions.py", line 241, in setup_Firefox
    dwnld_zip_file(dwnldLink, './webdriver')
  File "C:\Users\<Username>\Desktop\Edu-Mail-Generator-main\__dwnldDrivers\versions.py", line 188, in dwnld_zip_file
    with ZipFile(BytesIO(r.content)) as my_zip_file:
  File "C:\Users\<Username>\AppData\Local\Programs\Python\Python310\lib\zipfile.py", line 1258, in __init__
    self._RealGetContents()
  File "C:\Users\<Username>\AppData\Local\Programs\Python\Python310\lib\zipfile.py", line 1325, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file
PS C:\Users\<Username>\Desktop\Edu-Mail-Generator-main>

yes i am having same error

Now fixed

Its still not working!


Pluralization issue fix

Package imports with a singular package will no longer have plural associations.

lgtm!

Merging.


haskellPackages: improve LLVM handling for GHC

This Merge
This PR is the regular merge of the haskell-updates branch into master.
This branch is being continually built and tested by hydra at https://hydra.nixos.org/jobset/nixpkgs/haskell-updates. You may be able to find an up-to-date Hydra build report at cdepillabout/nix-haskell-updates-status.
We roughly aim to merge these haskell-updates PRs at least once every two weeks. See the @NixOS/haskell team calendar for who is currently in charge of this branch.
haskellPackages Workflow Summary
Our workflow is currently described in pkgs/development/haskell-modules/HACKING.md.
The short version is this:

We regularly update the Stackage and Hackage pins on haskell-updates (normally at the beginning of a merge window).
The community fixes builds of Haskell packages on that branch.
We aim at at least one merge of haskell-updates into master every two weeks.
We only do the merge if the mergeable job is succeeding on hydra.
If a maintained package is still broken at the time of merge, we will only merge if the maintainer has been pinged 7 days in advance. (If you care about a Haskell package, become a maintainer!)

This is the follow-up to #146621. Come to #haskell:nixos.org if you have any questions.
I'll focus on working on GHC in this iteration and maybe even skip bumping hackage. The things I'm looking into are:

[x] Use latest supported LLVM version for every GHC
[x] Simplify how llvmPackages is reexposed in the package set
[ ] Fix pkgsCross.*.buildPackages.ghc (is not a cross-compiler counter-intuitively)
[ ] Unify NCG-detection mechanism for withPackages and GHC derivations
[ ] Investigate if we can avoid propagating LLVM in the GHC derivations and instead add it to the wrapper
[ ] #146770

Fixes to existing packages / unbreaking packages are always welcome of course!

Sadly upgrading LLVM to 12 for GHC 8.10.7 doesn't solve some instances of LLVM's opt failing during compilation for aarch64-darwin that were reported on Matrix. This seems to be happening when compiling haskellPackages.happy_1_19_12 which is required for ghcjs.

Sorry for asking here but matrix won't let me in...
What's the best way of making sure that haskellPackages.ShellCheck always points to the latest version? IIRC there used to be a hack somewhere in the haskell tooling that did this but 0.8.0 has been out for three weeks and ShellCheck is still at 0.7.2?

@zowoq Seems like we used to delete the version constraint on ShellCheck coming from stackage. This was deleted in a later cleanup trimming down the sed expression.
Not having the latest version of something due to stackage is an annoyance with most packages that are “user facing”, so shellcheck, pandoc, xmonad etc. This should get better when we switch to stackage nightly again, but is an inherent flaw of our packaging approach.
Since we do have the latest version of every package available, there should be nothing stopping us from using them for the top level packages (except for XMonad which is a bit tricky) with a bit of manual effort.

Seems like everything worked out. Aside from some flaky test suites, there is only one regression, namely haskellPackages.risc386 on aarch64-linux which fails due to a crash (bug) in LLVM. This seems to be due to our upgrade to LLVM 12, not our changes to GHC's packaging which is great.
Some stuff I have in mind is still pending which I'll tackle some other time:

ghcWithPackages/ghcWithHoogle refactors (#146770)
Set absolute paths in the binary GHCs' settings files as well (this is not urgent since our binary GHCs just need to be good enough to build the proper GHCs, but still nicer overall; since it's easy enough to test, it's just a matter of slipping in a set rebuild at some point (maybe even via staging?))
Have an absolute path for touch in settings (this is not supported by upstream at the moment, maybe we should try to get a patch in?)

Some of the changes here will be backported as #147404, but mostly the LLVM upgrades. 17b8b5a4dc5a49d83d508cb66c6ac2f0bb9fa39b would be very cool to backport, but I'd like to see if there are any downstream issues first. Backporting after release has the issue that this is a subtle breaking change since LLVM would no longer be in PATH in derivations that use a GHC with LLVM, but not sure if anyone relies on this behavior.

Seems like we used to delete the version constraint on ShellCheck coming from stackage.

Knew it used to be around somewhere, thanks for finding it.

Since we do have the latest version of every package available, there should be nothing stopping us from using them for the top level packages (except for XMonad which is a bit tricky) with a bit of manual effort.

ShellCheck does have a top level package that installs justStaticExecutables haskellPackages.ShellCheck and docs/manpage.
PR: https://github.com/NixOS/nixpkgs/pull/147780


Enhance Support Stats reporting - using an S3 bucket to store and append .csv file

Replaces https://github.com/ministryofjustice/operations-engineering/issues/4209 which has been broken down into smaller tickets.
Related to https://github.com/ministryofjustice/operations-engineering/issues/4350 and https://github.com/ministryofjustice/operations-engineering/issues/4353 which change and test the data format Support Request stats are reported in.
This ticket is to look at the possibility of storing the Support Requests stats reporting data in an S3 bucket.
To achieve this:

create an S3 bucket to store the .csv file in
set up a GitHub workflow action to read/write from this .csv file
create a method for appending data to the .csv file to record the previous days Support Requests
update the stats reporting script to use the new file instead of the one currently stored in the repo
test that this new process works and we can still report data to the KPI dashboard

Considered done when:

S3 bucket exists and is accessible to GitHub workflow action
.csv file can be appended on a daily basis
data can be pulled from the new .csv file to provide a breakdown of yesterdays Support Requests
Breakdown of stats is reported into the Slack channel

Investigating how to create an s3 bucket via code.
Started writing script to pull csv file from s3, upload amended csv to s3 from repo.
Figuring out the workflow - best way to update the csv file with new data, is this via the Google sheet (current method) or find a new way (maybe with web front end and db, edit csv in the repo?)

Example of how data can be added as a new file and Athena is used to create a table from all the files:
https://github.com/moj-analytical-services/lookup_offence/blob/main/.github/scripts/deploy.py

Understanding the method used in the example above and how to implement this for support stats

Initial PR for first version of the s3 upload script #72

Have now updated the script with changes:

check file path to the csv file
check if db and table exist
push csv file to s3 bucket amended
update table from new csv file upload

Added a timestamp variable.
Use this variable to create in the write_path for the latest csv update

Create s3 bucket for support stats:
operations-engineering-support-request-stats

Added support_request_stats.json to create the table schema

After a discussion with Connor it was decided that using this solution for uploading the support stats is over engineered and too complex.  It would require creating AWS s3 IAM roles and credentials, and would require using Athena to query the data.  As the KPI dashboard already utilises a database and has the necessary connections in place I am going to close this ticket off and look at using the existing infrastructure instead.


Issue creating new project

Hello everyone, and thanks for helping.  Here is the information you'll need.
The command doesn't work from the website.  I'm not new to react, and this is a brand new project.  I only did git init prior to running this command in a fresh directory.
Versions:
Camerons-MacBook-Pro:~ camerondixon$ npm -v
8.4.1
Camerons-MacBook-Pro:~ camerondixon$ npx -v
8.4.1
Camerons-MacBook-Pro:~ camerondixon$ node -v
v17.5.0

Output:
Camerons-MacBook-Pro:puzzles camerondixon$ npx create-react-app crossword

Creating a new React app in /Users/camerondixon/Desktop/puzzles/crossword.

Installing packages. This might take a couple of minutes.
Installing react, react-dom, and react-scripts with cra-template...

added 1355 packages in 10s

171 packages are looking for funding
  run `npm fund` for details

Installing template dependencies using npm...
npm ERR! code ERESOLVE
npm ERR! ERESOLVE unable to resolve dependency tree
npm ERR!
npm ERR! While resolving<EMAIL_ADDRESS>npm ERR! Found<EMAIL_ADDRESS>npm ERR! node_modules/react
npm ERR<EMAIL_ADDRESS>from the root project
npm ERR!
npm ERR! Could not resolve dependency:
npm ERR! peer<EMAIL_ADDRESS>from<EMAIL_ADDRESS>npm ERR! node_modules/@testing-library/react
npm ERR<EMAIL_ADDRESS>from the root project
npm ERR!
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force, or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.
npm ERR!
npm ERR! See /Users/camerondixon/.npm/eresolve-report.txt for a full report.

npm ERR!      /Users/camerondixon/.npm/_logs/2022-04-12T06_08_59_197Z-debug-0.log
`npm install --no-audit --save<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>failed
Camerons-MacBook-Pro:puzzles camerondixon$ ls -al
total 0
drwxr-xr-x   4 camerondixon  staff  128 Apr 12 02:08 .
drwxr-xr-x@ 23 camerondixon  staff  736 Apr 12 02:02 ..
drwxr-xr-x  12 camerondixon  staff  384 Apr 12 02:07 .git
drwxr-xr-x   9 camerondixon  staff  288 Apr 12 02:08 crossword
Camerons-MacBook-Pro:puzzles camerondixon$ rm -rf crossword/
Camerons-MacBook-Pro:puzzles camerondixon$ npx create-react-app crossword --legacy-peer-deps

Creating a new React app in /Users/camerondixon/Desktop/puzzles/crossword.

Installing packages. This might take a couple of minutes.
Installing react, react-dom, and react-scripts with cra-template...

added 1355 packages in 21s

171 packages are looking for funding
  run `npm fund` for details

Installing template dependencies using npm...
npm ERR! code ERESOLVE
npm ERR! ERESOLVE unable to resolve dependency tree
npm ERR!
npm ERR! While resolving<EMAIL_ADDRESS>npm ERR! Found<EMAIL_ADDRESS>npm ERR! node_modules/react
npm ERR<EMAIL_ADDRESS>from the root project
npm ERR!
npm ERR! Could not resolve dependency:
npm ERR! peer<EMAIL_ADDRESS>from<EMAIL_ADDRESS>npm ERR! node_modules/@testing-library/react
npm ERR<EMAIL_ADDRESS>from the root project
npm ERR!
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force, or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.
npm ERR!
npm ERR! See /Users/camerondixon/.npm/eresolve-report.txt for a full report.

npm ERR!      /Users/camerondixon/.npm/_logs/2022-04-12T06_09_54_767Z-debug-0.log
`npm install --no-audit --save<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>failed

I have the same error

Create react app is broken at the moment: https://github.com/facebook/create-react-app/issues/12278

I found the solution.
create-react-app is not updating index.js as per the new react V18
so you need to modify index.js file as follow
import React from 'react';
import {createRoot} from 'react-dom/client';
import './index.css';
import App from './App';
// import reportWebVitals from './reportWebVitals';
const container = document.getElementById('root');
const root = createRoot(container)
root.render(

)
// If you want to start measuring performance in your app, pass a function
// to log results (for example: reportWebVitals(console.log))
// or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals
// reportWebVitals();

When rerunning the create-react-app command on the website I now get a corrected installation.  Good to go.


[Question]: cannot import name 'LinearAdapterEmbeddingModel' from 'llama_index.core.embeddings'

Question Validation

[X] I have searched both the documentation and discord for an answer.

Question
from llama_index.core.embeddings import LinearAdapterEmbeddingModel
embed_model2 = LinearAdapterEmbeddingModel(base_embed_model, "/content/drive/MyDrive/model_output_test")
error:
ImportError                               Traceback (most recent call last)
 in <cell line: 1>()
----> 1 from llama_index.core.embeddings import LinearAdapterEmbeddingModel
2 embed_model2 = LinearAdapterEmbeddingModel(base_embed_model, "/content/drive/MyDrive/model_output_test")
ImportError: cannot import name 'LinearAdapterEmbeddingModel' from 'llama_index.core.embeddings' (/usr/local/lib/python3.10/dist-packages/llama_index/core/embeddings/init.py)

NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.
To view examples of installing some common dependencies, click the
"Open Examples" button below.
Does someone know how to fix this?

Nomic has their own HF embedding class (it requires a special pooling method as well)
pip install llama-index-embeddings-nomic
from llama_index.embeddings.nomic import NomicHFEmbedding

pip install llama-index-embeddings-adaptor
from llama_index.embeddings.adaptor import AdapterEmbeddingModel
Not sure where you saw LinearAdapterEmbeddingModel -- it doesn't exist when I ctrl-f the entire repo


Push notifications

What need to be done
Currently, there are only push notifications from chat.
We need to accomplish notifications in Android for next things:

New posts from subscriptions
New comments on user posts
New replies on user comments

comment

comment

PR #621 closes this issue

PR #721 disabled the functionality added in #621. More work needed.


nixos/prometheus-wireguard-exporter: incorporante changes in CLI arguments

In Version 3.6.1 (Merged in #164090) a braking change regarding the configuration was merged. Boolean options now need either true or false passed to them. Thus, using cfg.{verbose,singleSubnetPerField,withRemoteIp} would make the systemd service crash.
Commit containing the breaking changes: https://github.com/MindFlavor/prometheus_wireguard_exporter/commit/5b709b19f10c0a8a96f53167f4df28517584e450
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Yeah, currently can't use withRemoteIp = true; in the module.
Causes:
error: The argument '--export_remote_ip_and_port <export_remote_ip_and_port>' requires a value but none was supplied

Sorry, didn't realize that there was another PR for that. Fixed in #177430.


Develop into master

appears shell linter borked since https://github.com/D-Nice/vindi/pull/17/commits/92968e89c7ac132cc12490621aa37b685e1e90e4

will test this for a few days before merging in


returned data doesn't go through ember-data serializers

Don't know if this is intended, but the extract and normalizeHash function never fire when a request is answered by ember-cli-mirage.

sorry, made a typo which caused some errors to get swallowed up


HTGR pin lattice not well mapped during vis-file

I'm attempting to model an HTGR minicore and having a few issues / sticking points.  The output of armi vis-file minicore.h5 yields
                       ---------------------------------------------------
                      |             _      ____     __  __    ___         |
                      |            / \    |  _ \   |  \/  |  |_ _|        |
                      |           / _ \   | |_) |  | |\/| |   | |         |
                      |          / ___ \  |  _ <   | |  | |   | |         |
                      |         /_/   \_\ |_| \_\  |_|  |_|  |___|        |
                      |         Advanced  Reactor  Modeling Interface     |
                       ---------------------------------------------------
                                      Version 0.1.6

[info] Creating visualization file for cycle 0, time node 0...
[info] Loading reactor state for time node (0, 0)
[info] Applying <Migration from 0.1.2: Removes now-invalid `centers:` lines fro...
[info] Removing `centers:` sections.
[info] Applying <Migration from 0.1.2: Update elemental nuclide flags....
=========== Initializing Mesh, Assembly Zones, and Nuclide Categories ===========
[info] Nuclide categorization for cross section temperature assignments:
       ------------------  -----------------------------------------------------------
       Nuclide Category    Nuclides
       ------------------  -----------------------------------------------------------
       Fuel                CM245, LFP41, PU236, PU241, DUMP1, ZR90, U238, ZR96, CM242,
                           PU238, U234, ZR91, PU239, U235, LFP35, AM242, AM243,
                           CM244, LFP40, NP238, ZR94, PU240, AM241, LFP38, NP237,
                           LFP39, ZR92, CM247, DUMP2, CM243, U236, PU242, CM246
       Coolant             C
       Structure           MO97, SI30, NI62, W183, NI58, FE56, MN55, CR52,
                           CR53, MO92, NI60, NI61, MO98, CR50, MO96,
                           SI28, SI29, FE58, NI64, V, W184, FE54,
                           MO100, CR54, MO95, W182, MO94, FE57, W186
       ------------------  -----------------------------------------------------------
[info] Assembly zone definitions:
       --------  ------  ------------  -----------------------------------------  ----------  ------
           Zone                                                                     Symmetry  Hot
         Number  Name      Assemblies  Locations                                      Factor  Zone
       --------  ------  ------------  -----------------------------------------  ----------  ------
              1  ring-1            31  A1001, A2001, A2002, A3001, A3002, A3003,           3  False
                                       A3012, A4001, A4002, A4003, A4004,
                                       A4005, A4018, A5001, A5002, A5003,
                                       A5004, A5005, A5006, A5023, A5024,
                                       A6001, A6002, A6003, A6004, A6005,
                                       A6006, A6007, A6008, A6029, A6030
       --------  ------  ------------  -----------------------------------------  ----------  ------
[info] The setting `splitZones` is enabled. Building subzones from core zones:
       --------  ------------------  ----------  ------------  -----------  ----------  ------
           Zone                      Original                                 Symmetry  Hot
         Number  Name                Name          Assemblies  Locations        Factor  Zone
       --------  ------------------  ----------  ------------  -----------  ----------  ------
              1  ring-1-reflector-1  ring-1                 1  A4005                 3  False
       --------  ------------------  ----------  ------------  -----------  ----------  ------

gives some spurious messages about the assemblies. I think this also causes the vtu file to not reflect the core geometry, or what I believe the core geometry should indicate

As a note, the current HEAD 1fd21f8236763f43c3bc25b866122aa9789486bc produces similar issues, but with different assembly naming
              1  ring-1            31  001-001, 002-001, 002-002, 003-001, 003-002, 003-003,           3  False
                                       003-012, 004-001, 004-002, 004-003, 004-004,
                                       004-005, 004-018, 005-001, 005-002, 005-003,
                                       005-004, 005-005, 005-006, 005-023, 005-024,
                                       006-001, 006-002, 006-003, 006-004, 006-005,
                                       006-006, 006-007, 006-008, 006-029, 006-030

I've thought about not using the lattice map for the pin grid and instead using grid contents which may let me specify the pin position with tuples (ring, pos) if I'm understanding the docs right - https://terrapower.github.io/armi/.apidocs/armi.reactor.blueprints.gridBlueprint.html#armi.reactor.blueprints.gridBlueprint.GridBlueprint.gridContents
As a consistency check, if you load this reactor up, the fuel lattice appears to be present in each block with the correct lattice structure. Checked based on the spatialLocator for circular components in a fuel block.

That is also not what I would expect from the vtu file...
Hmmm.... yes the assemblies appear to be in the right place as you say:
In [2]: for a in o.r.core:
    ...:     print(a.spatialLocator.getRingPos())
(2, 5)
(2, 6)
(2, 4)
(1, 1)
(2, 1)
(2, 3)
(2, 2)

Its possible you might need a minus sign in front of the first line:
  lattice map: |
   -  F  F
    F  F  F
      F  F

By I don't know why it would only effect the vtu... I'll try to look into it more later...

I added the minus sign and it looks like I have slightly different art work than you....

I might get someone more familiar with grids to take a look.

Fascinating! Thanks for the report. Yeah I just am doing some sanity checks with a simple settings file:
import armi
armi.configure()
o = armi.init()
core = o.r.core
import matplotlib.pyplot as plt
plt.figure()
x = []
y = []
for a in core:
    xi,yi,z = a.spatialLocator.getGlobalCoordinates()
    x.append(xi)
    y.append(yi)
plt.plot(x,y,'o')

Same with blocks as the plotter. And from what you say sounds like components too. So the grid at least is represented in ARMI ok, so it seems, as you both already confirmed ;)
I'm guessing that something is throwing off the VTK writer related to the blocks actually having pin-level grids. That's not something we've shaken down so that's the likely suspect IMHO.
going into the VTK code and trying it out interactively and plotting the block mesh x and y, I get:
from armi.reactor import blocks
blk = o.r.getChildren(deep=True, predicate=lambda o: isinstance(o, blocks.Block))
from armi.bookkeeping.visualization import utils
for blki in blk:
    bmsh = utils.createBlockMesh(blki)
    plt.plot(bmsh.x, bmsh.y, 'o')
plt.show()

Still looking good way down here (these are the corners of the blocks).
Hmm and I can't even run the viz-file entry point on the case I end up making from this with my dummy settings file. Will need to work more on it. Sorry for the trouble.

Ah yes I remember this issue now. So this is related to the fact that block-level pin subassembly grids not getting stored correctly in the database3 hdf5 database. So everything is all good and well during the initial run, but upon loading from a database (as vis-file does), some of the geometric details are lost. This should actually produce a RuntimeError saying that it's not yet implemented since this kind of strange behavior is pretty unexpected and random.
I loaded from DB (as shown here) and then plotted block coords again and saw garbage:

@youngmit has been anticipating this a bit and I believe it's part of what's behind the big disclaimer on the LWR input tutorial, but I apologize that it wasn't really highlighted as an issue.
Obviously, storing grid-specified geometric pin details in the h5 database is something fundamental that we absolutely must support so now that we have some user demand maybe we can get it going.
For a temporary work-around, I believe if you just call the vis-file code during the initial run rather than in a follow-up step, it will probably show the blocks just fine. You can use the XdmfDumper class directly in a plugin or script perhaps.

@onufer @ntouran thanks for digging in to this so soon. I'm glad to see it's as baffling as it felt to me 😅
Thankfully, as you've both pointed out, it looks like the internal representation of the blocks and their pin lattices is maintained when reading from the yaml file. This means that any plugins should still be able to faithfully model the problem and either wait to use armi vis-file or add in some final hook like

For a temporary work-around, I believe if you just call the vis-file code during the initial run rather than in a follow-up step, it will probably show the blocks just fine.

Alright @drewejohnson, here is the work around. Before you let you of your reactor instance, run something like this:
from armi.bookkeeping.visualization import vtk
fileName = "myReactor"
dumper = vtk.VtkDumper(fileName , "")
with dumper:
    dumper.dumpState(o.r)

The file will be named: myReactor_blk_000000.vtu (assuming the reactor cycle/node are 0,0)
Its actually a lot faster than the post-process call.

Leaving this open, since we need to fix this for when there are pin lattices. If the block was defined another way (with multiplicity, instead of a lattice) the post-process vis-file call should work.

Excellent! That's working on my end too

I replicated the issue by adjusting one of the unit tests in https://github.com/ntouran/armi/commit/2f6f9e2fbd97d3944bbc41d8789f740b39ed9edd in this branch . We'll work off this branch to find and implement the fix.

@drewejohnson #256 should fix your issue. Mind pulling that branch and trying it out with your model to see if it fixes the issue? Thanks!

That did the trick! Thanks everyone for digging in and fixing this!


Skip Cutscene

What is your question?
how can I skip cutscenes in psych engine?

For dialogue, you can't. (Atleast not without editing the source code.)
For video cutscenes, I'm not sure.


measurements should be performed in a fast cycle

measurements should be performed every 5 s
https://github.com/zadasa-u/termostato-esp32-iot2023/blob/a4dfbacdcb6d5503d51f0a21575b4888b3673fe1/main.py#L166

solved


The Big Migration Thread

Post your issues and/or gotchas from migrating in here. We'll try and help out as much as we can.

    let msg = switch e {
      | JsError(err) =>
        switch Exn.message(err) {
        | Some(msg) => msg
        | None => ""
        }
      | _ => "Unexpected error occurred"
    }

After the migration to @rescript/core, the above code fragment has e with type exn and err now with type unknown which results in the following compile error in reference to err being passed into Exn.message:
 This has type: unknown
  Somewhere wanted: RescriptCore.Exn.t (defined as Js_exn.t)

This worked before the migration. Thanks.

EDIT: I modified JsError to Exn.Error and it compiled, but I don't think this code properly handles Js exceptions. I looked in the rescript documentation and did not find much.

This is the correct way to do it. Does it not work for you? We have tests with that same behavior: https://github.com/rescript-association/rescript-core/blob/main/test/PromiseTest.res#L46-L71

I migrated our entire corporate code base in a branch (for now) to see what came up.
Our starting point was -open Belt and -open ReScriptJs (bloodyowl's).
I decided to switch to -open RescriptCore only and not open Belt, to make it clearer where we still use Belt.
Some observations (not issues in my opinion):

I had expected ReScriptCore instead of RescriptCore to be the name of the module, consistent with project branding
Because we already used bloodyowl's rescript-js and -open Belt, I assumed we could not use the automated migrations. If this is incorrect, we may want to add a note about this for other users.
Because of our -opens, Js.String.Split used to refer to bloodyowl's binding and now refers to the binding shipped with the compiler, which has the same signature but a different implementation. This caused erroneous behaviour at runtime (fixed by changing to String.split)
After noticing this I decided to remove the Js. prefix everywhere
Belt.Array.concatMany (which was Array.concatMany in our codebase) and RescriptCore.Array.concatMany (which is now Array.concatMany) have the same name but different signatures; I had to migrate them to RescriptCore's Array.flat
Belt.Array.joinWith had an extra 't => string parameter which RescriptCore.Array.joinWith does not have (an improvement in my opinion; all but one place in our codebase passed x => x)
Belt.Array.mapWithIndex passes the index as the first argument, while RescriptCore.Array.mapWithIndex passes the index as the second argument
Belt.Array.slice takes ~offset and ~len, while RescriptCore.Array.slice takes ~start and ~end, which can be a gotcha
Same for ReScriptJs.String.substr (~start and ~length) versus RescriptCore.String.substring (~start and ~end)
Belt.Array.sliceToEnd takes a positional argument while RescriptCore.Array.sliceToEnd takes a labeled argument (~start)
I decided to use Array.getUnsafe instead of Array.getExn (which does not exist anymore in RescriptCore) in some places
There now exists a module Date and module Error in global scope, which are shadowed by some of my own modules
I passed Belt.Int.fromString to Array.map, but RescriptCore.Int.fromString takes a optional ~radix argument which created a warning
Some renamed types:

Js.Date.Time.t -> Date.time

Some renamed functions:

Js.String.replaceString -> String.replace
keep -> filter
keepMap -> filterMap
getBy -> find
getIndexBy -> findIndexOpt (would have preferred if findIndex returned an option<int>)
Js.JSON.Decode.classify -> JSON.Classify.classify
Js.Global.setTimeout -> setTimeout (don't really like these functions in the global scope)

Places where we still use Belt after the migration:

Belt.Map and functions (extensively)
Belt.Map.String and functions
Belt.Map.Int and functions
Belt.Set and functions (extensively)
Belt.Set.String and functions
Belt.Id (for Map and Set)
Belt.SortArray.stableSortBy
Belt.Array.zip
Belt.Array.unzip
Belt.Array.range
Belt.Array.partition
Belt.Array.make
Belt.Array.makeBy

@Minnozz fantastic and insightful write up, thank you!

I found out
Js.Math._PI is now Math.Constants.pi
I also was using open Belt in many files so I removed that and added Belt. to all the Array so combo can work better.
Array.map( -> Belt.Array.map
Combi missed to flip the index in Array.mapWithIndex.


Another atom-sample: TODO list

Add another sample - simplest web application "TODO list" to demonstrate Atom abilities of using existing 3rd party libraries and solutions in Java ecosystem.

Atom samples were deprecated.


Daniil: Watch email-<EMAIL_ADDRESS>

Daniil requests the watch of the watch_keyword email-<EMAIL_ADDRESS>See the MS search here and the Stack Exchange search here.

email-<EMAIL_ADDRESS>has been seen in 0 true positives, 0 false positives, and 0 NAAs.

Approved by tripleee in Charcoal HQ

Too quick to click - I'll fix this, sorry.


Add exception for SupabaseSaveState to ExtensionsValidatorExceptions.js

This needs to be merged before merging #1069.

Now ready for merge

Please, run the npm format.

Please run the npm format.

Done


rerun pipeline from cached pipeline

run a pipeline lets from the ui, when its finished, run the same pipeline from cached.
when that pipeline finished try to reRun the last chached pipeline
you get an error-"pipeline name must contain only alphanumeric, dash, dot or underscore"

tested on "simple" algorithm


deployment keycloak-postgresql gets stuck

I'm using kas-installer.sh and it's getting stuck with the below error:
deployment keycloak-postgresql still not created. Waiting 10s...
I'm using 4.12.0 OSD version. The keycloak custom resource shows the below message :
message: no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"

@MikeEdgar suggests that due to https://kubernetes.io/docs/reference/using-api/deprecation-guide/#poddisruptionbudget-v125 in 4.12 OSD the current mass-sso installation might be broken.


chore: fixed workflow runs

С данной конфигурацией экшены должны запускаться при выставлении ПРов из форков

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.007%) to 80.68%

Totals

Change from base Build<PHONE_NUMBER>:
-0.007%

Covered Lines:
7309

Relevant Lines:
8259

💛  - Coveralls


[firebase_messaging] hard to detect initial route when the app is opened from push

Problem
Clicking on push usually opens some specific route, however current api makes it hard and not effective. Currently, I have to create app with initial route, when onLaunch is use navigator to perform navigation to a correct page.
Solution
Ideal solution is to get the data before run off the app. So solution could be by extending api:
  final options = await _firebaseMessaging.launchOptions;
  if (options != null) {
    runApp(MyAwesomeApp.withInitialOptions(options));
  } else {
    runApp(MyAwesomeApp());
  }

Hey all 👋
As part of our roadmap (#2582) we've just shipped a complete rework of the firebase_messaging plugin that aims to solve this and many other issues.
If you can, please try out the dev release (see the migration guide for upgrading and for changes) and if you have any feedback then join in the discussion here.
Given the scope of the rework I'm going to go ahead and close this issue in favor of trying out the latest plugin. I'd recommend checking out the new APIs getInitialMessage and onMessageOpenedApp to help with this issue.
Thanks everyone 🤓


"Abuse" dnsServer to redirect all to AP_LOCAL_IP

On DNS Setup, you can use
dnsServer.start(DNS_PORT, "*", AP_LOCAL_IP);

to let every IP be your T-Dongle network. That way you can send people in that WiFi network directly to the webpage

oops. Already in there but not working;


customs declaration items description too long

If your customs declaration items description is longer then 30 characters long then you will get this API error:
data.shipments[0].customs_declaration.items[0].description - data.shipments[0].customs_declaration.items[0].description should be 30 characters or less - Shipment validation error
Label description has the same error, but this is solved by cutting the description at 42 characters and adding three dots.
If it is possible do the same for the item description.

fix: https://github.com/myparcelnl/sdk/pull/345

Fixed in latest release