Removing security rules with positional dependencies results in errors placing subsequent rules

Describe the bug
Removing a rule results in an error when placing rules dependent on other rules that were previously dependent on the removed ruled.
Expected behavior
Removing a rule should invalidate references to the removed rule so subsequent rules can be placed properly.
Current behavior
The following error is seen on an existing rule when a rule at least two spots above it is removed.
Error: Can't position group directly after "[Name of rule directly below removed rule]": rule is not present

Trying to back out the change by adding the removed rule back in results in the same error message. The tfstate file has to be deleted first and the original main.tf file run to restore the policies to the previous state.
Possible solution

Steps to reproduce

The behavior can be reproduced using the following main.tf files. To reproduce:

Run the first main.tf file to establish steady state.
Run the second main.tf file to generate the error.
(Optional) Re-run the first main.tf again to demonstrate the change cannot be rolled back.

Original state main.tf:
	provider "panos" {
	  hostname = "panorama.local.int"
	  username = "my_username"
	  password = "my_password"
	}

	resource "panos_panorama_security_rule_group" "pre_rule_1" {
	  position_keyword   = "top"
	  rulebase           = "pre-rulebase"
	  device_group       = "Test-device-group"

	  rule {
	    name                               = "Tenant-rule-1"
	    description                        = null
	    tags                               = null
	    source_zones                       = ["Tenant_Private"]
	    source_addresses                   = ["any"]
	    source_users                       = ["any"]
	    hip_profiles                       = ["any"]
	    destination_zones                  = ["Tenant_WAN"]
	    destination_addresses              = ["any"]
	    applications                       = ["any"]
	    services                           = ["any"]
	    categories                         = ["any"]
	    group                              = ""
	    log_end                            = true
	    log_setting                        = ""
	    action                             = "allow"
	    disable_server_response_inspection = false
	    disabled                           = false
	    icmp_unreachable                   = false
	    log_start                          = false
	    negate_destination                 = false
	    negate_source                      = false
	    negate_target                      = false
	  }

	  lifecycle {
	    create_before_destroy = true
	  }
	}

	resource "panos_panorama_security_rule_group" "pre_rule_2" {
	  position_keyword   = "directly after"
	  position_reference = "Tenant-rule-1"
	  rulebase           = "pre-rulebase"
	  device_group       = "Test-device-group"

	  rule {
	    name                               = "Tenant-rule-2"
	    description                        = null
	    tags                               = null
	    source_zones                       = ["Tenant_WAN"]
	    source_addresses                   = ["any"]
	    source_users                       = ["any"]
	    hip_profiles                       = ["any"]
	    destination_zones                  = ["Tenant_Private"]
	    destination_addresses              = ["any"]
	    applications                       = ["any"]
	    services                           = ["any"]
	    categories                         = ["any"]
	    group                              = ""
	    log_end                            = true
	    log_setting                        = ""
	    action                             = "allow"
	    disable_server_response_inspection = false
	    disabled                           = false
	    icmp_unreachable                   = false
	    log_start                          = false
	    negate_destination                 = false
	    negate_source                      = false
	    negate_target                      = false
	  }

	  lifecycle {
	    create_before_destroy = true
	  }

	  depends_on = [
	    panos_panorama_security_rule_group.pre_rule_1
	  ]
	}
	resource "panos_panorama_security_rule_group" "pre_rule_3" {
	  position_keyword   = "directly after"
	  position_reference = "Tenant-rule-2"
	  rulebase           = "pre-rulebase"
	  device_group       = "Test-device-group"

	  rule {
	    name                               = "Tenant-rule-3"
	    description                        = null
	    tags                               = null
	    source_zones                       = ["Tenant_WAN", "Tenant_Private"]
	    source_addresses                   = ["any"]
	    source_users                       = ["any"]
	    hip_profiles                       = ["any"]
	    destination_zones                  = ["Tenant_Public"]
	    destination_addresses              = ["any"]
	    applications                       = ["any"]
	    services                           = ["service-http"]
	    categories                         = ["any"]
	    group                              = ""
	    log_end                            = true
	    log_setting                        = ""
	    action                             = "allow"
	    disable_server_response_inspection = false
	    disabled                           = false
	    icmp_unreachable                   = false
	    log_start                          = false
	    negate_destination                 = false
	    negate_source                      = false
	    negate_target                      = false
	  }

	  lifecycle {
	    create_before_destroy = true
	  }

	  depends_on = [
	    panos_panorama_security_rule_group.pre_rule_2
	  ]
	}
	resource "panos_panorama_security_rule_group" "pre_rule_4" {
	  position_keyword   = "directly after"
	  position_reference = "Tenant-rule-3"
	  rulebase           = "pre-rulebase"
	  device_group       = "Test-device-group"

	  rule {
	    name                               = "Tenant-rule-4"
	    description                        = null
	    tags                               = null
	    source_zones                       = ["Tenant_WAN", "Tenant_Private"]
	    source_addresses                   = ["any"]
	    source_users                       = ["any"]
	    hip_profiles                       = ["any"]
	    destination_zones                  = ["Tenant_Public"]
	    destination_addresses              = ["any"]
	    applications                       = ["any"]
	    services                           = ["service-http"]
	    categories                         = ["any"]
	    group                              = ""
	    log_end                            = true
	    log_setting                        = ""
	    action                             = "allow"
	    disable_server_response_inspection = false
	    disabled                           = false
	    icmp_unreachable                   = false
	    log_start                          = false
	    negate_destination                 = false
	    negate_source                      = false
	    negate_target                      = false
	  }

	  lifecycle {
	    create_before_destroy = true
	  }

	  depends_on = [
	    panos_panorama_security_rule_group.pre_rule_3
	  ]
	}

Contents of second main.tf that generate an error:
	provider "panos" {
	  hostname = "panorama.local.int"
	  username = "my_username"
	  password = "my_password"
	}

	resource "panos_panorama_security_rule_group" "pre_rule_1" {
	  position_keyword   = "top"
	  rulebase           = "pre-rulebase"
	  device_group       = "Test-device-group"

	  rule {
	    name                               = "Tenant-rule-1"
	    description                        = null
	    tags                               = null
	    source_zones                       = ["Tenant_Private"]
	    source_addresses                   = ["any"]
	    source_users                       = ["any"]
	    hip_profiles                       = ["any"]
	    destination_zones                  = ["Tenant_WAN"]
	    destination_addresses              = ["any"]
	    applications                       = ["any"]
	    services                           = ["any"]
	    categories                         = ["any"]
	    group                              = ""
	    log_end                            = true
	    log_setting                        = ""
	    action                             = "allow"
	    disable_server_response_inspection = false
	    disabled                           = false
	    icmp_unreachable                   = false
	    log_start                          = false
	    negate_destination                 = false
	    negate_source                      = false
	    negate_target                      = false
	  }

	  lifecycle {
	    create_before_destroy = true
	  }
	}

	resource "panos_panorama_security_rule_group" "pre_rule_2" {
	  position_keyword   = "directly after"
	  position_reference = "Tenant-rule-1"
	  rulebase           = "pre-rulebase"
	  device_group       = "Test-device-group"

	  rule {
	    name                               = "Tenant-rule-3"
	    description                        = null
	    tags                               = null
	    source_zones                       = ["Tenant_WAN", "Tenant_Private"]
	    source_addresses                   = ["any"]
	    source_users                       = ["any"]
	    hip_profiles                       = ["any"]
	    destination_zones                  = ["Tenant_Public"]
	    destination_addresses              = ["any"]
	    applications                       = ["any"]
	    services                           = ["service-http"]
	    categories                         = ["any"]
	    group                              = ""
	    log_end                            = true
	    log_setting                        = ""
	    action                             = "allow"
	    disable_server_response_inspection = false
	    disabled                           = false
	    icmp_unreachable                   = false
	    log_start                          = false
	    negate_destination                 = false
	    negate_source                      = false
	    negate_target                      = false
	  }

	  lifecycle {
	    create_before_destroy = true
	  }

	  depends_on = [
	    panos_panorama_security_rule_group.pre_rule_1
	  ]
	}
	resource "panos_panorama_security_rule_group" "pre_rule_3" {
	  position_keyword   = "directly after"
	  position_reference = "Tenant-rule-3"
	  rulebase           = "pre-rulebase"
	  device_group       = "Test-device-group"

	  rule {
	    name                               = "Tenant-rule-4"
	    description                        = null
	    tags                               = null
	    source_zones                       = ["Tenant_WAN", "Tenant_Private"]
	    source_addresses                   = ["any"]
	    source_users                       = ["any"]
	    hip_profiles                       = ["any"]
	    destination_zones                  = ["Tenant_Public"]
	    destination_addresses              = ["any"]
	    applications                       = ["any"]
	    services                           = ["service-http"]
	    categories                         = ["any"]
	    group                              = ""
	    log_end                            = true
	    log_setting                        = ""
	    action                             = "allow"
	    disable_server_response_inspection = false
	    disabled                           = false
	    icmp_unreachable                   = false
	    log_start                          = false
	    negate_destination                 = false
	    negate_source                      = false
	    negate_target                      = false
	  }

	  lifecycle {
	    create_before_destroy = true
	  }

	  depends_on = [
	    panos_panorama_security_rule_group.pre_rule_2
	  ]
	}

Context

We are no longer able to delete unused rules reliably via code. We have to delete the rules manually, remove references to them from our Terraform files, and delete TFstate files in order to maintain a proper working environment.
Your Environment

Version used: Tested on module version v1.9.0 and v1.10.0
Environment name and version (e.g. Chrome 59, node.js 5.4, python 3.7.3): Panorama version 10.0.4

Ok, so I can reproduce this using the above steps, but I would argue that the above is "doing it wrong," and doing it "the right way" does not result in an error.
In the 1st stage of the above plan file, there are 4 security rules.  In the 2nd stage, this gets trimmed down to 3 security rules, but because an emphasis is placed on the Terraform names of individual resources being meaningful to a human reading the plan file, removing a rule in the middle has a ripple effect of causing every other rule defined after it to be touched.  If this was 100 rules, and you needed to delete the 2nd, that would be 1 (delete the 2nd rule) + 98 * 3 updates (you have to change 1) the Terraform name, 2) the position_reference, and 3) the depends_on).  This feels wrong to me.
If, instead, the Terraform names are not locked to the placement of the rule within the rulebase, then deleting the 2nd security rule only requires a 1 (delete the 2nd rule) + 1 (update reference of the 3rd rule) changes.  There's nothing wrong with having the Terraform name still being meaningful to humans reading the plan file, just don't tie the name to the rulebase location :)
Here is an initial deploy file that I'll be using.  You'll note that I'm forgoing the depends_on in order to directly reference the rule name.  If for some reason you really like using depends_on, then it would be 1 + 2 updates instead of 1 + 1.  You'll also note that I'm using a NGFW instead of a Panorama here, but the concept is the same for Panorama and does not change anything other than resource naming.
Here is my initial deploy config:
resource "panos_security_rule_group" "top" {
    position_keyword = "top"
    rule {
        name = "first"
        source_zones = ["any"]
        source_addresses = ["any"]
        source_users = ["any"]
        destination_zones = ["any"]
        destination_addresses = ["any"]
        applications = ["any"]
        services = ["any"]
        categories = ["any"]
        log_end = true
        action = "allow"
    }

    lifecycle {
        create_before_destroy = true
    }
}

resource "panos_security_rule_group" "one" {
    position_keyword = "directly after"
    position_reference = panos_security_rule_group.top.rule.0.name
    rule {
        name = "second"
        source_zones = ["any"]
        source_addresses = ["any"]
        source_users = ["any"]
        destination_zones = ["any"]
        destination_addresses = ["any"]
        applications = ["any"]
        services = ["any"]
        categories = ["any"]
        log_end = true
        action = "allow"
    }

    lifecycle {
        create_before_destroy = true
    }
}

resource "panos_security_rule_group" "two" {
    position_keyword = "directly after"
    position_reference = panos_security_rule_group.one.rule.0.name
    rule {
        name = "third"
        source_zones = ["any"]
        source_addresses = ["any"]
        source_users = ["any"]
        destination_zones = ["any"]
        destination_addresses = ["any"]
        applications = ["any"]
        services = ["any"]
        categories = ["any"]
        log_end = true
        action = "allow"
    }

    lifecycle {
        create_before_destroy = true
    }
}

resource "panos_security_rule_group" "three" {
    position_keyword = "directly after"
    position_reference = panos_security_rule_group.two.rule.0.name
    rule {
        name = "fourth"
        source_zones = ["any"]
        source_addresses = ["any"]
        source_users = ["any"]
        destination_zones = ["any"]
        destination_addresses = ["any"]
        applications = ["any"]
        services = ["any"]
        categories = ["any"]
        log_end = true
        action = "allow"
    }

    lifecycle {
        create_before_destroy = true
    }
}

Updated deploy config (removes the 2nd rule):
resource "panos_security_rule_group" "top" {
    position_keyword = "top"
    rule {
        name = "first"
        source_zones = ["any"]
        source_addresses = ["any"]
        source_users = ["any"]
        destination_zones = ["any"]
        destination_addresses = ["any"]
        applications = ["any"]
        services = ["any"]
        categories = ["any"]
        log_end = true
        action = "allow"
    }

    lifecycle {
        create_before_destroy = true
    }
}

resource "panos_security_rule_group" "two" {
    position_keyword = "directly after"
    position_reference = panos_security_rule_group.top.rule.0.name
    rule {
        name = "third"
        source_zones = ["any"]
        source_addresses = ["any"]
        source_users = ["any"]
        destination_zones = ["any"]
        destination_addresses = ["any"]
        applications = ["any"]
        services = ["any"]
        categories = ["any"]
        log_end = true
        action = "allow"
    }

    lifecycle {
        create_before_destroy = true
    }
}

resource "panos_security_rule_group" "three" {
    position_keyword = "directly after"
    position_reference = panos_security_rule_group.two.rule.0.name
    rule {
        name = "fourth"
        source_zones = ["any"]
        source_addresses = ["any"]
        source_users = ["any"]
        destination_zones = ["any"]
        destination_addresses = ["any"]
        applications = ["any"]
        services = ["any"]
        categories = ["any"]
        log_end = true
        action = "allow"
    }

    lifecycle {
        create_before_destroy = true
    }
}

I don't know Terraform internals well enough to know why doing it your way is causing it to break.  If I had to guess, I would say it's a combination of reusing the Terraform names while changing their contents, which in actuality overlaps with existing config on PAN-OS.  But doing it the way that is mentioned in the opening of this issue doesn't seem right to me from a Terraform perspective.
From a PAN-OS perspective, I also would not recommend doing it that way even if Terraform wasn't breaking on it.  This is because you are asking Terraform to delete and re-create every single security rule when you change the resource location like this.  This not only will make deployment of the config take longer than it needs to, but also will affect the operational state of your security rules as well, such as hit count counters and the such.
Finally, if it is at all possible, I would highly recommend that if the intent is that security rules be back-to-back, they should all be in a single panos_security_rule_group / panos_panorama_security_rule_group resource, as there are additional performance gains that take place within the confines of a single resource (as of panos v1.9.0).  AKA:
resource "panos_security_rule_group" "top" {
    position_keyword = "top"
    rule {
        name = "first"
        source_zones = ["any"]
        source_addresses = ["any"]
        source_users = ["any"]
        destination_zones = ["any"]
        destination_addresses = ["any"]
        applications = ["any"]
        services = ["any"]
        categories = ["any"]
        log_end = true
        action = "allow"
    }
    rule {
        name = "second"
        source_zones = ["any"]
        source_addresses = ["any"]
        source_users = ["any"]
        destination_zones = ["any"]
        destination_addresses = ["any"]
        applications = ["any"]
        services = ["any"]
        categories = ["any"]
        log_end = true
        action = "allow"
    }
    rule {
        name = "third"
        source_zones = ["any"]
        source_addresses = ["any"]
        source_users = ["any"]
        destination_zones = ["any"]
        destination_addresses = ["any"]
        applications = ["any"]
        services = ["any"]
        categories = ["any"]
        log_end = true
        action = "allow"
    }
    rule {
        name = "fourth"
        source_zones = ["any"]
        source_addresses = ["any"]
        source_users = ["any"]
        destination_zones = ["any"]
        destination_addresses = ["any"]
        applications = ["any"]
        services = ["any"]
        categories = ["any"]
        log_end = true
        action = "allow"
    }

    lifecycle {
        create_before_destroy = true
    }
}

Provided single resource is an option for your use case, doing it this way also has the benefit of increasing the human-readable factor of the security rules.
Hope this helps!


semi space seprate character

Thanks for the plugin.
It counts semi space as a seprate character in right to left languages.
Example:
می‌باشد: is a one word in persian. but the pluging counts it as two word. It's beacause of semi space between می and باشد.

Hi could you check if the count is same or different to the core word count plug-in? Thank you for the bug.

Hi could you check if the count is same or different to the core word count plug-in? Thank you for the bug report.

Hi. Thanks for reply.
I checked it. the count is same to the core work count plugin. both plugin count semi space as a seprate word.

This plugin tries to maintain the same word count plugin as the default. So if the default word count is count incorrectly please create a big report on the obsidian forums. When it is fixed, I will update. Sorry. I will keep this issue open until it the for the time being. Thank you.


Let all commenters reply to comments

Fixed https://github.com/potch/hax/issues/22.
Note - untested. @callahad
That condition had a nested condition for the "Edit" link, so it was basically -
if (a && b)
{
 /* do stuff */
 if (b)
 {
  /* do more stuff */
 }
}

That was probably just a mistake, so I followed the logic of https://github.com/potch/hax/blob/master/Hax/comments.php#L32.

@potch I can has review, please? :)


Remove clear latency derived statistic

Why?

because we found out it did not bring a lot of value to the user ?

Clear is asynchronous and not a CRUD operation.

Hum... Jsr107LatencyMonitor is used to provide latency to JSR107 JMX. It is a derived statistics. And it needs to be cleared.

we are not talking about the Jsr107LatencyMonitor (which by the way is not correct) - we are talking about the histogram stats that has been added recently with percentiles: Cache:ClearLatency#XXX

Then maybe the title and description of this issue should be improved :-)


Make tools/install-mypy exit on error.

Automated message from Dropbox CLA bot
@rwbarton, it looks like you've already signed the Dropbox CLA. Thanks!

Merged, after fixing this to use set -e for OpenBSD compatibility :)


Suggested source of entropy for random seed?

Moony is awesome! I've only just started using it to solve various midi filters. Thanks for creating it!
Since the os library isn't available for time(), do you have a suggested source of entropy to seed the random library with? Thanks! I've been working on this script: https://gist.github.com/ssj71/0e839f99e5163804ad005773827fc71b

On 26.10.22 20:55, ssj71 wrote:

Moony is awesome! I've only just started using it to solve various midi filters.

Glad you like it.

Since the |os| library isn't available for |time()|, do you have a suggested
source of entropy to seed the random library with?

Oh, I cannot think of anything inside Moony that could provide entropy. There's
definitely something missing here.
Some  time functions (gettimeofday, clock_gettime) are RT-safe, that could be an
option.
Maybe a dedicated function would even be better that would read from an RT-safe
ringbuffer filled up from /dev/(u)random.
 local seed = Entropy()

That sounds great! I was considering just using one of the handles passed into the once() function but that would be much better!

migrated to own forge https://todo.open-music-kontrollers.ch/~hp/issues/10


Document Viewer Search Highlight

HI, iam using mahapps metro plugin on my two apps, its great and working fine. I have added "DocumentViewer" to both the apps to view my reports in XPS and PDF format. DocumentViewer is doing great too, but when i try to search using inbuilt search of DocumentViewer, font doesnt get highlighted but page scrolls to searched term which means search is working but highlighting. When i tried to add DocumentViewer to simple app (without mahapps metro theming) it works absolutely fine without changing any code, so i think thers something messing up with highlighting text styling which i couldnt find. Can you guys help me out of it ? Thanx in advance.

Tried adding a WPF DocumentViewer with a XPS file created in MS Office Word 2016 and the search highlight worked (below). Does this happen in any specific situation? Can you send a sample XPS/PDF where this happens?


Enable instantiating component multiple times

Enable instantiating component multiple times as described in https://syn.tools/commodore/reference/architecture.html#_component_instantiation
use newest helm chart version 4.0.6

Checklist

[x] Keep pull requests small so they can be easily reviewed.
[x] Update the documentation.
[x] Update the ./CHANGELOG.md.

I updated the documentation and the CHANGELOG again to reflect the implementation changes.


Remove/unit of measurement

This PR will remove all need for us to determine what to display for the user. By reading the unit given to us by the API and telling HA what the NATIVE reading is, HA will automatically calculate and display the proper unit to the user.

You may want to install pre-commit on your local dev environment. It will fix minor code styling issued before the commit, so the git pipeline will not fail then.

Interestingly, I don’t usually work out of a “dev environment”. Most of my “play time” is spent on my ipad while sitting right on the couch. 🙈 I hope thats OK. I’ll try and work more in a teamate fashion using branches and PRs instead of straight master branch editing….

No problem at all, working from the browser should not be a problem.
I meant the GitHub check failures. The checks are mostly complaining about minor issues like extra space somewhere or unused import.
Let me think If I can fix it automatically since you are working from the browser.

Working this in #64


chore: manual rebase

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: KochC:x: GitHub ActionsGitHub Actions seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


chore: add more explicit-target logging

Codecov Report
Base: 63.93% // Head: 63.84% // Decreases project coverage by -0.10% :warning:

Coverage data is based on head (8827ef8) compared to base (50848d1).
Patch coverage: 39.00% of modified lines in pull request are covered.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1736      +/-   ##
==========================================
- Coverage   63.93%   63.84%   -0.10%
==========================================
  Files         132      132
  Lines       24481    24541      +60
==========================================
+ Hits        15651    15667      +16
- Misses       8830     8874      +44

Impacted Files
Coverage Δ

fedimint-core/src/lib.rs
90.53% <ø> (ø)

fedimint-server/src/config/mod.rs
62.27% <0.00%> (-1.06%)
:arrow_down:

fedimint-server/src/consensus/mod.rs
80.79% <0.00%> (-0.13%)
:arrow_down:

fedimint-server/src/logging.rs
54.09% <ø> (ø)

fedimint-core/src/task.rs
61.71% <16.66%> (-1.72%)
:arrow_down:

fedimint-server/src/net/peers.rs
88.00% <37.50%> (-3.62%)
:arrow_down:

fedimint-core/src/api.rs
80.35% <50.00%> (+0.17%)
:arrow_up:

fedimint-server/src/lib.rs
76.48% <60.00%> (-0.55%)
:arrow_down:

fedimint-core/src/db/mod.rs
88.41% <100.00%> (+0.04%)
:arrow_up:

... and 7 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


[Error message] Duplicate relation "access_list" in a relation expression. You should use "a.[b, c]" instead of "[a.b, a.c]"

Keep seeing this message in my Logs;
Duplicate relation "access_list" in a relation expression. You should use "a.[b, c]" instead of "[a.b, a.c]". This will cause an error in objection 2.0

Any idea what it means or how to get rid of it?
TIA

I had the same issue, the thing is one of my ACL was only one IP, and this IP was only present in another ACL.
Deleted this one ACL with one IP (didn't used it anymore), the warning disappeared.
From what I understand, use "a.[b, c]" instead of "[a.b, a.c]" means use ''rule a or rule b and c", instead of "rule a or b, and rule a or c" (simple boolean writing)

I had the same issue, the thing is one of my ACL was only one IP, and this IP was only present in another ACL. Deleted this one ACL with one IP (didn't used it anymore), the warning disappeared.
From what I understand, use "a.[b, c]" instead of "[a.b, a.c]" means use ''rule a or rule b and c", instead of "rule a or b, and rule a or c" (simple boolean writing, but don't hesitate to correct me if I'm wrong)

I'm sorry but I don't really understand what you mean. I have only one ACL for LAN ONLY access that contain this:
allow <IP_ADDRESS>/24
deny all (grayed out auto fill)

nothing further in there, and also no other ACL.
So I don't see how to correct this..

I had the same issue, the thing is one of my ACL was only one IP, and this IP was only present in another ACL. Deleted this one ACL with one IP (didn't used it anymore), the warning disappeared.

From what I understand, use "a.[b, c]" instead of "[a.b, a.c]" means use ''rule a or rule b and c", instead of "rule a or b, and rule a or c" (simple boolean writing, but don't hesitate to correct me if I'm wrong)

I'm sorry but I don't really understand what you mean. I have only one ACL for LAN ONLY access that contain this:

allow <IP_ADDRESS>/24

deny all (grayed out auto fill)

nothing further in there, and also no other ACL.
So I don't see how to correct this..

So, you're not in the same situation I was in, unfortunately my fix can't work for you...

I am getting the same error.  Same situation.  New install.  Can someone please provide some insight?

I am getting the same error. Same situation. New install. Can someone please provide some insight?

I recall never really getting around this, and switching to Caddy. Good luck.

I am getting the same error. Same situation. New install. Can someone please provide some insight?

I recall never really getting around this, and switching to Caddy. Good luck.

Thank you...  I'm switching now.  NPM just keeps having issues out of the blue, no solutions found just unresolved bugs.

Issue is now considered stale. If you want to keep it open, please comment 👍

The developer should be considered stale!


Add newTabWarning to Footer and update docs

What is the context of this PR?
This PR adds the ability to add a warning at the top of the Footer to warn users that links will open in a new tab and updates docs to reflect this.
It also:

updates the Footer docs to add a recent new param (crest)
fixes issues setting footerWarning and copyrightDeclaration using pageConfig
removes unused param serviceLinks from pageConfig
updates pageConfig docs for Footer
renames poweredByONS to poweredBy in docs

How to review

Check footer with warning line looks as it should
Check docs are up to date

 Add 'opens in new tab' lead-in line to footer

Inline footer links seem to be stacking too soon.  Should stack to a single column until they need to.  This example shows there's plenty of room to allow them to sit inline at this width.


ARROW2: fix csv schema inference records to read calculation

Describe the bug
Now that we have upstream support in https://github.com/jorgecarleitao/arrow2/pull/765, we should address the FIXME in datasource/file_format/csv.rs.
Additional context
This is a left over item for the arrow2 branch.

PR: https://github.com/jorgecarleitao/arrow2/pull/765

fixed by @Igosuki :rocket:


Standardize formatting of manifests/y/YaSuenag/SimpleCom/1.2.3

Microsoft Reviewers: Open in CodeFlow

@wingetbot run


[BUGCHECK] "Depth" capture cannot be performed.

How to set the screenshot function to capture depth image? Which effect needs to be enabled?

How to set the screenshot function to capture depth image? Which effect needs to be enabled?

Users are not required to do anything, but this is the bug check, so an unexpected error has caused somewhere.
Open the ReShade.log file and find all lines containing [Screenshot] and paste them here.
If these lines contain private information (e.g. user name or other), mask part of them.
C:\Users\cot6\Desktop
↓
C:\Users\********\Desktop

If no errors are logged, try using the experimental version from the following and reproduce the bug check.
https://ci.appveyor.com/project/cot6/reshade-addons/builds/49314638/artifacts
for 64-bit: ReShade64-Screenshot-By-seri14.zip

I tried using the experimental version, but there were no error records in the ReShade.log file.
Will the screenshot capture feature for normal images be added?

I tried using the experimental version, but there were no error records in the ReShade.log file.

That is odd. Maybe the add-on controlled effect has not compiled.
Currently I have no idea where the program make reach the state to cause bug check.
The effect file is located here.
%ALLUSERSPROFILE%\ReShade Addons\Seri\reshade-shaders\Shaders\__Addon_ScreenshotDepth_Seri14.addonfx

Will the screenshot capture feature for normal images be added?

No, but you can modify the add-on controlled effect.
Copy the file %ALLUSERSPROFILE%\ReShade Addons\Seri\reshade-shaders\Shaders\__Addon_ScreenshotDepth_Seri14.addonfx to %ALLUSERSPROFILE%\ReShade Addons\Seri\reshade-shaders\Shaders\__Addon_ScreenshotDepth_Seri14.addonfxgen and edit it.
Then the depth image will be saved as you modified it. (Require to restart the application)

@cot6
I'm having this same problem, I thought it was something wrong with my setup until I finally tried A/B testing v10.6.1 and 10.5.1.
So I was in the middle of writing out my settings when I think I stumbled on the problem here. Depth screenshots were not working for me sometimes in some games, and were working in others. Sometime switching versions helped, it seemed random.
Finally I noticed this setting that had been populated automatically, and once I saw it, removing this extra search path broke the depth screenshots, and re-adding it manually fixed them (naturally).

It looks like this setting and file are supposed to populate automatically, but I think this means is that the placement of the __Addon_ScreenshotDepth_Seri14.addonfx file as well as being able to find that file (auto adding the search path) is very fickle, and it doesn't work pretty often. And when it doesn't resolve, you can still take screenshots normally but the Depth image doesn't appear, silently failing with no warning, no logs or messages in Reshade.log that I can see.
I think @L020304 could try simply locating __Addon_ScreenshotDepth_Seri14.addonfx on their system and add that effect search path in their config to see if that was the issue, but for a larger fix, making the install process, or error messages more robust, or at least adding a note in the Readme about making sure that file can be found would help.

Thanks for the addon by the way, I have been looking for this exact "Screenshot + Depth Buffer" functionality for so long in a combined package, I love it.

The behavior of the Screenshot add-on has been changed to solve this issue.
See: https://github.com/cot6/reshade-addons/commit/b808136a714eff02f93e8df06d3850e8af65d805
Please get the binary from the following URL.
https://ci.appveyor.com/project/cot6/reshade-addons/builds/49499492/artifacts

The initial process that modifies the ReShade configuration file is still not correctly (Timing issue) because API not supported the some features.
I will make some suggestions for the next ReShade release.
So this issue will not be fixed immediately, but the depth capture feature will work if you manually set the Effect Search Paths, so please do that for now.

A useful option in this case is Turn On Effects.
Ignore
This feature does nothing. This is the default option.
While myset is active
Turn on effects state if it is disabled when taking screenshot.
When myset is deactivated, this feature will turn off the effects state if the state was switched to enabled.
When activate myset
Turn on effects state if it is disabled when activating myset.
When myset is deactivated, this feature will turn off the effects state if the state was switched to enabled.

I think @L020304 could try simply locating __Addon_ScreenshotDepth_Seri14.addonfx on their system and add that effect search path in their config to see if that was the issue, but for a larger fix, making the install process, or error messages more robust, or at least adding a note in the Readme about making sure that file can be found would help.

Thank you, （"Depth" capture cannot be performed） does not appear again.
But the tiff file is completely white. Screenshot  not running correctly.

If you have succeeded to create the TIFF file, the depth map should be captured. (If you can see the shadow in DisplayDepth effect on scene.)
Note the picture you can see on TIFF is not the same as with ReShade. The depth buffer data needs to be adjusted for the formula of the effects, but the saving data is not required.
Use a viewer or editor that has the ability to handle floating point color.
If the software can handle only 8-bit depth integer color, it may look like filled image.

That's right, the scaling for the depth buffer is very high. I worked on a script to reprocess the depth maps to be more like what DisplayDepth outputs.
Screen

Depth before

Depth after

rescale_tiff_depth.py
import argparse

from PIL import Image
import numpy as np

def parse_args():
    parser = argparse.ArgumentParser(description="Process a depth map stored in a TIFF file.")
    parser.add_argument("--input-path", "--i", type=str, required=True, help="The path to the input depth map TIFF file.")
    parser.add_argument("--output-path", "--o", type=str, required=False, default='', help="The path to save the processed depth map PNG file.")
    parser.add_argument("--no-invert", action="store_true", help="Do not invert the depth map.")
    parser.add_argument("--far-plane", type=float, default=1000.0, help="The far plane distance for the depth map.")

    return parser.parse_args()

def main(cmd_args: argparse.Namespace):

    input_path = cmd_args.input_path
    far_plane = 1000.0
    skip_invert = cmd_args.no_invert

    if cmd_args.output_path:
        output_path = cmd_args.output_path
    else:
        output_path = input_path.replace(".tiff", ".png")

    depth_map = Image.open(input_path)
    depth_array = np.array(depth_map)

    # Linearize
    linearized_depth = 1.0 / (1.0 - depth_array / far_plane)

    # Normalize
    normalized_depth = (linearized_depth - np.min(linearized_depth)) / (np.max(linearized_depth) - np.min(linearized_depth))

    # Invert
    if not skip_invert:
        normalized_depth = 1.0 - normalized_depth

    # Convert back to an image
    processed_depth_map = Image.fromarray((normalized_depth * 255).astype(np.uint8))

    processed_depth_map.save(output_path, compression=3)

    print(f"Saved processed depth map to: {output_path}")

if __name__ == '__main__':
    cmd_args = parse_args()
    main(cmd_args)

usage: rescale_tiff_depth.py [-h] --input-path INPUT_PATH [--output-path OUTPUT_PATH] [--no-invert]
                                   [--far-plane FAR_PLANE]

Process a depth map stored in a TIFF file.

options:
  -h, --help            show this help message and exit
  --input-path INPUT_PATH, --i INPUT_PATH
                        The path to the input depth map TIFF file.
  --output-path OUTPUT_PATH, --o OUTPUT_PATH
                        The path to save the processed depth map PNG file.
  --no-invert           Do not invert the depth map.
  --far-plane FAR_PLANE
                        The far plane distance for the depth map.

Used like (process in place and save to new .png, warning will overwrite if one exists already)
python rescale_tiff_depth.py -i path/to/image.tiff
or output to a new file
python rescale_tiff_depth.py -i "path/to/image.tiff" -o "path/to/output.png" --no-invert

Fixed.
Get updates from the following links.
ReShade
Binaries (Unofficial build): https://ci.appveyor.com/project/cot6/reshade/builds/49564258/artifacts
Commit: https://github.com/crosire/reshade/commit/358f45439edfbebd2fc91b4c135c762d16f7ad56
Add-ons (Preview)
Binaries: https://ci.appveyor.com/project/cot6/reshade-addons/builds/49564757/artifacts
Commit: https://github.com/cot6/reshade-addons/commit/18b11caa87e40c7dcb75a9067829a226c6076a12


Unhandled error during execution of scheduler flush

Vue version
3.2.38
Link to minimal reproduction
https://vkceyugu.cdn.bspapp.com/VKCEYUGU-fba5454d-3a27-4ea5-babf-ea74eb4644f5/649066cc-3178-4393-8064-b59df4403c29.zip
Steps to reproduce

npm install
npm run dev:test
open browser and console, login, username=admin, password=123456
then click left menu, goto Documentation page, open the file: src/views/documentation/index.vue
goto line 23, modify title value '基础篇1' to trigger hot reload
the console error: Unhandled error during execution of scheduler flush

What is expected?
no error, can hot reload
What is actually happening?
the console error: Unhandled error during execution of scheduler flush
Uncaught (in promise) TypeError: parentComponent.ctx.deactivate is not a function
System Info
No response
Any additional comments?
No response

#6222
use the latest version of chrome could avoid this bug temporarily

Duplicate of #6222

Please don’t send zips for reproductions. Host it on GitHub


🐛 [firebase_firestore] Flutter app crashes when opening a screen with a query in firestore/Storage only in iOS

Bug report
Describe the bug
I have a flutter app, and in one screen, I'm getting content from storage and firestore. With this code I'm able to see a list of links from firestore (if exists) and the list of images from storage (the controller do it together as you can see below)
myScreen.dart
//I have another method similar to this to show the content inside folders (read below)
loadBasicStructure(submenu ,callback, context){
    return FutureBuilder(
      future: ctrl.loadMain(submenu),
      builder: (ctx, snapshot) {
        if (snapshot.connectionState == ConnectionState.waiting) {
          return Center(child: CircularProgressIndicator());
        } else if (snapshot.error != null) {
          print(snapshot.error);
          return Center(child: Text('Error'));
        } else {
          return ListView.separated(
            separatorBuilder: (context, index) {
              return Divider();
            },
            itemCount: snapshot.data.length,
            itemBuilder: (ctx, i) {
              ItemLab item = snapshot.data[i];
              print(item);
              return ListTile(
                  leading: setIconLab(item),
                  title: Text(setText(item)),
                  trailing: Icon(Icons.chevron_right),
                  onTap: () async {
                      //mycode
                    }
                  },
                );
            },
          );
        }
      },
    );
  }

controller.dart

Future<List<ItemLab>> loadLab(String submenu) async {
    var snapshot = await _storage.ref().child(submenu);
    var ret = await snapshot.listAll();
    List<ItemLab> content = [];
    ret.prefixes.forEach((element) {
      content add(
        Item(
          type: 'FOLDER',
          elemento: element,
        ),
      );
    });

    ret.items.forEach((element) {
      print(element);
      conteudo.add(
        ItemLab(
          type: 'FILE',
          element: element,
        ),
      );
    });

    try{
      await databaseReference
          .collection("${submenu}_url")
          .get()
          .then((snapshot)  {

        snapshot.docs.forEach((f) async => {
          f.data().isNotEmpty ?
            content.add(
              Item(
                type: 'VIDEO',
                element: null,
                video: f.data(),
              ),
            )
          :
            null

        });
      });

      await databaseReference
          .collection("${submenu}_url")
          .doc("Folder")
          .collection("RESET")
          .get()
          .then((snap){
        snap.docs.isNotEmpty ? conteudo.add(
          Item(
              type: FOLDER',
              folderVideo: "RESET"
          ),
        ) : null;
      });

    }catch(e){
      print(e);
    }

    return Future.value(content);
  }

How this code works:
I have a screen with a list of submenus EX ("submenu1, submenu2"). Inside it I check in storage if in that folder ("submenu1") it has some files and if yes I show in a list, if has not I check if there are folders inside this one ("submenu1 > RESET"). If it exists I show the folder.
I do this exact check too in firestore (because there I store some links from youtube). In this case the error is throwing because I have another screen with only items from firestore (and with the same structure) and it works normally, but in the submenu "photos" when I click in the folder "RESET" the app crashes and show this error:
2021-02-01 09:28:22.129581-0300 Runner[391:22847] An error occurred while parsing query arguments, this is most likely an error with this SDK. (
	0   CoreFoundation                      0x0000000189afb9ec F3021642-E3C0-33F8-9911-DD303A6056D0 + 1157612
	1   libobjc.A.dylib                     0x000000019de81b54 objc_exception_throw + 56
	2   Runner                              0x00000001023cddf0 _ZN8firebase9firestore4util16ObjcThrowHandlerENS1_13ExceptionTypeEPKcS4_iRKNSt3__112basic_stringIcNS5_11char_traitsIcEENS5_9allocatorIcEEEE + 304
	3   Runner                              0x00000001023cd90c _ZN8firebase9firestore4util5ThrowENS1_13ExceptionTypeEPKcS4_iRKNSt3__112basic_stringIcNS5_11char_traitsIcEENS5_9allocatorIcEEEE + 20
	4   Runner                              0x00000001023dfcf8 _ZN8firebase9firestore4util20ThrowInvalidArgumentIJNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEEmEEEvPKcDpRKT_ + 48
	5   Runner                              0x00000001023b8238 _ZN8firebase9firestore3api19CollectionReferenceC2ENS0_5model12ResourcePathENSt3__110shared_ptrINS1_9FirestoreEEE + 716
	6   Runner                              0x00000001023e6684 _ZN8firebase9firestore3api9Firestore13GetCollectionERKNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEE + 104
	7   Runner                              0x00000001023fc0a4 -[FIRFirestore collectionWithPath:] + 112
	8   Runner                              0x00000001025319b4 -[FLTFirebaseFirestoreReader FIRQuery] + 344
	9   Runner                              0x0000000102531428 -[FLTFirebaseFirestoreReader readValueOfType:] + 1272
	10  Flutter                             0x0000000103914130 Flutter + 5685552
	11  Runner                              0x0000000102530fdc -[FLTFirebaseFirestoreReader readValueOfType:] + 172
	12  Flutter                             0x000000010391564c Flutter + 5690956
	13  Flutter                             0x00000001039121dc Flutter + 5677532
	14  Flutter                             0x00000001033d9f9c Flutter + 204700
	15  Flutter                             0x00000001036b1d88 Flutter + 3186056
	16  Flutter                             0x00000001033e342c Flutter + 242732
	17  Flutter                             0x00000001033e53a0 Flutter + 250784
	18  CoreFoundation                      0x0000000189a7c3e0 F3021642-E3C0-33F8-9911-DD303A6056D0 + 635872
	19  CoreFoundation                      0x0000000189a7bfe4 F3021642-E3C0-33F8-9911-DD303A6056D0 + 634852
	20  CoreFoundation                      0x0000000189a7b4c4 F3021642-E3C0-33F8-9911-DD303A6056D0 + 632004
	21  CoreFoundation                      0x0000000189a75850 F3021642-E3C0-33F8-9911-DD303A6056D0 + 608336
	22  CoreFoundation                      0x0000000189a74ba0 CFRunLoopRunSpecific + 572
	23  GraphicsServices                    0x00000001a07dd598 GSEventRunModal + 160
	24  UIKitCore                           0x000000018c3662f4 CC6E5AC7-8248-35F6-8B42-2E25C93DCF0A + 11723508
	25  UIKitCore                           0x000000018c36b874 UIApplicationMain + 164
	26  Runner                              0x000000010239af80 main + 64
	27  libdyld.dylib                       0x0000000189753568 0B475C78-3C12-3121-B7F8-2B95B83DAF44 + 5480
)
2021-02-01 09:28:22.130980-0300 Runner[391:22847] -[NSNull getDocumentsWithSource:completion:]: unrecognized selector sent to instance 0x1d9269ee0
2021-02-01 09:28:22.132261-0300 Runner[391:22847] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[NSNull getDocumentsWithSource:completion:]: unrecognized selector sent to instance 0x1d9269ee0'
*** First throw call stack:
(0x189afb9d8 0x19de81b54 0x189a0bbbc 0x189afe01c 0x189afff8c 0x102530514 0x10252e9d0 0x103912224 0x1033d9f9c 0x1036b1d88 0x1033e342c 0x1033e53a0 0x189a7c3e0 0x189a7bfe4 0x189a7b4c4 0x189a75850 0x189a74ba0 0x1a07dd598 0x18c3662f4 0x18c36b874 0x10239af80 0x189753568)
libc++abi.dylib: terminating with uncaught exception of type NSException
*** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[NSNull getDocumentsWithSource:completion:]: unrecognized selector sent to instance 0x1d9269ee0'
terminating with uncaught exception of type NSException

Expected behavior
I expect that when I tap in the folder inside submenu, it opens a list of links (like it do in the other submenu that only have storage files

Additional context
The problem only occurs in iOS... In android it works fine

Flutter doctor
Run flutter doctor and paste the output below:
Click To Expand
[✓] Flutter (Channel stable, 1.22.5, on Mac OS X 10.15.7 19H114 darwin-x64, locale pt-BR)

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)

[✓] Xcode - develop for iOS and macOS (Xcode 12.4)
[!] Android Studio (version 4.1)
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
[✓] VS Code (version 1.52.1)
[✓] Connected device (1 available)

Flutter dependencies
Run flutter pub deps -- --style=compact and paste the output below:
Click To Expand
Dart SDK 2.10.4
Flutter SDK 1.22.5
app_md 1.0.0+1

dependencies:
- cached_network_image 2.5.0 [flutter flutter_cache_manager octo_image]
- carousel_slider 2.3.1 [flutter]
- cloud_firestore 0.16.0 [flutter meta quiver firebase_core firebase_core_platform_interface cloud_firestore_platform_interface cloud_firestore_web]
- cpfcnpj 1.0.3
- cupertino_icons 1.0.0
- dio 3.0.10 [http_parser path]
- email_validator 1.0.6
- firebase_core 0.7.0 [firebase_core_platform_interface flutter quiver meta firebase_core_web]
- firebase_messaging 8.0.0-dev.14 [meta flutter firebase_core firebase_core_platform_interface firebase_messaging_platform_interface firebase_messaging_web]
- firebase_storage 7.0.0 [flutter firebase_core firebase_core_platform_interface firebase_storage_web firebase_storage_platform_interface quiver]
- flutter 0.0.0 [characters collection meta typed_data vector_math sky_engine]
- flutter_barcode_scanner 1.0.1 [flutter flutter_plugin_android_lifecycle]
- flutter_local_notifications 3.0.1 [flutter platform flutter_local_notifications_platform_interface timezone]
- flutter_localizations 0.0.0 [flutter intl characters collection meta path typed_data vector_math]
- flux_validator_dart 1.0.3
- hive 1.4.4 [meta crypto]
- hive_flutter 0.3.1 [flutter hive path_provider path]
- http 0.12.2 [http_parser path pedantic]
- image_picker 0.6.7+14 [flutter flutter_plugin_android_lifecycle image_picker_platform_interface]
- intl 0.16.1 [path]
- mask_shifter 0.1.0 [flutter]
- mask_text_input_formatter 1.1.0 [flutter]
- open_file 3.0.3 [flutter ffi]
- path_provider 1.6.27 [flutter path_provider_platform_interface path_provider_macos path_provider_linux path_provider_windows]
- pdf 1.12.0 [archive barcode crypto image meta path_parsing utf vector_math]
- permission_handler 5.0.1+1 [flutter meta permission_handler_platform_interface]
- printing 3.7.1 [flutter flutter_web_plugins image js meta pdf plugin_platform_interface]
- provider 4.3.2+2 [flutter nested collection]
- qr_code_scanner 0.0.13 [flutter]
- shared_preferences 0.5.12+4 [meta flutter shared_preferences_platform_interface shared_preferences_linux shared_preferences_macos shared_preferences_web shared_preferences_windows]
- table_calendar 2.3.1 [flutter intl simple_gesture_detector]
- url_launcher 5.7.10 [flutter url_launcher_platform_interface url_launcher_web url_launcher_linux url_launcher_macos url_launcher_windows]

dev dependencies:
- build_runner 1.10.0 [args async build build_config build_daemon build_resolvers build_runner_core code_builder collection crypto dart_style glob graphs http_multi_server io js logging meta mime path pedantic pool pub_semver pubspec_parse shelf shelf_web_socket stack_trace stream_transform timing watcher web_socket_channel yaml]
- flutter_test 0.0.0 [flutter test_api path fake_async clock stack_trace vector_math async boolean_selector characters charcode collection matcher meta source_span stream_channel string_scanner term_glyph typed_data]
- hive_generator 0.7.2+1 [build source_gen hive analyzer dartx]

transitive dependencies:
- _fe_analyzer_shared 6.0.0 [meta]
- analyzer 0.39.14 [_fe_analyzer_shared args charcode cli_util collection convert crypto glob html meta package_config path pub_semver source_span watcher yaml]
- archive 2.0.13 [crypto args path]
- args 1.6.0
- async 2.5.0-nullsafety.1 [collection]
- barcode 1.17.1 [meta qr]
- boolean_selector 2.1.0-nullsafety.1 [source_span string_scanner]
- build 1.3.0 [analyzer async convert crypto logging meta path glob]
- build_config 0.4.2 [checked_yaml json_annotation meta path pubspec_parse yaml]
- build_daemon 2.1.4 [built_collection built_value http_multi_server logging pedantic path pool shelf shelf_web_socket stream_transform watcher web_socket_channel]
- build_resolvers 1.3.11 [analyzer build crypto graphs logging path package_config pub_semver]
- build_runner_core 5.2.0 [async build build_config build_resolvers collection convert crypto glob graphs json_annotation logging meta path package_config pedantic pool timing watcher yaml]
- built_collection 4.3.2 [collection quiver]
- built_value 7.1.0 [built_collection collection fixnum quiver]
- characters 1.1.0-nullsafety.3
- charcode 1.2.0-nullsafety.1
- checked_yaml 1.0.2 [json_annotation source_span yaml]
- cli_util 0.1.4 [path]
- clock 1.1.0-nullsafety.1
- cloud_firestore_platform_interface 3.0.1 [flutter meta collection firebase_core plugin_platform_interface]
- cloud_firestore_web 0.3.0+1 [flutter flutter_web_plugins http_parser meta firebase_core firebase_core_web cloud_firestore_platform_interface js]
- code_builder 3.4.1 [built_collection built_value collection matcher meta]
- collection 1.15.0-nullsafety.3
- convert 2.1.1 [charcode typed_data]
- crypto 2.1.5 [collection convert typed_data]
- csslib 0.16.2 [source_span]
- dart_style 1.3.6 [analyzer args path source_span]
- dartx 0.5.0 [collection path crypto characters time meta]
- fake_async 1.2.0-nullsafety.1 [clock collection]
- ffi 0.1.3
- file 5.2.1 [intl meta path]
- firebase_core_platform_interface 3.0.1 [flutter meta plugin_platform_interface quiver]
- firebase_core_web 0.2.1+3 [firebase_core_platform_interface flutter flutter_web_plugins meta js]
- firebase_messaging_platform_interface 1.0.0-dev.10 [flutter meta firebase_core plugin_platform_interface]
- firebase_messaging_web 0.1.0-dev.5 [flutter flutter_web_plugins meta firebase_core firebase_core_web firebase_messaging_platform_interface js service_worker]
- firebase_storage_platform_interface 1.0.4 [flutter meta collection firebase_core plugin_platform_interface]
- firebase_storage_web 0.1.1+3 [async crypto firebase_core firebase_core_web firebase_storage_platform_interface flutter flutter_web_plugins http js meta]
- fixnum 0.10.11
- flutter_blurhash 0.5.0 [flutter meta]
- flutter_cache_manager 2.1.1 [flutter path_provider uuid http path sqflite pedantic clock file rxdart image]
- flutter_local_notifications_platform_interface 2.0.0 [flutter plugin_platform_interface]
- flutter_plugin_android_lifecycle 1.0.11 [flutter]
- flutter_web_plugins 0.0.0 [flutter characters collection meta typed_data vector_math]
- glob 1.2.0 [async collection node_io path pedantic string_scanner]
- graphs 0.2.0
- html 0.14.0+3 [csslib source_span]
- http_multi_server 2.2.0 [async]
- http_parser 3.1.4 [charcode collection source_span string_scanner typed_data]
- image 2.1.19 [archive xml meta]
- image_picker_platform_interface 1.1.1 [flutter meta http plugin_platform_interface]
- io 0.3.4 [charcode meta path string_scanner]
- js 0.6.2
- json_annotation 3.1.0
- logging 0.11.4
- matcher 0.12.10-nullsafety.1 [stack_trace]
- meta 1.3.0-nullsafety.3
- mime 0.9.7
- nested 0.0.4 [flutter]
- node_interop 1.1.1 [js]
- node_io 1.1.1 [node_interop path]
- octo_image 0.3.0 [flutter flutter_blurhash]
- package_config 1.9.3 [path charcode]
- path 1.8.0-nullsafety.1
- path_parsing 0.1.4 [vector_math meta]
- path_provider_linux 0.0.1+2 [path xdg_directories path_provider_platform_interface flutter]
- path_provider_macos 0.0.4+4 [flutter]
- path_provider_platform_interface 1.0.3 [flutter meta platform plugin_platform_interface]
- path_provider_windows 0.0.4+1 [path_provider_platform_interface meta path flutter ffi win32]
- pedantic 1.9.0
- permission_handler_platform_interface 2.0.1 [flutter meta plugin_platform_interface]
- petitparser 3.1.0 [meta]
- platform 2.2.1
- plugin_platform_interface 1.0.2 [meta]
- pool 1.4.0 [async stack_trace]
- process 3.0.13 [file intl meta path platform]
- pub_semver 1.4.4 [collection]
- pubspec_parse 0.1.5 [checked_yaml json_annotation pub_semver yaml]
- qr 1.3.0 [meta]
- quiver 2.1.3 [matcher meta]
- rxdart 0.24.1
- service_worker 0.2.4 [js]
- shared_preferences_linux 0.0.2+4 [file flutter meta path path_provider_linux shared_preferences_platform_interface]
- shared_preferences_macos 0.0.1+11 [shared_preferences_platform_interface flutter]
- shared_preferences_platform_interface 1.0.4 [meta flutter]
- shared_preferences_web 0.1.2+7 [shared_preferences_platform_interface flutter flutter_web_plugins meta]
- shared_preferences_windows 0.0.1+3 [shared_preferences_platform_interface flutter ffi file meta path path_provider_platform_interface path_provider_windows]
- shelf 0.7.9 [async collection http_parser path stack_trace stream_channel]
- shelf_web_socket 0.2.3 [shelf web_socket_channel stream_channel]
- simple_gesture_detector 0.1.4 [flutter]
- sky_engine 0.0.99
- source_gen 0.9.7+1 [analyzer async build dart_style glob meta path pedantic source_span]
- source_span 1.8.0-nullsafety.2 [charcode collection path term_glyph]
- sqflite 1.3.2+1 [flutter sqflite_common path]
- sqflite_common 1.0.2+1 [synchronized path meta]
- stack_trace 1.10.0-nullsafety.1 [path]
- stream_channel 2.1.0-nullsafety.1 [async]
- stream_transform 1.2.0
- string_scanner 1.1.0-nullsafety.1 [charcode source_span]
- synchronized 2.2.0+2
- term_glyph 1.2.0-nullsafety.1
- test_api 0.2.19-nullsafety.2 [async boolean_selector collection meta path source_span stack_trace stream_channel string_scanner term_glyph matcher]
- time 1.3.0
- timezone 0.5.9 [path]
- timing 0.1.1+2 [json_annotation]
- typed_data 1.3.0-nullsafety.3 [collection]
- url_launcher_linux 0.0.1+4 [flutter]
- url_launcher_macos 0.0.1+9 [flutter]
- url_launcher_platform_interface 1.0.9 [flutter meta plugin_platform_interface]
- url_launcher_web 0.1.5+1 [url_launcher_platform_interface flutter flutter_web_plugins meta]
- url_launcher_windows 0.0.1+3 [flutter]
- utf 0.9.0+5
- uuid 2.2.2 [crypto convert]
- vector_math 2.1.0-nullsafety.3
- watcher 0.9.7+15 [async path pedantic]
- web_socket_channel 1.1.0 [async crypto stream_channel]
- win32 1.7.3 [ffi]
- xdg_directories 0.1.0 [path process flutter]
- xml 4.5.1 [collection convert meta petitparser]
- yaml 2.2.1 [charcode collection string_scanner source_span]

I don't know if i'm being clear, if there are a question, tell me
Xcode Error Print

Hi @GilbertoGmobil
This sounds like complicated use case. Could you provide the query that is throwing the error or a minimal complete reproducible code sample?
Thank you

Could I send it privately? I forgot to tell that the error only appear in iOS building from xcode. In android it works well

Would you be able to boil this down to a minimal complete reproducible code sample where you only execute the exact query that crashes the app (for example with just an ElevatedButton) so that we could try it as well?

The code I typed in the question is what you need. I'm justing referencing the loadBasicStructure at the body of my scaffold

I tried with xcode 12.3 too and still doesn't working

The code you provided contains no imports or definitions for all of the widgets you are using. Without a complete example we would just have to guess what kind of implementation you are using and we have no way of debugging the issue.

I changed a bit my query and now its working, if it persists I open again


Fix case-sensitive search term in example.

Converting both the name and search value to lowercase solves case sensitive searching.

thanks!


[Bug]:  Chromium issues "Aw snap" error code 5 page while navigating pages and grabbing urls from those pages

Bug expectation
for most urls we scrape the code works fine.
for a few urls chromium issues the  "Aw snap" error code 5 page  and seems to hang the puppeteer script.
I am guessing chromium isnt returning back to puppeteer
The chromium issue happens in a different page in each script run on the await page.$x
console.log("before jobLookUpCriteria: " + jobLookUpCriteria);
let urlElements = await page.$x(jobLookUpCriteria);
console.log("after jobLookUpCriteria: " + jobLookUpCriteria);
Here is the end of the output
exact numeric page match//a[.='36'] length[1]
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
<--- Last few GCs --->
[66116:0x130009a0000]   147366 ms: Scavenge 285.3 (308.8) -> 279.8 (309.3) MB, 1.18 / 0.00 ms  (average mu = 0.989, current mu = 0.986) task;
[66116:0x130009a0000]   147853 ms: Scavenge 285.6 (309.3) -> 280.2 (309.5) MB, 1.00 / 0.00 ms  (average mu = 0.989, current mu = 0.986) task;
[66116:0x130009a0000]   148452 ms: Scavenge 286.1 (309.8) -> 280.9 (310.3) MB, 1.28 / 0.00 ms  (average mu = 0.989, current mu = 0.986) task;
<--- JS stacktrace --->
[66116:259:0215/083233.334034:ERROR:v8_initializer.cc(750)] V8 process OOM (Oilpan: Reserving memory.).
before jobLookUpCriteria: //a
Bug behavior

[X] Flaky
[ ] PDF

Minimal, reproducible example
const puppeteer = require("puppeteer");

(async () => {
const options = {
    headless: false,
    dumpio: true,
    args: [
        "--no-sandbox",
        "--disable-setuid-sandbox",
        "--disable-dev-shm-usage"
    ]
};
const browser = await puppeteer.launch(options);

let page = await browser.newPage();

await page.setDefaultNavigationTimeout(0); // A value of 0 means an unlimited amount of time

// set user agent (override the default headless User Agent)
await page.setUserAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4427.0 Safari/537.36');

// indicate we are a desktop page
const device_width = 1920;
const device_height = 1080;
await page.setViewport({width: device_width, height: device_height})

page.on('console', consoleObj => console.log(consoleObj.text()));

let url = "https://jobs.vioc.com/";
//let url =  "https://17012017.staging.softgarden.io/es/vacancies";
let response;

console.log('url is ' + url);

// 1st try a goto waitUntil networkidle0
try {
    response = await page.goto(url, {timeout:  45000, waitUntil: "networkidle0"});
} catch (err) {
    // 2nd try a goto waitUntil load
    console.log("Failed to connect to url:" + url + " with networkidle0 error:" + err + " will try goto waitUntil load");
    try {
        response = await page.goto(url, {timeout: 45000, waitUntil: "load"});
    } catch (err) {
        // at this point we give up
        console.log("Failed to connect to url:" + url + " error:" + err);
        throw err;
    }
}

console.log(`url is ${url} opened`);

let companyCrawlerDto =  await crawlByCompany(
    url,
    page
);

console.log("NumCrawledUrls:" + companyCrawlerDto.getNumCrawledUrls());
console.log("NumPagesCrawled:" + companyCrawlerDto.getNumPagesCrawled());
console.log("CrawledUrls:" + companyCrawlerDto.getCrawledUrls());

if (page) {
    await page.close();
}
if (browser) {
    await browser.close();
}
})();

const crawlByCompany = async (
    url,
    page
) => {

    let dto = new CompanyCrawlerDto();
    dto.setUrl(url);
    let crawledUrls = [];
    let pageNum = 0;
    let isPagePaginated = false;
    let expectedNumPages = 0;
    let expectedNumJobs = null;
    let jobLookUpCriteria = "//a";
    let paginationStrategyName = null;
    let redirectedUrl = null;

    try {
        // record the final url we may have been redirected to
        redirectedUrl = await page.url();

        let extractUrlsResult = await extractUrls(page,
            jobLookUpCriteria,
            2500
        );

        pageNum = extractUrlsResult.numPagesCrawled;
        crawledUrls = extractUrlsResult.crawledUrls;
        paginationStrategyName = extractUrlsResult.paginationStrategyName;
        expectedNumPages = extractUrlsResult.expectedNumPages

    } catch (error) {
        console.log("companycrawler.crawlByCompany() url: " + url + "error: ", error);
        if (error instanceof PuppeteerLaunchError) {
            throw error;
        }
        dto.addError(error.message);
    }

    dto.setNumPagesCrawled(pageNum);
    dto.setExpectedNumPages(expectedNumPages);
    dto.setExpectedNumJobs(expectedNumJobs);
    dto.setNumCrawledUrls(crawledUrls.length);
    dto.setCrawledUrls(crawledUrls);
    dto.setPaginationStrategyName(paginationStrategyName);
    dto.setIsPagePaginated(isPagePaginated);
    dto.setRedirectedUrl(redirectedUrl)
    return dto;
};

async function extractUrls (page,
                            jobLookUpCriteria,
                            timeout
) {
    let crawledUrls = [];
    let pageNum = 0;
    let nextPage = null;
    let nextPageElementHandle = null;

    //expected no pages here based on flag - no for google bing but yes for company crawler
    do {
        pageNum++;

        nextPage = null;
        let crawledUrlsFromCurrentPage = await getCrawledUrlsByCriteria(
            page,
            jobLookUpCriteria
        );

        // filter out non job urls
        crawledUrlsFromCurrentPage = await filterUrls(crawledUrlsFromCurrentPage)

        if (compareArraysInclude(crawledUrlsFromCurrentPage, crawledUrls)) {
            console.log("Current Page urls match existing collected urls, terminating pagination");
            let extractUrlsResult = {
                numPagesCrawled: pageNum - 1,
                crawledUrls: [...new Set(crawledUrls)]
            };
            return extractUrlsResult;
        }

        crawledUrls.push(...crawledUrlsFromCurrentPage);

        // get the next numeric page element
        nextPageElementHandle = await getNextPageElementHandle(
            page,
            pageNum
        );

        if (nextPageElementHandle) {
            nextPage = true;

            // companycrawling use click or navigation promise.race
            await applyClickToPage(page,nextPageElementHandle,timeout);
        }
    } while (nextPage);

    //TODO: revisit if takes too much time for ats with too many urls - already also done on java side
    crawledUrls = [...new Set(crawledUrls)];

    console.log("crawlerhelper.extractUrls() numPagesCrawled: " +
        pageNum +
        ", crawledUrls size: " +
        crawledUrls.length);

    let extractUrlsResult = {
        numPagesCrawled: pageNum,
        crawledUrls: crawledUrls
    };

    return extractUrlsResult;
};

async function getNextPageElementHandle(page, pageNum) {
    // using context item expression "."
    // order is important here. Originally we only looked for //a
    const elementsToFind = ['//a', '//button','//li', '//input'];
    const nextPageNum = pageNum + 1;
    let nextPageElementHandle = [];
    let nextPageContainsElementHandles = [];

    for(let elementToFind of elementsToFind) {

        // exact match for elementsToFind equals nextPageNum
        const nextPageCriteria = `${elementToFind}[.='${nextPageNum}']`;
        nextPageElementHandle = await page.$x(nextPageCriteria);
        if(nextPageElementHandle.length) {
            console.log("exact numeric page match" + nextPageCriteria + " length[" + nextPageElementHandle.length + "]");
            return nextPageElementHandle[0];
        }
        console.log(nextPageCriteria + " not matched");
    }

    // means we didnt find a matching html elment
    return nextPageElementHandle[0];
}

async function applyClickToPage(page, elementHandle,timeout) {

    await Promise.race([

        page.evaluate(nextPageLink => nextPageLink.click(), elementHandle),
        page.waitForNavigation({ timeout:  timeout, waitUntil: 'networkidle0' })
    ]);

}

async function filterUrls (urlsFromPage
) {
    let filterUrls = [];
    //remove urls starting with bing - these are unique urls generated by bing
    filterUrls=urlsFromPage.filter(urls => !urls.startsWith('https://www.bing.com'));

    //remove urls starting with tel:
    filterUrls=filterUrls.filter(urls => !urls.startsWith('tel:'));

    //remove urls starting with mailto:
    filterUrls=filterUrls.filter(urls => !urls.startsWith('mailto:'));

    //remove urls starting with javascript:
    filterUrls=filterUrls.filter(urls => !urls.startsWith('javascript:'));

    return filterUrls;
}

async function getCrawledUrlsByCriteria(page, jobLookUpCriteria) {
    let crawledUrls = [];

    await new Promise(r => setTimeout(r, 2500));

    console.log("before jobLookUpCriteria: " + jobLookUpCriteria);
    let urlElements = await page.$x(jobLookUpCriteria);
    console.log("after jobLookUpCriteria: " + jobLookUpCriteria);
    for (let urlElement of urlElements) {
        let crawledUrl = await page.evaluate((el) => el.href, urlElement);
        crawledUrls.push(crawledUrl);
    }
    // check if there are childframes and if yes, try to get urls from those child frames
    // we can also try page.frames() if we are seeing issues here
    const childFrames = page.mainFrame().childFrames();
    //const childFrames = await page.$$('iframe');
    if (childFrames && childFrames.length > 0) {
        console.log("Child Frames:", childFrames.length);
        for (let childFrame of childFrames) {

            let childPageFrameName = childFrame.name();
            // note all iframes might not have names
            console.log("Child Frame Name:", childPageFrameName);
            // checking for presence of url, might need to add check for document content of iframe as well
            if (!childFrame.url()) {
                continue;
            }
            let urlElements = await childFrame.$x(jobLookUpCriteria);
            for (let urlElement of urlElements) {
                let crawledUrl = await childFrame.evaluate((el) => el.href, urlElement);
                crawledUrls.push(crawledUrl);
            }
        }
    }
    return crawledUrls;
}

function compareArraysInclude(array1, array2) {
    if (
        array1.every((value, index) => array2.includes(value))
    ) {
        return true;
    }
    return false;
}

class CompanyCrawlerDto {
    constructor() {
        this.siteId = "";
        this.url = "";
        this.numPagesCrawled = 0;
        this.expectedNumPages = 0
        this.numCrawledUrls = 0;
        this.crawledUrls = [];
        this.expectedNumJobs = {
            min: null,
            max: null
        };
        this.totalDiscoverableJobsLookUpCriteria = "";
        this.totalDiscoverableJobsElement = 0;
        this.errors = [];
        this.paginationStrategyName = "";
        this.isPagePaginated = null;
        this.redirectedUrl = "";
    }
    getSiteId() {
        return this.siteId;
    }
    setSiteId(input) {
        this.siteId = input;
    }
    getUrl() {
        return this.url;
    }
    setUrl(input) {
        this.url = input;
    }

    getRedirectedUrl() {
        return this.redirectedUrl;
    }
    setRedirectedUrl(input) {
        this.redirectedUrl = input;
    }

    getNumPagesCrawled() {
        return this.numPagesCrawled;
    }
    setNumPagesCrawled(input) {
        this.numPagesCrawled = input;
    }
    getExpectedNumPages() {
        return this.expectedNumPages;
    }
    setExpectedNumPages(input) {
        this.expectedNumPages = input;
    }
    getNumCrawledUrls() {
        return this.numCrawledUrls;
    }
    setNumCrawledUrls(input) {
        this.numCrawledUrls = input;
    }
    getCrawledUrls() {
        return this.crawledUrls;
    }
    setCrawledUrls(input) {
        this.crawledUrls = input;
    }
    getExpectedNumJobs() {
        return this.expectedNumJobs;
    }
    setExpectedNumJobs(input) {
        this.expectedNumJobs = input;
    }
    getMinExpectedNumJobs() {
        return this.expectedNumJobs.min;
    }
    setMinExpectedNumJobs(input) {
        this.expectedNumJobs.min = input;
    }
    getMaxExpectedNumJobs() {
        return this.expectedNumJobs.min;
    }
    setMaxExpectedNumJobs(input) {
        this.expectedNumJobs.min = input;
    }
    getTotalDiscoverableJobsLookUpCriteria() {
        return this.totalDiscoverableJobsLookUpCriteria;
    }
    setTotalDiscoverableJobsLookUpCriteria(input) {
        this.totalDiscoverableJobsLookUpCriteria = input;
    }
    getTotalDiscoverableJobsElement() {
        return this.totalDiscoverableJobsElement;
    }
    setTotalDiscoverableJobsElement(input) {
        this.totalDiscoverableJobsElement = input;
    }
    getErrors() {
        return this.errors;
    }
    setErrors(input) {
        this.errors = input;
    }
    addError(input) {
        this.errors.push(input);
    }

    getPaginationStrategyName() {
        return this.paginationStrategyName;
    }
    setPaginationStrategyName(input) {
        this.paginationStrategyName = input;
    }

    getIsPagePaginated() {
        return this.isPagePaginated;
    }
    setIsPagePaginated(input) {
        this.isPagePaginated = input;
    }
}

class PuppeteerLaunchError extends Error {
    constructor(message) {
        super(message);
        this.name = "PuppeteerLaunchError";
    }
}

Error string
no error
Puppeteer configuration
No response
Puppeteer version
19.7.0
Node version
v16.19.0
Package manager
npm
Package manager version
8.19.3
Operating system
macOS

Can I get more information on why the bug needs more feedback?

@dwillisgh the automatic analyser was not able to run https://github.com/puppeteer/puppeteer/actions/runs/4184679853/jobs/7250702120
Can you please provide a minimal reproduction? we want be able to debug such a long script for you. Thanks.

I shrunk it down more and added some comments.

@dwillisgh do you dispose element handles? If you don't call .dispose() on the element handles they might cause the memory to retained and it looks like it fails with out of memory errors.

I didnt know that was a thing :-) Thank you

I updated the example with the dispose in case it helps someone else

Hi. I have a similar issue. Sometimes I'm getting "Aw, Snap! Something went wrong while displaying this webpage. Error code: 11". I'll try using the dispose function, as my code is getting stucked when i get this error.

Hi. I have a similar issue. Sometimes I'm getting "Aw, Snap! Something went wrong while displaying this webpage. Error code: 11". I'll try using the dispose function, as my code is getting stucked when i get this error.

However, I would like to know if there is away throw an error or set a timeout when this happens.


Clicking on type in Firefox

In Firefox, if you click on a type in the sidebar or in the graph, instead of centring the view on the type, it centres somewhere off near the start of the graph.

@LaurenceGA Do you use our online demo or NPM package?

npm package. Live demo seems to work fine.

@RomanGotsiy I suspect it's duplicate of #76 which already have reproduce sample.

Hmm yes, may be a duplicate actually.
Yeah I can try throw a sample together sometime this weekend. There's nothing special in the project it's in.
For some extra detail, the bug was found on latest Firefox on the latest Mac OSX (no bug with latest chrome on the same machine).

@LaurenceGA Sorry for the delay.
Fixed in 1.0.0-rc.26


server does not support the reflection API when celestia-appd starts with broken config

Context
@renaynay and @cristaloleg had a validator with a broken config (app.toml, config.toml)
Problem
The broken config resulted in this symptom:
$ grpcurl -plaintext -d '{"subspace": "gov", "key": "votingparams"}' <IP_ADDRESS>:9090 cosmos.params.v1beta1.Query/Params
Error invoking method "cosmos.params.v1beta1.Query/Params": failed to query for service descriptor "cosmos.params.v1beta1.Query": server does not support the reflection API

Basically all GRPC queries to the server failed with something like: server does not support the reflection API.
Proposal

Get the broken configs from their validator so that we can repro the issue. I looked but couldn't find them on the Scaleway instance. cc: @cmwaters
When celestia-app starts up, if it can't register reflection on the GRPC server, panic with an error message stating that config is invalid instead of starting up.

The config on that node looks deleted. @evan-forbes reproed the issue with different config. Any chance you still have those and can attach them here?


Get rid "gcs" terminology + mv tests to the correct folder.

Testbed commit -> I just did that to remove coav_gcs without break it while we dont have coav-sim merged because we will have ppl looking at the project in between. So is temporary but not pointless. So I'm going to merge the commits except the Move test that needs to be re-done.

Yeah, i figured as much. That's fine, there is no harm in merging it.

polarhist_test.cc should be renamed to polarhist_sample.cc and keept at the samples folder. It is more of a sample on how to use polarhist than a polar hist test itself.

In this case you are totally write, its a small app that shows the usage of a given detector and has a simple output used for demonstration.

Commit "Move test apps out of 'samples' folder" is all wrong. test.cc should be renamed to  vdebug and moved into the tools directory.
The idea is to keep tests that can be automated in tests/ folder, and thats not the case with polarhist_test.

But here is where I have to disagree, test.cc is a horrible sample, it actually has little to no value as a sample at all. It really isn't an automated test but its a testing tool more than a sample. Its only purpose is to help one "test" if the simulated camera input is working. Maybe it should be coav-control's neighbor under tools?

well, that's what I wrote there... "test.cc should be renamed to vdebug and moved into the tools directory"

I'm getting old fast :-(


Vickel: Watch remarkhr.com

Vickel requests the watch of the watch_keyword remarkhr\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

remarkhr\.com has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Rejected by Jeff Schaller in Charcoal HQ. Reason: 'duplicate of 12150, at author's request '

Rejected by Jeff Schaller in Charcoal HQ. Reason: 'duplicate of 12150, at author's request '


Sidebar highlight overflowing

seems to affect chrome mostly

fixed by this PR:
https://github.com/cennznet/ui/pull/134


Migrate Baremetal Chainquery db to production

This is for a full resync of Chainquery database. It will fix older data issues that we see. Please attach fixed issues that are good going forward but need a resync for old data.

Test after resync: https://spee.ch/@NRR:97db3b6c7444a8a4c95cca3611ce88c85d3cfcbd
Chainquery has no claims with publisher_id = 97db3b6c7444a8a4c95cca3611ce88c85d3cfcbd

test after resync: https://explorer.lbry.io/tx/de23687d057583d33a44f8e39803bbebd4c313050d50d1e5753e887226b0b818 - check link to spent, right now it goes to https://explorer.lbry.io/tx/#input-

It's done, and the above now works i believe