Don't convert inputs to env vars. Instead, generate terragrunt.tfvars.json and use as -var-file

I've been reading documentation for terraform 0.12 in regards to handling env vars, and https://www.terraform.io/docs/configuration/variables.html states that:

Some special rules apply to the -var command line option and to environment variables. For convenience, Terraform defaults to interpreting -var and environment variable values as literal strings, which do not need to be quoted

and

However, if a root module variable uses a type constraint to require a complex value (list, set,  map, object, or tuple), Terraform will instead attempt to parse its value using the same syntax  used within variable definitions files, which requires careful attention to the string escaping  rules in your shell

And it's so error-prone and counter-intuitive! Just take a look at https://github.com/gruntwork-io/terragrunt/issues/740 and https://github.com/gruntwork-io/terragrunt/issues/738
Instead, terraform directly supports loading variables from json files: https://www.terraform.io/docs/configuration/variables.html#variable-definitions-tfvars-files
Terraform 0.11 supports it as well, although it's not properly documented:

Variables files use HCL or JSON syntax to define variable values

The main point here is that types are not lost and terraform doesn't have to "guess" them. Also, the whole logic of converting inputs to env vars can be simplified to one json.Marshal.
Benefits:

Simplified logic
Supported by terraform 0.12 and properly documented
Types are not lost in translation, no need to set constraints in variable {} blocks
Works with terraform 0.11 nicely, for compatibility sake!

I can make a PR should it be needed.

Oh yes ! This is definitely the way to go for input variables.
Furthermore, as discussed https://github.com/gruntwork-io/terragrunt/issues/737#issuecomment-500895952 with @brikis98 we should also stick to Terraform, by defining variables explicitly.
# In root terragrunt.hcl
terraform {
  extra_arguments "terragrunt_generated_vars" {
    commands = "${get_terraform_commands_that_need_vars()}"

    required_inputs_files = [
      "${get_terragrunt_dir()}/${find_in_parent_folders("common.hcl")}",
    ]
  }
}

# stage/frontend-app/terragrunt.hcl
terraform {
  source = "..."
}

# Include all settings from the root terragrunt.hcl file
include {
  path = "${find_in_parent_folders()}"
}

inputs = {
  aws_region = get_input("aws_region")
  remote_state_bucket = get_input("remote_state_bucket")

  instance_type = "t2.micro"
  instance_count = 10
}

This will generate terragrunt-12334.tfvars.json and use terraform plan -var-file terragrunt-12334.tfvars.json or terragrunt.auto.tfvars.json loaded automatically by Terraform.

I agree that Terraform's parsing of env vars is not ideal... But generating files on disk is always a bit messy:

What file name to use? What if the file exists already? Do we generate a randomly named temp file?
Do we delete the file after?
What if the values in the inputs block are secrets? Writing them to disk may be a bad idea. E.g., consider:inputs = {
  # Intentionally read value from env vars so it's not written to disk
  db_password = get_env("DB_PASSWORD", "")
}

What if the user has -var-file arguments already in extra_arguments? Does the generated file go before or after those?
What if the user has -var-file arguments already when running terragrunt (e.g., terragrunt apply -var-file=foo? Does the generated file go before or after those?

@brikis98 I understand your point, but with all the benefit we can get, I think it worth that Terragrunt generates tfvars.json files.

What file name to use? What if the file exists already ? Do we generate a randomly named temp file? What about on subsequent re-runs of terragrunt commands?

Maybe in  .terragrunt-cache we can generate single file (terragrunt.auto.tfvars.json) with all needed inputs, and set Terragrunt working directory to the .terragrunt-cache. Terraform will load automatically this tfvars like before.

Do we delete the file after?
What if the values in the inputs block are secrets?

Yes. For consistency and to mitigate secrets exposure. Even clean before every run.

What if the user has -var-file arguments already in extra_arguments? Does the generated file go before or after those?
What if the user has -var-file arguments already when running terragrunt (e.g., terragrunt apply -var-file=foo? Does the generated file go before or after those?

Same as today. It'll follow Terraform variable precedence
PS: Even Terraform Entreprise is using this for workspace variables.

I agree that Terraform's parsing of env vars is not ideal... But generating files on disk is always a bit messy:

What file name to use? What if the file exists already? Do we generate a randomly named temp file? What about on subsequent re-runs of terragrunt commands?

Let's make it terragrunt.tfvars.json? It's unlikely that it can be taken. I would explicitly use -var-file instead of .auto.tfvars.json, firstly because the old terraform doesn't load it automatically, and secondly it's better to keet it explicit!

Do we delete the file after?

Not sure about this one. Why do we need to delete it? It will be regenerated on every terragrunt run.

What if the values in the inputs block are secrets? Writing them to disk may be a bad idea. E.g., consider:inputs = {
  # Intentionally read value from env vars so it's not written to disk
  db_password = get_env("DB_PASSWORD", "")
}

Well, if somebody had secrets in terraform.tfvars before, then it doesn't change the state at all.
Otherwise, TF_VAR_xxx env var will just work as before.

What if the user has -var-file arguments already in extra_arguments? Does the generated file go before or after those?

To keep the previous behaviour it needs to go as the last argument.

What if the user has -var-file arguments already when running terragrunt (e.g., terragrunt apply -var-file=foo? Does the generated file go before or after those?

I think same as above? Variables from terragrunt.tfvars used to overrided everything.
Hmm, I answered almost the same as @barryib  :)

Just another point: the JSON format is meant to be machine-generated, and it's a perfect use case for terragrunt.

This syntax is useful when generating portions of a configuration programmatically, since existing JSON libraries can be used to prepare the generated configuration files.

Well, if somebody had secrets in terraform.tfvars before, then it doesn't change the state at all.
Otherwise, TF_VAR_xxx env var will just work as before.

This isn't a secret in the file. It's a secret read using a Terragrunt helper that would only exist in memory... Unless we start writing var files to disk.
One other thought:
As of Terraform 0.12, you can no longer set a variable in  .tfvars file if that variable isn't defined in your Terraform (.tf) code. Is that true of .tfvars.json? If so, doesn't that significantly limit reuse of common variable files across different modules?

Not sure about secrets.
The same rules apply to both .tfvars and tfvars.json, so yes, it will limit reuse of common variables in the current state of terraform.

The same rules apply to both .tfvars and tfvars.json, so yes, it will limit reuse of common variables in the current state of terraform.

I suspect that will be a serious problem for many users... So we'd be trading one mild problem (variables without a type being parsed incorrectly) to a bigger problem (reuse of values becoming very difficult).

How about parsing the resulting terraform files and only setting variables that are defined there?

This isn't a secret in the file. It's a secret read using a Terragrunt helper that would only exist in memory... Unless we start writing var files to disk.

What if we can define env vars and var files ? Something like:
env_inputs = {
  password = ${get_env("MY_SECRET_PASSWORD")}
}

inputs = {
  instance = "t2.nano"
  region     = "eu-west-1"
}

env_inputs will be treated like Terraform env vars TF_VARS_password and inputs will be converted in json.

I suspect that will be a serious problem for many users... So we'd be trading one mild problem (variables without a type being parsed incorrectly) to a bigger problem (reuse of values becoming very difficult).

I think that reuse of values will be difficult in Terraform 0.12, because they take the decision to move to explicite variable definition. This is neither good nor bad move, it's juste a direction they took.
So should we follow them or not ? I'll say yes, because Terragrunt is a wrapper. It should add value/feature but it's not meant to change Terraform behavior. This could be confusing for users.
Saying that, how can Terragrunt become explicit without loosing DRY concept ? I think get_input() or a map with all merged inputs is a good direction. This map can be used to explicitly define inputs in terragrunt.hcl :
inputs = {
  aws_region = local.region
  # or
  aws_region = get_input("region")
}

@ekini @brikis98 How can we move forward on this discussion ?

The upstream issue seems to be ignored  by Terraform team, which is a blocker.

@barryib I have not had time to dig into a solution yet. Hoping to have more thoughts to share in the next couple weeks.

Since locals (or local) has been supported in Terragrunt, how do we reference the data source?
data "aws_ami" "example" {
  executable_users = ["self"]
  most_recent      = true
  name_regex       = "^myami-\\d{3}"
  owners           = ["self"]

  filter {
    name   = "name"
    values = ["myami-*"]
  }
}

how do we reference data.ami_ami.example.id in terragrunt.hcl?

@ozbillwang I run into the same issue few days ago. Problem is that Terragrunt only allows terragrunt.hcl file in the directory where it is run. I tried leaving a .tf file with the data sources but it didn't work. Think this will never get implemented as, lookin at what we wanted to do, Terragrunt would have to copy those data source definitions on the side and run Terraform to get the outputs, since we wanted to instruct it to assign those outputs to it's inputs (at least that was what I was trying to achieve).
In the end we went with a new module, with only data sources defined inside of it :( ugly as hell, since you need to still pass values to the module (e.g. for google_kms_secret data source - a key and a ciphertext), but I don't think there's a better way of doing this currently.

I don't think we will ever implement a way to reference data or resources in terragrunt directly, as that requires implementing terraform directly within terragrunt, making it more than a wrapper.
With that said, I am having a hard time understanding the use case, so could help if I had a concrete example with ideal terragrunt.hcl and terraform code snippet. What I am having a hard time understanding is the benefit of doing the processing of inputs via data sources within terragrunt, as opposed to doing it in the module in terraform.

You can almost do this today, with the new generate block. The problem is that terragrunt currently injects a comment into the file with no way to disable that feature, and json does not allow comments. My workaround was to write a templated *.auto.tvars file instead. See https://github.com/gruntwork-io/terragrunt/issues/1121 for the approach.

Of course you'll still run into the undeclared var problem, but you can use a .tfvars file instead of ENVs... ;)

@yorinasub17 Sorry for neglecting to respond :(
I was (and still am) opposed to putting data source definitions in modules. I think of them rather as something that provides input values to a module. Our usage was to provide a list of IPs to a firewall configuration in out Terraform module. The list of IPs was being fetched using the http data source. Putting the data source in the module felt more like hardcoding, which is what we went for in the end.
Looking at the new generate block, I suppose this can be handled more or less the way I wanted it initially, but I haven't tested it yet.

@kcatro Since you are using the http data source, could you use run_cmd to accomplish the same thing?

@yorinasub17 Seems like a valid idea! :)

Relabeled as a feature enhancement that needs design. The main concerns around this are documented in the discussion on this PR: https://github.com/gruntwork-io/terragrunt/pull/1267

I'd like to link an issue that could be solved by generating .tfvars with inputs:  https://bytemeta.vip/repo/gruntwork-io/terragrunt/issues/2132

I guess using Terragrunt config to store variables instead of .tfvars, then load it into the Terragrunt config in local block by read_terragrunt_config(get_env("PATH_TO_TG_VARS_FILE")) is the workaround solution for this, just need to set some ENV var when running the terragrunt commands


install_cbmc.sh: build from source on non-x86_64

CBMC only provides binaries for x86_64, but it runs fine on other arches (like aarch64) Build from source in that case
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.

Thanks @bennofs for the contribution!
@adpaco-aws can you review this PR since you're the most familiar with this code?
The failure is unrelated to the PR, and is due to a CBMC viewer/python version issue.


How training is done ?

Hello, thanks for this project :) it's very helpful
This is not an issue but can anyone explain to me how the training is done exactly?
Any help is appreciated thanks!

Hello @GhadaJouini, I'm glad to read that :D
First, I've not test anything with Tesseract 4.0 so I dont know if it's the same as the version 3.0. But here is how I've achieved the training.
In my case, I work in a place where I have multiple stubs and cheques. So the goal is to build an image file with multiple OCR line. When it's done, I've loaded the image in jTessBoxEditor. And then, for every Character the application has detected, verify the result and correct it if there is a wrong recognition. Do this for every Character. And then, you'll have your trainedData for OCR. If you want MICR trained_data as well, repeat all these processes. The more you train your data, more accurate results you'll get from the engine.
I believe jTessBoxEditor is not supported for Tesseract 4.0. There is certainly a way to achieve the same processes. You'll need some research if you choose the most recent version of Tesseract.
Good luck!


nim: 0.18.0 -> 0.19.0

remove sqlite as dependency (I don't see why it's needed)
run checkPhase after installPhase (at least one test assumes the
standard library in ../lib relative to the nim binary)
the broken tests pass now or don't exist anymore
two of the tests requiring network access pass now without network access
the tests in manyloc do not download dependencies and are passing now

Motivation for this change
Update nim to 0.19.0.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg build nim

Failure on aarch64-linux (full log)
Attempted: nim
Partial log (click to expand)
PASS: cyclecollector C -d:release                                  (3.40048289 secs)
PASS: cyclecollector C -d:release -d:useRealtimeGC                 (3.43674994 secs)
PASS: cyclecollector C --gc:markAndSweep                           (2.61317205 secs)
PASS: cyclecollector C -d:release --gc:markAndSweep                (2.97520089 secs)
PASS: cyclecollector C --gc:boehm                                  (1.25764084 secs)
PASS: cyclecollector C -d:release --gc:boehm                       (0.91316295 secs)
/build/nim-0.19.0/tests/testament/tester html
tests failed
builder for '/nix/store/ci71mgmf37hb10k3pjyd5dblak95wy2i-nim-0.19.0.drv' failed with exit code 1
error: build of '/nix/store/ci71mgmf37hb10k3pjyd5dblak95wy2i-nim-0.19.0.drv' failed

Timed out, unknown build status on x86_64-darwin (full log)
Attempted: nim
Partial log (click to expand)
  clang++ '-DV8_GYP_BUILD' '-D_DARWIN_USE_64_BIT_INODE=1' '-DV8_TARGET_ARCH_X64' '-DENABLE_DISASSEMBLER' '-DV8_PROMISE_INTERNAL_FIELD_COUNT' '-Dv8_promise_internal_field_count' '-DV8_INTL_SUPPORT' '-DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_STATIC' '-DUCONFIG_NO_SERVICE=1' '-DUCONFIG_NO_REGULAR_EXPRESSIONS=1' '-DU_ENABLE_DYLOAD=0' '-DU_STATIC_IMPLEMENTATION=1' '-DU_HAVE_STD_STRING=0' '-DUCONFIG_NO_BREAK_ITERATION=0' -I../deps/v8 -I../. -I/private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj/gen -I/nix/store/qscisgacfgykr63pgmjp1s65rndma3wy-zlib-1.2.11-dev/include -I/nix/store/59c6bz08jxcyxyi962fhf56jzph3f333-libuv-1.23.1/include -I/nix/store/j3cqsdyfdzv21kkgzwv19f7cxzz00znz-openssl-1.0.2p-dev/include -I../deps/v8/include -I../deps/icu-small/source/i18n -I../deps/icu-small/source/common  -O3 -gdwarf-2 -fstrict-aliasing -mmacosx-version-min=10.7 -arch x86_64 -Wall -Wendif-labels -W -Wno-unused-parameter -std=gnu++0x -stdlib=libc++ -fno-rtti -fno-exceptions -fno-threadsafe-statics -fno-strict-aliasing -MMD -MF /private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/.deps//private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj.target/v8_base/deps/v8/src/compiler/js-create-lowering.o.d.raw   -c -o /private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj.target/v8_base/deps/v8/src/compiler/js-create-lowering.o ../deps/v8/src/compiler/js-create-lowering.cc
  clang++ '-DV8_GYP_BUILD' '-D_DARWIN_USE_64_BIT_INODE=1' '-DV8_TARGET_ARCH_X64' '-DENABLE_DISASSEMBLER' '-DV8_PROMISE_INTERNAL_FIELD_COUNT' '-Dv8_promise_internal_field_count' '-DV8_INTL_SUPPORT' '-DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_STATIC' '-DUCONFIG_NO_SERVICE=1' '-DUCONFIG_NO_REGULAR_EXPRESSIONS=1' '-DU_ENABLE_DYLOAD=0' '-DU_STATIC_IMPLEMENTATION=1' '-DU_HAVE_STD_STRING=0' '-DUCONFIG_NO_BREAK_ITERATION=0' -I../deps/v8 -I../. -I/private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj/gen -I/nix/store/qscisgacfgykr63pgmjp1s65rndma3wy-zlib-1.2.11-dev/include -I/nix/store/59c6bz08jxcyxyi962fhf56jzph3f333-libuv-1.23.1/include -I/nix/store/j3cqsdyfdzv21kkgzwv19f7cxzz00znz-openssl-1.0.2p-dev/include -I../deps/v8/include -I../deps/icu-small/source/i18n -I../deps/icu-small/source/common  -O3 -gdwarf-2 -fstrict-aliasing -mmacosx-version-min=10.7 -arch x86_64 -Wall -Wendif-labels -W -Wno-unused-parameter -std=gnu++0x -stdlib=libc++ -fno-rtti -fno-exceptions -fno-threadsafe-statics -fno-strict-aliasing -MMD -MF /private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/.deps//private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj.target/v8_base/deps/v8/src/compiler/js-generic-lowering.o.d.raw   -c -o /private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj.target/v8_base/deps/v8/src/compiler/js-generic-lowering.o ../deps/v8/src/compiler/js-generic-lowering.cc
  clang++ '-DV8_GYP_BUILD' '-D_DARWIN_USE_64_BIT_INODE=1' '-DV8_TARGET_ARCH_X64' '-DENABLE_DISASSEMBLER' '-DV8_PROMISE_INTERNAL_FIELD_COUNT' '-Dv8_promise_internal_field_count' '-DV8_INTL_SUPPORT' '-DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_STATIC' '-DUCONFIG_NO_SERVICE=1' '-DUCONFIG_NO_REGULAR_EXPRESSIONS=1' '-DU_ENABLE_DYLOAD=0' '-DU_STATIC_IMPLEMENTATION=1' '-DU_HAVE_STD_STRING=0' '-DUCONFIG_NO_BREAK_ITERATION=0' -I../deps/v8 -I../. -I/private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj/gen -I/nix/store/qscisgacfgykr63pgmjp1s65rndma3wy-zlib-1.2.11-dev/include -I/nix/store/59c6bz08jxcyxyi962fhf56jzph3f333-libuv-1.23.1/include -I/nix/store/j3cqsdyfdzv21kkgzwv19f7cxzz00znz-openssl-1.0.2p-dev/include -I../deps/v8/include -I../deps/icu-small/source/i18n -I../deps/icu-small/source/common  -O3 -gdwarf-2 -fstrict-aliasing -mmacosx-version-min=10.7 -arch x86_64 -Wall -Wendif-labels -W -Wno-unused-parameter -std=gnu++0x -stdlib=libc++ -fno-rtti -fno-exceptions -fno-threadsafe-statics -fno-strict-aliasing -MMD -MF /private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/.deps//private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj.target/v8_base/deps/v8/src/compiler/js-graph.o.d.raw   -c -o /private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj.target/v8_base/deps/v8/src/compiler/js-graph.o ../deps/v8/src/compiler/js-graph.cc
  clang++ '-DV8_GYP_BUILD' '-D_DARWIN_USE_64_BIT_INODE=1' '-DV8_TARGET_ARCH_X64' '-DENABLE_DISASSEMBLER' '-DV8_PROMISE_INTERNAL_FIELD_COUNT' '-Dv8_promise_internal_field_count' '-DV8_INTL_SUPPORT' '-DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_STATIC' '-DUCONFIG_NO_SERVICE=1' '-DUCONFIG_NO_REGULAR_EXPRESSIONS=1' '-DU_ENABLE_DYLOAD=0' '-DU_STATIC_IMPLEMENTATION=1' '-DU_HAVE_STD_STRING=0' '-DUCONFIG_NO_BREAK_ITERATION=0' -I../deps/v8 -I../. -I/private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj/gen -I/nix/store/qscisgacfgykr63pgmjp1s65rndma3wy-zlib-1.2.11-dev/include -I/nix/store/59c6bz08jxcyxyi962fhf56jzph3f333-libuv-1.23.1/include -I/nix/store/j3cqsdyfdzv21kkgzwv19f7cxzz00znz-openssl-1.0.2p-dev/include -I../deps/v8/include -I../deps/icu-small/source/i18n -I../deps/icu-small/source/common  -O3 -gdwarf-2 -fstrict-aliasing -mmacosx-version-min=10.7 -arch x86_64 -Wall -Wendif-labels -W -Wno-unused-parameter -std=gnu++0x -stdlib=libc++ -fno-rtti -fno-exceptions -fno-threadsafe-statics -fno-strict-aliasing -MMD -MF /private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/.deps//private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj.target/v8_base/deps/v8/src/compiler/js-inlining.o.d.raw   -c -o /private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj.target/v8_base/deps/v8/src/compiler/js-inlining.o ../deps/v8/src/compiler/js-inlining.cc
  clang++ '-DV8_GYP_BUILD' '-D_DARWIN_USE_64_BIT_INODE=1' '-DV8_TARGET_ARCH_X64' '-DENABLE_DISASSEMBLER' '-DV8_PROMISE_INTERNAL_FIELD_COUNT' '-Dv8_promise_internal_field_count' '-DV8_INTL_SUPPORT' '-DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_STATIC' '-DUCONFIG_NO_SERVICE=1' '-DUCONFIG_NO_REGULAR_EXPRESSIONS=1' '-DU_ENABLE_DYLOAD=0' '-DU_STATIC_IMPLEMENTATION=1' '-DU_HAVE_STD_STRING=0' '-DUCONFIG_NO_BREAK_ITERATION=0' -I../deps/v8 -I../. -I/private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj/gen -I/nix/store/qscisgacfgykr63pgmjp1s65rndma3wy-zlib-1.2.11-dev/include -I/nix/store/59c6bz08jxcyxyi962fhf56jzph3f333-libuv-1.23.1/include -I/nix/store/j3cqsdyfdzv21kkgzwv19f7cxzz00znz-openssl-1.0.2p-dev/include -I../deps/v8/include -I../deps/icu-small/source/i18n -I../deps/icu-small/source/common  -O3 -gdwarf-2 -fstrict-aliasing -mmacosx-version-min=10.7 -arch x86_64 -Wall -Wendif-labels -W -Wno-unused-parameter -std=gnu++0x -stdlib=libc++ -fno-rtti -fno-exceptions -fno-threadsafe-statics -fno-strict-aliasing -MMD -MF /private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/.deps//private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj.target/v8_base/deps/v8/src/compiler/js-inlining-heuristic.o.d.raw   -c -o /private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj.target/v8_base/deps/v8/src/compiler/js-inlining-heuristic.o ../deps/v8/src/compiler/js-inlining-heuristic.cc
  clang++ '-DV8_GYP_BUILD' '-D_DARWIN_USE_64_BIT_INODE=1' '-DV8_TARGET_ARCH_X64' '-DENABLE_DISASSEMBLER' '-DV8_PROMISE_INTERNAL_FIELD_COUNT' '-Dv8_promise_internal_field_count' '-DV8_INTL_SUPPORT' '-DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_STATIC' '-DUCONFIG_NO_SERVICE=1' '-DUCONFIG_NO_REGULAR_EXPRESSIONS=1' '-DU_ENABLE_DYLOAD=0' '-DU_STATIC_IMPLEMENTATION=1' '-DU_HAVE_STD_STRING=0' '-DUCONFIG_NO_BREAK_ITERATION=0' -I../deps/v8 -I../. -I/private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj/gen -I/nix/store/qscisgacfgykr63pgmjp1s65rndma3wy-zlib-1.2.11-dev/include -I/nix/store/59c6bz08jxcyxyi962fhf56jzph3f333-libuv-1.23.1/include -I/nix/store/j3cqsdyfdzv21kkgzwv19f7cxzz00znz-openssl-1.0.2p-dev/include -I../deps/v8/include -I../deps/icu-small/source/i18n -I../deps/icu-small/source/common  -O3 -gdwarf-2 -fstrict-aliasing -mmacosx-version-min=10.7 -arch x86_64 -Wall -Wendif-labels -W -Wno-unused-parameter -std=gnu++0x -stdlib=libc++ -fno-rtti -fno-exceptions -fno-threadsafe-statics -fno-strict-aliasing -MMD -MF /private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/.deps//private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj.target/v8_base/deps/v8/src/compiler/js-intrinsic-lowering.o.d.raw   -c -o /private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj.target/v8_base/deps/v8/src/compiler/js-intrinsic-lowering.o ../deps/v8/src/compiler/js-intrinsic-lowering.cc
  clang++ '-DV8_GYP_BUILD' '-D_DARWIN_USE_64_BIT_INODE=1' '-DV8_TARGET_ARCH_X64' '-DENABLE_DISASSEMBLER' '-DV8_PROMISE_INTERNAL_FIELD_COUNT' '-Dv8_promise_internal_field_count' '-DV8_INTL_SUPPORT' '-DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_STATIC' '-DUCONFIG_NO_SERVICE=1' '-DUCONFIG_NO_REGULAR_EXPRESSIONS=1' '-DU_ENABLE_DYLOAD=0' '-DU_STATIC_IMPLEMENTATION=1' '-DU_HAVE_STD_STRING=0' '-DUCONFIG_NO_BREAK_ITERATION=0' -I../deps/v8 -I../. -I/private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj/gen -I/nix/store/qscisgacfgykr63pgmjp1s65rndma3wy-zlib-1.2.11-dev/include -I/nix/store/59c6bz08jxcyxyi962fhf56jzph3f333-libuv-1.23.1/include -I/nix/store/j3cqsdyfdzv21kkgzwv19f7cxzz00znz-openssl-1.0.2p-dev/include -I../deps/v8/include -I../deps/icu-small/source/i18n -I../deps/icu-small/source/common  -O3 -gdwarf-2 -fstrict-aliasing -mmacosx-version-min=10.7 -arch x86_64 -Wall -Wendif-labels -W -Wno-unused-parameter -std=gnu++0x -stdlib=libc++ -fno-rtti -fno-exceptions -fno-threadsafe-statics -fno-strict-aliasing -MMD -MF /private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/.deps//private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj.target/v8_base/deps/v8/src/compiler/js-native-context-specialization.o.d.raw   -c -o /private/tmp/nix-build-nodejs-slim-8.12.0.drv-0/node-v8.12.0/out/Release/obj.target/v8_base/deps/v8/src/compiler/js-native-context-specialization.o ../deps/v8/src/compiler/js-native-context-specialization.cc
building of '/nix/store/gdcvliy19dcvcsrckg687jk1b82mhqq0-nodejs-slim-8.12.0.drv' timed out after 1800 seconds
cannot build derivation '/nix/store/msxydnlij99p5m7k08ja42g846dbwjl0-nim-0.19.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/msxydnlij99p5m7k08ja42g846dbwjl0-nim-0.19.0.drv' failed

Timed out, unknown build status on x86_64-linux (full log)
Attempted: nim
Partial log (click to expand)
PASS: cycleleak C -d:release --gc:markAndSweep                     (3.16289592 secs)
PASS: cycleleak C --gc:boehm                                       (1.32490492 secs)
PASS: cycleleak C -d:release --gc:boehm                            (1.19588614 secs)
PASS: closureleak C                                                (3.68987894 secs)
PASS: closureleak C -d:release                                     (3.80776811 secs)
PASS: closureleak C -d:release -d:useRealtimeGC                    (3.86992311 secs)
PASS: closureleak C --gc:markAndSweep                              (3.14992309 secs)
PASS: closureleak C -d:release --gc:markAndSweep                   (3.86592388 secs)
building of '/nix/store/n834y56z1brj3ml7npm1zpyfm71ijxcy-nim-0.19.0.drv' timed out after 1800 seconds
error: build of '/nix/store/n834y56z1brj3ml7npm1zpyfm71ijxcy-nim-0.19.0.drv' failed

I tried to fix the failure on Aarch64.
What can we do about the timeouts?

On x86_64 the tests were interrupted after 1800 seconds, I don't think they were hanging.
On darwin building nodejs took too long. I could try to disable nodejs for darwin, it's only needed for the tests that are not run anyway.

If ofborg timeouts this is ok. Our hydra has longer timeouts.

@GrahamcOfBorg build nim

Success on aarch64-linux (full log)
Attempted: nim
Partial log (click to expand)
PASS: stackrefleak C -d:release --gc:boehm                         (1.88382483 secs)
PASS: cyclecollector C                                             (3.06635785 secs)
PASS: cyclecollector C -d:release                                  (3.41082907 secs)
PASS: cyclecollector C -d:release -d:useRealtimeGC                 (3.43448997 secs)
PASS: cyclecollector C --gc:markAndSweep                           (2.60068607 secs)
PASS: cyclecollector C -d:release --gc:markAndSweep                (2.96925998 secs)
PASS: cyclecollector C --gc:boehm                                  (1.24932003 secs)
PASS: cyclecollector C -d:release --gc:boehm                       (0.91260719 secs)
/build/nim-0.19.0/tests/testament/tester html
/nix/store/7nxv376k3ry7khwmrd5axy7mbn8nnnlg-nim-0.19.0

Timed out, unknown build status on x86_64-linux (full log)
Attempted: nim
Partial log (click to expand)
PASS: unicode.nim C                                                (3.06197500 secs)
PASS: unittest.nim C                                               (2.31589603 secs)
PASS: uri.nim C                                                    (2.17861605 secs)
PASS: volatile.nim C                                               (1.59678316 secs)
PASS: xmldom.nim C                                                 (1.88742185 secs)
PASS: xmldomparser.nim C                                           (2.13288188 secs)
PASS: xmlparser.nim C                                              (2.51197982 secs)
PASS: xmltree.nim C                                                (2.50686479 secs)
building of '/nix/store/838ldag9fs6p9lvsajhqm3vw8v1fz1ly-nim-0.19.0.drv' timed out after 1800 seconds
error: build of '/nix/store/838ldag9fs6p9lvsajhqm3vw8v1fz1ly-nim-0.19.0.drv' failed

Build + tests successful on nixos-18.09 otherwise
Full log here

thanks for reviewing. I moved the packages you mentioned to nativeBuildInputs and nodejs-slim, too because it's only needed for the checkPhase.
Also I updated nodejs-slim-8_x to nodejs-slim-10_x.

@royneary You need to add the HOME=$TMPDIR fix to the other nim packages as well.
@GrahamcOfBorg build nrpl
(mosdepth doesn't seem to compile with the new version.)

No attempt on aarch64-linux (full log)
The following builds were skipped because they don't evaluate on aarch64-linux: nrpl
Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

Timed out, unknown build status on x86_64-linux (full log)
Attempted: nrpl
Partial log (click to expand)
PASS: unittest.nim C                                               (2.39380908 secs)
PASS: uri.nim C                                                    (2.15112114 secs)
PASS: volatile.nim C                                               (1.61674714 secs)
PASS: xmldom.nim C                                                 (1.96292019 secs)
PASS: xmldomparser.nim C                                           (2.10824990 secs)
PASS: xmlparser.nim C                                              (2.68674994 secs)
PASS: xmltree.nim C                                                (2.47554517 secs)
building of '/nix/store/p48w1n52q5nl4xbs8frwa4ppxmxsnn20-nim-0.19.0.drv' timed out after 1800 seconds
cannot build derivation '/nix/store/swd0ggw35z30miqx2qkhk2vbwri7gnbs-nrpl-20150522.drv': 1 dependencies couldn't be built
error: build of '/nix/store/swd0ggw35z30miqx2qkhk2vbwri7gnbs-nrpl-20150522.drv' failed

@gebner fixed.

nrpl builds for me now.


Update Wiki docs for vwan

Overview/Summary
This PR adds new content to the Wiki and updates existing to provide the latest updates regard vwan connectivity and to provide more consistent branding in line with the move from "Enterprise-scale" to "Azure landing zones".
We have also added collapsible content sections to hide large code blocks in the docs to improve readability.
This PR includes a cosmetic update to variables.tf to move the in-line comments for configure_connectivity_resources to a block of comments above the variable block. This improves readability and removes the comments from the schema in auto-generated docs on Terraform Registry.
Additionally, stricter linting rules have been enabled for markdown content, and correction made to existing Wiki pages where necessary.
This PR fixes/adds/changes/removes

No GitHub Issues are fixed.
This PR addresses outstanding documentation updates as part of our backlog.

Breaking Changes
N/A
Testing Evidence
N/A
As part of this Pull Request I have

[x] Checked for duplicate Pull Requests
[x] Associated it with relevant issues, for tracking and closure.
[x] Ensured my code/branch is up-to-date with the latest changes in the main branch
[x] Performed testing and provided evidence.
[x] Updated relevant and associated documentation.
[ ] Updated the "What's New?" wiki page (located in the Enterprise-Scale repo in the directory: /docs/wiki/whats-new.md)

/azp run unit


Failed to deploy resources when using built-in k3s cluster: Could not resolve host

Describe the bug

Steps to reproduce

Run Walrus by docker-compose and using built-in k3s cluster.
Create a resource.

Result

Failed to deploy.

Error: Failed to download module

  on main.tf line 40:
  40: module "mysql" {

Could not download module "mysql" (main.tf:40) source code from
"git::https://github.com/walrus-catalog/terraform-kubernetes-mysql?ref=v0.3.0":
error downloading
'https://github.com/walrus-catalog/terraform-kubernetes-mysql?ref=v0.3.0':
/usr/bin/git exited with 128: Cloning into '.terraform/modules/mysql'...
fatal: unable to access
'https://github.com/walrus-catalog/terraform-kubernetes-mysql/': Could not
resolve host: github.com

Check the coredns on the k3s cluster.

NAMESPACE       NAME                                              READY   STATUS              RESTARTS   AGE
kube-system     pod/coredns-77ccd57875-sgscv                      0/1     ContainerCreating   0          38s
kube-system     pod/local-path-provisioner-957fdf8bc-kpnnr        1/1     Running             0          38s
walrus-system   pod/walrus-mirror-hermitcrab-0                    2/2     Running             0          38s
walrus-system   pod/walrus-workflow-controller-57f768b478-rmqpj   1/1     Running             0          38s

k describe pod/coredns-77ccd57875-sgscv -n kube-system
Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    11m                  default-scheduler  Successfully assigned kube-system/coredns-77ccd57875-sgscv to local
  Warning  FailedMount  99s (x13 over 11m)   kubelet            MountVolume.SetUp failed for volume "config-volume" : configmap references non-existent config key: NodeHosts
  Warning  FailedMount  43s (x5 over 9m52s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[config-volume], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition

Expected behavior

Environment

Walrus version: main 59df6ecf375f04b1783c2c7023b246bb6d228aa3
UI version: dev-874ecc7

Additional context
Add any other context about the problem here.
Another error logs after deployment.
Error refreshing state: Failed to get state: GET https://xxx/v1/projects/503607005865312258/environments/503607005882089474/resources/503613659642068994/runs/503613659675623426/terraform-states giving up after 11 attempt(s): Get "https://xxx/v1/projects/503607005865312258/environments/503607005882089474/resources/503613659642068994/runs/503613659675623426/terraform-states": dial tcp: lookup xxx on <IP_ADDRESS>:53: write udp <IP_ADDRESS>:39668-><IP_ADDRESS>:53: write: operation not permitted

https://github.com/k3s-io/k3s/issues/9381

Environment

Walrus version: main 96dd76526ad969a2776aecc30fbe79c812b81c2f
UI version: dev-874ecc7

Test Result: pass


Feature/metric point generator

#307  revised datapack and point_generator.

Codecov Report

Merging #313 into 2.0 will increase coverage by 0.13%.
The diff coverage is 100%.

@@           Coverage Diff            @@
##              2.0   #313      +/-   ##
========================================
+ Coverage   99.86%   100%   +0.13%
========================================
  Files          28     28
  Lines         732    735       +3
========================================
+ Hits          731    735       +4
+ Misses          1      0       -1

Impacted Files
Coverage Δ

matchzoo/datapack.py
100% <100%> (ø)
:arrow_up:

matchzoo/engine/base_generator.py
100% <100%> (ø)
:arrow_up:

matchzoo/models/dssm_model.py
100% <100%> (ø)
:arrow_up:

matchzoo/models/cdssm_model.py
100% <100%> (ø)
:arrow_up:

matchzoo/preprocessor/dssm_preprocessor.py
100% <100%> (ø)
:arrow_up:

matchzoo/engine/base_preprocessor.py
100% <100%> (ø)
:arrow_up:

matchzoo/generators/point_generator.py
100% <100%> (ø)
:arrow_up:

matchzoo/engine/base_model.py
100% <0%> (+1.23%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 98142e2...468c209. Read the comment docs.


Replace zapr with slog

Slog solves all the feature requirements that we so no need to keep zapr.

I would like to remove logr in the future but the issue is that I do like the fact that it is possible to keep the logging object in the context. This is obviously not something that everyone thinks is a good idea. I just think that it is better than a global logger.


Image Similarity Documentation is still unclear

https://github.com/apple//turicreate/blob/master/src/python/turicreate/toolkits/image_similarity/image_similarity.py#L44

This sentence sounds like it's a patchwork of partial sentences:
   feature : string
        indicates that the SFrame has only column of Image type and that will
        Name of the column containing the input images. 'None' (the default)
        be used for similarity.

How should it read?

My attempt to fix in PR #2510

Fixed by #2510


WIP theme assets pipeline

Pre-Flight checklist

[ ] Screenshots and design review for any changes that affect layout or styling

[ ] Desktop screenshots
[ ] Mobile width screenshots

[ ] Migrations

[ ] Migration is backwards-compatible with current production code

[ ] Testing

[ ] Code is tested
[ ] Changes have been manually tested

[ ] Settings

[ ] New settings are documented and present in app.json
[ ] New settings have reasonable development defaults, if applicable
[ ] Opened issue for DevOps regarding necessary configuration changes to deployed environments

What are the relevant tickets?
(Required)
What's this PR do?
(Required)
How should this be manually tested?
(Required)
Where should the reviewer start?
(Optional)
Any background context you want to provide?
(Optional)
Screenshots (if appropriate)
(Optional)
What GIF best describes this PR or how it makes you feel?
(Optional)

Codecov Report

Merging #945 (96fca37) into master (89f5862) will increase coverage by 3.29%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #945      +/-   ##
==========================================
+ Coverage   91.74%   95.03%   +3.29%
==========================================
  Files         190       94      -96
  Lines        7096     2498    -4598
  Branches     1270      551     -719
==========================================
- Hits         6510     2374    -4136
+ Misses        509      124     -385
+ Partials       77        0      -77

Impacted Files
Coverage Δ

content_sync/api.py

content_sync/pipelines/base.py

content_sync/pipelines/concourse.py

content_sync/tasks.py

main/settings.py

videos/youtube.py

main/urls.py

news/urls.py

content_sync/utils.py

content_sync/apps.py

... and 86 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 89f5862...96fca37. Read the comment docs.

@mbertrand Yep, I'll merge it after https://github.com/mitodl/salt-ops/pull/1496 is reviewed and merged.


Tutorial issues

Thanks for the great work! Currently I do not use ArduCam and I just want to test LCD
After following steps described in the tutorial, I can compile without errors but run with some errors:
(1) No source available for "d_expression_1() at 0x8001058"
(2) Even if I move lcdsetup() before "SystemClock_Config()" and try to display string on LCD, it doesn't show anything onto the LCD screen. Do we need to set something else? Or is it just related to the cam device?
(3) Is there any method we can print message in console mode to verify if we enter main() function?
Furthermore, the table of "Measure results", how can we get the time measurements? Use HAL_getTick() before and after invoke()?
Could you please help comment that? Thanks

Hi @waquey, thanks for your interest in our work!
For questions 1 and 2, I believe the errors came out due to the lack of ArduCam. To support your request and also other users who do not have an ArduCam, we will revise our tutorial ASAP to support testing TinyEngine without an ArduCam.
At this time, to test TinyEngine without an ArduCam, you could try to comment out the codes related to camera initialization functions in main.cpp. Also, please import a testing image as your input image.
For question 3, if you are using STM32CubeIDE, you can just get into the Debug Mode in the IDE, and use its built-in functions (such as adding breakpoints or using Run to Line function). After setting up and starting the debug process in the Debug Mode, you should be able to verify whether you enter main() function or not.
For question 4, yes, please use HAL_GetTick() before and after invoke() to obtain the inference latency.
If you have any further questions, please feel free to let us know!

Hi @waquey, thanks for your interest in our work!
For questions 1 and 2, I believe the errors came out due to the lack of ArduCam. To support your request and also other users who do not have an ArduCam, we will revise our tutorial ASAP to support testing TinyEngine without an ArduCam.
At this time, to test TinyEngine without an ArduCam, you could try to comment out the codes related to camera initialization functions in main.cpp. Also, please import a testing image as your input image.
For question 3, if you are using STM32CubeIDE, you can just get into the Debug Mode in the IDE, and use its built-in functions (such as adding breakpoints or using Run to Line function). After setting up and starting the debug process in the Debug Mode, you should be able to verify whether you enter main() function or not.
For question 4, yes, please use HAL_GetTick() before and after invoke() to obtain the inference latency.
If you have any further questions, please feel free to let us know!

Close due to inactivity. Feel free to reopen.


Templates and Tools - DAO and Grant Tools

re-open

turn list into links, add a short description of each so people know why they might want to use the tool.

redunandnt, removing this section


EnderIO-1.10.2-<IP_ADDRESS>_beta Conduit lag

Issue Description:
This issue has been present in both All The mods pack and FTB Present hermitpack (current)
What happens:
After some time the enderIO conduits start effecting the server tickrate.
What you expected to happen:
Not affecting the server performance that much.
Steps to reproduce:

Place down plenty of energy conduits.
Put something that produce in one end and a few consumers. (net negative)

Affected Versions (Do not use "latest"):

EnderIO: 1.10.2-<IP_ADDRESS>_beta
EnderCore: 1.10.2-<IP_ADDRESS>-beta
Minecraft: 1.10.2
Forge: forge-2092

Your most recent log file where the issue was present:
https://timings.aikar.co/v2/?id=e438b783044147af939debeea708d409&section=all

looks to me like environmentaltech:controller_void_res_miner is causing chunk generation.
Is there any way to get that graph with a bit more detail? Ender IO::ConduitNetworkTickHandler_onServerTick() is a method that has basically one line of code...

I would love to get a more granular view of the issue. However I haven't found a profiler that will provide such info. You can go to chunk view and find all items placed in the world. however. I'm pretty certain it's related to energy conduits. Had most success causing lag with that.

If you need access to the server or any other detail, let me know.

One problem is that ConduitNetworkTickHandler_onServerTick handles all conduits: all blocks, in all worlds, of all types.
I'll probably add some debug timing output for this.

This from HermitPack modpack.
 I'll do better next time, I promise!

Time: 10/11/16 7:01 PM
Description: Watching Server

java.lang.Error
	at net.minecraft.world.World.func_189508_F(World.java:2421)
	at net.minecraft.world.World.func_175625_s(World.java:108)
	at com.enderio.core.common.BlockEnder.getTileEntity(BlockEnder.java:160)
	at crazypants.enderio.conduit.BlockConduitBundle.func_189540_a(BlockConduitBundle.java:868)
	at org.spongepowered.common.event.tracking.CauseTracker.notifyBlockOfStateChange(CauseTracker.java:316)
	at net.minecraft.world.WorldServer.func_175685_c(WorldServer.java:1263)
	at crazypants.enderio.conduit.TileConduitBundle.doConduitsDirty(TileConduitBundle.java:255)
	at crazypants.enderio.conduit.TileConduitBundle.doUpdate(TileConduitBundle.java:238)
	at com.enderio.core.common.TileEntityBase.func_73660_a(TileEntityBase.java:36)
	at org.spongepowered.common.event.tracking.TrackingUtil.tickTileEntity(TrackingUtil.java:194)
	at net.minecraft.world.WorldServer.redirect$onUpdateTileEntities$zji000(WorldServer.java:1290)
	at net.minecraft.world.World.func_72939_s(World.java:1255)
	at net.minecraft.world.WorldServer.func_72939_s(WorldServer.java:709)
	at net.minecraft.server.MinecraftServer.func_71190_q(MinecraftServer.java:709)
	at net.minecraft.server.dedicated.DedicatedServer.func_71190_q(DedicatedServer.java:387)
	at net.minecraft.server.MinecraftServer.func_71217_p(MinecraftServer.java:613)
	at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:471)
	at java.lang.Thread.run(Thread.java:745)

A detailed walkthrough of the error, its code path and all known details is as follows:
---------------------------------------------------------------------------------------

-- Head --
Thread: Server Watchdog
Stacktrace:
	at net.minecraft.world.World.func_189508_F(World.java:2421)
	at net.minecraft.world.World.func_175625_s(World.java:108)
	at com.enderio.core.common.BlockEnder.getTileEntity(BlockEnder.java:160)
	at crazypants.enderio.conduit.BlockConduitBundle.func_189540_a(BlockConduitBundle.java:868)
	at org.spongepowered.common.event.tracking.CauseTracker.notifyBlockOfStateChange(CauseTracker.java:316)
	at net.minecraft.world.WorldServer.func_175685_c(WorldServer.java:1263)
	at crazypants.enderio.conduit.TileConduitBundle.doConduitsDirty(TileConduitBundle.java:255)
	at crazypants.enderio.conduit.TileConduitBundle.doUpdate(TileConduitBundle.java:238)
	at com.enderio.core.common.TileEntityBase.func_73660_a(TileEntityBase.java:36)
	at org.spongepowered.common.event.tracking.TrackingUtil.tickTileEntity(TrackingUtil.java:194)
	at net.minecraft.world.WorldServer.redirect$onUpdateTileEntities$zji000(WorldServer.java:1290)
	at net.minecraft.world.World.func_72939_s(World.java:1255)
	at net.minecraft.world.WorldServer.func_72939_s(WorldServer.java:709)
	at net.minecraft.server.MinecraftServer.func_71190_q(MinecraftServer.java:709)
	at net.minecraft.server.dedicated.DedicatedServer.func_71190_q(DedicatedServer.java:387)
	at net.minecraft.server.MinecraftServer.func_71217_p(MinecraftServer.java:613)
...

Stacktrace:
	at net.minecraft.server.dedicated.ServerHangWatchdog.run(ServerHangWatchdog.java:63)
	at java.lang.Thread.run(Thread.java:745)

-- System Details --
Details:
	Minecraft Version: 1.10.2
	Operating System: Linux (amd64) version 3.14.32-xxxx-grs-ipv6-64
	Java Version: 1.8.0_101, Oracle Corporation
	Java VM Version: OpenJDK 64-Bit Server VM (mixed mode), Oracle Corporation
	Memory:<PHONE_NUMBER> bytes (1966 MB) /<PHONE_NUMBER> bytes (5078 MB) up to<PHONE_NUMBER> bytes (5120 MB)
	JVM Flags: 20 total; -Xms3G -Xmx5G -XX:+UseG1GC -XX:+UseStringDeduplication -XX:ParallelGCThreads=2 -XX:+CMSClassUnloadingEnabled -XX:+CMSIncrementalPacing -XX:+AggressiveOpts -XX:NewRatio=3 -XX:SurvivorRatio=3 -XX:TargetSurvivorRatio=80 -XX:MaxTenuringThreshold=8 -XX:MaxGCPauseMillis=10 -XX:GCPauseIntervalMillis=50 -XX:+ExplicitGCInvokesConcurrent -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+BindGCTaskThreadsToCPUs -Xnoclassgc -Xss4M
	IntCache: cache: 0, tcache: 0, allocated: 12, tallocated: 94
	FML: MCP 9.32 Powered by Forge <IP_ADDRESS>5 104 mods loaded, 104 mods active
	States: 'U' = Unloaded 'L' = Loaded 'C' = Constructed 'H' = Pre-initialized 'I' = Initialized 'J' = Post-initialized 'A' = Available 'D' = Disabled 'E' = Errored
...

This is with EnderIO-1.10.2-<IP_ADDRESS>_beta

From the same profiling:

crazypants.enderio.capacitor.CapacitorKey.get() takes ~6% CPU by itself, out of 22% taken by crazypants.enderio.conduit.power.PowerConduitNetwork.doNetworkTick().
A opened view of the item conduit lag:

Just putting a comment here that im running version .120 and im getting extreme conduit lag. After taking apart my mods its come down to me leading to ender io. I had to create an rfworld to escape the lag but of course eventually built conduits. At that point even that world is lagged (only contains cobble, ender io spawners and conduits). After removing ender io the whole world is seemless with no lag.
Ive only used the green (20krf/t) conduit and item conduit if that matters.

I'm unable to reproduce the lag in single player with just EnderIO.
@sherkas can you share pictures of your setup?

The problem causing this lag is Sponge. Update it or follow the steps mentioned in this issue

I currently am not using Sponge and am experiencing this issue. Shutting off an EnderIO autocrafting set up it stops the lag on the server that has been slowly building up and crashing things.

Sorry for not being able to send a WarmRoast report and/or an other profiling proof now, but my server is having lag spikes, and EnderIO seems to be causing that. The class is the same. Here is the timings: https://timings.aikar.co/?id=3f7c72898cf64f8eb6b5b43b26224c1d

I wouldn't believe a single timing from that Aikar tool, it has the tendency to measure complete nonsense.

The reason is that I'm reporting the SAME issue as the people above, so it should matter.

So, you report that you have the same issue as people had 3 months ago but cannot give any data that would enable me to do anything about it. Not even the Ender IO version you are using.
That should matter? No, it doesn't.
The data you present is useless on so many levels. I see that half your tick time is used up by all of your world's conduits combined. So if you have a world with 8000 conduits constantly shuffling around items, fluids and energy and using the latest Ender IO, I'd call that an extreme success for all those countless optimisations I added in the last months. If you have it with the Ender IO version from 4 months ago and a single conduit in your world, I'd call it a bad case of not updating. That is if that measurement even is correct. Just yesterday I got an Aikar report that claimed the conduit code spending its time adding energy capabilities to item stacks---which is just one example of the nonsense it measures.

PS: Which reminds me...: Closing this issue for being outdated. The code in question has changed too much since November.

Sorry, you could have replied me about the mod versions. I replied here because of I don't like opening duplicated issues, so I first search for already opened ones. In the Timings you can see the count of conduits, and my server is new and doesn't have chunk loaders, my guess is that there are 200 +/- conduits in the world.
EnderIO-1.10.2-3.1.164
EnderCore-1.10.2-<IP_ADDRESS>-beta
Forge 1.10.2-<IP_ADDRESS>2
My modpack is this one, and you can see the mod list here.

also the version is directly in the report..
https://timings.aikar.co/?id=3f7c72898cf64f8eb6b5b43b26224c1d&section=plugins

Still, that output is next to useless.
First, it says you have 1,283 conduits. Then it says the processing for all conduits takes "8.56% of tick". And in another view it says it takes "50.31% of tick", which it presumable spends mostly in "EnergyUpgradePowerAdapter_attachCapabilities"---a method that has nothing to do with conduits at all.


[SLM] Chat CLI

Chat with Llama-2-7b-chat-hf q4f16
python -m  mlc_chat chat HF://junrushao/Llama-2-7b-chat-hf-q4f16_1-MLC \
    --device "cuda:0" \
    --overrides context_window_size=1024 \
    --opt "O2"

special commands in CLI:
  /help               print the special commands
  /exit               quit the cli
  /stats              print out the latest stats (token/sec)
  /reset              restart a fresh chat
  /set [overrides]    override settings in the generation config. For example,
                      `/set temperature=0.5;max_gen_len=100;stop=end,stop`
                      Note: Separate stop words in the `stop` option with commas (,).
  Multi-line input: Use escape+enter to start a new line.

You may want to add @Sing-Li as the co-author of this PR as it's inspired by his :))


Rename this project to steal-component

This project is a steal plugin, so I would prefer to rename it to what it really does.
Maybe rename it to steal-can-component
@matthewp what do you think.

DoneJS is (mostly) Steal + Can so I think the name is appropriate as done-component. It combines the two.  steal-can-component is the same as done-component 😉

Let's close this unless you feel really strongly about it..


Fix #220

This makes it possible to run build_membership.py on Python 3.5 which doesn't allow from PySide2 import * on anything but module level. See #220.
Also, I don't think (?) that the QtOpenGL module should be unsupported, so this is now included in the members dict.
Here's the output I'm getting on Ubuntu 16.04 Xenial, Python 3.5.1, PySide2 built against the 5.9 branch, which could potentially prompt for a dictionary update in Qt.py (related: #219).
 So the tests on Python 2.7 failed but they were fine on 3.5.
I checked on Gitter/PySide2 and QtOpenGL is supported but won't get much attention as it is deprecated in Qt itself: https://gitter.im/PySide/pyside2?at=5979e6dcc101bc4e3ac62bb2
And I can successfully use QtOpenGL in Python 2.7 using a wheel built recently against the 5.6 branch.

@thopiekar ping!

@fredrikaverpil Yes, you seem to be right: https://launchpad.net/~thopiekar/+archive/ubuntu/pyside-git/+packages
You will have the best chance in having something working, when using Ubuntu Zesty or Artful!
If you like to fix this for your Ubuntu version feel free to contribute at: https://github.com/thopiekar/PySide-packaging

@thopiekar thanks! Having given this some thought, I think this issue will come up again so perhaps we need to find a different solution altogether.
@mottosso hm. What do you think?
I'm starting to think we should build Qt ourselves and then also build PySide2 ourselves. We could do this by making use of Docker images which are chained using FROM commands. One Dockerfile to build Qt which is stored as an image. Then another Dockerfile to build PySide2 using the image of the previous step. Then finally, our current Dockerfiles would not use FROM ubuntu:16.04 but instead they would use the PySide2-built image. I could mock something up.

I'm starting to think we should build Qt ourselves and then also build PySide2 ourselves.

That would be great, there are just two problems with that.

Time - Travis will time out after an hour of activity, and building each of those can take some time. We could solve this by building elsewhere and uploading it somewhere, but that's a lot of work
Compatibility - The thing we build and test against is unlikely to match what others use. At least now we use something others are also using, increasing the area of compatibility. What I had in mind here was to follow the VFX Platform, and stick to whatever versions they list. If we build towards that, then this might not be an issue either.

@mottosso Actually, we probably don't need to build Qt. But building PySide2 ourselves could be useful...

Time - Travis will time out after an hour of activity, and building each of those can take some time. We could solve this by building elsewhere and uploading it somewhere, but that's a lot of work

Not really, actually. We could just create a Dockerfile and store the image over at Dockerhub. We run the container and update the image locally when required. Then Travis just uses that image instead of ubuntu:16.04.

Compatibility

Well, true. But we just test on one version of Ubuntu. I believe most people use CentOS in the VFX world.

What I had in mind here was to follow the VFX Platform, and stick to whatever versions they list. If we build towards that, then this might not be an issue either.

Yes. Makes sense.

@mottosso so how does this sound:

Create Dockerfile using CentOS7
Install pre-built Qt 5.6
Git clone the pyside-setup, 5.6 branch and build PySide2
Commit image to Dockerhub
Have our current Dockerfiles (in this repo) use our custom image

Then our Dockerfiles, currently in this repo, start by pip-installing the PySide2 wheel which exists on the image. The image already has pre-built Qt installed so we don't have to worry about the wheel being non-portable.
Whenever something needs to be changed, we can re-build the Dockerfile which compiles PySide2.

Sounds good to me!
Could we build the version listed on this years VFX Platform? Do you think we could build each binding this way?

And what do you think about having Dockerfile.2017, Dockerfile.2016 etc. so that we can test against all VFX Platform specifications?

Yes, this sounds good. I'll take a stab at it.

Well, I'm currently building nightly builds in my PPA. Also the test suite is enabled, so what I have is a kind of docker setup. Whenever the build incl tests fails, no package is built.

@thopiekar I can see metions of "pyside2-2.0.0+dev" in the build log. QtC changed their git repo so that you need to use their 5.6 branch instead of the dev one.
I quickly glanced through the log but could see no mention of a git clone. Do you clone their git repo?
This is what I do:
git clone --recursive --branch 5.6 https://codereview.qt-project.org/pyside/pyside-setup

Yes, it mentions dev but 5.6 is being used. This is the build recipe for
5.6:
https://code.launchpad.net/~thopiekar/+recipe/pyside2-5.6-daily
However, all these builds fail I think because the Shiboken builds are
broken, too.
https://code.launchpad.net/~thopiekar/+recipe/shiboken2-5.6-daily

@thopiekar Do you mean you think something is wrong with the git source?
I'm unfamiliar with launchpad and how that works... but I can build PySide2 (and shiboken2) just fine from the PySide2 5.9 branch.

@fredrikaverpil No, this is not what I mean.
First of all, I'm building all packages without PySide2-setup, but with my own scripts.
As you can see from the URLs I sent before there are builds on some Ubuntu releases which fail.
For example this buildlog from Ubuntu Xenial amd64 https://launchpadlibrarian.net/326735108/buildlog_ubuntu-xenial-amd64.shiboken2_2.0.0.dev0-0~201707041042~rev1707~pkg30~ubuntu16.04.1_BUILDING.txt.gz
says fatal error: QtCore/QVersionNumber: No such file or directory.

@mottosso I built a Docker image which follows CY2018, builds and installs Autodesk-modified Qt 5.6.1 and PySide2. It's around 4 GB in size so it would clearly slow Travis down but it shouldn't come close to timing out.
https://github.com/fredrikaverpil/vfxplatform-docker
I should probably also build and install PyQt5 from this Qt installation and add that to the image.
We could then create new Dockerfiles for Qt.py which starts with loading such an image and adds PySide(1) and PyQt4 as well as xvfb etc to it (for these we don't have to build from source). However, I also wanted to try to just up the Ubuntu version in the current Dockerfiles just to see if that could perhaps finish this PR. But upping to 17.04 didn't go well:
$ docker run --rm -v $(pwd):/Qt.py mottosso/qt.py35
Waiting for Xvfb...
#
# Testing implementation..
#
# Running tests with PyQt4..
#
Tests require all bindings to be installed ... ok
load_ui returns an instance of QObject ... ok
Tests to see if the baseinstance loading loads widgets on properly ... ok
Tests to see if the baseinstance loading loads widgets on properly ... ok
Tests to see if loadUi successfully fails on invalid paths ... ok
Tests to see if loadUi successfully fails on invalid ui files ... ok
PyQt4/5 loadUi functiion will fail if the widget has a preexisting ... ok
Preferring None shouldn't import anything ... ok
Qt.py may be bundled along with another library/project ... ok
python -m Qt --convert works in general ... ok
Converting a converted file produces an identical file ... ok
Converting produces a backup ... ok
Fix #133, `from Qt.QtWidgets import XXX` works ... ok
import Qt.QtCore works on all bindings ... ok
Arguments of QtCompat.translate are correct ... ok
Qt's __binding_version__ and __qt_version__ populated ... ok
Tests to see if the Qt binding enum states are set properly ... ok
Qt.py is available from the command-line ... ok
.wrapInstance and .getCppPointer is identical across all bindings ... ok
.wrapInstance doesn't need the `base` argument ... ok
QT_PREFERRED_BINDING = PyQt4 properly forces the binding ... ok
Preferred binding PyQt4 should have sip version 2 ... ok

----------------------------------------------------------------------
Ran 22 tests in 8.875s

OK
#
# Running tests with PySide..
#
Tests require all bindings to be installed ... ok
load_ui returns an instance of QObject ... ok
Tests to see if the baseinstance loading loads widgets on properly ... ok
Tests to see if the baseinstance loading loads widgets on properly ... ok
Tests to see if loadUi successfully fails on invalid paths ... ok
Tests to see if loadUi successfully fails on invalid ui files ... ok
PyQt4/5 loadUi functiion will fail if the widget has a preexisting ... ok
Preferring None shouldn't import anything ... ok
Qt.py may be bundled along with another library/project ... ok
python -m Qt --convert works in general ... ok
Converting a converted file produces an identical file ... ok
Converting produces a backup ... ok
Fix #133, `from Qt.QtWidgets import XXX` works ... ok
import Qt.QtCore works on all bindings ... ok
Arguments of QtCompat.translate are correct ... ok
Qt's __binding_version__ and __qt_version__ populated ... ok
Tests to see if the Qt binding enum states are set properly ... ok
Qt.py is available from the command-line ... ok
QT_PREFERRED_BINDING = PySide properly forces the binding ... ok
QT_PREFERRED_BINDING = more than one binding excludes others ... ok

----------------------------------------------------------------------
Ran 20 tests in 8.279s

OK
#
# Running tests with PyQt5..
#
Tests require all bindings to be installed ... ok
load_ui returns an instance of QObject ... ERROR
Tests to see if the baseinstance loading loads widgets on properly ... ERROR
Tests to see if the baseinstance loading loads widgets on properly ... ERROR
Tests to see if loadUi successfully fails on invalid paths ... ERROR
Tests to see if loadUi successfully fails on invalid ui files ... ERROR
PyQt4/5 loadUi functiion will fail if the widget has a preexisting ... ERROR
Preferring None shouldn't import anything ... ok
Qt.py may be bundled along with another library/project ... ok
python -m Qt --convert works in general ... ok
Converting a converted file produces an identical file ... ok
Converting produces a backup ... ok
Fix #133, `from Qt.QtWidgets import XXX` works ... ok
import Qt.QtCore works on all bindings ... ok
Arguments of QtCompat.translate are correct ... ok
Qt's __binding_version__ and __qt_version__ populated ... ok
Tests to see if the Qt binding enum states are set properly ... ok
Qt.py is available from the command-line ... ok
.wrapInstance and .getCppPointer is identical across all bindings ... ERROR
.wrapInstance doesn't need the `base` argument ... ERROR
QT_PREFERRED_BINDING = PyQt5 properly forces the binding ... ok

======================================================================
ERROR: load_ui returns an instance of QObject
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 134, in run
    self.runTest(result)
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 152, in runTest
    test(result)
  File "/usr/local/lib/python3.5/dist-packages/nosepipe.py", line 179, in __call__
    (header + data).decode("latin1"))
Exception: short message body (want<PHONE_NUMBER>, got 97)
Something went wrong
Message: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'

addSuccesstopTest

======================================================================
ERROR: Tests to see if the baseinstance loading loads widgets on properly
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 134, in run
    self.runTest(result)
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 152, in runTest
    test(result)
  File "/usr/local/lib/python3.5/dist-packages/nosepipe.py", line 179, in __call__
    (header + data).decode("latin1"))
Exception: short message body (want<PHONE_NUMBER>, got 97)
Something went wrong
Message: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'

addSuccesstopTest

======================================================================
ERROR: Tests to see if the baseinstance loading loads widgets on properly
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 134, in run
    self.runTest(result)
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 152, in runTest
    test(result)
  File "/usr/local/lib/python3.5/dist-packages/nosepipe.py", line 179, in __call__
    (header + data).decode("latin1"))
Exception: short message body (want<PHONE_NUMBER>, got 97)
Something went wrong
Message: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'

addSuccesstopTest

======================================================================
ERROR: Tests to see if loadUi successfully fails on invalid paths
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 134, in run
    self.runTest(result)
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 152, in runTest
    test(result)
  File "/usr/local/lib/python3.5/dist-packages/nosepipe.py", line 179, in __call__
    (header + data).decode("latin1"))
Exception: short message body (want<PHONE_NUMBER>, got 97)
Something went wrong
Message: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'

addSuccesstopTest

======================================================================
ERROR: Tests to see if loadUi successfully fails on invalid ui files
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 134, in run
    self.runTest(result)
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 152, in runTest
    test(result)
  File "/usr/local/lib/python3.5/dist-packages/nosepipe.py", line 179, in __call__
    (header + data).decode("latin1"))
Exception: short message body (want<PHONE_NUMBER>, got 97)
Something went wrong
Message: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'

addSuccesstopTest

======================================================================
ERROR: PyQt4/5 loadUi functiion will fail if the widget has a preexisting
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 134, in run
    self.runTest(result)
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 152, in runTest
    test(result)
  File "/usr/local/lib/python3.5/dist-packages/nosepipe.py", line 179, in __call__
    (header + data).decode("latin1"))
Exception: short message body (want<PHONE_NUMBER>, got 97)
Something went wrong
Message: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'

addSuccesstopTest

======================================================================
ERROR: .wrapInstance and .getCppPointer is identical across all bindings
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 134, in run
    self.runTest(result)
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 152, in runTest
    test(result)
  File "/usr/local/lib/python3.5/dist-packages/nosepipe.py", line 179, in __call__
    (header + data).decode("latin1"))
Exception: short message body (want<PHONE_NUMBER>, got 97)
Something went wrong
Message: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'

addSuccesstopTest

======================================================================
ERROR: .wrapInstance doesn't need the `base` argument
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 134, in run
    self.runTest(result)
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 152, in runTest
    test(result)
  File "/usr/local/lib/python3.5/dist-packages/nosepipe.py", line 179, in __call__
    (header + data).decode("latin1"))
Exception: short message body (want<PHONE_NUMBER>, got 97)
Something went wrong
Message: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'

addSuccesstopTest

----------------------------------------------------------------------
Ran 21 tests in 8.461s

FAILED (errors=8)
#
# Running tests with PySide2..
#
Tests require all bindings to be installed ... ok
load_ui returns an instance of QObject ... ERROR
Tests to see if the baseinstance loading loads widgets on properly ... ERROR
Tests to see if the baseinstance loading loads widgets on properly ... ERROR
Tests to see if loadUi successfully fails on invalid paths ... ERROR
Tests to see if loadUi successfully fails on invalid ui files ... ERROR
PyQt4/5 loadUi functiion will fail if the widget has a preexisting ... ERROR
Preferring None shouldn't import anything ... ok
Qt.py may be bundled along with another library/project ... ok
python -m Qt --convert works in general ... ok
Converting a converted file produces an identical file ... ok
Converting produces a backup ... ok
Fix #133, `from Qt.QtWidgets import XXX` works ... ok
import Qt.QtCore works on all bindings ... ok
Arguments of QtCompat.translate are correct ... ok
Qt's __binding_version__ and __qt_version__ populated ... ok
Tests to see if the Qt binding enum states are set properly ... ok
Qt.py is available from the command-line ... ok
.wrapInstance and .getCppPointer is identical across all bindings ... ERROR
.wrapInstance doesn't need the `base` argument ... ERROR
QT_PREFERRED_BINDING = PySide2 properly forces the binding ... ok
Qt.py may be use alongside the actual binding ... ok
QT_PREFERRED_BINDING = more than one binding excludes others ... ok

======================================================================
ERROR: load_ui returns an instance of QObject
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 134, in run
    self.runTest(result)
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 152, in runTest
    test(result)
  File "/usr/local/lib/python3.5/dist-packages/nosepipe.py", line 179, in __call__
    (header + data).decode("latin1"))
Exception: short message body (want<PHONE_NUMBER>, got 97)
Something went wrong
Message: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'

addSuccesstopTest

======================================================================
ERROR: Tests to see if the baseinstance loading loads widgets on properly
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 134, in run
    self.runTest(result)
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 152, in runTest
    test(result)
  File "/usr/local/lib/python3.5/dist-packages/nosepipe.py", line 179, in __call__
    (header + data).decode("latin1"))
Exception: short message body (want<PHONE_NUMBER>, got 97)
Something went wrong
Message: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'

addSuccesstopTest

======================================================================
ERROR: Tests to see if the baseinstance loading loads widgets on properly
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 134, in run
    self.runTest(result)
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 152, in runTest
    test(result)
  File "/usr/local/lib/python3.5/dist-packages/nosepipe.py", line 179, in __call__
    (header + data).decode("latin1"))
Exception: short message body (want<PHONE_NUMBER>, got 97)
Something went wrong
Message: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'

addSuccesstopTest

======================================================================
ERROR: Tests to see if loadUi successfully fails on invalid paths
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 134, in run
    self.runTest(result)
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 152, in runTest
    test(result)
  File "/usr/local/lib/python3.5/dist-packages/nosepipe.py", line 179, in __call__
    (header + data).decode("latin1"))
Exception: short message body (want<PHONE_NUMBER>, got 97)
Something went wrong
Message: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'

addSuccesstopTest

======================================================================
ERROR: Tests to see if loadUi successfully fails on invalid ui files
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 134, in run
    self.runTest(result)
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 152, in runTest
    test(result)
  File "/usr/local/lib/python3.5/dist-packages/nosepipe.py", line 179, in __call__
    (header + data).decode("latin1"))
Exception: short message body (want<PHONE_NUMBER>, got 97)
Something went wrong
Message: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'

addSuccesstopTest

======================================================================
ERROR: PyQt4/5 loadUi functiion will fail if the widget has a preexisting
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 134, in run
    self.runTest(result)
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 152, in runTest
    test(result)
  File "/usr/local/lib/python3.5/dist-packages/nosepipe.py", line 179, in __call__
    (header + data).decode("latin1"))
Exception: short message body (want<PHONE_NUMBER>, got 97)
Something went wrong
Message: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'

addSuccesstopTest

======================================================================
ERROR: .wrapInstance and .getCppPointer is identical across all bindings
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 134, in run
    self.runTest(result)
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 152, in runTest
    test(result)
  File "/usr/local/lib/python3.5/dist-packages/nosepipe.py", line 179, in __call__
    (header + data).decode("latin1"))
Exception: short message body (want<PHONE_NUMBER>, got 97)
Something went wrong
Message: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'

addSuccesstopTest

======================================================================
ERROR: .wrapInstance doesn't need the `base` argument
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 134, in run
    self.runTest(result)
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 152, in runTest
    test(result)
  File "/usr/local/lib/python3.5/dist-packages/nosepipe.py", line 179, in __call__
    (header + data).decode("latin1"))
Exception: short message body (want<PHONE_NUMBER>, got 1028)
Something went wrong
Message: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
\xadaddFailure:\x80cbuiltins
AssertionError
qX\xcf
-------------------- >> begin captured stdout << ---------------------
Order: 'PySide2'
Trying PySide2
'QtCore.QMetaType' was missing.

--------------------- >> end captured stdout << ----------------------qcnosepipe
Traceback
q)\x81q}q(X     tb_linenoqK;tb_frameqcnosepipe
Frame
q)\x81}q        (Xf_codeq
cnosepipe
Code
q
 )\x81q
(X     }q
  co_filenameqX#/usr/lib/python3.5/unittest/case.pyqXco_nameqXtestPartExecutorqubX      f_globalsq}q__file__qX#/usr/lib/python3.5/unittest/case.pyqsubXtb_nextqh)\x81q}q(hMYhh)\x81q}q(h
h
 )\x81q(hhhXrunqubh}qhhsubhh)\x81q}q (hK\xc6hh)\x81q!}q"(h
h
 )\x81q#}q$(hX3/usr/local/lib/python3.5/dist-packages/nose/case.pyq%hXrunTestq&ubh}q'hh%subhh)\x81q(}q)(hM\xe1hh)\x81q*}q+(h
h
 )\x81q,}q-(hX/workspace/Qt.py/tests.pyq.hXtest_implicit_wrapInstanceq/ubh}q0hX/workspace/Qt.py/tests.pyq1subhNubububub\x87q2stopTest

----------------------------------------------------------------------
Ran 23 tests in 10.335s

FAILED (errors=8)
Traceback (most recent call last):
  File "run_tests.py", line 54, in <module>
    raise Exception("%i binding(s) failed." % errors)
Exception: 2 binding(s) failed.

For some reason, there's a problem with QtCompat.loadUi and only on PyQt5. This happens with both Python 2 and 3. Hm...

Is that output from changing to 17, or from this new Docker image?

That log was from our current Docker images in this repo, but with FROM ubuntu:17.04.

Ah ok. Yeah, I chose 16 specifically because it had the right combination of available versions of each binding.
About your VFX Platform image, does it work? Should we give it a go?

About your VFX Platform image, does it work? Should we give it a go?

Seems to work, yes :)
You can see from the build log (attached as .zip to release) that Qt5WebEngine wasn't included in the build. But apart from that, PySide2 finds everything it's looking for. I'll have to investigate why that's not included.

We'd have to install Python 3 and build PySide2 though. So we should probably create a new intermediate image which our Qt.py Dockerfiles are based off.

Closing, as I will deal with this in https://github.com/mottosso/Qt.py/pull/227 instead.


constraint form clean methods validation to only forms within the project

this modification constrains the default behavior for checking Form classes' clean methods to only those forms that exist within the project. forms which would fail this test but exist in third-party libraries are ignored. I added a configuration to enable strict validation if one would desire it.
a practical use case is django-oscar, which does indeed contain a form that fails the form clean validity check, thus breaking the project through third-party code, which is something we probably want to avoid.
ps: don't hate me but I reformatted with isort and ruff/black. it seemed nice to do ;(

I can't accept this change due to the reformat. It's unreviewable. If you propose to reformat, you need to have a separate commit for the reformat, separate from the reformat.

okilydokily. I have undone the formatting and stuck with the basics. this has been formatted fully in another branch called blacked that I can PR if you like (really, isort and black make maintainability a lot easier by enforcing standard stylization imho).


QGIS project & projection

I created a new project in QGIS, added an openStreetMap layer and changed the projection (for instance to World_Aitoff). Then I saved my QGIS project and opened it in JupyterGIS. The QGIS project opens (in my local installation of JupyterGIS) but the layer does not appear. Below is what I have in QGIS:

in #196

We may want to start looking into using https://github.com/proj4js/proj4js to support other projection types
There is an example here https://openlayers.org/en/latest/examples/wms-image-custom-proj.html using proj4js with OpenLayers

https://github.com/josueggh/proj4-list/blob/master/src/list.js


Remove pip as the primary installation method

airflowctl is affected by the chicken-and-egg problem of Python version. No surprisingly it has the same burden as other Python environment & project managers (e.g., hatch, pipx, pyenv) :
If we start with pip, the version used in airflowctl built must match the same version as your pip environment:
  $ python -V
  Python X.Y.Z

  $ python -m pip install airflowctl
  $ airflowctl init --python-version=X.Y.Z .
  $ airflowctl build .

Otherwise pyenv is invoked to get the correct python version (forcing users to use a specific version manager).
The above is the ideal instruction if we're going to use pip. With the current instructions, you can run into a lot of issues if you have one of these situation:

--python-version is not the same as the version used in pip install airflowctl
--python-version is not specified
pyenv is not available
python is a different version than the one installed with pip. For example:

In a broken environment where python and pip are related
User later on upgraded Python and did not reinstall airflowctl.

Here are my suggestions:

Installation instructions should remove pip as the primary method.
Ideally, airflowctl behaviors should not depend on the Python version it is installed to.
airflowctl build should fail if python_version does not exist in $PATH. Error message should ask users to have the correct Python version.

Alternatively, users can override which Python path to build the virtual environment with.

Provide a Python-agnostic installation method (e.g., brew, apt, yum, winget, choco, etc.)

Alternatively, although a bigger rewrite, switch to using hatch or rye.

I have switched to using uv and created 0.2.11. Let me know if you try it out

$ uv tool install airflowctl
$  airflowctl init my_airflow_project --build-start
$ ls -la my_airflow_project
total 1416
drwxr-xr-x  15 keto  staff     480 Nov 18 10:28 .
drwx------   3 keto  staff      96 Nov 18 10:27 ..
drwxr-xr-x   3 keto  staff      96 Nov 18 10:27 .airflowctl
-rw-r--r--   1 keto  staff     316 Nov 18 10:27 .env
-rw-r--r--   1 keto  staff     150 Nov 18 10:27 .gitignore
drwxr-xr-x   9 keto  staff     288 Nov 18 10:27 .venv
-rw-------   1 keto  staff   87628 Nov 18 10:27 airflow.cfg
-rw-r--r--   1 keto  staff  581632 Nov 18 10:28 airflow.db
drwxr-xr-x   4 keto  staff     128 Nov 18 10:28 dags
drwxr-xr-x   4 keto  staff     128 Nov 18 10:28 logs
drwxr-xr-x   2 keto  staff      64 Nov 18 10:27 plugins
-rw-r--r--   1 keto  staff       0 Nov 18 10:27 requirements.txt
-rw-r--r--   1 keto  staff     717 Nov 18 10:27 settings.yaml
-rw-------   1 keto  staff      16 Nov 18 10:27 standalone_admin_password.txt
-rw-r--r--   1 keto  staff    4762 Nov 18 10:28 webserver_config.py

That's much better. Thank you!


Publish fails when project contains linked Content (dup of #359)

I'm still hitting exactly the issue described in #359: if there's linked content then publishing through Visual Studio fails.  Actually, even previewing through the Publish wizard fails, with the error

Error: InvalidParameter
Parameter name: index

Was there a commit that was supposed to have fixed that issue (there's no discussion there, so it's hard to tell whether it was directly addressed), or could it have regressed?

Closing. We're re-evaluating publishing from VS, for now if you run into issues please use Azure CLI


UVI Request

--- UVI JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from  to before v5.10.53",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=a4a54c54af2516caa9c145015844543cfc84316a"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "a4a54c54af2516caa9c145015844543cfc84316a",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "scsi: libfc: Fix array index out of bound exception\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v5.10.53 by commit a4a54c54af2516caa9c145015844543cfc84316a. For more details please see the references link."
}
--- UVI JSON ---

This issue has been assigned UVI-2021-1001254


When training classifier_train.py with my own dataset, the loss never drops and the acc is very low!

I trained the classifier on my own dataset, which has a total of 100 images divided into 5 classes with 20 images in each class. Each image is in the form of label1_xxx.jpg, label2_xxx.jpg named. But when training, the loss is always big and not decreasing, the acc is always low and my batch_size is 4. Why is this? Is the dataset too small?

hi,have you solved the problem?
and is it ok for you to share your commands  ,thanks.


WIP: Add option to fix value associated with the HF bitstring

This looks like a correct and fine addition. I believe my only comment is that it's not clear to me what it means precisely to "fix" the first HF value (the code/comment implies it is dropped from the tensor prep circuits?), and why one might want to do this. I wonder if there's an improvement that could be made to another section of the documentation other than the docstrings (e.g., explanatory material).

Yes, there is some good documentation on this feature in the prototype. I will move it over in this PR