Large quantity of schedules/backups cause high memory usage over time

We are creating schedules for each namespace - around 100 of them. Each schedule runs a backup every hour, with a ttl of 7 days. The schedules are offset by one minute each, so at most two run at the same minute.
Current backup counts:
Completed: 13286
Failed: 949
After running for about 5 days the memory usage of the ark container is 4gb and continues to increase.
Setting a memory limit of 1gb causes the container to restart with OOM errors after about 12 hours. Upping the limit to 1.5gb increases the time between restarts, but it still OOM crashes after a while. The OOM crashes cause failed backups.
Possible memory leak?
The logs show failures due to the OOM issue, but don't really indicate any issues.
Sample schedule and backup: https://gist.github.com/twforeman/f1e2289360a5c5c29701b284f7928ea4
The failed backups have no logs.
Environment:

Ark version (use ark version):

ark version
Version: v0.9.0
Git commit: v0.9.0
Git tree state: clean

Kubernetes version (use kubectl version):
We are running in Openshift 3.9:

oc version
oc v3.9.0+191fece
kubernetes v1.9.1+a0ce1bc657
openshift v3.9.30

Cloud provider or hardware configuration:
We are running in AWS and backups are stored in S3

OS (e.g. from /etc/os-release):
NAME="Red Hat Enterprise Linux Server"
VERSION="7.5 (Maipo)"

Thanks for sharing this information @twforeman.  We'll need to do some investigation here to see if we can figure out what's causing the runaway memory usage.  Could you also provide the count of DeleteBackupRequests (these are Ark-defined CRDs)?

@skriss we should also take advantage of the Golang metrics that are exposed through Prometheus.  This will give memory & GC details for the go runtime.
There area few Grafana dashboards as well: https://grafana.com/dashboards/3574 and https://grafana.com/dashboards/240

@twforeman kubectl get deletebackuprequests -n heptio-ark | wc -l should do the trick

Looks like oc get deletebackuprequests.ark.heptio.com -n heptio-ark should work (in Openshift) but it returns No resources found
The other CRDs return values, so I think there aren't any deletebackuprequests

OK, well that's a good sign.  We had issues in previous releases with a huge number of those CRDs piling up, so just wanted to rule that out as an issue.

@twforeman
We'll probably need to enable pprof support for the ark server and get some heap dumps from it. I can resurrect my PR for that and we can do some local simulations where we create ~15k backups and see how Ark behaves.
Does the average backup include PV snapshots? Could you possibly share an example backup yaml (redact any sensitive information, if applicable) - ark backup get <name> -o yaml?
Do you have any restores?

We are not currently backing up PVs. Mainly because the back end in the cluster is GlusterFS and I can't be bothered to figure out how to make that work. :) It's our test cluster. In prod we are using EBS volumes and I'll be backing those up.
 > ark get backup recommendations-test-20180818040501 -o yaml
apiVersion: ark.heptio.com/v1
kind: Backup
metadata:
  creationTimestamp: 2018-08-18T04:05:02Z
  labels:
    ark-schedule: recommendations-test
  name: recommendations-test-20180818040501
  namespace: heptio-ark
  resourceVersion: "79137277"
  selfLink: /apis/ark.heptio.com/v1/namespaces/heptio-ark/backups/recommendations-test-20180818040501
  uid: e180f6e4-a29b-11e8-93d6-0e811deaacd0
spec:
  excludedNamespaces: null
  excludedResources: null
  hooks:
    resources: null
  includeClusterResources: null
  includedNamespaces:
  - recommendations-test
  includedResources: null
  labelSelector: null
  ttl: 168h0m0s
status:
  completionTimestamp: 2018-08-18T04:05:07Z
  expiration: 2018-08-25T04:05:04Z
  phase: Completed
  startTimestamp: 2018-08-18T04:05:04Z
  validationErrors: null
  version: 1
  volumeBackups: null

We haven't had to do any restores yet, other than some testing to make sure it actually worked. It did. :)

It appears that the some of the underlying data structures used by the gzip Writer that we create for each backup are persisting in the heap instead of being garbage collected:
heap profile: 182: 86310864 [1738: 205693080] @ heap/1048576
115: 76308480 [115: 76308480] @ 0x1269e71 0x1275961 0x1aef655 0x1aefa5c 0x1de5998 0x1f675e7 0x1f6643d 0x1f83c8e 0x1f65cb1 0x1f65bbb 0x1f83cea 0x14237e4 0x14230cd 0x1422ffd 0x1f80d58 0x105b7e1
#	0x1269e70	compress/flate.NewWriter+0x30												/usr/local/Cellar/go/1.10.3/libexec/src/compress/flate/deflate.go:667
#	0x1275960	compress/gzip.(*Writer).Write+0x3b0											/usr/local/Cellar/go/1.10.3/libexec/src/compress/gzip/gzip.go:191
#	0x1aef654	github.com/heptio/ark/vendor/github.com/sirupsen/logrus.Entry.log+0x354							/Users/andy/go/src/github.com/heptio/ark/vendor/github.com/sirupsen/logrus/entry.go:114
#	0x1aefa5b	github.com/heptio/ark/vendor/github.com/sirupsen/logrus.(*Entry).Info+0xab						/Users/andy/go/src/github.com/heptio/ark/vendor/github.com/sirupsen/logrus/entry.go:141
#	0x1de5997	github.com/heptio/ark/pkg/backup.(*kubernetesBackupper).Backup+0x3c7							/Users/andy/go/src/github.com/heptio/ark/pkg/backup/backup.go:236
#	0x1f675e6	github.com/heptio/ark/pkg/controller.(*backupController).runBackup+0x3d6						/Users/andy/go/src/github.com/heptio/ark/pkg/controller/backup_controller.go:365
#	0x1f6643c	github.com/heptio/ark/pkg/controller.(*backupController).processBackup+0x5fc						/Users/andy/go/src/github.com/heptio/ark/pkg/controller/backup_controller.go:279
#	0x1f83c8d	github.com/heptio/ark/pkg/controller.(*backupController).(github.com/heptio/ark/pkg/controller.processBackup)-fm+0x3d	/Users/andy/go/src/github.com/heptio/ark/pkg/controller/backup_controller.go:102
#	0x1f65cb0	github.com/heptio/ark/pkg/controller.(*backupController).processNextWorkItem+0xd0					/Users/andy/go/src/github.com/heptio/ark/pkg/controller/backup_controller.go:191
#	0x1f65bba	github.com/heptio/ark/pkg/controller.(*backupController).runWorker+0x2a							/Users/andy/go/src/github.com/heptio/ark/pkg/controller/backup_controller.go:178
#	0x1f83ce9	github.com/heptio/ark/pkg/controller.(*backupController).(github.com/heptio/ark/pkg/controller.runWorker)-fm+0x29	/Users/andy/go/src/github.com/heptio/ark/pkg/controller/backup_controller.go:165
#	0x14237e3	github.com/heptio/ark/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x53					/Users/andy/go/src/github.com/heptio/ark/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
#	0x14230cc	github.com/heptio/ark/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc						/Users/andy/go/src/github.com/heptio/ark/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
#	0x1422ffc	github.com/heptio/ark/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c						/Users/andy/go/src/github.com/heptio/ark/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
#	0x1f80d57	github.com/heptio/ark/pkg/controller.(*backupController).Run.func2+0x67							/Users/andy/go/src/github.com/heptio/ark/pkg/controller/backup_controller.go:165

I'm continuing to investigate.

This has been fixed in master. In 0.9.x, commenting out https://github.com/heptio/ark/blob/release-0.9/pkg/backup/item_backupper.go#L324 appears to make this leak go away. I'd ideally like to fix the leak in 0.9.x, but I'm not yet sure what's holding on to the references here.

I've found out what is causing the leak. Our base go-plugin config is this:
https://github.com/heptio/ark/blob/release-0.9/pkg/plugin/manager.go#L44-L48
Managed: true is the source of the problem. When this is set, it instructs the go-plugin library to store a reference to all plugin clients ever instantiated. The expectation is that you'll eventually call the library's CleanupClients() function. We do call it, but unfortunately in 0.9.x it's a deferred call that only ever executes at server shutdown. In master, we don't set Managed: true and we handle killing plugin clients at the appropriate times.
In 0.9.x, to fix this, we'll need to set Managed: false and make sure that we're killing the plugin clients at all the right times. I think the only change required is to adjust manager.CleanupClients to kill all clients instead of calling go-plugin's function to clean up all managed clients.

This was fixed by #797 which has been released in v0.9.4+


Integrating with volar 2

This commit in nvim-lspconfig mentions that vue integration is now done via a tsserver plugin instead of using volar in take-over-mode. I think I got that working with typescript-tools.nvim, by installing @vue/typescript-plugin globally and adding this to my setup:
require("typescript-tools").setup {
  filetypes = {
    "javascript",
    "javascriptreact",
    "typescript",
    "typescriptreact",

    "vue", -- This needed to be added.
  },
  settings = {
    tsserver_plugins = {
      -- Seemingly this is enough, no name, location or languages needed.
      "@vue/typescript-plugin",
    },
  },
}

From what I can tell I think I go it working, but I am running into this problem as well. It seems to only happen in vue files, so it might just as well be volar, that's misbehaving and not typescript-tools.
Is there anything else we need to do for this setup? Also it might be good to add this setup to typescript-tools.nvim's README once vue integration fully works.

This commit in nvim-lspconfig mentions that vue integration is now done via a tsserver plugin instead of using volar in take-over-mode. I think I got that working with typescript-tools.nvim, by installing @vue/typescript-plugin globally and adding this to my setup:
require("typescript-tools").setup {
  filetypes = {
    "javascript",
    "javascriptreact",
    "typescript",
    "typescriptreact",

    "vue", -- This needed to be added.
  },
  settings = {
    tsserver_plugins = {
      -- Seemingly this is enough, no name, location or languages needed.
      "@vue/typescript-plugin",
    },
  },
}

From what I can tell I think I go it working, but I am running into this problem as well. It seems to only happen in vue files, so it might just as well be volar, that's misbehaving and not typescript-tools.
Is there anything else we need to do for this setup? Also it might be good to add this setup to typescript-tools.nvim's README once vue integration fully works.

I am using the same configuration as you.but after create a vue project with yarn create vite, i found some mistakes.

Same problem.
I plan to use tsserver to observe whether the issue occurs in typescript-tools.

This commit in nvim-lspconfig mentions that vue integration is now done via a tsserver plugin instead of using volar in take-over-mode. I think I got that working with typescript-tools.nvim, by installing @vue/typescript-plugin globally and adding this to my setup:
require("typescript-tools").setup {
  filetypes = {
    "javascript",
    "javascriptreact",
    "typescript",
    "typescriptreact",

    "vue", -- This needed to be added.
  },
  settings = {
    tsserver_plugins = {
      -- Seemingly this is enough, no name, location or languages needed.
      "@vue/typescript-plugin",
    },
  },
}

From what I can tell I think I go it working, but I am running into this problem as well. It seems to only happen in vue files, so it might just as well be volar, that's misbehaving and not typescript-tools.
Is there anything else we need to do for this setup? Also it might be good to add this setup to typescript-tools.nvim's README once vue integration fully works.

I am using the same configuration as you.but after create a vue project with yarn create vite, i found some mistakes.

Same problem.
I plan to use tsserver to observe whether the issue occurs in typescript-tools.

link: https://github.com/pmizio/typescript-tools.nvim/issues/250

You can now use takeover mode again in volar v2

You can now use takeover mode again in volar v2

If you could tell us the method, we would be very grateful.

現在您可以在 volar v2 中再次使用接管模式

如果您能告訴我們方法，我們將不勝感激。

volar = {
filetypes = { 'typescript', 'javascript', 'javascriptreact', 'typescriptreact', 'vue' },
init_options = {
vue = {
hybridMode = false,
},
},
},
But I still hope this issue can be resolved. I prefer a hybridMode.

現在您可以在 volar v2 中再次使用接管模式

如果您能告訴我們方法，我們將不勝感激。

volar = {
  filetypes = { 'typescript', 'javascript', 'javascriptreact', 'typescriptreact', 'vue' },
  init_options = {
    vue = {
      hybridMode = false,
    },
  },
 },

But I still hope this issue can be resolved. I prefer a hybridMode.

Does this mean we need to give up the typescript-tools.nvim plugin and use volar exclusively?

Or does it mean that typescript-tools works only with TypeScript files, while vue files are handled by volar?

Or does it mean that typescript-tools works only with TypeScript files, while vue files are handled by volar?

Turning off hybridMode means reverting to the old version 1.8 mode, where takeover mode needs to be enabled for Volar to handle TS and Vue files.
Re-importing the TS server is due to some unresolved compatibility issues. Perhaps for now, discontinuing hybridMode can lead to a better development experience, but hybridMode is what Volar aims for in the future.
I prefer using tsserver or typescript-tools + Volar for development.

Or does it mean that typescript-tools works only with TypeScript files, while vue files are handled by volar?

Turning off hybridMode means reverting to the old version 1.8 mode, where takeover mode needs to be enabled for Volar to handle TS and Vue files.
Re-importing the TS server is due to some unresolved compatibility issues. Perhaps for now, discontinuing hybridMode can lead to a better development experience, but hybridMode is what Volar aims for in the future.
I prefer using tsserver or typescript-tools + Volar for development.

I would prefer to use typescript-tools as well, but when I use hybridMode with typescript-tools I have the issue described here:
https://github.com/pmizio/typescript-tools.nvim/issues/250

You can temporarily disable semantic token for the client until it's fixed.
client.server_capabilities.semanticTokensProvider = false

You can temporarily disable semantic token for the client until it's fixed.
client.server_capabilities.semanticTokensProvider = false

When I do that I get no tsserver diagnostic:

@catgoose you can try out vtsls if you don't mind changing a tool. https://github.com/yioneko/vtsls/issues/148#issuecomment-2015051938

@catgoose you can try out vtsls if you don't mind changing a tool. yioneko/vtsls#148 (comment)

Thanks I will try it out!

Can temporarily solve the problem
on_attach = function(client)
  client.server_capabilities.semanticTokensProvider = false
end,

But I still hope @pmizio  can take some time to look into these issues 🙏

Hey! I've fixed semantic tokens issue. Is there anything else that is wrong with with vue? You can check out linked PR

I'm new to typescript (but not neovim/lsps, etc)... What exactly do I have to do to get this working with a fresh nuxt 3 project?
This is my config:
return {
  "pmizio/typescript-tools.nvim",
  dependencies = {
    "nvim-lua/plenary.nvim",
    "neovim/nvim-lspconfig",
  },
  config = function()
    require("typescript-tools").setup({
      filetypes = {
        "javascript",
        "typescript",
        "vue",
      }
    })
  end,
}

I can see popups if I do K to view info

But do I have to do anything to show errors?
Right now they show in vue-tsc, but they dont show in my editor

@9mm You shouldn't need to, here's my config. I've also installed @vue/typescript-plugin globally with npm and volar with mason.nvim & mason-lspconfig.nvim.
require("mason-lspconfig").setup({
	ensure_installed = {
                -- ...
		"volar",
                -- ...
	},
})

local capabilities = require("cmp_nvim_lsp").default_capabilities()
local lspconfig = require("lspconfig")
local servers = {
        -- ...
	"volar",
        -- ...
}
for _, lsp in ipairs(servers) do
	lspconfig[lsp].setup({
		capabilities = capabilities,
	})
end

require("typescript-tools").setup({
	filetypes = {
		"javascript",
		"javascriptreact",
		"typescript",
		"typescriptreact",
		"vue",
	},
	settings = {
		capabilities = capabilities,
		root_dir = root_pattern_exclude({
			root = { "package.json" },
			exclude = { "deno.json", "deno.jsonc" },
		}),
		single_file_support = false,
		tsserver_plugins = {
			"@vue/typescript-plugin",
		},
	},
})

You might try running :LspInfo to see if typescript-tools is running on that buffer and not some other language server?

thanks @Drew-Daniels ! it was actually 2 separate issues in volar and some other plugin that were both fixed simultaneously. just bad luck would have it that it was the DAY that i switched over, so i thought that it was something in my config. works perfect now!


Some vorbis files distorted over USB audio on macOS Big Sur

I have an Ogg Vorbis file that sounds distorted when played back with rodio over an USB audio device (have tried 2 different ones), but works over the headphone jack (macOS 11.3.1 on a Mac mini 2018; no problem on MacBook Pro 2015 with macOS 10.15.7).
I can reproduce on the current master, I just change the path to my file in examples/music_ogg.rs and run cargo run --example music_ogg. If I oggdec the file and use it with examples/music_ogg.wav, it works. Somehow, there must be a problem with decoding vorbis data under some circumstances (it works in VLC, ffplay, Audacity, so the Vorbis data itself should be no problem).
Here is the track in Vorbis and as WAV: https://easyupload.io/m/crad1n

It might be some sampling rate issue? No idea.

I did a bisect. Before commit 3bc614fc12fae8afcb81f568efe215fe6a968993, it was crackling way more (and the .wav was also crackling), so the changes seem to have done something, but not resolved it completely. But could also be unrelated.

It might be some sampling rate issue? No idea.

Seems so. The external USB interfaces were set to 48 kHz, whereas the internal one was set to 44.1 kHz. When using 44.1 kHz for the USB interface, the Vorbis file (which also is 44.1 kHz) works without crackling/distortion.
Will change the title accordingly.


🧱 Upgrade Terraform Version

Aim for the latest version.
Information here: https://www.terraform.io/language/upgrade-guides/1-1
(including resolving all warnings)

Hey team! Please add your planning poker estimate with ZenHub @ASTRobinson @bagg3rs @kyphutruong @mitchdawson1982 @mTouhid @moontune @wanieldilson

For the purposes of this upgrade we will use the newly spun up 'gh' terraform workspace.

This branch has been created in relation to this work. A PR will be created using it when it is complete.
This issue was also created for later resolution.

This branch is now completely upgraded to v0.14. I will liase with the team before working on the next upgrade to v1.1.8

Terraform upgraded to v0.14 successfully, commits merged to main, pipelines ran to completion.

New branch created, tested and merged into the pipeline successfully. The upgrade is complete.

Made a further commit in relation to the AWS VPC module which also needed updated, this had a variety of refactoring in it. Confirmed that metrics are still appearing in Production Grafana, to confirm the changes made in relation to monitoring had not broken anything.


Turn off scib if no metrics provided

I noticed scib script runs even if there are no cell type labels provided. I was wondering if we could add run: False param in the yaml or add checks in the pipeline_integration.py script to not submit unnecessary jobs to the cluster if no reference is provided.

I think scob can run in a celltype label or batch specific manner

Without celltypenlables it runs based on batch

I think run: True/False would be useful to prevent crashes - especially on smaller datasets.

sure i agree addding a run param is not a bad idea, as users can decide if they want it run at all, but all i was trying to say is without celltype label columns not provided it runs on batch

im happy to do this @bio-la @wlason when i work on panpipes related stuff tomorrow. unless you are already doing a PR @wlason

Happy for you to do it @deevdevil88 !

i have added a PR  #321 to add a run parameter for both LISI and scib metrics.


Remove kv.rangefeed.catchup_scan_concurrency cluster setting from the changefeed docs

Fixes DOC-9451
The kv.rangefeed.catchup_scan_concurrency cluster setting has been removed. This PR removes and adjusts the Advanced Changefeed Configuration page to also remove that setting. (No new content here.)

TFTRs!


Lockbox error when I'm on my rogue

23x NOP/nop-core.lua:502: bad argument #1 to 'gsub' (string expected, got table)
[string "=[C]"]: in function `gsub'
[string "@NOP/nop-core.lua"]:502: in function `CompressText'
[string "@NOP/nop-button.lua"]:340: in function `ButtonShow'
[string "@NOP/nop-item.lua"]:419: in function `ItemShow'
[string "@NOP/nop-item.lua"]:477: in function `?'
[string "@WeakAuras/Libs/AceTimer-3.0-17/AceTimer-3.0.lua"]:55: in function <WeakAuras/Libs/AceTimer-3.0/AceTimer-3.0.lua:50>
Locals:
(*temporary) =  {
castTime = 1500
name = "Pick Lock"
minRange = 0
originalIconID = 136058
iconID = 136058
maxRange = 5
spellID = 1804
}
(*temporary) = "
"
(*temporary) = "/n"
(*temporary) = "string expected, got table"

You seem to be on an older version. Could you please update?

Should be fixed with the next version.


Roll engine 64f18f2cb686..4dbb9ba388e1 (1 commits)

https://github.com/flutter/engine/compare/64f18f2cb686...4dbb9ba388e1
git log 64f18f2cb686649e7ba330e8254053f13d1526cd..4dbb9ba388e133f42fde26e722e7fc2dd4c1644d --no-merges --oneline
4dbb9ba38 Expose pointer type and buttons in embedder.h (flutter/engine#9034)
The AutoRoll server is located here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md
If the roll is causing failures, please contact the current sheriff (garyq@google.com), and stop
the roller if necessary.

Trybots failed. These were the failed builds: https://cirrus-ci.com/task/5681088106594304 , https://cirrus-ci.com/task/5118138153172992

Commit queue failed; closing this roll.


No tags

Hi,
Can you tag the latest release so the version can be locked down in bower?
Thanks

i'm not very familiar with git conventions for releases management, not sure what is "add tag"
i've sent invitation to you to project, feel free to add required things

I have added the tag and everything is working as expected now, thanks.


Lead mapper explanation

Usage friction. Users will not know what lead mapper is so they will likely do one of 2 things

ignore and leave it unchecked
stop the process to search around for an answer
Neither is good

@excelsior - Please add the following text:  "Only lead mappers can publish completed mappings."

Resolved


GET DOCKER ROOT DIR FROM SHELL

Coverage increased (+11.6%) to 63.428% when pulling 5fd66f34c7af078f83ebdafe3499ebfff76c5d82 on yuye/bug_fix into 45fb9385db65538d16b5feb2929e67773bc0eeb8 on master.


[Fix-10522] [API] Fix get executable time null pointer exception

Purpose of the pull request

this pr closes: #10522
Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

Codecov Report

Merging #10523 (25ae248) into dev (64cee03) will decrease coverage by 0.04%.
The diff coverage is 33.33%.

:exclamation: Current head 25ae248 differs from pull request most recent head b95a208. Consider uploading reports for the commit b95a208 to get more accurate results

@@             Coverage Diff              @@
##                dev   #10523      +/-   ##
============================================
- Coverage     40.88%   40.84%   -0.05%
+ Complexity     4855     4847       -8
============================================
  Files           886      887       +1
  Lines         36039    36039
  Branches       3999     3999
============================================
- Hits          14734    14719      -15
- Misses        19849    19860      +11
- Partials       1456     1460       +4

Impacted Files
Coverage Δ

...nscheduler/api/configuration/AppConfiguration.java
94.11% <0.00%> (ø)

...lphinscheduler/common/thread/BaseDaemonThread.java
0.00% <0.00%> (ø)

...ver/master/consumer/TaskPriorityQueueConsumer.java
0.00% <0.00%> (ø)

...ter/processor/queue/StateEventResponseService.java
0.00% <0.00%> (ø)

...uler/server/master/runner/EventExecuteService.java
0.00% <0.00%> (ø)

...er/server/master/runner/FailoverExecuteThread.java
0.00% <0.00%> (ø)

...r/server/master/runner/MasterSchedulerService.java
0.00% <0.00%> (ø)

.../server/master/runner/StateWheelExecuteThread.java
0.47% <0.00%> (-0.01%)
:arrow_down:

...pache/dolphinscheduler/service/corn/CronUtils.java
62.50% <0.00%> (-2.28%)
:arrow_down:

...eduler/api/configuration/TrafficConfiguration.java
83.33% <83.33%> (-16.67%)
:arrow_down:

... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 64cee03...b95a208. Read the comment docs.

@SbloodyS thx.


Spec Representation of Catalog Entries (Apps) within the Control Plane

Mon, Mar 23, '20 status in this story: https://github.com/giantswarm/roadmap/issues/95
Wed, Apr 8, '20 Spec: Representation of apps in the CP
Mon, Apr 20, '20 update: When spec is done, team needs a call to spec out implementation.
User Story
This is an internal story to solve for several internal technical needs and requirements of user facing stories.
This is the basis. The user stories depending on this are:

Listing of Apps
Updating of Apps (update events/notifications)
Defaulting of App CR fields (user just says I want prometheus and we default to latest version)
Validation of App CRs (specifically validate if app name and version are exisiting in said catalog)

Latter 2 stories are concerned with havng functionality baked into our Control Plane and its Kubernetes API so that we are not dependent on the to-be-deprecated GS API.
Future stories depending on this might be:

Dependency management between apps and other apps
Dependency management between apps and GS versions
Making required config fields explicit
Other metadata to apps (Screenshots, long descriptions, example configs)

Current State
Currently, an App Catalog is represented solely by its appcatalog CR in the CP. There's no information about its contents in the CP.
Happa talks directly to the storage of the Helm repository depicted in the appcatalog CR (i.e. in our cases Github) to get the index.yaml, parse it, and use it for some defaulting and representation cases.
The functionality to list apps is not present in our API. Thus, UX is being able to see the SSoT, i.e. Github, and following it to check out which apps are present.
Requirements
What we need is a way to get the SSoT, i.e. a list of apps, their versions, as well as metadata, from a definite place inside the Control Plane.
Options
As we are extending the Kubernetes API of the CP there's basically 2 ways that are native:

CRD + Controller (i.e. Operator Pattern)
Microservice that offers an aggregated API

While 1 has been our goto solution for most issues, we often have mitigated the impact of such an asynchronous, event-driven API towards the user by adding a microservice or API logic of our own. Latter patterns we want to get rid of as our API and the microservices that "translate" towards CRDs are something we want to deprecate mid-term.
For 2 we do not have any production experience. It is used upstream for the metrics API and Team Ludacris is considering using it for giantswarm/roadmap#13 (Health of TCs). There's a PoC that they have built in the Hackathon last week.
Team Batman has extensively evaluated Option 1 (see also https://docs.google.com/document/d/1LCd4XeLh8fYKILjTfyvBILrWZc07Gn0zdrbpC-MZUOg/edit), so we will describe this and also a lower fidelity spec of Option 2 in the following.
Keep in mind that no matter what option we go with the SSoT stays within the Helm Repository, no matter if that repository is blob storage with an index.yaml like right now, or might have a better API that can offer such information as in chart-museum and Harbor. Currently, I do not see a way around having a local representation somehow (more on this in the harbor case below). Correct me if this assumption is wrong.
1) AppCatalogEntry CRD
The idea is to have a cluster-scoped AppCatalogEntry CR per App in a Catalog. This CR is to be created by a new controller in app-operator based on the SSoT that is the appCatalog storage, i.e. github or other helm repo and its index.yaml
Sample CR (details can be discussed in apiextensions PR once we have a better plan here)
metadata:
  name: kong-incubator
  labels:
   application.giantswarm.io/catalog: giantswarm-incubator
spec:
  name: kong-app
  description: The Cloud-Native Ingress and Service Mesh for APIs and Microservices
  engine: gotpl
  home: https://KongHQ.com/
  Icon: https://s3.amazonaws.com/downloads.kong/universe/assets/icon-kong-inc-large.png
  sources:
    - https://github.com/Kong/kong
  urls:
    - https://giantswarm.github.com/giantswarm-incubator-catalog/kong-app-0.2.0.tgz
versions:
  - version: 0.2.1
    appVersion: 1.2.3
    created: "2019-09-28T13:18:25.002458784Z"
  - version: 0.2.0
    appVersion: 1.2.2
    created: "2019-08-28T13:18:22.004458784Z"

This CRD can then be used for all possible use cases mentioned above. It can be extended to include additional metadata with time, but the idea would be to start small.
2) appcatalog-service
A microservice serving an aggregated API that does one of the 2 following things:

In case of blob storage for Helm Repository, it downloads the index.yaml, keeps a purged representation of that in its "cache" and answers requests for apps, versions, and metadata.
In case of chart-museum/harbor it proxies (and maybe caches) requests to said chart registry.

The reason for a proxy in case 2 is that it abstracts away from the actual representation and storage in case those change. Another reason is that for a better UX a single place to get the information about app and version availability and install would make more sense. Having the user switch to a Chart Registry frontend to explore which apps they want and then come back and install and configure them in clusters sounds icky. Again correct me if I'm wrong.

@puja108 and @piontec will bring the spec to markdown and ask people to review. @tomahawk28 will work on this.
Maybe this cycle, we'll aim to get the CRD merged. We have to think about the design of the controller.

hi @puja108 @piontec , I guess we forgot (?) to work on this last cycle...? first step was to bring the spec to markdown?

Yes, on my plate :| As mentioned today in the testing meeting with Halo, I'll try to work on this soon (next 1.5 week)

Related issue about keeping metadata in Halo team: https://github.com/giantswarm/roadmap/issues/95

@oponder I had this on my todo list ""Test versions" need to be in the meta-data thing w lukasz"
Just to confirm that is for this task in this issue https://github.com/giantswarm/giantswarm/issues/8021 right?

[ ] Happa allows selecting between stable and stable+resting quality apps

Let's be careful with "testing" and "pre-release".
"Testing" is what we agreed to go to "-testing-catalog" catalogs and is never visible to customers; these are dev-time builds.
"Pre-release" are "-alpha", "-beta*", "-rc*" builds that are to go to normal catalog, but are meant to show the quality is pre-release. I believe this is what we want to show in happa.

This will be a shared work between batman and halo. Please have a look at the first draft (CR naming to be adjusted): https://www.dropbox.com/scl/fi/vo6a6x89k5vuediy91q43/Application-metadata-for-app-catalog-entries.paper?dl=0&rlkey=n1t51gtt2w4ae2exqzix2ddni.
Please comment in the doc, @giantswarm/team-batman

Update 2020-03-20:

I created a spec from scratch, while there was one already created; merged them, shared with both teams, waiting for feedback

Spec is ready. Move to implementation.

@piontec do you have a ticket for implementing this story? Or should I create one?

@piontec will commit spec. Then create issues for implementation.


Customize warning notice timers

Users should be able to set a custom length for each warning timer to something suitable for them. I suspect the only constraint necessary is that the second notice must be longer than the first.
To eliminate that validation requirement, the alternative would be to set the second warning to be time after the first warning. For example, to have the timer fire at 10 seconds and then 60 seconds after you stop writing, you would select 10 and 50 seconds, respectively. The first warning would fire at 10 seconds and then the second warning 50 seconds after the first warning.

will be available in 0.0.10


jfrog rt gp doesn't publish .info file.

According to https://research.swtch.com/vgo-module#publishing, repository should contain 4 files:

../@v/list
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
But it uploads only .mod and .zip so my build fails with, for example:
go<EMAIL_ADDRESS>unexpected status (https://artifactory.example.com/api/go/go-local/github.com/user/example/v2/@v/v2.25.0.info): 404 Not Found

Content of such files is. for example:
{"Version":"v2.5.0","Time":"2018-03-26T05:07:29Z"}

I believe jfrog rt gp should publish such file as well.

Thank you @edganiukov.
We intend to add this soon.

Hey @eyalbe4 just wondering what your ETA/plan is for including this functionality?

@andrewhoff,
Actually, version 1.26.0 has just been released and includes publishing info files.
The support for this has been added in Artifactory 6.10.0, so you'll need both the CLI and Artifactory to be at those versions for this to work.
Looking forward to your feedback for this.

@eyalbe4 Thanks for the quick response! I've actually noticed it working fine with cli version 1.25 against a hosted artifactory instance, but it doesn't seem to work work cli version 1.25 against docker.bintray.io/jfrog/artifactory-pro:6.10.3. Also just curious when the 1.26 mojave package will be published to homebrew? Thanks!

@andrewhoff,
This should work for all scenarios with JFrog CLI 1.26.0 and above and Artifactory 6.10.0 and above.
Please let us know if you see any issues with those versions.
Version 1.26.1 is available on homebrew now.

ahhh it works great now, thanks!

Happy to hear this @andrewhoff. Thanks :)


Polkadot node warning "unexpected canonical chain state…"

I synced a Polkadot from a Polkachu snapshot, which worked fine.
Then after some time it printed this:
Error importing block 0x62fbc8c30cabdab4d0a92234c1556a26c6bc090cf4c14b9eafca6e4ca30311bc: consensus error: Import failed: Parent block of 0x62fb…11bc has no associated weight

It kept importing blocks but would print the following warning every now and then:
unexpected canonical chain state or malformed finality notification

Here the -lbabe=trace log with these warnings in them: log.txt.
@bkchr pobably knows what this means :smiley:

With https://github.com/paritytech/substrate/pull/11263

unexpected canonical chain state or malformed finality notification

should be gone.
I think the other error may comes from having a fork block in the queue while we already have finalized past this fork and then we get this error?
WDYT @andresilva @davxy

I think the other error may comes from having a fork block in the queue while we already have finalized past this fork and then we get this error?

I think this can definitely happen. One option would be to add some leeway and prune N blocks after last finalized.

Yeah, we could probably store the finalized heads/stale heads in the aux store and then remove them later when something else gets finalized.


import: remove last dependency on symlink

Closes #3479.
Signed-off-by: Christian Brauner<EMAIL_ADDRESS>

Can we get a test for this? should just be a matter of removing the symlink as well as the container entry right?

Looks good. Can you maybe split this into two separate commits though?
The commit message doesn't really cover everything that's happening in there :)


Update installation documentation

I noticed some issues in these docs.

[x] Refer to docker compose instead of docker-compose.
[x] Add note about currently supporting iRODS 4.2
[x] Update deployment docs
[x] Add note about sodar-docker-compose versioning

Done.


Release PRs Sync

Description

Schema fixes for rnv new
schema fixes for bootstrap questions
support for dynamic engine configurations
support for engine specific template overrides
refactor c into getContext

Related issues
Included PRs to avoid larger conflicts due to API signature changes and streamlined QA:

https://github.com/flexn-io/renative/pull/1451

https://github.com/flexn-io/renative/pull/1438 (fixed in this PR)

https://github.com/flexn-io/renative/pull/1447 (additional fixes in this PR)

https://github.com/flexn-io/renative/pull/1453

https://github.com/flexn-io/renative/pull/1439

https://github.com/flexn-io/renative/pull/1450 (additional fixes in this PR)

GH issues

Npm releases
n/a

kaios sim launch isn't fixed

this is solved


SELFDESTRUCT warning

Overview
Issue a warning when selfdestruct is used, as recommended in the Shanghai spec.
It interacts weirdly with the spinner but it works ^^
% huffc temp.huff
⠋ Compiling...Warning: "selfdestruct" has been deprecated. The underlying opcode will eventually undergo breaking changes, and its use is not recommended.
⠙ Compiling...

Fixes #271. Inspired by https://github.com/ethereum/solidity/pull/13884.
I ran all the checks.

Done @cheethas :-)

Great idea to add a warning, only concern here is that this breaks foundry-huff for all contracts that use SELFDESTRUCT. Would be great to add a flag that suppresses warnings or add this as a tracing warning.

I fear that if it is not shown by default, people will not notice it. Is the issue with foundry-huff a big deal ? We could even throw an error each time (with potentially a flag that can deactivate this error), as selfdestruct is really not supposed to be used. Wdyt ?

I fear that if it is not shown by default, people will not notice it. Is the issue with foundry-huff a big deal ? We could even throw an error each time (with potentially a flag that can deactivate this error), as selfdestruct is really not supposed to be used. Wdyt ?

Good catch by vex I didnt think of this! I agree people won't see it if it is not default, however as vex said, a suppress warning flag would be required to not break foundry huff (which we can leave on by default in the foundry-huff package).

As EIP-6780 deactivating SELFDESTRUCT will be deprecated in Cancun (source), I don't think that it's worth adding this to huffmate now in the end.


New replaygain backend bs1770gain failing

Hi Folks
trying the new backend out - very handy to at last have an easier option on Windows.  However, hitting a hurdle pretty quick on testing:
Traceback (most recent call last):
  File "C:\Python27\Scripts\beet-script.py", line 9, in <module>
    load_entry_point('beets==1.3.11', 'console_scripts', 'beet')()
  File "C:\Python27\lib\site-packages\beets\ui\__init__.py", line 1081, in main
    _raw_main(args)
  File "C:\Python27\lib\site-packages\beets\ui\__init__.py", line 1071, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "C:\Python27\lib\site-packages\beets\ui\commands.py", line 892, in import_func
    import_files(lib, paths, query)
  File "C:\Python27\lib\site-packages\beets\ui\commands.py", line 869, in import_files
    session.run()
  File "C:\Python27\lib\site-packages\beets\importer.py", line 316, in run
    pl.run_parallel(QUEUE_SIZE)
  File "C:\Python27\lib\site-packages\beets\util\pipeline.py", line 301, in run
    out = self.coro.send(msg)
  File "C:\Python27\lib\site-packages\beets\util\pipeline.py", line 183, in coro
    func(*(args + (task,)))
  File "C:\Python27\lib\site-packages\beets\importer.py", line 1343, in plugin_stage
    func(session, task)
  File "C:\Python27\lib\site-packages\beets\plugins.py", line 123, in wrapper
    return func(*args, **kwargs)
  File "C:\Python27\lib\site-packages\beetsplug\replaygain.py", line 849, in imported
    self.handle_album(task.album, False)
  File "C:\Python27\lib\site-packages\beetsplug\replaygain.py", line 790, in handle_album
    album_gain = self.backend_instance.compute_album_gain(album)
  File "C:\Python27\lib\site-packages\beetsplug\replaygain.py", line 133, in compute_album_gain
    return AlbumGain(output[-1], output[:-1])
IndexError: list index out of range

My config is:
library: C:\Utilities\beets\library.db
directory: F:\TestOut
import:
    copy: yes
    write: yes
    resume: ask
    quiet_fallback: skip
    timid: no
    log: beets.log
ignore: .AppleDouble ._* *~ .DS_Store
art_filename: folder
original_date: yes
ui:
    color: yes
#plugins: scrub chroma lyrics fetchart embedart ftintitle missing
plugins: scrub chroma lyrics fetchart embedart replaygain ftintitle missing
threaded: yes
id3v23: yes
paths:
    default: $albumartist/$year - $album%aunique{}/$track $title
    singleton: Singles/$year/$artist - $title
    comp: Various Artists/$year - $album%aunique{}/$track $title (by $artist)
lastgenre:
    count: 3
#bs1770gain.exe must be in $PATH
replaygain:
    backend: bs1770gain
    overwrite: yes
    albumgain: yes
fetchart:
    cautious: true
    cover_names: front back cover folder

Error happens with albumgain enabled or not.  Not sure if this is still used, couldn't see it in the docs.
Any clues on this one?

@jmwatte, would you mind taking a look? I added you as a collaborator so I could assign this to you. :smiley:
It looks like the output parser may need to tolerate malformed output. Just logging an error in this case and failing would be fine—but having the actual output from the underlying tool in the debug log would be useful to understand exactly what's going wrong.

I'm looking into it... got a free day tomorrow. Do not have a windows-machine. I'll see what we can do.

Happy to help with any output etc, can run from git if required no worries.
Doesn't work at all on windows currently, tried with 4 albums that all import fine without this.

Just a quick one ...
can you run
bs1770gain.exe path_to_album_folder --replaygain --file "testresults.txt"
and post me the output.
This should put an analyses of your album with replaygain in a textfile
named testresults.
Just to make sure that bs1770gain is working out of the box.
I'm not at my machine now, so this is from my head ;)

On Tue, Apr 7, 2015 at 10:21 AM, bossanova808<EMAIL_ADDRESS>wrote:

Happy to help with any output etc, can run from git if required no worries.
Doesn't work at all on windows currently, tried with 4 albums that all
import fine without this.
—
Reply to this email directly or view it on GitHub
https://github.com/sampsyo/beets/issues/1398#issuecomment-90455293.

Yep, here's the output
Inhuman Rampage
analyzing "Inhuman Rampage" ...
  [1/9] "01 Through The Fire And Flames.mp3":
      integrated:  -7.5 LUFS / -10.5 LU
  [2/9] "02 Revolution Deathsquad.mp3":
      integrated:  -6.9 LUFS / -11.1 LU
  [3/9] "03 Storming The Burning Fields.mp3":
      integrated:  -7.0 LUFS / -11.0 LU
  [4/9] "04 Operation Ground And Pound.mp3":
      integrated:  -7.3 LUFS / -10.7 LU
  [5/9] "05 Body Breakdown.mp3":
      integrated:  -8.0 LUFS / -10.0 LU
  [6/9] "06 Cry For Eternity.mp3":
      integrated:  -7.6 LUFS / -10.4 LU
  [7/9] "07 The Flame Of Youth.mp3":
      integrated:  -7.8 LUFS / -10.2 LU
  [8/9] "08 Trail Of Broken Hearts.mp3":
      integrated:  -8.6 LUFS / -9.4 LU
  [9/9] "09 Lost Souls In Endless Time (inhuman Rampage Special Edition) (Inhuman Rampage Special Edition).mp3":
      integrated:  -7.6 LUFS / -10.4 LU
  [ALBUM]:
      integrated:  -7.5 LUFS / -10.5 LU
done.

And another for good measure
MM
analyzing "MM" ...
  [1/15] "01 - Strangers To Ourselves.flac":
      integrated:  -9.7 LUFS / -8.3 LU
  [2/15] "02 - Lampshades On Fire.flac":
      integrated:  -7.6 LUFS / -10.4 LU
  [3/15] "03 - Shit In Your Cut.flac":
      integrated:  -7.6 LUFS / -10.4 LU
  [4/15] "04 - Pistol (A. Cunanan, Miami, FL. 1996).flac":
      integrated:  -8.0 LUFS / -10.0 LU
  [5/15] "05 - Ansel.flac":
      integrated:  -7.7 LUFS / -10.3 LU
  [6/15] "06 - The Ground Walks, With Time In A Box.flac":
      integrated:  -5.9 LUFS / -12.1 LU
  [7/15] "07 - Coyotes.flac":
      integrated:  -8.3 LUFS / -9.7 LU
  [8/15] "08 - Pups To Dust.flac":
      integrated:  -7.4 LUFS / -10.6 LU
  [9/15] "09 - Sugar Boats.flac":
      integrated:  -6.3 LUFS / -11.7 LU
  [10/15] "10 - Wicked Campaign.flac":
      integrated:  -7.4 LUFS / -10.6 LU
  [11/15] "11 - Be Brave.flac":
      integrated:  -7.5 LUFS / -10.5 LU
  [12/15] "12 - God Is An Indian And You're An Asshole.flac":
      integrated:  -10.2 LUFS / -7.8 LU
  [13/15] "13 - The Tortoise And The Tourist.flac":
      integrated:  -6.5 LUFS / -11.5 LU
  [14/15] "14 - The Best Room.flac":
      integrated:  -6.2 LUFS / -11.8 LU
  [15/15] "15 - Of Course We Know.flac":
      integrated:  -7.4 LUFS / -10.6 LU
  [ALBUM]:
      integrated:  -7.2 LUFS / -10.8 LU
done.

quickly between the soup and the potatoes...
If you get no other output on your commandline when importing (like:
analyzing this file...) then it seems to me that no items are getting to
the replaygain module.
If I look at the code it returns when there are no items to analyse and
then it goes to that error message you see
return AlbumGain(output[-1], output[:-1])
While still being away from my machine...
Have you tried it with another backend instead of the bs1770gain?
At least then we know that items are getting in the replaygain plugin ...
Back to my potatoes
;)
On Tue, Apr 7, 2015 at 11:53 AM, bossanova808<EMAIL_ADDRESS>wrote:

And another for good measure
MM
analyzing "MM" ...
[1/15] "01 - Strangers To Ourselves.flac":
integrated:  -9.7 LUFS / -8.3 LU
[2/15] "02 - Lampshades On Fire.flac":
integrated:  -7.6 LUFS / -10.4 LU
[3/15] "03 - Shit In Your Cut.flac":
integrated:  -7.6 LUFS / -10.4 LU
[4/15] "04 - Pistol (A. Cunanan, Miami, FL. 1996).flac":
integrated:  -8.0 LUFS / -10.0 LU
[5/15] "05 - Ansel.flac":
integrated:  -7.7 LUFS / -10.3 LU
[6/15] "06 - The Ground Walks, With Time In A Box.flac":
integrated:  -5.9 LUFS / -12.1 LU
[7/15] "07 - Coyotes.flac":
integrated:  -8.3 LUFS / -9.7 LU
[8/15] "08 - Pups To Dust.flac":
integrated:  -7.4 LUFS / -10.6 LU
[9/15] "09 - Sugar Boats.flac":
integrated:  -6.3 LUFS / -11.7 LU
[10/15] "10 - Wicked Campaign.flac":
integrated:  -7.4 LUFS / -10.6 LU
[11/15] "11 - Be Brave.flac":
integrated:  -7.5 LUFS / -10.5 LU
[12/15] "12 - God Is An Indian And You're An Asshole.flac":
integrated:  -10.2 LUFS / -7.8 LU
[13/15] "13 - The Tortoise And The Tourist.flac":
integrated:  -6.5 LUFS / -11.5 LU
[14/15] "14 - The Best Room.flac":
integrated:  -6.2 LUFS / -11.8 LU
[15/15] "15 - Of Course We Know.flac":
integrated:  -7.4 LUFS / -10.6 LU
[ALBUM]:
integrated:  -7.2 LUFS / -10.8 LU
done.
—
Reply to this email directly or view it on GitHub
https://github.com/sampsyo/beets/issues/1398#issuecomment-90491737.

FWIW : the albumgain part in
replaygain:
backend: bs1770gain
overwrite: yes
albumgain: yes
I can not find it mentioned when doing a search on this repository.
So I think it is fair to say that  you can delete it. But best to check it with sampsyo.
I searched around and came to this:
Replaygain tags set but always 0 issue #628
https://github.com/sampsyo/beets/issues/628
I wonder if those 2 issues are connected.
Anyway ...
Tomorrow I'll confiscate my daughters' windows machine and see if I can get beets working on that. In the mean time, I'll also put in some debuglines for some more info.

@jmwatte Ah, cool. If that option is obsolete, would you mind removing it from the docs? Thanks!

That's the question.... I haven't found it anywhere. It's not in the docs as far
as I can tell.
Sent using CloudMagic [https://cloudmagic.com/k/d/mailapp?ct=pa&cv=6.1.28&pv=4.4.2]
On di, apr. 07, 2015 at 4:01 PM, Adrian Sampson <<EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:
@jmwatte [https://github.com/jmwatte] Ah, cool. If that option is obsolete, would you mind removing it from the docs?
Thanks!
—
Reply to this email directly or view it on GitHub
[https://github.com/sampsyo/beets/issues/1398#issuecomment-90567440] .[https://github.com/notifications/beacon/AAu4cEXZirWyCnHAsDVJfXRR5CIOyNQ0ks5n89qrgaJpZM4D7Udc.gif]

Aha, great—yes, I think the option was removed. Thanks for checking.

an update...
The primary reason that you have no output on windows has to do with
https://github.com/sampsyo/beets/issues/628
It's the prefix for filenames that exceed the MAX_PATH length that gets us into trouble.
That I can fix with something like:
output = call(cmd + [syspath(i.path) for i in items])
    if not output :
        output = call(cmd + [syspath(i.path, prefix=False) for i in items])

That will get you into trouble with long filenames :(
I'm contacting the Peter Belkner to see if it is possible to make bs1770gain accept filenames with the prefix extension.
I'll keep you updated on that one.
But...
There is another problem now ...
Let me explain :
In linux the regex and parser work correct. In windows they don't.
this piece of code :
regex = ("(\s{2,2}\[\d+\/\d+\].*?|\[ALBUM\].*?)(?=\s{2,2}\[\d+\/\d+\]"
             "|\s{2,2}\[ALBUM\]:|done\.$)")
    results = re.findall(regex, data, re.S | re.M)

gives me the albumgain in linux, but in windows I don't get that one. So the program gives a replaygainerror because 'the tracks that go in' mismatch 'the items that come out'  with 1... :(
The file that Bsgain returns is the same in linux and windows :
.
analyzing "." ...
[1/19] "01. Welcome To New York.mp3":
integrated:  -7.9 LUFS / -15.1 LU
[2/19] "02. Blank Space.mp3":
integrated:  -8.3 LUFS / -14.7 LU
[3/19] "03. Style.mp3":
integrated:  -7.5 LUFS / -15.5 LU
[4/19] "04. Out Of The Woods.mp3":
integrated:  -9.1 LUFS / -13.9 LU
[5/19] "05. All You Had To Do Was Stay.mp3":
integrated:  -7.8 LUFS / -15.2 LU
[6/19] "06. Shake It Off.mp3":
integrated:  -8.2 LUFS / -14.8 LU
[7/19] "07. I Wish You Would.mp3":
integrated:  -8.4 LUFS / -14.6 LU
[8/19] "08. Bad Blood.mp3":
integrated:  -8.8 LUFS / -14.2 LU
[9/19] "09. Wildest Dreams.mp3":
integrated:  -9.9 LUFS / -13.1 LU
[10/19] "10. How You Get The Girl.mp3":
integrated:  -8.4 LUFS / -14.6 LU
[11/19] "11. This Love.mp3":
integrated:  -9.0 LUFS / -14.0 LU
[12/19] "12. I Know Places.mp3":
integrated:  -7.8 LUFS / -15.2 LU
[13/19] "13. Clean.mp3":
integrated:  -9.9 LUFS / -13.1 LU
[14/19] "14. Wonderland.mp3":
integrated:  -8.2 LUFS / -14.8 LU
[15/19] "15. You Are In Love.mp3":
integrated:  -10.7 LUFS / -12.3 LU
[16/19] "16. New Romantics.mp3":
integrated:  -7.8 LUFS / -15.2 LU
[17/19] "17. I Know Places (Voice Memo).mp3":
integrated:  -19.4 LUFS / -3.6 LU
[18/19] "18. I Wish You Would (Voice Memo).mp3":
integrated:  -18.4 LUFS / -4.6 LU
[19/19] "19. Blank Space (Voice Memo).mp3":
integrated:  -17.9 LUFS / -5.1 LU
[ALBUM]:
integrated:  -9.2 LUFS / -13.8 LU
done.
Yeah, it's my daughters' windows machine ;)
The regex is the same.
It's just the [ALBUM]-part does not turn up in the results from regex.findall...
So I'm a bit stuck and am trying to get more info on that one...

Wow; it's deeply disturbing that regex matching seems to differ on Windows vs. Linux. :scream:
@jmwatte, perhaps if we can narrow down a short string that shows off this match failing, we can start to understand how it's going wrong. If you paste the repr() version of that output string above, for example, maybe I can paste it into a Python interpreter here and see if I can reproduce the problem.
Nice T-Swift, BTW. :smiley:

I am following but don't have a great deal to add here - happy to help if you need any outputs or whatever, and working on windows is tough...I have a windows machine running all the time

T-Swift ... yeah... daughters' machine... I'm more of a T-Waits-er ;)
Anyway update...
Remember I never ran Beets on windows-machines.
So, almost everything is fixed. The regex is done... '\s' instead of 'b' ...
I build in some subroutines to make sure that when albums are really big I get no out-of-memory problems (splitting albums up in chuncks, settable in config)...
But now there is this...
beet ls snoop check -p:
C:\Users\myrthe\jm\test\Rap\Snoop Dogg\Doggystyle\17 CheckinÔÇÖ.1.mp3

If I run that through bs1770gain I get errors.
It goes through this routine
try:
            output = call(cmd + [syspath(i.path) for i in items])
            if not output:
                output = call(cmd + [syspath(i.path, prefix=False) for i in items])
except:

So I thought these paths would be 'wellformed' ...but I'm probably forgetting something here...
Anyway I'll look into it but any help is appreciated...

So...
I pushed a version. I checked it on linux and windows for the last hours with a lot of variations. It works as far as I can tell.
To summarize(AFAICT):

long path names are a windows/bs1770gain-problem. Best solution is support for long pathnames from bs1770gain.(waiting for an answer on that one)
fix: throw the long-path prefix out of the window if the prefix does not work.
On the plus-side: it works for short filepaths. Down-side: long filepaths.
Also on windows ...I ran into problems with some filenames(unicode?)...
fix: asciify_paths: true in the config.
Within those limitations it works.
So BossaNova ... maybe you can check it out. I may not be the perfect solution but till something better comes along...

@jmwatte Since the tool currently can't handle the long-path prefix, shouldn't we just disable the prefix entirely for now? We can then bring it back if the tool starts to support it.

Sure could do that. This was just to be save if they changed it without us
knowing. And when someone finds out they did I could just remove the "no-prefix"
condition.
What is your opinion on the asciify in the config ?
Sent using CloudMagic [https://cloudmagic.com/k/d/mailapp?ct=pa&cv=6.1.28&pv=4.4.2]
On vr, apr. 10, 2015 at 2:07 AM, Adrian Sampson <<EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:
@jmwatte [https://github.com/jmwatte] Since the tool currently can't handle the long-path prefix, shouldn't we just
disable the prefix entirely for now? We can then bring it back if the tool
starts to support it.
—
Reply to this email directly or view it on GitHub
[https://github.com/sampsyo/beets/issues/1398#issuecomment-91385277] .[https://github.com/notifications/beacon/AAu4cEpmnnsVcu4gwEfLV096zCEaeR_Gks5n9wvDgaJpZM4D7Udc.gif]

OK, good to know! Let's leave the fallback out for now (for simplicity) and add it back if something changes.
Recommending asciify_paths seems OK as a temporary workaround, since Python's handling of Unicode arguments to subprocesses on Windows is broken. (See also similar issues with convert and chroma, via audioread).

Ok will do..
Sent using CloudMagic [https://cloudmagic.com/k/d/mailapp?ct=pa&cv=6.1.28&pv=4.4.2]
On vr, apr. 10, 2015 at 7:50 AM, Adrian Sampson <<EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:
OK, good to know! Let's leave the fallback out for now (for simplicity) and add
it back if something changes.
Recommending asciify_paths seems OK as a temporary workaround, since Python's
handling of Unicode arguments to subprocesses on Windows is broken. (See also
similar issues with convert and chroma, via audioread).
—
Reply to this email directly or view it on GitHub
[https://github.com/sampsyo/beets/issues/1398#issuecomment-91437997] .[https://github.com/notifications/beacon/AAu4cGhBhQCs8uI1AAQu72POgEGps-cIks5n91wbgaJpZM4D7Udc.gif]

Great; thanks. I implemented a few other small cleanups.
@bossanova808 Can you confirm that this works—or at least does not crash—on your machine now?

I will be able to try tomorrow maybe...sorry got caught up by some dental surgery :(

No hurry! Dental surgery sounds much more important.

(Much less pleasant anyway).
Ok I tried running from source on two albums.
C:\GIT\beets>beet version
beets version 1.3.12
plugins: chroma, embedart, fetchart, ftintitle, lyrics, missing, replaygain, scrub
No crash, but no replaygain tags in the output either?
Config section is simply:
replaygain:
    backend: bs1770gain
    overwrite: yes

bs1770gain.exe is in my path
Any ideas??

Got the verbose output from running the replaygain command?

As in manually?  I see no output about RG running beets anymore...
f:\Unsorted\Music\Inhuman Rampage (9 items)
Correcting tags from:
    Dragonforce - Inhuman Rampage (Special Edition)
To:
    DragonForce - Inhuman Rampage
URL:
    http://musicbrainz.org/release/2f5489f2-edc4-3e36-9dae-8298c254e29a
(Similarity: 96.9%) (album, year, tracks) (CD, 2005, JP)
 * Through The Fire And Flames (#0) ->
   Through the Fire and Flames (#1)
 * Revolution Deathsquad (#0) ->
   Revolution Deathsquad (#2)
 * Storming The Burning Fields (#0) ->
   Storming the Burning Fields (#3)
 * Operation Ground And Pound (#0) ->
   Operation Ground and Pound (#4)
 * Body Breakdown (#0) ->
   Body Breakdown (#5)
 * Cry For Eternity (#0) ->
   Cry for Eternity (#6)
 * The Flame Of Youth (#0) ->
   The Flame of Youth (#7)
 * Trail Of Broken Hearts (#0) ->
   Trail of Broken Hearts (#8)
 * Lost Souls In Endless Time (inhuman Rampage Special Edition) (Inhuman Rampage
 Special Edition) (#0) ->
   Lost Souls in Endless Time (#9) (title)

f:\Unsorted\Music>

Sample output from running the Rg manually is above...

Yes, with the non-import RG command, but in verbose mode:
$ beet -v replaygain

Ok so then I do get output. Although it's happening on the unprocessed files that were imported, not the copied files in the output directory
F:\Unsorted\Music>beet -v replaygain
user configuration: C:\Users\X\AppData\Roaming\beets\config.yaml
data directory: C:\Users\X\AppData\Roaming\beets
plugin paths:
Sending event: pluginload
library database: C:\Utilities\beets\library.db
library directory: F:\TestOut
Sending event: library_opened
replaygain: analyzing Doves - The Places Between: The Best Of - There Goes the Fear
Sending event: database_change
Sending event: write
Sending event: after_write
replaygain: analyzing Doves - The Places Between: The Best Of - Black And White Town
Sending event: database_change
Sending event: write
Sending event: after_write
replaygain: analyzing Doves - The Places Between: The Best Of - Snowden
Sending event: database_change
Sending event: write
Sending event: after_write

Just an explanation - at the moment before really getting in to beets, I am using it on my downloads/mess folder and output the processed files (via copy) to TestOut...rather than working with my actual final library...until such a time as I know it's all working well.
So my expectation is that all the processing happens on the initial import (including RG) - and the output files go in to TestOut and are added to my beets library.  At the moment I then manually copy them over to my master library once I have checked them, all separate from beets, and have been manually doing replaygain on those as a last step (this was the main missing thing for me).
Periodically I have been doing a beets up to get rid of things now absent from my downloads folder and that I've moved from TestOut to my master library - so my beets library remains basically empty and I am just using it as a processor.
Anyway - am I misunderstanding here?  Should it not be doing RG on input?

You're not misunderstanding. That should work. I just asked you to run the explicit command so we could see the debugging output.
That all looks OK; no indication that anything's amiss. There are really no RG tags written?

Definitely not - I inspect with mp3tag and have a custom columns for replaygain that works will all replaygain outs I have seen - the column is empty.  I have also used it to inspect all extended tags and they're not there.  Checked an mp3 album and a flac one.
(and I can see the output from the ones being processed in the source folder itself)

How about with beet info (a plugin you may need to enable)? Does that show the values for the imported files?

It does not

How strange. Since the tool is reporting success, I'm not sure what else to suggest. Please post again if you discover any evidence about what could be going wrong.

hmm, so I got rid of everything from both the in and out folder
beet up - deletes a bunch of stuff
beet info - shows nothing
beet import -v Whatever
beet import  -v Whatever2
imports successfully, outputs to the TestOut folder, but definitely no RG tags and no output about it either.
beet -v replaygain
Processes Whatever and Whatever2 in the TestOut folder and adds the tags.
So basically it just appears to be not running RG at all now on import. It was before, but crashed, so something in these changes seems to mean it is bailing earlier...

So basically it just appears to be not running RG at all now on import. It was before, but crashed, so something in these changes seems to mean it is bailing earlier...

Wow; that's quite strange. Thanks for investigating. Is there anything suspicious in the verbose (-vv) output during import?

beet info - shows nothing

This command is used with a path.

Nothing in -vv that I can see, no reference to replaygain at all (can see others, chroma, lyrics, scrubbing etc).
I do notice if RG tags exists in a file they are not removed.

Aha, thanks! It looks like there was a bug that was preventing debug output from the plugin in import mode. Can you give the latest few commits a try and see if beet -vv import ... is any more helpful now?

ok progress
replaygain: analyzing DragonForce - Inhuman Rampage
replaygain: executing bs1770gain --replaygain -it F:\Unsorted\Music\Inhuman Rampage\01 Through The Fire And Flames.mp3 F:\Unsorted\Music\Inhuman Rampage\02 Revo
lution Deathsquad.mp3 F:\Unsorted\Music\Inhuman Rampage\03 Storming The Burning Fields.mp3 F:\Unsorted\Music\Inhuman Rampage\04 Operation Ground And Pound.mp3 F
:\Unsorted\Music\Inhuman Rampage\05 Body Breakdown.mp3 F:\Unsorted\Music\Inhuman Rampage\06 Cry For Eternity.mp3 F:\Unsorted\Music\Inhuman Rampage\07 The Flame
Of Youth.mp3 F:\Unsorted\Music\Inhuman Rampage\08 Trail Of Broken Hearts.mp3 F:\Unsorted\Music\Inhuman Rampage\09 Lost Souls In Endless Time (inhuman Rampage Sp
ecial Edition) (Inhuman Rampage Special Edition).mp3
replaygain: analysis finished: analyzing ...
  [1/9] "01 Through The Fire And Flames.mp3":
      integrated:  -7.5 LUFS / -10.5 LU
       true peak:  0.9 TPFS / 1.105080
  [2/9] "02 Revolution Deathsqua        1%
      integrated:  -6.9 LUFS / -11.1 LU
       true peak:  0.7 TPFS / 1.080454
  [3/9] "03 Storming The Burning Fields.mp3":
      integrated:  -7.0 LUFS / -11.0 LU
       true peak:  1.1 TPFS / 1.137201
  [4/9] "04 Operation Ground And Pound.mp3":
      integrated:  -7.3 LUFS / -10.7 LU
       true peak:  0.7 TPFS / 1.084129
  [5/9] "05 Body Breakdown.mp3":
      integrated:  -8.0 LUFS / -10.0 LU
       true peak:  0.9 TPFS / 1.112334
  [6/9] "06 Cry For Etern        1%1%
      integrated:  -7.6 LUFS / -10.4 LU
       true peak:  0.7 TPFS / 1.082500
  [7/9] "07 The Flame Of Youth.mp3":
      integrated:  -7.8 LUFS / -10.2 LU
       true peak:  0.7 TPFS / 1.086795
  [8/9] "08 Trail Of Broken Heart        1%
      integrated:  -8.6 LUFS / -9.4 LU
       true peak:  0.6 TPFS / 1.067787
  [9/9] "09 Lost Souls In Endless Time (inhuman Rampage Special Edition) (Inhuman Rampage Special Edition).mp3":
      integrated:  -7.6 LUFS / -10.4 LU
       true peak:  0.6 TPFS / 1.077090
  [ALBUM]:
      integrated:  -7.5 LUFS / -10.5 LU
       true peak:  1.1 TPFS / 1.137201
done.

replaygain: 9 items, 9 results
replaygain: ReplayGain error: ReplayGain backend failed for some tracks in album DragonForce - Inhuman Rampage
Sending event: item_copied

Looks like there is still some issues with paths maybe?
(I am using the asciify_paths option)

Goodmorning,
Been busy I see.
I think I know what is causing the problem.
I split up the long regex on multiple lines and then it doesn't get the last album tag. 9 items 9 results should be 10 results. I just wished I could split it up correctly. I am out now for a few hours but I'll try again when I get back. Sorry for that. But I'm sure it works I just tested it. It's the split regex...aaaahhhhhh

I fixed it. It's tested. I pushed it. If it gets through, try it. Got to go. Back in a few hours.

Yep one quick test here works!  Yay!
Will report back again after some more testing

Yep seems to work ok.  The asciify paths thing is a shame to have to use (I actually use unicode in python win regularly without issue - learnt from - http://blog.notdot.net/2010/07/Getting-unicode-right-in-Python
But 4 albums processed successfully with RG.

Woohoo! Thanks, @jmwatte. And thanks to @bossanova808 for testing.
The remaining issue is not Unicode in general but a bug in Python with calling subprocesses on Windows with Unicode arguments: http://bugs.python.org/issue1759845


Azimuth angle in degrees

Allow the azimuth angle to be given in string or in degrees.
A couple of other fixes were included in this PR because the error messages I got while debugging were wrong.

(and I think we can ignore the failed unit test - this is the DB issue with multiple connections)

(and I think we can ignore the failed unit test - this is the DB issue with multiple connections)

I agree we can ignore this.
I just need your OK about the azimuth angle (mostly to make sure I am not wrong about it) and then I will push the fixes and merge. Thanks for the review!

OK, thanks! I will merge once the tests pass.

Merging despite the failing tests since they are related to the DB and the issue was thought to be solved. It is apparently not, but I will look into it again elsewhere.


How use ProviderListener with StateNotifierProvider and freezed?

I use riverpod with StateNotifierProvider and freezed in my project. I need to show dialog. Do i need to use ProviderListener ?
Any example? I  couldn't find anything and got a little confused.

Can you post your code? Been having issues with it so i might be able to help

@FetFrumos
Here is a example from the documentation. It uses StateProvider but it work with StateNotifier in same way.
Widget build(BuildContext context) {
  return ProviderListener<StateController<int>>(
    provider: counterProvider,
    onChange: (context, counter) {
      if (counter.state == 5) {
        showDialog(...);
      }
    },
    child: Whatever(),
  );
}

Here is an example from a personal project.
The State with @freezed:
@freezed
class MyFormState with _$MyFormState {
  const factory MyFormState({
    @Default(false) bool isSaving,
    @Default(false) bool isSuccess,
    String? errorText,
  }) = _MyFormState;
}

The StateNotifier:
class MyFormController extends StateNotifier<MyFormState> {
  MyFormController() : super(MyFormState);

The StateNotifierProvider:
final formProvider = StateNotifierProvider
    .autoDispose<MyFormController, MyFormState>((ref) => MyFormController();
});

Anf finally the widget:
```dart
class MyFormPage extends StatelessWidget {
  const MyFormPage({Key? key}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    return ProviderListener<MyFormState>(
      provider: formProvider,
      onChange: (context, state) async {
        if (state.isSuccess) {
          Navigator.pop(context);
        } else if (state.errorText != null) {
          showAlertDialog(
            context,
            title: "An error occurred",
            content: state.errorText,
          );
        }
      },
      child: const MyFormPageBuilder(),
    );
  }
}

I generally use freezed to create sealed classes. After, you can use ProviderListener and call showDialog().

Closing as this was answered


Dynamic selection of inline HTML elements

It would be very useful to be able to specify a function to the options that would take an HTML node object as a parameter, and returns a boolean value to specify whether the element should be beautified (or conversely, whether it should be considered inline). Returning "undefined" would fall back to the static include/include+/include- properties in options. This feature would allow you to only inline for example ul.classA > li elements (to use CSS notation). Adding support for such CSS/jQuery notation to the existing include family of properties is an alternative solution.
cf. https://github.com/beautify-web/js-beautify/issues/1668

@alexreg
Thanks for filing this.
This can be done, but it will probably be difficult and error prone until after the beautifier does a much better job of parsing. The html beautifier only recently reached the point of parsing tokens before beautifying.  Before that it would have been impossible to even do a ul.classA query, let alone ul.classA > li.
Allowing  some query notation in the existing inline setting would be preferable IMO - that would allow us to keep the behavior well defined and keep us from having to lock down APIs that are currently internal and in-progress.  Implementing some querying would be involved but would be less of a bug farm than taking a function.

Do you use htmlparser2 internally for parsing? If so (or even if not), cheerio is a great layer over that, which gives you a jQuery-like interface to the DOM, and allows querying/checking of the form ul.classA > li very easily. :-)


Tool use guidelines

Hi everyone,
Thanks for open-sourcing your code; this is an amazing framework.
My question concerns tool use, i.e., providing predefined functions/methods so Trace can generate code using them (calling instead of defining the functions). Are there any guidelines on how to do it? How did you envision this type of scenario?
From my attempts, I understand it is possible to provide a list of tool descriptions as part of Feedback or even as part of the objective prompt to the optimizer. Still, is there a proper way more aligned with what you envisioned?

Ching-An can chime in here, but you can write an orchestration function like this.
@bundle()
def call_tool_1(param1, param2, ...):
  """
  Tool 1 is about .... (description of tool 1)
  input, output to Tool 1
  """
  return calling_external_api()

@bundle()
def call_tool_2(param1, param2, ...):
  """
  Description of tool 2
  """
  return calling_external_api()

@bundle(trainable=True)
def act(query):
   """
   Decide which tool to call, available tools: tool_1, tool_2
   """
   result = call_tool_1(param1, param2)
   return

Note that, both call_tool_1 and call_tool_2 are not changed / fixed by Trace, but act can change!
The docstrings you provide to all three functions are visible to the Trace optimizers. These became your "prompts".
You don't have to provide a list of tool descriptions as feedback.

Exactly -- the docstring is the prompt/explanation on what the tool is about and how it should be used. It can be long :)

Your approach worked great in my toy example :) Providing the list of tools as part of the docstring I get something like
def act() -> bool:
    """Perform car maintenance for the cheapest price possible. Use the available tools below if necessary.
   ...
    - setup_vehicle(): Sets up the vehicle for deployment (incurr a cost)
    - calibrate_sensors(): Calibrates vehicle sensors (incurr a cost).
    - perform_safety_checks(): Performs safety checks on the vehicle (incurr a cost).
    - deploy_vehicle(): Deploys the vehicle if ready (incurr a cost).
    - perform_maintenance(): Performs maintenance on the vehicle if necessary (incurr a cost).
    - validate_vehicle(): Validates the vehicle before deployment (incurr a cost).
    - calibrate_and_check_sensors(): Calibrate sensors and perform safety checks in parallel (incurr a cost).
    """
    if not is_vehicle_ready():
        setup_vehicle()
        print('Vehicle was set up.')
        return False
    if not are_sensors_calibrated() or not are_safety_checks_passed():
        calibrate_and_check_sensors()
        print('Sensors calibrated and safety checks performed.')
    if is_maintenance_required():
        perform_maintenance()
        print('Maintenance performed.')
    if not is_vehicle_deployed():
        deploy_vehicle()
        print('Vehicle deployed.')
    return True)

where the tools/functions are being used correctly.
However, in my previous version (including the list in the feedback string), I could dynamically include the list of tools, while here, they are hardcoded in the docstring. Do you have any suggestions about that? (I tried to update the act.info[doc] to change the list of tools in the docstring dynamically but did not work )

Hi @jeanpm , if you want to dynamically change the description of the bundled operator, you can change the description attribute, i.e., act.description. Note when overwriting it, you need to follow its syntax of '[$op_name]  $description' The attribute was constructed from docstring if it's not manually provided (see https://github.com/microsoft/Trace/blob/4e6fad28d3b4c044baae8ce1ab670b45204041e9/opto/trace/bundle.py#L124C9-L124C25). Let me know if it works.

Hi @chinganc , thanks for the suggestion but I couldnt make it work that way (seems like the optimizer is unaware of the tools if I include them as part of the description only). I'll dig deeper in the code later, but for now I'll move forward using the feedback (or objective) string.


Sending binary data; can only send Blobs; how to send ArrayBuffer?

Include the following information (where applicable):

What operating system are you running?
Linux
Are you running any CPU architechute out of the normal?
I run 32-bit in the year 2017
What platform/language are you using?
C++
Are you using SSL, if so, how?
SSL is not part of the equation
What version of uws are you using?

master

What features of µWS are involved in the issue?
Server support (WebSocket)
How does the issue reproduce?
All the time
Can you show me some short code that triggers your issue or any other important information?

#include <uWS.h>
#include <Node.h>
#include <iostream>
#include <cmath>
#include <vector>

struct state {
    uWS::WebSocket<uWS::SERVER>* sock;
    bool sock_open;
};

int main() {
    uWS::Hub h;
    size_t framesize = 2000;
    size_t connections = 0;
    struct state s;
    s.sock_open = false;

    std::cout << "Server started." << std::endl;

    h.onHttpRequest([](uWS::HttpResponse *res, uWS::HttpRequest req, char *data, size_t length, size_t remainingBytes) {
        res->end("KEKE", 4);
    });

    h.onConnection([&s, &connections](uWS::WebSocket<uWS::SERVER> *ws, uWS::HttpRequest req) mutable {
        s.sock = ws;
        s.sock_open = true;
        std::cout << "[New Connection] # clients: " << ++connections << std::endl;
    });

    h.onMessage([](uWS::WebSocket<uWS::SERVER> *ws, char *message, size_t length, uWS::OpCode opCode) {
        // if isBinary:
        //     print("Binary message received: {} bytes".format(len(payload)))
        // else:
        //     payload = payload.decode('utf8')
        //     print("Text message received: {}".format(payload))
        //     cmd = json.loads(payload)
        //     if 'frameSize' in cmd:
        //         self.frameSize = int(cmd['frameSize'])
        std::cout << "Got message: " << message << std::endl;
    });

    h.onDisconnection([&s, &connections](uWS::WebSocket<uWS::SERVER> *ws, int code, char *message, size_t length) mutable {
        s.sock_open = false;
        std::cout << "[Connection lost] # clients: " << --connections << std::endl;
    });

    Timer *timer = new Timer(h.getLoop());
    timer->setData(&s);
    timer->start([](Timer *timer) {
        std::vector<unsigned short> sinus(2000);
        for(size_t i = 0; i < 2000; i++){
            sinus.push_back(16384 + 7000 * sin(i/2000*40*M_PI));
        }
        struct state* s = (struct state*)timer->getData();
        std::cout << "Try Send: " << std::endl;
        if(s->sock_open){
            std::cout << "Send: " << sinus.size() << std::endl;
            s->sock->send((char*)&sinus[0], sinus.size(), uWS::OpCode::BINARY);
        }
    }, 0, 350);

    h.listen(50090);
    h.run();
}

This is the server side. It works great as far as it can open and close connections properly. It also receives messages properly. Where I am stuck is the line: s->sock->send((char*)&sinus[0], sinus.size(), uWS::OpCode::BINARY);. It get's transmitted and I get a message on the JS-Client side.
If I just open a WebSocket with
var sock = WebSocket('ws://localhost:9000');

the data contained is
data: Blob {
    size: 4000
    type: ""
}

but if I want to convert the data to an array buffer and I set
var sock = WebSocket('ws://localhost:9000');
socket.binaryType = 'arraybuffer';

i will get an ArrayBuffer with length === undefined.
When I use the Autobahn python library this works properly when I send binary data, so I assume that it's something I do not understand with uWS.
Could you give me a hint?
Everything works fine and it was really easy to code a simple sever :)
Thank you!

It is called byteLength

If the property in JS was the wrong one why would it work with the Autobahn server? I don't understand.

No idea, that is a front-end question. Server side doesn't deal with ArrayBuffer or Blob - it's just binary or text.

No, that is my point. The client side is the same, the only thing I changed was the server side. Anyways, works like a charm now, thank you very much kind sir!


Add include checks for several Boost utilities

Add #include checks for boost::atomic|exception_ptr|intrusive_ptr|make_shared|shared_ptr. Fix violations.

Wow, thanks! LGTM!


Correctly handle the Proxy IsCallable and IsConstructor information

The IsCallable(target) and IsConstructor(target) info
can't be stored in the target/handler values.
If the input for the ProxyCreate was a revocable Proxy the original target's
callable/constructor information must be retained even after the
Proxy was revoked.
Depends on: #4261

We in general use the least used object member for tagging, i.e. the scope pointer of byte code functions. Is the property list the least used member?

Based on the tests and investigation the property list is not used in case of proxy objects.


Require Python 3.11

Description
The CI is only testing Python 3.11, and the Conda feedstock requires Python 3.11 and 3.12, so I assume that we only intend to support >=3.11. This reflects this intention in the PyPI metadata.

Probably the better way to go here is #563

Closing in favor of #563


On chain balance disappeared

Zapping out/transferring ecash from freedom one to another wallet and midway through I closed the app and refreshed but now my onchain is gone!
I resynced a couple times but it just said 0

If a resync is not working then you can go through the recovery flow here: https://github.com/MutinyWallet/mutiny-web/wiki/Recovery#on-chain

Thank you


Save JSON file

I want to save a JSON file, but run into an error.
This is my code:
  Future<void> _saveSubjects() async {
    Directory directory = await getApplicationDocumentsDirectory();
    File file = File(join(directory.path, 'assets/data/subjects.json'));
    await file.create(recursive: true);
    await file.writeAsString(jsonEncode(subjects.map((subject) {
      return subject.toJSON();
    })));
  }

The error is:
JsonUnsupportedObjectError (Converting object to an encodable object failed: Instance of 'MappedListIterable<Subject, Map<String, dynamic>>')
This is the model file:
import 'dart:convert';
import './topic.dart';

class Subject {
  String name;
  int order;
  bool isMajor;
  List<Topic> topics;

  Subject({this.name, this.order, this.isMajor, this.topics});

  factory Subject.fromJSON(Map<String, dynamic> json) {
    if (json != null) {
      return Subject(
          name: json['name'],
          order: json['order'],
          isMajor: json['isMajor'],
          topics: List<Topic>.from(
              json['topics'].map((topic) => Topic.fromJSON(topic))));
    } else {
      return null;
    }
  }

  Map<String, dynamic> toJSON() {
    return {
      'name': name,
      'order': order,
      'isMajor': isMajor,
      'topics': topics.map((topic) => jsonEncode(topic.toJSON())).toList(),
    };
  }
}

Maybe someone can help?

You need to .toList() after .map, since you can't covert an Iterable into a JSON object.
In general, questions like these are better suited for Q&A sites like StackOverflow or on one of the flutter community sites at https://flutter.dev/community . The issue tracker is for bugs or feature requests

ok, sorry, I will change that! Thank you!

No problemo

Could you please have a look at this:
Future<File> _saveSubjects() async {
    final Directory directory = await getApplicationDocumentsDirectory();
    final File file = File(join(directory.path, 'assets/data/subjects.json'));
    await file.create(recursive: true);
    final String jsonString = jsonEncode(subjects.map((subject) {
      return subject.toJSON();
    }).toList());
    return await file.writeAsString(jsonString, flush: true);
  }

Somehow this doesn't write the file.


[test] delegate crash when receiving transaction of game_update_operation

This only happen in bdd_tests, if run it manually, it goes ok.

https://github.com/bitsuperlab/cpp-play/commit/b40e15530751352459a828629ddcb13d231f0a4c


⚠ T:Riza Community Forum has degraded performance

In 0d906c7, T:Riza Community Forum (https://community.triza.dev) experienced degraded performance:

HTTP code: 200
Response time: 1275 ms

Resolved: T:Riza Community Forum performance has improved in 59b3a07.


chore: change how we deal with dependencies

⚠ WIP
Okay, I’ve learned a ton about dependencies and npm today. Based on the learnings I’ve had today, I’d like to suggest a new approach on how we deal with dependencies:
devDependencies
Dependencies that are only required for development, should live in the devDependencies. New: This includes @types/ packages. Those are not required to be installed in user projects.
dependencies
Dependencies that are required to run the code, should live in dependencies. In user projects, those dependencies will be installed, because they are in our package.json.
🆕 This includes vue.
🆕 Those shouldn’t have a fixed version number, that would force users to have the exact same version installed.
peerDependencies
peerDependencies should only be used in rare cases, to make sure something is really required in a specific version (or above). That makes sense for frameworks (React, Vue), plugin systems (CodeMirror).
Depending on the package manager, the package manager will make sure to install them as specified or complain if they are not installed as specified.
🆕 I removed most of the packages from the peerDependencies.
externals
There isn’t really a good reason to bundle dependencies with our packages. npm (and other managers) make sure to install the dependencies anyway. Having them bundled leads to all sorts of version conflicts.
Also, bundling dependencies increases the risk of ending up with a huge bundle (if tree shaking doesn’t work, which is too often the case).
🆕 The externals array is automatically filled with the list of dependencies in the package.json. No need to maintain multiple lists, but this one list needs to be correct.
Will it be the last change in that regard? I doubt it. I feel like we’re still learning. But let’s try it. :)
Tasks

[ ] Looks like the bundle includes multiple Vue versions, probably missing in an external array

I feel like I don't really have a good enough understanding of how this works / the motivations to weigh in on it. I'm in favour of anything that reduces our conflict issues though.


text.match is not a function

Describe the bug
Similar error to https://stackoverflow.com/questions/59061937/postwoman-api-tester-response-is-always-empty.
I keep getting a 500 text.match is not a function when I run a get api request

Please share the steps to reproduce the issue

Closing due to inactivity.

I had the same issue. I try to GET https://live.bilibili.com/21403609 with Header: Accept: /
It'll return 200 but nothing in Postwoman's Response area, and I get a TypeError: text.match is not a function.
I tried curl and postman, both works well, so I guess it's postwoman's issue

There's a known issue with the current published version of Postwoman Chrome extension (v. 012). Latest version is under review. To fix the issue - try disabling extension by going to Settings > Extensions (turn off "Use extensions").
Let me know if the issue is fixed.

@liyasthomas The request will be blocked by CORS policy if I turn off extension
(index):1 Access to XMLHttpRequest at 'https://live.bilibili.com/21403609' from origin 'https://postwoman.io' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

can you try turning on Proxy mode (Settings > Proxy)?

@liyasthomas Yes via proxy it works well, the Response shows properly. BTW, I used to think the proxy is socks proxy or http proxy, but I found it's proxywoman


BUG: compatibility with plotly.express.choropleth

Modin version checks

[X] I have checked that this issue has not already been reported.

[X] I have confirmed this bug exists on the latest released version of Modin.

[ ] I have confirmed this bug exists on the main branch of Modin. (In order to do this you can follow this guide.)

Reproducible Example
df = pd.read_csv("/kaggle/input/united-nations-refugee-data-unhcr/United Nations Refugee Data.csv")

df_ = df.groupby(["Year", "Country of origin (ISO)"])[["Refugees under UNHCR's mandate"]].sum().reset_index()

import plotly.express as px

fig_world_map = px.choropleth(
    data_frame=df_,  # ._to_pandas()
    locations="Country of origin (ISO)",
    color="Refugees under UNHCR's mandate",
    animation_frame="Year",
)

Issue Description
the plot just crash
Expected Behavior
showing interactive map
when passing data_frame=df_._to_pandas() helps
Error Logs

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[16], line 1
----> 1 fig_world_map = px.choropleth(
      2     data_frame=df_,  # ._to_pandas()
      3     locations="Country of origin (ISO)",
      4     color="Refugees under UNHCR's mandate",
      5     animation_frame="Year",
      6     title="Refugees Origine Over Time",
      7     range_color=[0, int(max_all_years)],
      8     color_continuous_scale="Reds",
      9     labels={'Refugees under UNHCR\'s mandate': '#Refugees',
     10             'Country of origin (ISO)': 'Origin'},
     11     projection="natural earth",
     12     category_orders={"Year": sorted(df_["Year"].unique())}
     13 )
     15 fig_world_map.update_layout(height=800, width=1200)
     16 fig_world_map.show()

File /opt/conda/lib/python3.10/site-packages/plotly/express/_chart_types.py:1091, in choropleth(data_frame, lat, lon, locations, locationmode, geojson, featureidkey, color, facet_row, facet_col, facet_col_wrap, facet_row_spacing, facet_col_spacing, hover_name, hover_data, custom_data, animation_frame, animation_group, category_orders, labels, color_discrete_sequence, color_discrete_map, color_continuous_scale, range_color, color_continuous_midpoint, projection, scope, center, fitbounds, basemap_visible, title, template, width, height)
   1051 def choropleth(
   1052     data_frame=None,
   1053     lat=None,
   (...)
   1085     height=None,
   1086 ) -> go.Figure:
   1087     """
   1088     In a choropleth map, each row of `data_frame` is represented by a
   1089     colored region mark on a map.
   1090     """
-> 1091     return make_figure(
   1092         args=locals(),
   1093         constructor=go.Choropleth,
   1094         trace_patch=dict(locationmode=locationmode),
   1095     )

File /opt/conda/lib/python3.10/site-packages/plotly/express/_core.py:2076, in make_figure(args, constructor, trace_patch, layout_patch)
   2073 layout_patch = layout_patch or {}
   2074 apply_default_cascade(args)
-> 2076 args = build_dataframe(args, constructor)
   2077 if constructor in [go.Treemap, go.Sunburst, go.Icicle] and args["path"] is not None:
   2078     args = process_dataframe_hierarchy(args)

File /opt/conda/lib/python3.10/site-packages/plotly/express/_core.py:1485, in build_dataframe(args, constructor)
   1482     args["color"] = None
   1483 # now that things have been prepped, we do the systematic rewriting of `args`
-> 1485 df_output, wide_id_vars = process_args_into_dataframe(
   1486     args, wide_mode, var_name, value_name
   1487 )
   1489 # now that `df_output` exists and `args` contains only references, we complete
   1490 # the special-case and wide-mode handling by further rewriting args and/or mutating
   1491 # df_output
   1493 count_name = _escape_col_name(df_output, "count", [var_name, value_name])

File /opt/conda/lib/python3.10/site-packages/plotly/express/_core.py:1210, in process_args_into_dataframe(args, wide_mode, var_name, value_name)
   1208         if argument == "index":
   1209             err_msg += "\n To use the index, pass it in directly as `df.index`."
-> 1210         raise ValueError(err_msg)
   1211 elif length and len(df_input[argument]) != length:
   1212     raise ValueError(
   1213         "All arguments should have the same length. "
   1214         "The length of column argument `df[%s]` is %d, whereas the "
   (...)
   1221         )
   1222     )

ValueError: Value of 'animation_frame' is not the name of a column in 'data_frame'.  Which version of plotly are you using?

Hello, thanks for the contribution! Which version of plotly are you using?

plotly                                   5.16.1
plotly-express                           0.4.1

I can reproduce the problem. I'll see how to fix this.

https://github.com/plotly/plotly.py/pull/4442

@Borda the pull request in plotly.py has been merged. It will be available, I believe, from the v5.19.0 version.

@Borda the pull request in plotly.py has been merged. It will be available, I believe, from the v5.19.0 version.

Thank you!


Polish Documentation

[x] Update any needed changes for gobolinux 017
[x] Add pages for any empty or broken links
[x] Update any page stubs

All three points have been adressed recently. If there is something specific missing be free to open an issue for that.


Pipeline variable works in one stage but not the other

Required Information
Type: Bug
Enter Task Name: AzureResourceGroupDeployment@2
Environment

Server - Azure Pipelines

Acct: locumtenensdotcom1
Team Project Name: clinician portal
Build Def Name: Release Pipeline
Release-75

Agent - Hosted

Agent Version: 2.160.1
Agent Machine Name: fv-az433

Issue Description
We are using an ARM Template to deploy an Azure SQL Elastic Pool with a default admin password.  In the dev stage, the pool deploys without issue using a variable group to store the secured password and parameter overrides to insert it into the ARM Template.  In the UAT stage, if the same variable is supplied, or the correct variable (uat var instead of dev var), the deploy fails, stating that the password does not meet the complexity requirements.  If instead the password is overridden in plain text in the override params field, the template works without issue.
Examples
These overrides fail:

While this works:

All variable values meet the complexity requirements, and the dev variable works in the dev stage.
Task logs
Debug logging doesn't help, but is attached for completion.
tasklog_15 (1).zip
Troubleshooting
We have attempted to wrap the variable overrides in quotations, but that didn't help.  We have tried re-implementing the task from scratch, no joy.

Hey @gneely-credera from the logs it seems like there is no variable defined by  uat-sql-server-admin-password. I am referring to this line in the attached logs

2019-11-18T22:37:43.6323703Z ##[debug]overrideParameters=-administratorLoginPassword $(uat-sql-server-admin-password)

Usually, the variables are substituted by their respective values in the logs, unless the variable is not defined in the pipeline. **Note:** Please mark your passwords as 'secret variable' to avoid it getting logged as plain text.
Let me know if you are still facing this.

Howdy @jalajmsft !
Here's the variable in the Library:

Additionally, in my troubleshooting above, I get a similar issue running a dev version of that variable (dev-sql-server-admin-password), even though that dev variable works successfully in the development environment deployment tasks.

Hey @gneely-credera did you link Library variable group to Release Pipeline variable group? If that is done can you please also check for stage wise linking to variable group? You can read more about variable group here
Let me know if you are still facing this.

Howdy @jalajmsft, thanks for pointing out the stagewise linking!  We had only given the values scope for our dev and QA stages, not UAT.  Thanks so much for your help!


HTTP extension recommedations

It would be ideal to allow simple extensions of the HTTP class for other methods such as DELETE.
This would be a clean extension for 'delete' but ClientRequestOptions does allow a full URL:
extension HTTP {
    class func delete(_ url: String, callback: ClientRequest.Callback) {
        var options: [ClientRequest.Options] = []
        options.append(.method("delete"))
        options.append(.url(url))   // Error: Type 'ClientRequestOptions' has no member 'url'
        let req = HTTP.request(options, callback: callback)
        req.end()
    }
}

Another suggestion would to create an request function that allows for both url and options.  This would require extending the underlying ClientRequest class.
extension HTTP {
    class func delete(_ url: String, callback: ClientRequest.Callback) {
        var options: [ClientRequest.Options] = []
        options.append(.method("delete"))
        // Error: Cannot invoke 'request' with an argument list of type ...
        let req = HTTP.request(url, options: options, callback: callback)
        req.end()
    }
}```

In your first example can you not use the .hostname and .path options?

Yes, you could.  However if you get a full URL, for example, from another web service, you need to destruct it and re-construct it as a [ClientRequest.Options].
Our existing workaround looks like this:
extension HTTP {

    class func urlToOptions(_ url: String) -> [ClientRequest.Options] {

        let nsURL = NSURL(string: url)!
        var options: [ClientRequest.Options] = []

        #if os(Linux)
        if let scheme = nsURL.scheme {
            options.append(.schema("\(scheme)://"))
        }
        #else
        options.append(.schema("\(nsURL.scheme)://"))
        #endif
        if let host = nsURL.host {
            options.append(.hostname(host))
        }
        if let path = nsURL.path {
            var fullPath = path
            // query strings and parameters need to be appended here
            if let query = nsURL.query {
                fullPath += "?"
                fullPath += query
            }
            if let parameters = nsURL.parameterString {
                fullPath += ";"
                fullPath += parameters
            }
            options.append(.path(fullPath))
        }
        if let port = nsURL.port {
            let portInt16 = port.int16Value
            options.append(.port(portInt16))
        }
        if let username = nsURL.user {
            options.append(.username(username))
        }
        if let password = nsURL.password {
            options.append(.password(password))
        }
        return options
    }

    class func delete(_ url: String, callback: ClientRequest.Callback) {

        var options = urlToOptions(url)
        options.append(.method("delete"))
        let req = HTTP.request(options, callback: callback)
        req.end()
    }
}

Well we'd be very appreciative if you submit a PR for this.

I will look at this one we get up to the 0725 toolchain.

Actually, we're migrating to the 07/28 toolchain.

@ianpartridge what is the procedure to close issues?  Should we wait until a release is made or close it now?

Close once the fix is merged. Thanks!


jsDump treats any object with a length property as an array

jsDump doesn't even agree with jQuery.isArray, treating anything with a length parameter as an array, causing inappropriate output, most commonly empty arrays or arrays containing undefined for as many times as the length property value. Real Array, and possibly NodeList objects can be detected by the same means that the objectType or jQuery.isArray methods work.

I've a fork of jsDump that doesn't have this problem. See 1d9563a.

Fixed by #164


Made JSON encoder a public function

This one is pretty simple.  It's just hidden.

Two questions:
First, I wasn't sure if Decode should also accept []byte, to have some symmetry with Encode.
Second, I wasn't sure if I should add some tests. I started to add some, but realized I was testing basically everything that's already being tested.

First, I wasn't sure if Decode should also accept []byte, to have some symmetry with Encode.

Yes, I think it should.

Second, I wasn't sure if I should add some tests. I started to add some, but realized I was testing basically everything that's already being tested.

You could add one test to ensure that a value can be encoded and then decoded to ensure there are no gross errors, but I wouldn't bother testing all the cases of JSON.


docs: update example code

What kind of change does this PR introduce? (check at least one)

[ ] Bugfix
[ ] Feature
[ ] Code style update
[ ] Refactor
[x] Docs
[ ] Underlying tools
[ ] Other, please describe:

Does this PR introduce a breaking change? (check one)

[ ] Yes
[x] No

Other information:
I think this is the correct PR 🤒
#4925

@NataliaTepluhina 🙈🎅

@whoisjorge Thank you! And sorry for making it this long 😅


Bump senzing from 3.2.0 to 3.3.0

See https://github.com/Senzing/senzingapi-runtime/issues/31

Pull requests:

https://github.com/Senzing/docker-installer/pull/41


can't find crate for std

Hi there, thanks for this package! I was following the install instructions and ran into this error:
~/scm/dev/rustplay/meow ➜  wargo build
  Setup wasm checking dependencies...
  Setup wasm found emsdk installation in ~/.emsdk
  Setup wasm setting environment...
emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 1.37.22
clang version 4.0.0  (emscripten 1.37.22 : 1.37.22)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir: /Users/danielpazsoldan/.emsdk/clang/e1.37.22_64bit
INFO:root:(Emscripten: Running sanity checks)
  Setup wasm running 'cargo build --target=wasm32-unknown-emscripten'
   Compiling meow v0.1.0 (file:///Users/danielpazsoldan/scm/dev/rustplay/meow)
error[E0463]: can't find crate for `std`
  |
  = note: the `wasm32-unknown-emscripten` target may not be installed

error: aborting due to previous error

error: Could not compile `meow`.

To learn more, run the command again with --verbose.

Are you on macOS? Try running rustup target add wasm32-unknown-emscripten
The newest wargo in master fixes this automatically, I just need to cut a new release.

Yup I'm on macOS and that worked for me. Thanks!