Partial aggregation easily causes OOM

Description
Partial AGG currently doesn't support reclaiming/spilling so user should manually control its memory usage through several memory limit settings for partial AGG.
However the limits are hard to be determined in production use cases since the plan may contain many nodes that are all consuming memory at the same time. As a result to avoid OOM we can only disable partial AGG (e.g by always abandoning it) but that way the final AGG will be under huge pressure.
In the case of Velox + Spark, speaking of Vanilla Spark it does seem to support spill-to-disk no matter an AGG is partial or not. So probably supporting spilling is an option to Velox's partial AGG either.
I was working on a patch to make Velox flush AGG intermediate result on OOM but that way doesn't seem to cover some corner cases. So we need a complete solution for the issue.

@xiaoxmeng @mbasmanova Do you have any comment on this issue? Thanks!

@zhztheplayer Our thinking is that partial agg doesn't require spilling since it can flush on memory pressure. We are also thinking that partial agg doesn't consume much memory before flush, only 16 MB [1]. When Task runs multi-threaded each thread may consume up to 16MB, but I believe Spark runs only one thread.
Still, it sounds like you are seeing something different. You mentioned that there can be many partial aggregations within a single task. This is true, but I wonder how many are you seeing? Would you share a query plan, the OOM error message, and the amount of memory used by partial aggs at OOM time.
[1] https://facebookincubator.github.io/velox/configs.html#memory-management

@mbasmanova Thanks for your comments.

Our thinking is that partial agg doesn't require spilling since it can flush on memory pressure.

Yes this is also my understanding about the design when I was working with it. But partial agg spilling could still be valuable in the case of N mappers + M reducers, N >> M. We should leverage as much resource as possible from mappers since there are just a few of reducers can be used.
A simple example, using the following SQL query (on TPC-DS partitioned data, SF 1000):
select cast(rand() * 2000000 as integer) as key, count(1) as cnt from catalog_sales group by key order by cnt desc limit 100

In the case of N = 72, M = 1, compare the query performance among 3 profiles:

Details about the profiles:

PARTIAL_MODE SPILL: Spill when partial aggregation runs out of memory (I use this patch which seems to be functional to enable spilling for partial aggregation)
PARTIAL_MODE FLUSH: Spill flush intermidiate data within memory limit = ~20 MiB per task
PARTIAL_MODE ABANDON: Abandon partial aggregation immediately

Thus in this case, mode SPILL is far faster than FLUSH and ABANDON.
Reducer shuffle read size of partial mode SPILL:

Reducer shuffle read size of partial mode FLUSH:

Reducer shuffle read size of partial mode ABANDON:

You mentioned that there can be many partial aggregations within a single task. This is true, but I wonder how many are you seeing? Would you share a query plan, the OOM error message, and the amount of memory used by partial aggs at OOM time.

Consecutive partial aggregations are not normal in our workloads. However We did encounter the case including several consecutive joins + one partial aggregation. Whether to OOM is up to our settings, like said, we can just abandon the partial aggregation or adopt more frequent partial flushes by setting lower memory limit numbers. But the way doesn't make sense if in the N+M case I have explained above.

I've opened a draft PR https://github.com/facebookincubator/velox/pull/7558, you can directly see the code changes I used in it.

@zhztheplayer Thank you for explaining. In this case, with a single reducer, I can see how spilling partial agg is beneficial. In this case spilling writes to disk (as shuffle does), but then unspilling processes data on N nodes instead of 1 (in case of shuffle). Since cardinality reduces significantly after partial agg with spilling, the query runs faster with spilling.
At first I was wondering why is there only 1 reducer, but then I noticed that there is a LIMIT clause in the query and Spark has a known inefficiency when it comes to running LIMIT queries. Spark doesn't plan partial and final limit and requires all the data to be collected on a single node to apply the LIMIT. I wonder if newer versions of Spark have addressed this problem already. Do you know?
I also have another question. How much memory do you specify for the query and for partial aggregation to trigger spilling? With default settings, 16MB for partial agg, a few GB per query, spilling won't be triggered because partial agg will be flushing.
My assumption is that allowing spilling for partial agg like in #7558 would have no effect unless one also disables flushing by increasing the memory limit for partial agg from the default 16MB to a few GB (or close to query memory limit). If that's the case, then it means that each query needs to be configured separately, which is not scalable.

At first I was wondering why is there only 1 reducer, but then I noticed that there is a LIMIT clause in the query and Spark has a known inefficiency when it comes to running LIMIT queries. Spark doesn't plan partial and final limit and requires all the data to be collected on a single node to apply the LIMIT. I wonder if newer versions of Spark have addressed this problem already. Do you know?

Aha, the single partition doesn't relate to any of Spark's limitation. It's because I manually set spark's shuffle partition number to 1. Spark's scan stage (the first stage to read data from sources) doesn't always respect shuffle partition number settings so it's normal that we have more scan tasks and less reducer tasks in one query.

I also have another question. How much memory do you specify for the query and for partial aggregation to trigger spilling? With default settings, 16MB for partial agg, a few GB per query, spilling won't be triggered because partial agg will be flushing.

I manually set it to a very large number, something like int.max, in this case 1024 * total host memory. To make sure that partial flushing never gets triggered.

If that's the case, then it means that each query needs to be configured separately, which is not scalable.

IIUC the UT case in #7511 successfully spilled data and didn't require setting mem limit because the memory pool capacity is 10MB which is smaller than 16MB. Anyway the UT should be revived (move to SharedArbitratorTest, probably) later on.
BTW, if setting a very large memory limit number doesn't seem to be a good way, maybe we can use an independent config option to disable partial flush / abandon? If we decided to add spilling support to partial aggregation.

if host has limited memory space, flushing would not reduce cardinality much.

This is true, but it also means we cannot assess whether partial agg will reduce cardinality even with spilling and may hit another edge case where it doesn't reduce cardinality. In this case, we'll incur the costs of spilling for partial agg and will still have to shuffle all the data.

Why couldn't flushing and spilling work together? Say the memory pool capacity is 100MiB and flushing limit is set to 10MiB. When free capacity of the pool is smaller than 10MiB (say, other operators allocated 95MiB and there is only 5MiB left for partial aggregation),

In this case, memory arbitration should ask other operators to spill first. Partial agg that uses only 5% of the memory should not be the one to spill. Am I missing something?

then partial aggregation should spill 5MiB data out and process another 5MiB, then flush 10MiB together because it reaches 10MiB flushing limit. The challenge here is that (correct me if I was wrong) the spilled 5MiB is not counted to partial aggregation's memory usage.

This is an interesting idea. You are correct spilled data doesn't count toward memory usage because it doesn't use memory.

I noticed in presto you use a fixed 16 MiB partial aggregation memory limit, is this a trade-off between cardinality reduction and OOM risk? It looks like too small for big data scenario to me. I am curious how far it's better than just disabling partial aggregation in your real workloads.

This is a good question. I'll need to do some research to find an answer.

In this case, memory arbitration should ask other operators to spill first. Partial agg that uses only 5% of the memory should not be the one to spill. Am I missing something?

Umm I just took this simple case as an example... In Gluten we have been setting the memory limits to 0.5 * task avaliable memory by default. Which means if other operators used more than a half of total memory, then partial aggregation could go to OOM eaisily. But technically speaking, I think even in the case of 95 MiB + 5 MiB, the 95 MiB allocation is still possible to be from a lot of small consumers, e.g. 95 consecutive joins in which each one allocates 1 MiB... So there can still be some chance for OOM anyway.

is still possible to be from a lot of small consumers, e.g. 95 consecutive joins in which each one allocates 1 MiB... So there can still be some chance for OOM anyway.

While this is possible, it is quite unlikely. If this is the only case that would benefit from the new logic, then it feels that complexity and maintenance costs of this new logic do not justify the benefit. Am I missing something? It would be nice to come up with more realistic use cases that may require spilling for partial aggregation and then figure out the best way to support them.

is still possible to be from a lot of small consumers, e.g. 95 consecutive joins in which each one allocates 1 MiB... So there can still be some chance for OOM anyway.

While this is possible, it is quite unlikely. If this is the only case that would benefit from the new logic, then it feels that complexity and maintenance costs of this new logic do not justify the benefit. Am I missing something? It would be nice to come up with more realistic use cases that may require spilling for partial aggregation and then figure out the best way to support them.

I understand your concern. The realistic use case is actually the one I mentioned: We set 0.5 * task available memory as the partial aggregation memory limit (the one in presto you set to 16 MiB by default, I think).
The code is here:
https://github.com/oap-project/gluten/blob/d2980b73afbf28800727172200e36a84c4c83536/cpp/velox/compute/WholeStageResultIterator.cc#L341-L342
The issue we faced is that if we set the factor to 0.5 or higher, it's high risk to get into OOM. If set set the factor to a low value, say 0.05, there would nearly be no cardinality reduction and partial aggregation is usually skipped.
Preventing OOM is only one of the benefits we can get from partial aggregation spilling. As I explained in previous comments, it can considerably reduce cardinality then to reduce shuffle size in Spark.

It would be nice to come up with more realistic use cases that may require spilling for partial aggregation and then figure out the best way to support them.

Hi @mbasmanova , I am feeling that our concern here about adding spilling support to partial aggregation is based on the assumption that 16 MiB memory limit is enough for user. Let's think about somehow a higher memory limit is needed, e.g. 128 MiB, 1 GiB, etc. The spill is reasonable (even the only option) since we don't want OOM. Is that correct?
If yes, the problem will become why we need large number of memory limit. So I recommend running some tests against prestodb to see if 16 MiB memory limit actually helped to reduce data size. In my case you can see 20 MiB didn't reduce data size.
What I could assume is that, small aggregation buffer like 16 MiB can still help a lot in the case of aggregating pre-ordered (or nearly pre-ordered) keys. For example aggregation on time key in time-series data in streaming SQL or time-series database. But this case is only one of the usages of databases.

@zhztheplayer Hi Hongze, thank you for providing further context.

Let's think about somehow a higher memory limit is needed, e.g. 128 MiB, 1 GiB, etc. Then spill is reasonable (even the only option) since we don't want OOM. Is that correct?

Yes, spilling might be a reasonable option, but it is not the only one. Flushing is another viable option. I understand that it doesn't work well when there are very small number of reducer tasks (in the original example there is only 1 reducer task), but I don't quite yet understand why this is a common scenario. Even in that scenario, I assume total query CPU time is higher with spilling, while wall time is lower. Is this the case?
Spilling is expensive as it requires sorting data, copying data from row-wise representation into columnar, writing data to disk, then reading data back, merge-sorting, converting to row-wise representation. And after doing all this for partial aggregation we still need to shuffle the partial results (copying data again) and aggregate then again. This might be justified If spilling allows us achieve higher cardinality reduction than flushing.
For example, if we have an unfortunate case of receiving many rows with unique keys first before we start seeing repeated values. In this case though, we'd detect partial aggregation as non-cardinality reducing and disable it. Hence, spilling won't have a chance to kick in.
Are you on Slack? It might be easier to discuss there? Otherwise, maybe create a Google Doc and let's continue discussion there as it might be easier. It would be helpful to come up with a few common use cases that require spilling and explain for each why flushing doesn't work and how you plan to avoid having partial agg disabled before spilling has a chance to kick in. It would be nice to also discuss how you plan to avoid the edge case of triggering spilling for the case of mostly unique keys.

Closing as not planned. See background here. Thanks everyone.


Discover safekeepers via broker request

We had an incident where pageserver requests timed out because pageserver couldn't fetch WAL from safekeepers. This incident was caused by a bug in safekeeper logic for timeline activation, which prevented pageserver from finding safekeepers.
This bug was since fixed, but there is still a chance of a similar bug in the future due to overall complexity.
This issue proposes to add a new broker message to "signal interest" for timeline. This signal will be sent by pageservers wait_lsn, and safekeepers will receive this signal to temporary activate this timeline. This will start broadcasting broker messages from safekeepers, and then every broker subscriber will be able to find the safekeepers and connect to them (to start fetching WAL).
Overall, this feature should work just like a normal network discovery through broadcasts.
This feature is not limited to pageservers and any service that wants to download WAL from safekeepers will be able to use this discovery request.

Update: I'm going to work on this issue, since it's very useful for pageserver sharding. We want to allow multiple pageservers to fetch WAL from safekeepers for a single timeline, and this is the easiest way to get started. We still can have issues with remote_consistent_lsn updates and non-efficient keeping/offloading/downloading WAL segments from S3, but that can be addressed later.


chore: add missing test dependency for docker

#13 21.46 npm ERR! If you are on Ubuntu, you can install with:
#13 21.46 npm ERR!
#13 21.46 npm ERR!  sudo apt install chromium
#13 21.46 npm ERR!
#13 21.46 npm ERR!
#13 21.46 npm ERR!  sudo apt install chromium-browser
#13 21.46 npm ERR!
#13 21.46 npm ERR! ERROR: Failed to set up Chromium r1022525! Set "PUPPETEER_SKIP_DOWNLOAD" env variable to skip download.

I tried building the Docker Image on ARM. There are a couple of problems. The image is almost 4GB big 😢

Happy to have more improvements done if you have any suggestion 👍

I will iterate with small changes :)

/rtm

https://github.com/all-contributors please add @Ferror for bug, infra

@all-contributors please add @Ferror for bug, infra


How did you do the implementation?

I am always curious on how people do the "conversion" from one language to another?
Did you take the original source code and translated it one file after another or did you take the concepts and try to implement it or another way?
Sorry for my curious mind. Thanks.

I took the Redis commands and configuration, stored as a todo list and implemented how I felt it was the Rust way. I only did literal translation for small pieces of code, like configuration parsing.


Clicking a series doesn't open it in the viewport. [v2.x]

Instead, one must drag and drop the study into the viewport.
If I remember correctly, in the Meteor version, one could click (or double click) the thumbnail in the series vertical bar, and the series would appear in the viewport.

I agree that this would align better with the expected experience. My only call-out is that we need to be careful on touch devices to differentiate between a scroll and a tap.

Scoped and added to backlog: #895


Add support for Cake.Net

Cake.Net creates a tools folder at the root which contains downloaded packages and should not be checked in. See documentation for reference to the tools folder where NuGet packages are downloaded.

@RehanSaeed thanks!


1.0 Soft multi-tenancy support

Describe the bug
We have been trying to follow https://istio.io/blog/2018/soft-multitenancy/ with 2 installs of istio on the same EKS cluster. We are using two separate namespaces for each istio install. When we deploy the bookinfo app (just into the default namespace) it depolys fine, the pod comes up, and all services are created, but we cannot reach the bookinfo productpage.
Expected behavior
If soft multi-tenancy is still supported since this blog was written, it should work as the blog describes, or updated documentation should be created to help users deploy istio in this manner. If it's not supported, then this becomes a feature request.
Steps to reproduce the bug
Deploy istio into istio-system-a.
Deploy istio isto istio-system-b.
Depoly bookinfo app (manual sidecar injection) with the -i istio-system-a flag to point to that install of istio.
Test the endpoint for productpage and observe no response from the app.
Version
istio 1.0, EKS K8s version 1.10.3.
Is Istio Auth enabled or not?
Did you install the stable istio.yaml, istio-auth.yaml.... or if using the Helm chart please provide full command line input.
Environment
EKS AWS
Cluster state
If this is needed I can provide later. I don't know what test cluster my colleague was using so I can't create the file until I speak to him tomorrow.

I'd say that this functionality is broken. I've tried to deploy istio to another namespace so I generated a yaml from helm with this block:
helm template $istio_path/install/kubernetes/helm/istio \
     --name istio \
     --namespace TEST \    <------ NEW NAMESPACE
     --set sidecarInjectorWebhook.enabled=false,galley.enabled=false,servicegraph.enabled=false,prometheus.enabled=false,grafana.enabled=false,global.mtls.enabled=true \
     > ./istio-install.yaml

And still see this in the istio-egressgateway configuration:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: istio-egressgateway
  namespace: istio-system-blue
  labels:
    app: egressgateway
    chart: gateways-1.0.0
    release: istio
    heritage: Tiller
    app: istio-egressgateway
    istio: egressgateway
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: istio-egressgateway
        istio: egressgateway
      annotations:
        sidecar.istio.io/inject: "false"
        scheduler.alpha.kubernetes.io/critical-pod: ""
    spec:
      serviceAccountName: istio-egressgateway-service-account
      containers:
        - name: egressgateway
          image: "201777367430.dkr.ecr.us-east-1.amazonaws.com/platform/istio/components/proxyv2:1.0.0"
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
            - containerPort: 443
          args:
          - proxy
          - router
          - -v
          - "2"
          - --discoveryRefreshDelay
          - '1s' #discoveryRefreshDelay
          - --drainDuration
          - '45s' #drainDuration
          - --parentShutdownDuration
          - '1m0s' #parentShutdownDuration
          - --connectTimeout
          - '10s' #connectTimeout
          - --serviceCluster
          - istio-egressgateway
          - --zipkinAddress
          - zipkin:9411
          - --statsdUdpAddress
          - istio-statsd-prom-bridge:9125
          - --proxyAdminPort
          - "15000"
          - --controlPlaneAuthPolicy
          - NONE
          - --discoveryAddress
          - istio-pilot.istio-system:8080     <----------- LOOK HERE
          resources:
            requests:
              cpu: 10m

Which is definitely wrong...

cc @rshriram @ZackButcher @costinm

Wondering if there has been any internal discussion about this yet. Thanks!

I have the same problem. When I added the arg ("--appNamespace   ns-0") for pilot, the service of bookinfo app could not be accessed.

I have the same problem too. Any update for this issue?

Note, from experimentation I found that the pilot option --appNamespace cannot refer to a different namespace than the istio control plane NS since it limits discovery to a single NS. In order for the policy pods to be in the mesh --appNamespace needs to be istio-system (or whenever you have installed istio control plane). In effect you need to install your workload in the control plane NS.


Ariston Genus One

Hi, you integration is promising! My heater has option to set 2 temps - daily and night which can be different and programmed separately. Any chance to add this? Ariston Genus One. Let me know so we can hook up on whatsup (login/pass).

Hi @VoyteckPL
Sorry I am a bit slow. Can you tell me which device type do you see on the "Device info" section? My guess is "GALEVO". What is your suggestion to add a new temperature? Should I create a new number entity or maybe something else?


[PF-541] refactor kube pod listener

I was studying this class and had a little trouble following what it does, so I broke it into some  smaller functions. Also added final, since some variables are imperative and some  aren't, some stream and Optional usage, and a couple of comments.

Looks like there's no unit test coverage for this guy, so I'm thinking I should add some. The mocking might be wonky but I think it should work.

I don't think it is worth trying to unit test this. Seems to me a lot of effort for little benefit. The whole purpose of the code is to interact with K8s, so I believe in testing it with K8s.


Support for Safari

I am submitting a

[x] Bug Report

What is the expected behavior?
Be able to get the user id through await OneSignal.getUserId()
What is the current behavior?
It returns a bug when subscribing as shown:

What are the steps to reproduce?
As far as I can see just creating the project as the example in the docs.
What is the use-case or motivation for changing an existing behavior?
Which versions are you using for the following packages?
Angular: 10.2.0
Angular CDK: 10.2.6
Angular CLI: 10.2.0
Nx (option):
ngx-onesignal: 10.0.0
Is there anything else we should know?
This is my app.module:
NgxOneSignalModule.forRoot({
      appId: 'my-app-id,
      autoRegister: false,
      allowLocalhostAsSecureOrigin: true,
      notifyButton: {
        enable: true,
      },
    }),

And my app.component:
import { Component, OnInit } from '@angular/core';
import { OneSignalService } from 'ngx-onesignal';

@Component({
  selector: 'app-root',
  templateUrl: './app.component.html',
  styleUrls: ['./app.component.scss']
})
export class AppComponent implements OnInit {

  constructor(public readonly onesignal: OneSignalService) {
    (window as any).ngxOnesignal = this.onesignal;
  }

  ngOnInit(): void {
    setTimeout(() => {
      this.onesignal.subscribe();
      console.log(`Initiated`);
    }, 2000);
  }
}

Also, I couldn't see anything about the safari_web_id property as described in https://documentation.onesignal.com/docs/safari-web-push-setup, so, does this wrapper supports it?
My Safari version is 14.0 (156<IP_ADDRESS>.9, 15610) and I'm on a Mac OS Catalina 10.15.7.
You can access my example here https://onesignal-testing.surge.sh and also its repository on Github https://github.com/nncl/ngx-onesignal-test
I really appreciate your help :)

@nncl
Yes, I support safari_safari_web_id in safari.
I have been suffering from the same phenomenon since the day before yesterday, but now it seems to be able to call the API successfully.
Are still experiencing the problem in your environment?
If the problem is still there, I'll find a reproducible environment and look deeper into it.

@MSakamaki yes, unfortunately still happening when testing on Safari this following environment https://onesignal-testing.surge.sh/ Have you tried? If so, this error is shown for you?
Thanks for your time.

Temporary Report
After investigating, I found that there is a problem with the Driver inside ngsw-worker.js.
It appears that the server is returning a CORS error due to the server hooking up the "https://onesignal.com/" communication when fetching onesignal communications and executing from the application's domain.
We have been able to confirm that it can be worked around by temporarily disabling this feature, but we are still investigating the underlying solution.
https://github.com/angular/angular/blob/master/packages/service-worker/worker/src/driver.ts#L404

@MSakamaki thanks, I appreciate your time. I'll try to disable and test as you did, and also if I come up with something else I'll let you know.


Unable to access menus, flickering while trying to open

https://u.pomf.io/vnmsmb.mp4
https://u.pomf.io/ngchfe.mp4
https://u.pomf.io/mutdqv.mp4
https://u.pomf.io/axbgnh.mp4
https://u.pomf.io/hmkkyd.mp4
https://u.pomf.io/fnukst.png

Are you up to date on everything? In particular, make sure you're up to date on both wlc and sway.

Yes, all system is up to date.

Could you humour me by recompiling wlc? This was a known bug that was fixed in https://github.com/Cloudef/wlc/commit/359cfe776a17a0fae5f8179e8a89e9b47256f602

I recorded this videos after recompiling, so...

Recompiling sway or wlc? What distro are you on?

Yes, i recompile sway, wlc, and packages on which they depend after seeing a bug. Gentoo unstable.

Which version of xwayland?

1.17.4

Can you try 1.18.0?

Still flickering with 1.18.0.

Strange. I can't reproduce. You're on your own here.

I've had this issue for some weeks.  Chromium generates it easy enough, just hover over any tabs and you'll get flickering when it fails to display the tooltip (hover-over text).

I just installed orbment and the same thing happens. It's probably wlc.

I just installed orbment and the same thing happens. It's probably wlc.

My current understanding of the issue is that the fault is shared between all three.

My current understanding of the issue is that the fault is shared between all three.


k8s API v1.21 and cdk8s-plus-21

This is just a question.
To use k8s API v1.19, v1.20 and cdk8s together,
Do I have to create my own cdk8s-plus-19, cdk8s-plus-20?
Currently, this repo provides only cdk8s-plus-17.
I was just wondered but still wondered even with the documentation.
Thank you.

Based on decision described here - we need to come up with a fork-merge strategy to manually push necessary changes to each of the library version lines.

As a first step, we decided we are going to create a single cdk8s-plus repository and have multiple release branches.
Each branch will correspond to a different k8s spec version.
So we will have, for example:

k8s.21
k8s.20
k8s.19

The library name will be derived from the branch name, and versions are completely independent between those libraries.
This requires some projen hackery to support a different package name per release branch.

Related: https://github.com/projen/projen/pull/1079


directx_samplegrabber_callback: Add missing DirectShow include.

Without this include I am not able to compile this module, it results in a ton of undefined identifier errors.

What version of the Windows SDK are you using, out of curiosity?

For the qedit header I'm using 7.1A.

Ah, sure, I apparently went overboard cleaning up includes. Not sure why it worked on my system. BITMAPINFOHEADER isn't defined by magic :)


Update docs with other typical errors

There are common errors that we explain to our users, such as
Console request failed and all its different options

issues related to exceeding max_connections

@danieltprice, @bgrenon
can you pick this item up? @neondatabase/proxy team is read to help you explaining such cases if you have questions


Will this integration work with Audio Events?

I'd like an event or state variable in home assistant when the cat event is triggered (see below, it is an audio event), and for it to be a different event than detecting a cat with the camera.  If  a cat is making noise near the microphone, it should trigger, assuming the model works.
# yaml-language-server: $schema=http://ha.local:5000/api/config/schema.json
detectors:
#  cpu1:
#    type: cpu
  coral:
    type: edgetpu
    device: usb

record:
  enabled: true
mqtt:
  # Required host name
  host: ha
  port: 1883
  #topic prefix must be  unique if running multiple instances
  topic_prefix: frigate
  stats_interval: 60
  user: mqttuser
  password: hotdog
environment_vars:
  LIBVA_DRIVER_NAME: i965

go2rtc:
  streams:
  # https://www.reddit.com/r/frigate_nvr/comments/1dkd2ua/doorbell_exceeded_fps_limit_exiting_ffmpeg/
  #possibly need to change settings on doorbell to use flv
    reolink_doorbell_det:
#      - "ffmpeg:http://doorbell.local/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=doug&password=ransom#video=copy#audio=copy"
      - rtsp://doug:ransom@doorbell.local:554/h264Preview_01_sub

    reolink_doorbell_rec:
#      - "ffmpeg:http://doorbell.local/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=doug&password=ransom#video=copy#audio=copy"
      - rtsp://doug:ransom@doorbell.local:554/h264Preview_01_main

# Optional: logger verbosity settings
logger:
  # Optional: Default log verbosity (default: shown below)
  default: info
  # Optional: Component specific logger overrides
  logs:
    frigate.event: debug

ffmpeg:
  hwaccel_args: preset-intel-qsv-h264

audio:
  enabled: True

cameras:

  reolink_doorbell_det:
    motion:
      mask:
        - 0.461,0.507,0.465,0.543,0.618,0.536,0.612,0.506
        - 0.333,-0.001,0.333,0.056,0.662,0.056,0.66,0
    ffmpeg:
      inputs:
        - path: rtsp://localhost:8554/reolink_doorbell_det
          roles:
            - detect
            - audio
          input_args: preset-rtsp-restream
        - path: rtsp://localhost:8554/reolink_doorbell_rec
          roles:
            - record
          input_args: preset-rtsp-restream
    audio:
      enabled: True
      listen:
        - cat
    detect:
      enabled: true # <---- disable detection until you have a working camera feed
      # width: 640 # <---- update for your camera's resolution
      # height: 480 # <---- update for your camera's resolution
    record:
      enabled: true
    snapshots:
      enabled: true

    zones:
      default_zone:
        coordinates:
          0.307,0.103,0.359,0.48,0.34,0.615,0.209,0.597,0.198,0.979,0.959,0.987,0.949,0.585,0.853,0.569,0.824,0.156,0.534,0.075
        loitering_time: 0
birdseye:
  enabled: true
  mode: continuous
version: 0.14

@blakeblackshear also do you want to turn on discussions for the various frigate github repos?

discussions are on in the main frigate repo

There are three different sounds that seem related to cat
https://github.com/blakeblackshear/frigate/blob/ed9c67804a33e93db518a487cbf27624e4b6c779/audio-labelmap.txt#L77-L80

I did find the entity related to the sound notification.  I guess I didn't see it in the search.


extend timeout for cloud build

What type of PR is this?
/kind bug

What this PR does / why we need it:
Your build timed out. Use the [--timeout=DURATION] flag to change the timeout threshold.
ERROR: (gcloud.builds.submit) build 1aaf01c6-a2fc-4a43-8e66-5aed461643b8 completed with status "TIMEOUT"

Not enough time for frontend build.
https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-kube-scheduler-simulator-push-images/1472567754648195072
Which issue(s) this PR fixes:

Ref #19
Special notes for your reviewer:
/label tide/merge-method-squash

/lgtm


Using GTIN for archive id

This is a hacky-but-it-works fix for https://github.com/9-FS/nhentai_archivist/issues/29. Komga parses the GTIN as the ISBN number here then makes the ISBN number searchable here. When parsing the GTIN, Komga does try to validate the number so I had to force it to be 10 digits and make the last digit (the checksum) valid.

Interesting from a technical point of view, but the additional digit for the checksum makes this jankier and less intuitive than the present solution.

Sure, understandable. Hopefully the Komga author comes up with a workable solution.

I wouldn't put my hope on it... https://github.com/gotson/komga/discussions/1723


cargo clippy is no-op when edition = 2018

Problem

cargo clippy is not running clippy when edition = 2018. It's also not providing any feedback. It works correctly on edition = 2015.
# set edition to 2018 in Cargo.toml
$ cargo clippy -vv
       Fresh reproclippy v0.1.0 (/home/ks/workdir/rust/reproclippy)
    Finished dev [unoptimized + debuginfo] target(s) in 0.00s

# Empty output - meanwhile, calling "clippy-driver" directly (implicitly in 2015 mode) generates non-empty output:

$ clippy-driver src/main.rs
warning: casting a character literal to `u8` truncates
[...]

Various component versions are listed below - it's nightly.
Note that it could as well be caused by clippy misbehaving on version = 2018, but I have no way to test this, since neither cargo nor clippy doesn't expose any information on calling clippy directly with non-default edition. Either way I blame cargo, either for a bug, or for lack of feedback that I could use to prove that it's clippy fault.
It worked around January/February, I'm not sure when it stopped (likely a good few months ago, because I've seen empty output but haven't checked if it's working correctly).
$ cat Cargo.toml
[package]
name = "reproclippy"
version = "0.1.0"
authors = ["redacted"]
edition = "2018"
[dependencies]

$ rustup show active-toolchain
nightly-x86_64-unknown-linux-gnu (default)

$ rustup show profile
default

$ clippy-driver --version
clippy 0.0.212 (346aec9 2020-07-11)

$ cargo --version
cargo 1.46.0-nightly (4f74d9b2a 2020-07-08

Steps

Just set up a new default project using Nightly toolchain, add something like let foo = 3; and test using cargo clippy and clippy-driver src/main.rs.

Side issue - there is no way to drill into what cargo clippy does on the inside. I would expect such option from a wrapper-like tool.
Is it feasible feature request to add logging (in like a 2nd or 3rd verbosity level) for all commands called from cargo master process?

I believe you are seeing that clippy shares the cache with cargo check.    You can run cargo +nightly clippy -Zunstable-options to avoid the caching. There are more details at #8143 and https://github.com/rust-lang/rust-clippy/issues/4612 and https://github.com/rust-lang/rust-clippy/issues/3837.
As for logging, the CARGO_LOG environment variable can be used to provide debugging information about cargo's behavior.  For example CARGO_LOG=cargo::core::compiler::fingerprint=trace will display information about cargo's fingerprinting.

@ehuss I see... Any chance for quick solution to avoid confusion? E.g. extra prompt message after running "cargo clippy"? Like "Compiling modified modules using clippy" (or whatever is happening actually). I appreciate there is complete solution in pipeline, but it seems months away, while the problem (misunderstanding) is actually trivial to fix.

You will need to follow up with the clippy team (either in one of the linked issues above, or file a new issue on their repo, or ask on Discord).
As for CARGO_LOG, it is mostly undocumented, as it is normally only used for debugging. It is mentioned here for contributors. It tends to be very low-level, and not easily usable by end-users.

I'm going to close this as the relevant issues are tracked in the issues linked above.