AsYouTypeFormatter

Thanks for the work you put in this! Nice utility.
Formatting
How would you incorporate AsYouTypeFormatter from google-libphonenumber into it?
I have:
import { PhoneNumberUtil, AsYouTypeFormatter } from "google-libphonenumber";
const phoneUtil = PhoneNumberUtil.getInstance();
const formatter = new AsYouTypeFormatter('US');

And it will indeed format the number as we type if we change to
onChangeText = (text) => {
    this.setState({ number: text });
    const { onChangeText, onChangeFormattedText } = this.props;
    if (onChangeText) {
      const formatted = formatter.inputDigit(text.slice(text.length - 1))
      onChangeText(formatted);
    }
    if (onChangeFormattedText) {
      const { code } = this.state;
      if (code) {
        onChangeFormattedText(text.length > 0 ? `+${code}${text}` : text);
      } else {
        onChangeFormattedText(text);
      }
    }
  };

But it gets screwed up once we type delete. It's probably due to saving the number in state.
How would you recommend handling this?
Pasting
This is less important, but could be nice to have.
I'm passing textContentType: 'telephoneNumber', via textInputProps.
textContentType (on iOS) allows one click number paste. Any suggestions on handling pasting of numbers and then formatting if needed? Maybe some type of paste(it's probably the same as type) > strip > format ?

I got parsePhoneNumber working from libphonenumber-js https://gitlab.com/catamphetamine/libphonenumber-js which is a similar formatter to AsYouType
`import PhoneInput from 'react-native-phone-number-input';
import { parsePhoneNumber} from 'libphonenumber-js'
onChange = (text) => {
const phoneInput = this.mRef
const countryCode = phoneInput.current.getCountryCode()
console.log('country code ===>', countryCode)
    console.log("PHONE INPUT===>", phoneInput);
    if(text.length > 5){
        const phoneNumber = parsePhoneNumber(text)
        this.setState({
            setFormattedMobile:
             phoneNumber.format("INTERNATIONAL").replace(/\s/g, '')
        })
    }
}

render() {
return(
<PhoneInput
ref={this.mRef}
onChangeText={text =>
this.setState({mNumber:text})}
onChangeFormattedText={this.onChange}
placeholder="Mobile Number"
defaultValue={this.state.mNumber}
/>
)
}
`

@KerrySng99 thanks for sharing your code, but this is not "as you type" solution. You only parse when the length is 5. Also, it doesn't account for pasting a whole number from the phone. For instance, with textContentType="telephoneNumber" option passed to the TextInput, it will allow for a quick one-click pasting of your phone number. That number may contain non-numeric characters as well, as in<PHONE_NUMBER>.
Still hoping @garganurag893 will drop a word on this. Not sure if this library is maintained though.

Hey all.
i was able to do that using libphonenumber-js.
You need:
1 - import { AsYouType } from "libphonenumber-js";
2 - const phoneFormatAsYouType = new AsYouType(phoneInput.current.getCountryCode());
3- onChangeText={(text) => {
setPhoneValue(phoneFormatAsYouType.input(text));
}}

@theodelebarre I haven't tested it thoroughly and on different devices, but it seems to work. thanks for sharing

It working nicely, however, once I set the default value to a formatted value, it gets all messed up when you start editing the number.
What I mean is this:
Say I have a user phone number as<PHONE_NUMBER>. when the user loads the update your phone number screen, I want to show the phone number already formatted, so I do this:
<PhoneInput
    ref={phoneInput}
    defaultValue={phoneFormatAsYouType.input(value)}
    defaultCode="US"
    onChangeText={(text) => {
        setValue(phoneFormatAsYouType.input(text));
    }}
    onChangeFormattedText={(text) => {
        setFormattedValue(text);
    }}
    textInputProps={{
        maxLength: 14,
    }}
/>

I'm passing a formatted value with defaultValue={phoneFormatAsYouType.input(value)}. It shows up nicely, but once you start typing it gets screwed up.
What works is passing defaultValue={value} - It will not show the formatted value right away, but once the user starts editing, it will function correctly. But that's not so nice.
Any ideas how to fix this?

@theodelebarre His solution works excellently for me, however when I go over the max length it jumps to being un-formatted.  Is there a way to dynamically set the maxLength of the text input (based on country) so that this can't happen?

@kickbk @ajlamarc Hey, sorry I haven't seen those comments before,
I think you retrieved a value that was previously sent unformatted, I had that issue before, but then I cleared my test users, started again with that input set everywhere I needed it, the phone number saved is formatted, and when returned it perfectly displays itself.
For the other issue where it gets unformatted if too long, I think libphonenumber.js does that by itself, maybe it is just a visual feedback saying here you're phone number's wrong, we can't format it, but I'm sure you can play around with this library to detect the country of the phone number typed in and format it differently.
Have fun coding! :P

Merci Théo. That definitely helped. It now mostly works. So far, what doesn't work, while testing on the simulator, is deleting a digit in the middle of the phone number using the "Delete to end of line" with Ctrl-D, or the "delete ->" button on a Mac keyboard. It removes a digit, adds an additional bracket in the beginning of the line instead of trimming the number.
However, I may be able to live with this since on a mobile device you cannot delete in such a way.
If you can think of why it does it and if there's a solution, that would be much better of course.

@kickbk Yeah I found out as well.. I'm still trying to enhance it from time to time until I won't have any issue. I will share asap if I'm able to find something :P

I got it working with google-libphonenumber by patching the library code:
onChangeText = (text) => {
    const originalNumber = (text.match(/\d/g) ?? []).join("");
    const { onChangeText, onChangeFormattedText } = this.props;
    const formatter = new AsYouTypeFormatter(this.state.countryCode || 'US');
    let formatted = '';
    formatter.clear();
    for (let c of originalNumber) {
      formatted = formatter.inputDigit(c);
    }
    this.setState({ number: formatted });
    if (onChangeText) {
      onChangeText(originalNumber);
    }
    if (onChangeFormattedText) {
      const { code } = this.state;
      if (code) {
        onChangeFormattedText(originalNumber.length > 0 ? `+${code}${originalNumber}` : originalNumber);
      } else {
        onChangeFormattedText(originalNumber);
      }
    }
  };

I submitted a PR: #51

I am afraid this code does not work :/

Thanks @WangHansen, worked like a charm


amalgamate aqua as the artisan

One thing that came to realization while working on https://github.com/ignatiusmb/aqua/pull/37, as everything's starting to come together there and docs are getting updated, was that aqua is too low-level to be useful on its own and it didn't make much of a sense that the docs relied on marqua that depends on an older version of the package that is being built. It was a cyclical mess that I considered having a monorepo to house both of these packages so that this problem would be easier to tackle, until I decided to sleep on it.
I'm glad I did because amalgamating aqua here is a much better solution, why try to solve or workaround a problem when we can just make the problem obsolete, basically nonexistent in this case. This would however mean a breaking change, which the v0.4.0 milestone has been reserved for a lot of stuff that needs revamping. We could either shift all of them to v0.5.0, or take some more time to tackle everything in the milestone and overhaul the codebase, which could be answered later on.
Closes #35
Closes #37
Ref https://github.com/ignatiusmb/aqua/pull/37
Deprecate https://github.com/ignatiusmb/aqua after merging

review and proofread docs as a follow-up


Use mapInPandas in DataFrame.apply

This PR proposes to use mapInPandas which is available from Spark 3.0 at DataFrame.apply. I manually ran the existing tests against Apache Spark 3.0.

LGTM

Thank you @ueshin and @itholic.


46 write test cases

Issue:
Previously, our repository was lacking automated testing for our python files
What changed:
Added a python-testing.yml file under .github/workflows and a tests folder under the root directory which contains a file to configure the pathing for our python tests (conftest.py) and a test for the build where text method (test_postgres_connector.py) which can be expanded on for further postgres connector testing.
How to experience the change:
Make a branch and corresponding PR, then push anything to the branch and check the "checks" tab of the PR to see the github action running. Alternatively, check out this branch and navigate to the root directory, then run 'python -m pytest -v ' which will execute the current test and output a '1 passed' message indicating the test passed.
Note: While all commits were pushed by Evan Richmond, all work for this PR was done collaboratively

Automated tests seem to fail when run through yml, runs and passes when testing locally, we are working to resolve this issue by modifying the yml file and looking into other path solutions. The primary issue is that our code is not packaged as python wants it to be so importing from backend into the tests folder is difficult.

Had to add a requirements.txt file, any dependencies for unit testing will need to be specified separately here due to the automated testing environment. Build now runs correctly and all tests pass!


Barrels not recognized as job site

Only the vanilla Oak Barrel seems to be recognized by villagers as a fisherman job block

i think i read somewhere that this was intentional. a feature, not a bug.


[QUESTION] Can this run on microcontroller such as Raspberry Pi Pico?

Can EmbeddedRPC be used on Raspberry Pi Pico for example?
I see that there is article https://github.com/EmbeddedRPC/erpc/wiki/Porting-Guide, but with no pseudo code or something similar, where interface classes are used?
I have their SDK for development applications, now I want to create RPC server on it and trigger it via UART.
RPC server runs on Raspberry Pi Pico, client runs on Linux PC. Microcontroller and PC are connected via UART.
If this can be done on Raspberry Pi Pico, is there some guide how to do it, not for Raspberry Pi Pico directly, but any generic microcontroller with some simple example?

Hello @cvele1995 , if we are talking about Linux OS with Linux serial and pthreads support, then i don't see any issue why it should not work on rpi pico.
Documentation is here: https://github.com/EmbeddedRPC/erpc/wiki/Getting-Started
As an example can serve tests (e.g. initialization files in test/common folder) or nxp examples for nxp devices where you need register and then you can download examples for some board.

The pico is an RP2040 microcontroller, it does not run on linux. But the controller is spreading like a wildfire :)

So then depends, if it is running OS which is not supported, you need implement serial/uart and thread. It should be just these two files.

Wonder if i could make the py version run with circuitpython, threading might be an issue since it only provides hardcoded asyncio not threading itself afaik

We actually didn't specify language and i was thinking that you mean C/C++. In case of Python maybe you need just implement new transport if current doesn't fit.

@Hadatko Thanks for responding.
Reason why I mentioned Raspberry Pi Pico was that I wanted to make a test program for EmbeddedRPC to see how it works on other simpler platform. Make a simple program.
I have NXP board and in the past I used mcux-sdk for developing firmware on it.
Now I am using some other product made by NXP that is not mcux-sdk (but it has a similar role).
For UART communication with the CPU I am using LPUART.
When I was looking at the board examples of Embedded RPC, for example matrix multiply
I noticed it uses _ARM_DRIVER_USART  structure, meaning it uses CMSIS USART driver.
Can Embedded RPC work with LPUART?
If it can, what would be the steps to integrate it?

Regards

HI, no problem. I think we used to have LPuart and uart implementation. But to make implementation easier we swtiched to use cmsis uart which in case of NXP could be used with both lpuart and uart device.
CMSIS: https://github.com/EmbeddedRPC/erpc/blob/develop/erpc_c/transports/erpc_uart_cmsis_transport.hpp

This means I would have to implement my own transport class using LPUART, by inheriting FramedTransport class?

@MichalPrincNXP Does cmsis uart works with nxp lpuart driver or only with full uart driver?

@Hadatko
When I look at the UartTransport class and its implementation, it inherits "FramedTransport" interface.
Is it mandatory for functions "underlyingReceive" and "underlyingSend" to block until next frame comes?

Hi @cvele1995 , cmsis uart driver is generic API interface that can be used with different underlying uart drivers, incl. lpuart. If you have the sdk package for you nxp board, I would recommend to look at boards//cmsis_driver_examples  folder to see how it is used in projects.
I would also recommend to download the sdk package for either evkmimxrt1170 or lpcxpresso55s36 or frdmk32l2b board that contain erpc_server_matrix_multiply_uart multiprocessor example showing the erpc communication between the PC running python and MCU running C via the uart transport. CMSIS UART transport is used there but different underlying uart drivers are utilized, based on the particular MCU uart peripheral.

I think thisproblematic was explained.


ApplicationSets

Tasks

[ ] What are the value props of applicationsets (why?)
[ ] How does an app-of-app pattern look like for a Applications
[ ] A bit about generators, understanding that apps are created through appsets (not in a git repo)

https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Use-Cases/

Close:
https://github.com/eficode-academy/argocd-katas/blob/main/applicationsets.md


I am facing an issue while creating a web app directory in system32?

Problem encountered on https://dotnet.microsoft.com/learn/aspnet/hello-world-tutorial/create
Operating System: windows
Provide details about the problem you're experiencing. Include your operating system version, exact error message, code sample, and anything else that is relevant.

Try creating the app in your user folder. System32 is intended for Windows' system files and should not be used for documents or project files.


bug: cannot suppress Resource::* on IAM Role with managed policy that uses *-Resource

What is the problem?
I get an AwsSolutions-IAM5[Resource:**] error on a Role that is using the AWS managed policy AmazonECSTaskExecutionRolePolicy, even though I am suppressing it. The managed policy is using the *-Resource, which is the error I want to suppress.
Reproduction Steps
const executionRole = new iam.Role(this, 'TaskExecutionRole', {
  roleName: `FrontendExecutionRole`,
  assumedBy: new iam.ServicePrincipal('ecs-tasks.amazonaws.com'),
  managedPolicies: [{ managedPolicyArn: 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy' }]
});
NagSuppressions.addResourceSuppressions(executionRole,
  [
    { id: 'AwsSolutions-IAM4', reason: 'Needed for every task execution role.' },
    { id: 'AwsSolutions-IAM5', reason: 'Managed policy uses * resource identifier.', appliesTo: ['Resource::*'] }
  ],
  true
);

cdk synth --quiet

[Error at /FrontendStack/FrontendService/TaskExecutionRole/DefaultPolicy/Resource] AwsSolutions-IAM5[Resource::*]: The IAM entity contains wildcard permissions and does not have a cdk-nag rule suppression with evidence for those permission. Metadata explaining the evidence (e.g. via supporting links) for wildcard permissions allows for transparency to operators. This is a granular rule that returns individual findings that can be suppressed with 'appliesTo'. The findings are in the format 'Action::<action>' for policy actions and 'Resource::<resource>' for resources. Example: appliesTo: ['Action::s3:*'].

What did you expect to happen?
The error should be supressed.
What actually happened?
The error is still showing up, even with the applyToChildren option. The only way I could find to suppress it is to provide the full path with NagSuppressions.addResourceSuppressionsByPath.
cdk-nag version
2.27.167
Language
Typescript
Other information
No response

I'm unable to reproduce the error with the given snippet as the default policy has not been generated at this point (see below CloudFormation)
 {
 "Resources": {
  "TaskExecutionRole250D2532": {
   "Type": "AWS::IAM::Role",
   "Properties": {
    "AssumeRolePolicyDocument": {
     "Statement": [
      {
       "Action": "sts:AssumeRole",
       "Effect": "Allow",
       "Principal": {
        "Service": "ecs-tasks.amazonaws.com"
       }
      }
     ],
     "Version": "2012-10-17"
    },
    "ManagedPolicyArns": [
     "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
    ],
    "RoleName": "FrontendExecutionRole"
   },
   "Metadata": {
    "aws:cdk:path": "test-iam-stack/TaskExecutionRole/Resource",
    "cdk_nag": {
     "rules_to_suppress": [
      {
       "reason": "Managed policy uses * resource identifier.",
       "id": "AwsSolutions-IAM5",
       "applies_to": [
        "Resource::*"
       ]
      }
     ]
    }
   }
  },

You will need  to move the suppression to after the Policy has been created, which in this case is probably after the ECS Task using this role has been initialized

I see. I moved the supression down and that solved my issue. Thank you very much!


crawl service crawl duplicate

Its not crawl duplicated movie,but didn't crawl because of this line=))

Recheck logic to get number of movie to crawl


Support Android Studio 3.3 Stable

Update flutter tooling and android templates to Gradle 4.9.

https://androidstudio.googleblog.com/2018/07/android-studio-33-canary-3-available.html
Gradle compatibility: Android Gradle Plugin 3.3.0-alpha3 and higher now requires Gradle 4.9 RC1 or higher.

https://androidstudio.googleblog.com/2018/09/android-studio-33-canary-10-available.html
The Android Gradle plugin now requires Gradle 4.10 or higher.

gentle bump since it's been a few months since last update, as we have 3.3 RC1 now and 3.4 is already at canary 6.

Flutter uses Gradle 4.10.2, so this part is fixed.
Please create a new issue for Android Studio in https://github.com/flutter/flutter-intellij (if none exists yet)


[AREA]: Create and process promissory notes

Contact details
<EMAIL_ADDRESS>Organization type
Microsoft employee
End-to-end business process
Procure to pay
Specify the business process area name for the article.
Create and process promissory notes
Enter any additional comments or information you want us to know about this business process area.
Draft in progress
Specify the date you expect the article to be completed and ready for review.
7/31/2023
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Merged into Issue and settle vendor payments.


Trouble installing and running - Please help

I have installed on local machine via Docker image and the "getting started" steps from openpim.org.
I installed Postgres DB, ran the Docker command with variables
I can go to my Localhost on browser, and get the login screen.
If I type anything or click anywhere I get an error about "Invalid JSON" expected '<'.
In the chrome inspector>networking there is a GraphQL error.
I don't care where I have to install this, but I need help getting it working. Where ever is the best place to deploy, Google cloud, digitalocean, heroku, where ever. please help, I can pay someone but don't have much.

I have emailed<EMAIL_ADDRESS>and haven't received any response.

Hi @Zakb8,
We received your email yesterday but we are not able to answer asap, usually we are answering in 1 day.
Regarding your question, seems you have wrong configuration so the system can not connect to database.
We have 2 parts inside our docker image. First is the web server (Nginx) that is hosted web page that you see (for example login page in this case) and second part is an application server that is connected to the database that you specified.
Here we have a situation where the web server is started and you can see the page but the application server is not started. Usually such an error is because the server can not connect to the database.
First that you can make is to take a look at server logs. You should make the following to see them:

Execute command "sudo docker container ls" and you will see all started docker processes (if you have only our system inside docker you will see only one process). And you should copy
Execute command "sudo docker exec -it  /bin/bash".  I think you will see an error that the system can not connect to the database in the error log.

Let me describe why you can have this. Problem is that Postgres must be configured right.

First problem is that usually Postgres is bound to the "localhost" interface only. To change this you must go to postgresql.conf and set listen_addresses = '*'. And restart postgres.
Second problem is that system is running inside docker VM (so it has it own IP address),
So you should allow connections from docker to postrges. To make this you should change postgres pg_hdba.conf and add the following line and also restart postgres:

host    all             all             /32           md5
or make it as
host    all             all             <IP_ADDRESS>/0           md5
to allow connection from all IPs
Also please make sure that firewall is also configured to allow the connection between docker and the database.
After this configuration you can restart our docker image and application server should be started. You can monitor this in the application server log file.
Please let me know if this helps.
Thanks,
Slava Pedak

Thank you so much for the answer !! :)
My docker install keeps failing, I will keep trying.
Would it be possible to deploy this to something else online? Heroku, DigitalOcean, or google cloud? I would much rather have it that way if possible?

It is possible to deploy the system to any cloud where docker is working, but the idea of deploy will be the same.
What errors do you have in the log? Also what command are you using to start the system? Can you please provide the details and I can try to help you.
Thanks,
Slava Pedak

Thank you so much for the answer !! :) My docker install keeps failing, I will keep trying.
Would it be possible to deploy this to something else online? Heroku, DigitalOcean, or google cloud? I would much rather have it that way if possible?

#cloud-config
users:
  - name: pim
    ssh-authorized-keys:
      - ssh-rsa AAA....
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    groups: sudo
    shell: /bin/bash
ssh_pwauth: false
disable_root: true

packages:
  - ca-certificates
  - curl
  - gnupg
  - lsb-release
  - postgresql
  - postgresql-contrib
  - ufw

package_update: true
# package_upgrade: true

write_files:
  - content: |
      listen_addresses = '*'
    path:
      /etc/postgresql/12/main/conf.d/01pim.conf

runcmd:
  # install docker
  - mkdir -p /etc/apt/keyrings
  - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
  - echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  - apt-get update
  - apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin
  # configure postgres
  - echo "host all all <IP_ADDRESS>/0 md5" >> /etc/postgresql/12/main/pg_hba.conf
  - sudo -u postgres psql -U postgres --command="ALTER USER postgres WITH PASSWORD '123';"
  - systemctl restart postgresql.service
  # configure firewall
  - ufw allow 22
  - ufw allow 80
  - ufw enable
  # install openpim
  - curl -O https://openpim.org/sql/init.sql
  - sudo -u postgres psql -U postgres -d postgres < init.sql
  # run docker as network host mode
  - docker run -d --network=host -v /mnt:/filestorage --env OPENPIM_DATABASE_ADDRESS=<IP_ADDRESS> --env OPENPIM_DATABASE_NAME=postgres --env OPENPIM_DATABASE_USER=postgres --env OPENPIM_DATABASE_PASSWORD=123 openpim/production:1.5
hostname: localhost
prefer_fqdn_over_hostname: false


Visual feedback for copying bibliographies

Either or:

System notifications
audio cue (system bell)
flash app

Should refactor all bibliography code to send emit() when clicking from each *Form.vue. This will make it easier to add new functionality like this

Added the notification code


Remove summarizer, ground truth deployment, and Ray serve

What's changing
We're working towards migrating from Ray Serve to offline inference via vLLM for our ground truth generation. This involves three moving pieces:

Getting rid of summarizer deployment and all related ground truth service assets since we won't be serving the model as a stand-alone web application but running in-process batch async evaluation.  <-- this PR
Moving experiments logically to jobs since we now have granular inference and evaluation jobs instead of full-length evaluation experiments.
Creating a new module, inference, that allows for only the generation of ground truth in the same way as an evaluation job.

I won't be merging this PR until the other ground truth functionality is ready and working, including tests, but wanted to get it approved in small chunks
How to test it
You should no longer see a /groundtruth endpoint and not be able to hit it via cURL

Related Jira Ticket
Additional notes for reviewers
I already...

[X] added some tests for any new functionality - removed tests for deployment components
[X] updated the documentation.

cc @javiermtorres @peteski22 FYI

Thanks Vicki! I ran ack on the codebase and found the following extra occurrences of summarizer:
pants.toml
45:root_patterns = ["python/mzai/backend", "lumigator/python/mzai/sdk", "lumigator/python/mzai/summarizer", "infra"]

README.md
32:  + Online Evaluations with [Ray Serve Deployments](lumigator/python/mzai/summarizer/README.md)

.env
1:PYTHONPATH=./lumigator/infra:./lumigator/python/mzai/backend:./lumigator/python/mzai/sdk:./lumigator/python/mzai/summarizer:$PYTHONPATH

lumigator/infra/mzai/helm/lumigator/README.md
59:| summarizerWorkDir | `""` | S3-compatible storage system folder where the summarizer is located |

lumigator/infra/mzai/helm/lumigator/templates/deployment.yaml
65:              value: {{ required "Missing Summarizer work directory name" .Values.summarizerWorkDir | quote }}

notebooks/lumigator_demo.py
316:def get_summarizer_deployment_id():
317:    """Gets the summarizer deployment id from ray serve."""
318:    path = "applications/summarizer/deployed_app_config/args/description"

notebooks/walkthrough.ipynb
486:    "deployment_id = ld.get_summarizer_deployment_id()\n",

Perhaps a few of these are lower priority (e.g. we can wait to update the notebook when we have a mean to calculate GT), but I hope some of these pointers can be actually useful to you

Thanks @aittalam! I also ran git-grep after this and found even more instances, so probably my lesson is don't use PyCharm find functionality moving forward.  They should all be gone now, except for the notebook and .env, which is a local file we ignore via .gitignore.

Thanks Vicki! I ran ack on the codebase and found the following extra occurrences of summarizer:
pants.toml
45:root_patterns = ["python/mzai/backend", "lumigator/python/mzai/sdk", "lumigator/python/mzai/summarizer", "infra"]

README.md
32:  + Online Evaluations with [Ray Serve Deployments](lumigator/python/mzai/summarizer/README.md)

.env
1:PYTHONPATH=./lumigator/infra:./lumigator/python/mzai/backend:./lumigator/python/mzai/sdk:./lumigator/python/mzai/summarizer:$PYTHONPATH

lumigator/infra/mzai/helm/lumigator/README.md
59:| summarizerWorkDir | `""` | S3-compatible storage system folder where the summarizer is located |

lumigator/infra/mzai/helm/lumigator/templates/deployment.yaml
65:              value: {{ required "Missing Summarizer work directory name" .Values.summarizerWorkDir | quote }}

notebooks/lumigator_demo.py
316:def get_summarizer_deployment_id():
317:    """Gets the summarizer deployment id from ray serve."""
318:    path = "applications/summarizer/deployed_app_config/args/description"

notebooks/walkthrough.ipynb
486:    "deployment_id = ld.get_summarizer_deployment_id()\n",

Perhaps a few of these are lower priority (e.g. we can wait to update the notebook when we have a mean to calculate GT), but I hope some of these pointers can be actually useful to you

Thanks @aittalam! I ran git grep on this as well afterwards and found even more references, so lesson learned here not to use PyCharm for find and replace alone. I removed everything except the mentions in the notebook for now and the .env file since that should be local and in .gitignore.


Tools: Support .NET Core Class Library projects

Add support for running "dotnet ef" on a .NET Standard and desktop .NET class library project.
This is blocked by https://github.com/dotnet/cli/issues/2645 and/or https://github.com/dotnet/cli/issues/4005 (depending on our implementation approach).
Short explanation
If you cannot do dotnet run, you also cannot use dotnet ef on your project.
More explanation
.NET Core, unlike .NET Framework, does not have a globally installed framework. All .NET Core apps need to reference an app framework (e.g. Microsoft.NETCore.App) in order to run. .NET Core class library projects are built without a framework, so dotnet-ef cannot run on these projects.
Workarounds
See https://docs.efproject.net/en/latest/miscellaneous/cli/dotnet.html#targeting-class-library-projects-is-not-supported for workarounds.

Hi,
Just to be sure, we can't use EF within a class library project (aka tests project) in RC2?

@Kurira you can use EF within a class library. You can't use EF .NET Core CLI commands on a class library. See https://docs.efproject.net/en/latest/cli/dotnet.html more info

So I would use tools from my web project and apply them to test context which resides in test project?

I need to test this more, but here is my solution that seems to be working: https://github.com/aspnet/EntityFramework/issues/5460#issuecomment-221167712

Is there a published workaround for this issue? I've added:
"frameworks": {
    "net451": {
        "buildOptions": {
            "emitEntryPoint": true
        }
    },
    "netcoreapp1.0": {
        "imports": [ "portable-net451+win8" ],
        "buildOptions": {
            "emitEntryPoint": true
        },
        "dependencies": {
            "Microsoft.NETCore.App": {
                "type": "platform",
                "version": "1.0.0-*"
            }
        }
    }
},

and a main() in program.cs, but I still get the "..Entity Framework tools does not support targeting class library projects.." message.

@ToddThomson you are probably hitting https://github.com/dotnet/cli/issues/3164. Try setting the top-level emitEntryPoint build option to true.

@natemcmaster Yes. That was it.

When we have this implemented we need to verify that the right-click->Publish migrations feature works when the context/migrations is in a class library - I suspect it is not going to work by default because it will likely need to specify some of the target/startup project parameters.

Additionally, if your DbContext takes a parameter then you also need to add a simple startup.cs to the class library. I added:
public Startup(IHostingEnvironment env)
{
var builder = new ConfigurationBuilder()
.SetBasePath(env.ContentRootPath)
.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
.AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true);
        Configuration = builder.Build();
    }

    public IConfigurationRoot Configuration { get; }

    // This method gets called by the runtime. Use this method to add services to the container.
    public void ConfigureServices( IServiceCollection services )
    {
        // Add Identity framework services.
        services.AddDbContext<SecurityDbContext>( options =>
             options.UseSqlServer( Configuration.GetConnectionString( "DefaultConnection" ) ) );

    }

This was sufficient for me to add an initial migration for Identity services.

@natemcmaster Any change in this regard in latest tooling preview 2?

We are still blocked on .NET Core CLI support for this. https://github.com/dotnet/cli/issues/2645.
Preview 2 added --startup-project which may mitigate the limitation for some project setups (https://github.com/aspnet/EntityFramework/pull/5680).

Hi, where can i use this "--startup-project" to fix that?
I'm trying install EntityFrameworkCore.Tools on class library and get "Package Microsoft.EntityFrameworkCore.Tools 1.0.0-preview2-final is not compatible with netstandard1.6"
10x!

@arielcloud did you upgrade your version of .NET Core CLI to preview 2?

@natemcmaster i dont sure I have CLI at all.
I downloaded .net core by "Visual Studio users: The best way to develop with .NET Core on Windows is using Visual Studio. You will need Visual Studio 2015 Update 3 and then download the .NET Core for Visual Studio official MSI Installer"
https://www.microsoft.com/net/core#windows
where can i see if i have CLI and it's version? 10x!

@arielcloud execute dotnet.exe --version from the command line. Also, Microsoft.EntityFrameworkCore.Tools only supports "netcoreapp1.0". It cannot be installed into a class library. https://docs.efproject.net/en/latest/miscellaneous/cli/dotnet.html#targeting-class-library-projects-is-not-supported

I've had lots of questions about this, so I've updated the original issue description above ⏫  to provide a little more explanation about why this feature is blocked and how you can workaround it.
FWIW this is feature is not simple to fix. Class libraries are not "runnable" by design (i.e. that's what it means to be a library, not an app). dotnet-ef needs to run the code to inspect the model and migrations, which is why it requires the target project be "runnable". See https://github.com/dotnet/cli/issues/2645#issuecomment-214591028

I've written up how I worked around the issue for anyone who is curious. It's based on the link given in the OP, but with a couple of tweaks based on experience from my real life project.
http://benjii.me/2016/05/dotnet-ef-migrations-for-asp-net-core/

@natemcmaster my version is 1.0.0-preview2-003121.
@bjcull please add create date / update date to your great blogs :)
i'll try use those workarounds. 10x!

Works great, except that Resharper mark most of classLibrary-EF code in red... i noted them

Is there a problem with the way I went about doing this? https://github.com/aspnet/EntityFramework/issues/5460#issuecomment-221167712
Seems to work pretty well. Migrations are in my Api project... but everything else is in the class library and class library doesn't have to be turned into an app.

I'm hoping I have this wrong, but with EF6, I could have my DbContext in a separate class library from my ASP.NET MVC project and create migrations in my DbContext class library.  Am I reading correctly that I can no longer do this?  If this is the case, where do I go from here?  Back to .NET 4.6/MVC5/EF6?

Works for me. Lets say I have two assemblies:

MyProject, which contains ASP.NET MVC
MyProject.Entities, which contains DbContext and migrations

When working with dotnet, to add a migration, I just do this from a command line with MyProject as the working directory:
dotnet ef --project ../MyProject.Entities --startup-project . migrations add MyNewMigration
To update the database is similar:
dotnet ef --project ../MyProject.Entities --startup-project . database update

I'm using Visual Studio, so my preference is to use the Package Manager Console which does the same.  So do I have to execute those two commands every time I want to create a migration script against my Data project?

Just choose entities project in the drop-down on the top part of package manager console

I did get it to work, but not excited about the process I had to take.  I had to reference my data project in my web project before a DLL was built so that migrations could execute.

As far as I can tell, I'm able to make this scenario work in the Package Manager Console Tools (i.e. Add-Migration) for VS "15". I'll do some more testing, but I should also be able to use the same approach in the .NET Command Line Tools (i.e. dotnet ef).

I would like to be able to have my repository (Entity Framework) classes in a separate class library. Is this possible right now ? if not, when should I expect this to be added to Entity Framework Core library ?

Yes, you can have your model or DbContext in a class library. This issue is about making the Migrations tooling experience a little better. Right now, you have to use an app (non-library) startup project for them to work at all.
I'm working on changes that will largely remove this restriction, although maybe not always for dotnet database update.

@bricelam Which means we still have to reference EF Core to the web project ?
Is there any open issue related with this problem we can follow ?
Lots of people are waiting to remove EF references from their web projects I guess...

So there's good news and bad news...
The good news is that I got it "working".
The bad news is that, in practice, it doesn't really "work" because the .NETStandard framework isn't intended for execution. It will work if everything down the execution path has a pure .NETStandard implementation that doesn't depend on a runtime. So basically the only command that will always work is:

dotnet ef dbcontext list

None of the other commands will work on SQL Server because SqlClient is too framework-and-runtime-specific. You get:

Could not load file or assembly 'System.Data.SqlClient, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

SQLite is a little better because anything that doesn't actually open a connection will work:

dotnet ef migrations list
dotnet ef migrations add
dotnet ef migrations script
dotnet ef migrations remove --force

Anything else will throw:

Unable to load DLL 'sqlite3': The specified module could not be found. (Exception from HRESULT: 0x8007007E)

So, the question is, do we...

Allow execution and warn, "The project 'ClassLibrary1' targets .NETStandard. This framework is not intended for execution and may fail to find runtime dependencies. If so, specify a --startup-project that references this project and targets a different framework (e.g. .NETCoreApp)."
Prevent execution entirely and force you to use --startup-project.

If we do allow execution, we should automatically add -Force to Remove-Migration like we do on UWP.

We'll go with the warning approach

As a developer trying to use EFcore and having to deal with this issue because we are leveraging a common data model across multiple web portals and even a Xamarin project for Android - my team would prefer to have the option and the warning message.
It feels a lot more natural to do data modeling and schema work against the Data project, not the application.

@bricelam so you are saying there will be no issues with a .NET Core class library targeting only full .NET Framework instead of .NET Standard?

Correct. Class libraries targeting .NET Framework or .NET Core App should work just fine without specifying a separate startup project.

To ellaborate, there are two projects when you invoke a command:
The target project that you are operating on. Files will be added to this project.
The startup project. This is the runtime environment we'll emulate at design time. This project's config will be loaded; it's Startup.ConfigureServices will be used if it's an ASP.NET Core project; etc.
In PMC, the target project defaults to toe selected Default project and can be overridden using -Project. The startup project defaults the the selected startup project (the one that runs when you click Start with debugging) and can be overridden using -StartupProject.
On CLI, the target project defaults to the one in the working directory and can be overridden using --project. The startup project defaults to the target project and can be overridden using --startup-project.

For the first time I kind of understand this, how about adding this to the docs?

@ErikEJ good idea. We don't really have any detailed docs on the commands at the moment, just a reference on what parameters are available etc.

Bad news
The solution I had stopped working. You will have to use a separate startup project when working directly with .NET Standard projects. You will receive the following error.

Startup project 'ClassLibrary1' targets framework '.NETStandard'. This  framework is not intended for execution. Specify a project that targets a concrete framework like '.NETCoreApp' or '.NETFramework' using the --startup-project option and try again.

Good news
They're adding a .NET Core (in addition to the .NET Standard) Class Library in Visual Studio 2017. The Entity Framework Core Tools will work as expected on these projects.

Scratch that. It was an unrelated issue. .NET Standard class libraries will "work" (as much as they can)

Can we add a way to persist the startup project in our class library so we don't need to pass the argument on every call and also we can share which startup project should be used in source control ? (Thinking in multiple project sharing (or not) the same dbcontext.
maybe dotnet ef --set-startup-project ..\MyApp, a key on csproj like user secrets or just a StartupProject.txt file :)

discard my last comment. I found a better way that I think it should be the official way because it's easy and afaik there's no downside.
To make your class library be able to run dotnet ef it's as simple as targeting both frameworks.
Example:
<TargetFrameworks>netcoreapp1.1;netstandard1.4</TargetFrameworks>

6 hours.... worked....

edit *.csproject add ....EntityFrameworkCore.Tools&...EntityFrameworkCore.Tools.Design With "" tag. (PackageReference and DotNetCliToolReference)
in PMC ,use "cd" command rederict to project floder.
use "dotnet ef" command ,you'll see unicorn.
中文：
1）修改项目文件，添加项目Nuget包引用以及Cli引用：EFCore.Tools&EFCore.Tools.Design.
2）在PMC中（程序包控制台）通过cd命令将目录转移到对应的项目目录下（解决方案默认打开的都是比项目更高一级的解决方案目录，此时应该通过CD转换到项目目录下。
3）使用 dotnet ef命令，就可以看到独角兽了。


Added in version specification for app download tracking

Sorry if there was a better way to update the pull request. Changes have been made as per your requests in #95. The version is now passed in via the constructor, not a new method as before, and a unit test has been made.

Sorry if there was a better way to update the pull request.
Normally you would squash or revert your commits, then force push your changes towards your pullrequest branch. Google has more info on that.

I think setting the version via constructor is slightly restricting here. What about a method  setVersion(@Nullable String version) and then in the respective places of DownloadTracker make use of a getVersion() that pulls the version from the mVersion field if it is non null, and otherwise pulls it from the apps context. This would nice integration within the TrackHelper.
Make an extra test method for the above case.
I think then we can merge it, I might a small code polish PR, and then the current dev branch including your changes can be merged with master and we'll do a new release.

Totally right, the try/catches were a bit unnecessary other than to get the package (which I don't think is necessary anymore, but followed the pattern I saw). I've refactored it to only have the try/catch in the constructor - as long as the version getter is using the same pkgInfo object, the try/catch isn't needed. Also removed that debug line -- wasn't intending to commit that and am glad it was caught.

:+1: Looks good to me


HUGE memory leak problem

Environment details

OS: Mac OS 10.15 Beta (19A573a)
Node.js version: v12.10.0
npm version: 6.10.3
@google-cloud/firestore version: 2.3.0

I'm still experiencing a memory leak potentially related to #661. The following is from my yarn.lock:
"<EMAIL_ADDRESS>  version "0.5.2"
  resolved<EMAIL_ADDRESS>  integrity sha512-NE1tP/1AF6BqhLdILElnF7aOBfoky+4ZOdZU/0NmKo2d+9F9QD8zGoElpBk/5BfyQZ3u1Zs+wFbDOFpVUzDx1w==
  dependencies:
    semver "^6.0.0"

"<EMAIL_ADDRESS>"<EMAIL_ADDRESS>  version "0.5.1"
  resolved<EMAIL_ADDRESS>  integrity sha512-3y0FhacYAwWvyXshH18eDkUI40wT/uGio7MAegzY8lO5+wVsc19+1A7T0pPptae4kl7bdITL+0cHpnAPmryBjQ==
  dependencies:
    lodash.camelcase "^4.3.0"
    protobufjs "^6.8.6"

Quick note: because of the grpc problems that keep appearing here, we should probably put grpc version as part of the issue template
So it is clear that I'm actually using the "correct" grpc. My application makes many calls to firestore and I'm experiencing massive memory leaks.

I'm not actually seeing the error message described previously (MaxListenersExceededWarning), however, the memory usage on my application slowly and steady is increasing. I've taken several heap snapshots from local tests over the course of several hours and when comparing snapshots from a few hours apart, I notice that the top culprits for memory allocations are all related to grpc.
Here you can clearly see that the top 4 culprits for the memory delta between snapshots 12 and 4 are vastly more than everything else

Here are the contents of each from largest to smallest:

(string): This by far is the fastest growing constructor. As you can see, there are 920k new allocations in the comparison between the two snapshots. Practically every single string here is ""(function anonymous( ) { return function BeginTransactionRequest(p){ if(p)for(var ks=Object.keys(p),i=0;i<ks.length;++i)if(p[ks[i]]!=null) this[ks[i]]=p[ks[i]] } })". And this all comes from message.js:13. This is a a file from the package protobufjs which is a dependency of grpc.

(system): Essentially everything here seems to be coming from from the same message.js:13.

I also have tens of thousands of objects that look like this (from grpc)

(array): Practically every single array allocation here is a alloc size 32 and is linked to type.js:31 and namespace.js:95, both of which are also part of protobufjs.

"Field" is also related to the same line numbers and is actually directly linked to grpc

Array: There are 235k of these and they all look the same - Array with an alloc size of 32 with some connection to grpc

Update: I tried updating to 0.5.4 based on the comments in this thread and it did not help
Here is the new yarn.lock:
"<EMAIL_ADDRESS>  version "0.5.4"
  resolved<EMAIL_ADDRESS>  integrity sha512-aY4fTCz7jq7oKFmfAeZVqGzMCR5I9NLdY9E2fJ70QtGXwlJnTaN6cnbRmCk23/aKPx9UHqOtk2lyjpN6LbAaxw==
  dependencies:
    semver "^6.2.0"

"<EMAIL_ADDRESS>"<EMAIL_ADDRESS>  version "0.5.2"
  resolved<EMAIL_ADDRESS>  integrity sha512-eBKD/FPxQoY1x6QONW2nBd54QUEyzcFP9FenujmoeDPy1rutVSHki1s/wR68F6O1QfCNDx+ayBH1O2CVNMzyyw==
  dependencies:
    lodash.camelcase "^4.3.0"
    protobufjs "^6.8.6"

If it is of any help, the way I use firestore is as follows:
I have a file called firestore.js that looks like this:
import Firestore from '@google-cloud/firestore';

export default new Firestore();

I then import firestore and use it in other places like this:
import firestore from './modules/firestore';

const updateThing = async (req, res) => {
  try {
    const { docId, updatePayload } = req.body;

    await firestore.doc(`Things/${docId}`).update(updatePayload);

    return res.status(200).send();
  } catch (err) {
    console.error(err);
    return res.status(500).send();
  }
};

I can probably have a temporary workaround by just creating a new Firestore instance from within each function body, but fixing the underlying problem is a more sustainable approach.
Also, my application doesn't have any event listeners. The only operations it uses are get(), set(), and update()

@jakeleventhal Thanks for the very detailed issue! Can you describe your work load a bit? It would help to know how many requests you are sending and if the memory usage tapers off once those requests are processed.

I have a couple of observations here:
The string that is being constructed almost a million times looks like a generated variant of the protobuf.js Message constructor that is specific to various message types from the .proto files in the Firestore library. The most likely reason for that happening is that the .proto files are getting loaded repeatedly. Other memory usage from protobuf.js probably has the same cause. The library @grpc/proto-loader is essentially just a proxy for protobuf.js, so memory leaks there probably also have the same cause.
The Http2Subchannel objects from grpc are probably the result of repeatedly constructing new clients, and that version of the grpc library tends to keep channels and subchannels alive even when they are not being used. This is improved in @grpc/grpc-js version 0.6.x, mainly because subchannels with identical parameters will be reused.

@schmidt-sebastian So I have an API server that accepts HTTP requests from a another scheduling server. The scheduling server repeatedly sends jobs to the API to save user data.
There are many operations that occur but, for the most part, they can all be summarized as this:

Fetch a large data report from a 3rd party API
Parse that data to do one of or both of the following:

a) Create a document with set()
b) Modify an existing document with update()

There can be anywhere from 1-1000 or so firestore calls per minute per user. Each document that is updated is unique so there is no contention going on here.
I also have used log statements to see that functions are entering and exiting (and they all more or less follow a similar structure to what I wrote above). The scope of the firestore calls should be limited to just the function. Moreover, the firestore calls aren't throwing errors or anything and are updating documents successfully. Again, there are no event listeners anywhere in my code.
I have also tried not using a single instance of Firestore that I import throughout my code - I tried creating a new instance of Firestore within the body of each function that uses firestore. I tracked the number of instances of Firestore objects through memory snapshots and was able to see that the Firestore objects were successfully being deleted/deallocated at the end of each function; however, the metadata with all those strings/arrays/objects still persisted outside the scope of the functions.
I hope that clears up everything.

@jakeleventhal we've hit with this kinds of problems too in cloud functions: https://github.com/firebase/firebase-functions/issues/536
The problems originate from grpc-js, which is still experimental.
What worked for us in the end was to downgrade our dependencies so it still uses the binary grpc which is much more mature and stable.
There's also a mechanism to force firestore to use binary grpc with the newer client version described here: https://github.com/firebase/firebase-functions/issues/536#issuecomment-531923813, however I have not tested that one as it is unsupported / untested if I understood correctly. That note also explains a bit why we are in this mess.
So, after grpc-js matures a bit, we’ll hop back to the latest versions.

@swftvsn what functionality will be lost/what differences will there be?

@swftvsn @schmidt-sebastian why is firestore not using the binary method of grpc until the other version is stable? what’s to be gained?

@jakeleventhal The reasoning to use grpc-js is explained in this comment: firebase/firebase-functions#536
We didn't lose any functionality we need when switching to earlier version, but of course we just checked everything we need. (We use typescript, so basically everything compiles -> you're good to go. ymmw etc. ofcourse.)

Providing a different GRPC implementation via the constructor argument (as described in https://github.com/firebase/firebase-functions/issues/536#issuecomment-531923813) is fully supported. All currently available features in Firestore are compatible with both grpc and @grpc/grpc-js.
@murgatroid99 Do you have a rough idea of where in our stack this problem might best be addressed?

I'm only really familiar with my end of the stack, so I can only say that it's somewhere above that. Probably this firestore library. Loading the same .proto file repeatedly is not an intended usage of @grpc/proto-loader, so I recommend that you find a way to load it only once per process.
Similarly, constructing many clients for the same service and backend is not the intended usage of either grpc library, but that should be a much smaller problem with the new release of grpc-js that gax should be picking up soon.

I spent some more time on this, but I am still not able to pinpoint the leak. What is interesting though is that this only seems to happen with set()/update()/delete() and not with a get() call.
After 2000 operations, my memory Profile looks like this:

The memory usage does not go down even when all operations resolve.

Is this confirmed to be fixed using the regular grpc library? i.e.
import grpc from 'grpc'

const firestore = new Firestore({ grpc })

@schmidt-sebastian I don't know if it's related, but there is a similar issue in BigQuery: https://github.com/googleapis/nodejs-bigquery/issues/547

@jakeleventhal I see similar memory profiles with both grpc and @grpc/grpc-js.

It looks like the memory leak happens in one of our depend layers, so it might take a bit of work to track this down. The memory leak also exist when the Veneer client (which is an auto-generated SDK that we use for our request logic) is used directly. What is somewhat interesting is that we only seem to be leaking memory once we get responses from the backend - I see a steady increase of memory consumption only after requests are starting to get fulfilled.
With 2000 writes, FirestoreClient_innerApiCalls has a retained size of 99680 bytes that is mostly unaccounted for. There are also 2000 PromiseWrap and Promise objects on the stack that take up roughly this amount of memory.

Great to see the investigation resulting some fruit @schmidt-sebastian - and sorry that I jumped to conclusion and thus wrote bs above..

@schmidt-sebastian There is definitely a huge improvement using the legacy grpc (new Firestore({ grpc })).
These are tests run (with the exact same traffic/load
grpc-js

grpc

You'll notice the memory is still climbing in the second example - but its very very minimal compared to the grpc-js version

How are you measuring the memory usage? If you are using a JavaScript memory profiler, it may be omitting memory allocations by native code, which account for the majority of the memory usage of grpc.

The screenshots in the original post that show the memory profiler are using the chrome memory tool when running with the —inspect flag.
The screenshots provided of the memory graph are the memory usage of my application on GKE

Is this resolved on 2.4.0?

@jakeleventhal 2.4.0 seemed to have resolved the issue on my side.

@MysteryG were you experiencing the same thing? and are/were you using the legacy grpc as a workaround

@MysteryG were you experiencing the same thing? and are/were you using the legacy grpc as a workaround

@jakeleventhal I ended up reverting to version 2.2.9 as a temporary measure to avoid implementing a temporary work around.

@jakeleventhal Ignore last, we are still experiencing memory issues, and now loss in connection to Firestore.

I'm certain there's a memory leak in @grpc/grpc-js v0.6.x, which does not exist in v0.5.4. I have a reproduction for it, but it's not yet minimal (very small tho). I'd love to give access to this repo to some Google engineers, please let me know if you'd be interested. I can't share it publicly.

@MysteryG If you upgrade to @grpc/grpc-js 0.6.5 you should once again be able to connect to Firestore. See https://github.com/grpc/grpc-node/issues/1064
@merlinnot I would love to see your repro! I'll send you a message to the email listed on your profile.

Being that this is a paid service, why can’t @google-cloud/firestore be released with the legacy grpc until this is resolved? I’m sure many people are experiencing this issue.

@merlinnot I am the maintainer of grpc-js. If you can share that repository with me that would be very helpful; I haven't seen any other information or reports of memory leaks in that version.

@schmidt-sebastian @murgatroid99
I added you both as collaborators to the repository, hope GitHub sent you a link.
Reproduction steps:
git checkout repro
npm ci // Installs<EMAIL_ADDRESS>npm run test // Memory usage is logged, see it growing.
npm i<EMAIL_ADDRESS>npm run test // Memory usage is logged, it's perfectly stable.

@merlinnot I see the memory increasing in both cases, just by significantly smaller amounts with version 0.5. I believe you have mainly demonstrated that the Client class is more heavy-weight in version 0.6. In addition, some objects owned by the client may keep themselves alive for some amount of time, but we do not consider that to be a significant issue because users are expected to create and use a constant number of clients over the lifetime of a process, not create a new client for every request.

There is clearly a problem with 2.4.0
This is a graph of memory usage on my application over the last several days.
For the last week I had been using @google-cloud/firestore 2.3.0 with the legacy grpc in the constructor. I tried switching to 2.4.0 without the legacy grpc constructor and my memory usage skyrocketed.

I also was experiencing this error when fetching data on 2.4.0 with the default constructor
(node:43) UnhandledPromiseRejectionWarning: Error: connect ECONNREFUSED <IP_ADDRESS>:8080
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1126:14)

After deploying back to 2.3.0 with the default constructor my memory usage is way down again and i am no longer getting that error

Similarly I was running into connection errors after +- an hour of the service running on 2.4.0
Error: No connection established
    at Http2CallStream.call.on (/pipeline/source/node_modules/@grpc/grpc-js/build/src/call.js:68:41)
    at Http2CallStream.emit (events.js:203:15)
    at process.nextTick (/pipeline/source/node_modules/@grpc/grpc-js/build/src/call-stream.js:75:22)
    at process._tickCallback (internal/process/next_tick.js:61:11)

Here you can see the results of before/after moving back to 2.3.0

I have conducted my own test for possible memory leaks in @grpc/grpc-js version 0.6.6. For the test, I have a server running in another process. Then in the client, I call a single method, and when that method completes, I call it again in a loop.  I got these results with the following variations:

Construct a single client and use it to make all of the calls. In this case, I observe no evidence of any memory leaks. This is how we recommend using gRPC.

Construct one client per call. In this case I see a consistent leak of about 4KB per call+client, all various objects owned by Channels.

Construct one client per call, and close the client after the call completes. In this case I see a leak of approximately a few hundred bytes per call. So far I can't tell what is using that memory.

Load the proto file and construct a client per call. In this case I see a leak of about 50KB per call+client, similar to number 2, plus generated protobuf code. The value here likely depends strongly on the contents of the proto file(s).

Load the proto file and construct a client per call, and close the client after the call completes. In this case I see a leak of about 45KB per call+client. I think this is just the same as number 4, with the same savings as in number 3.

If I run the same tests with @grpc/grpc-js version 0.5.4 I see the following:

This case still leaks no memory.

In this case I see a leak of about 6KB per call+client.

In this case I see no memory leak, a difference from the 0.6.x

In this case I see a leak of about 53KB per call+client. This is consistent with the same test case for 0.6.x, plus the slightly higher leak in number 2.

In this case I see a leak of about 48KB per call+client. This is again consistent with the previous observations.

I am not really sure what is causing Firestore to use more memory when using grpc-js version 0.6, but I think it can be fixed by constructing a single client and using it to make all of the requests.

Thanks for the update!
Firestore is using multiple clients only if there are more than 100 concurrent requests. Since we have long-standing streams, we cannot just "wait" until the request count drops to zero before we can start another stream.
We have logic that stops referencing clients that are no longer needed, but we don't do anything special to clean up their state (https://github.com/googleapis/nodejs-firestore/blob/a4efa097ef8de9ca4944356ab8767ddaa94c4188/dev/src/pool.ts#L174). From your investigating, it looks like we should at least be closing the GRPC channel manually. I will work on that for the time being.

@murgatroid99 i use one Firestore client

@jakeleventhal The Firestore client internally spins up multiple GRPC clients if needed.

Hi guys 👋 I believe I discovered a memory and network socket leak that might possibly be a root cause of this issue, or at least one of the causes, in a @grpc/grpc-js dependency.
I opened an issue with a reproduction, but I'd appreciate if you could validate that my reproduction yields the same results on your systems and that the reproduction steps are clear and easy to follow, so that gRPC team will be able to fix it more easily. Thanks in advance!
https://github.com/grpc/grpc-node/issues/1085

@merlinnot Thank you! In the meantime, I am also trying to see what it takes to make sure that our clients release all resources when they get garbage collected. This touches a lot of dependencies, so please bear with us.

Is there anything I can do to help? Today another update of @grpc/grpc-js was released, for which I can't easily find any significant memory leaks anymore. I could start making PRs against other repositories, but I'd need some guidelines on what do you mean by "a lot of dependencies" @schmidt-sebastian.

Any update on when this will be fixed? Im pinned at 2.3.0 and can't update to get the new features like the IN operator until this is fixed since it will cost too much $ to upgrade

@jakeleventhal  May I know which tool/plugin you are using to profile the way you did?

@voxelbusters I was using the node --inspect flag with the chrome memory profiling tool

@merlinnot that is good to know that you're no longer seeing memory leaks with the latest @grpc/grpc-js. We are gradually rolling this dependency out to some of the libraries I work on, I will keep you in the loop this week.
I know @alexander-fenster had released a next release of gax that pulls in some of the fixes.

@merlinnot "A lot of dependencies" refers to the updates we have to plumb through for https://github.com/googleapis/gapic-generator-typescript/issues/68

Thanks, I'm already following the other thread. I'm eager to help if you need anything (PRs, tests, ...) 👍

I'm experiencing this as well. Please fix

Is there an update on this? I'm literally paying for 5x more resources than I need right now because of this issue. It's starting to cost me a lot of money.

We have fixed numerous small leaks in @grpc/grpc-js (please upgrade to the latest version of google-gax or re-install Firestore). Next week, we are also planning to migrate the internal Firestore client to Typescript, which will allow us to close the GRPC channels on cleanup. We will release that the moment it is ready.

@schmidt-sebastian Any update on "next week" or did you have one of these weeks: https://www.mentalfloss.com/article/51370/why-our-calendars-skipped-11-days-1752

But in all seriousness, I'm spending a doggone fortune because of this leak..

I am very sorry, we are still working on this. We are making progress every day and I think I can see the finish line.

Is there any update on this? Bleeding cash for me.

Yeah, this has been nearly 3 months... come on...

We are basically done: https://github.com/googleapis/nodejs-firestore/pull/779 is waiting for a final review.
We also have some timeout settings that we need to fix in https://github.com/googleapis/nodejs-firestore/pull/805, but then we will release new client.

Version 3.0.0 should be released any minute now. With this version, we close the GRPC client when a channel becomes idle. If you request load frequently causes new channels to be spawn and released, this should reduce the memory burden described in this issue.
I am sorry and somewhat embarrassed by how long this took. I hope this solves the root issue described in here.

Any idea how soon firebase-admin-node will be upgraded from 2.6v to start using 3.0v of nodejs-firestore? @schmidt-sebastian  I'm really glad a solution is in the pipe here, this has caused us a lot of grief, thank you for your efforts!

Experiencing over 15x reduction in memory footprint with 3.0.0 👍🏼

@bfitec Unfortunately, we may have another breaking change coming up shortly as Node 8 goes end of life end of the year. We still have to figure out what this means for us and if we have to bump the major version for Firestore again, but we don't want to bump the major version for Firebase Admin twice in short succession and hence we haven't updated Firebase Admin.
Right now, we advise you to manually override the Firestore version for Firebase Admin (via npm shrinkwrap for example). Let us know if this is a huge hassle and we can reconsider.

@schmidt-sebastian I changed package.lock to edit the version of @google-cloud/firestore to 3.0.0 on a project using firebase admin. However I continue to observe huge amounts of memory (+3Gb)
In my npm ls output only  v3.0.0 is shown as used. I also get this message npm ERR! invalid<EMAIL_ADDRESS>/Users/joaomacedo/Projects/firestore-cgd-gateway/node_modules/firebase-admin/node_modules/@google-cloud/firestore but I guess that is to be inspected since I altered the dependency forcefully to 3.0.0.

In our service we are listening for some dozens of thousands of streams. We noticed the more streams we need to be listening to the more memory our process needs. It is currently requiring +4Gb and increasing as new clients join. This should not happen as 98% of the streams have no messages being exchanged other that the first couple.  As I said above, we attempted forcing the 3.0.0 firestore cloud version by editing the package.lock.

@schmidt-sebastian You advised shrinkwrap. Is using npm shrinkwrap for this case functionally different from changing package.lock?

@jmacedoit I added a branch to firebase-admin-node that should eliviate some of the installation pain. You can install as follows:
 "dependencies": {
    "firebase-admin": "git+https://github.com/firebase/firebase-admin-node.git#firestore-30"
  }

The bad news is that this might not address your specific memory concerns. GRPC only supports 100 concurrent operations at a time and as such, we need to create new GRPC clients for every 100 operations. The changes in this PR merely address the memory usage of GRPC clients that are no longer needed, but if you have a steady count of thousands of listeners, then some high memory usage is expected (at least for now).

@schmidt-sebastian Any tips on how can this be addressed? The project in question is a chatboat gateway. It needs to listen to the streams of thousands of users (1 stream per user), in order to send the message to a core so that it is processed and the answer is placed back in the client stream. Naturally, most of the time the streams will have no activity. Is there anything that can be done to mitigate this?

Two options come to mind:

If latency allows, use Cloud Functions to trigger an operation whenever a new message is written.
Use the same collection ID as a parent to all messages and use a single listener via Collection Groups.

If you have engineering resources to spare, we could also add an option to issue multiple listeners over the same stream. This is supported by the backend and currently used by the Android, iOS and Web SDK. We deliberately decided to use one stream per listeners as this provides isolation for all operations in the Server SDKs, but we could optionally bundle listeners together in the Node Server SDK. Unfortunately, this is a significant effort and will likely not be staffed by us in the near term.
As for your last question - yes, a new operation is spun once onSnapshot is called.

@schmidt-sebastian Collection groups would be great, unfortunately the client applications are already deployed (and the collection names to which they write have names which differ for each user) and we need a short term solution to the problem. I guess we will have to go with the cloud functions route, although to be honest I prefer to avoiding cloud functions whenever possible...

Looks like this issue generated yet another issue 🤦🏻♂
https://github.com/googleapis/nodejs-firestore/issues/829

The problem is still not solved. I am using firebase functions 3.6.1.
My package.json file is:
{   "name": "functions",   "description": "Cloud Functions for Firebase",   "scripts": {     "lint": "eslint .",     "serve": "firebase serve --only v1",     "shell": "firebase functions:shell",     "start": "npm run shell",     "deploy": "firebase deploy --only v1",     "logs": "firebase functions:log"   },   "dependencies": {     "body-parser": "^1.19.0",     "camelcase": "^5.3.1",     "es6-promise-pool": "^2.5.0",     "express": "^4.17.1",     "firebase-admin": "~8.0.0",     "firebase-functions": "^3.6.1",     "firebase-functions-helper": "^0.7.5",     "glob": "^7.1.6",     "react-native-uuid": "^1.4.9"   },   "engines": {     "node": "10"   },   "devDependencies": {     "eslint": "^5.12.0",     "eslint-plugin-promise": "^4.0.1"   },   "private": true }
My function file is:
`
'use strict';
const functions = require('firebase-functions');
const admin = require('firebase-admin');
const firebaseHelper = require("firebase-functions-helper");
const db = admin.firestore();
const promisePool = require('es6-promise-pool');
const PromisePool = promisePool.PromisePool;
// Maximum concurrent account deletions.
const MAX_CONCURRENT = 2;
var FieldValue = require('firebase-admin').firestore.FieldValue;
/**

Run once a day at midnight, to cleanup the users
Manually run the task here https://console.cloud.google.com/cloudscheduler
*/
exports.autoAcceptFriendship = functions.runWith({memory: '2GB', timeoutSeconds: '360'}).pubsub.schedule('every 30 minutes').onRun(async context => {
// Fetch all user details.
const pendingFriendships = await pendingFriendship();
// Use a pool so that we delete maximum MAX_CONCURRENT users in parallel.
const promisePool = new PromisePool(() => approveFriendship(pendingFriendships), MAX_CONCURRENT);
await promisePool.start();
console.log('All bot friendship approved');
});

/**

Deletes one inactive user from the list.
*/
async function approveFriendship(pendingFriendships) {
if (pendingFriendships.length > 0) {
const friendshipToApprove = pendingFriendships.pop();
console.log('friendship accepted started for ', JSON.stringify(friendshipToApprove));
//accept Friendship
try {
const friendship = {
user1: friendshipToApprove.user1,
user2: friendshipToApprove.user2,
jointIds: [friendshipToApprove.user1, friendshipToApprove.user2].sort(),
created_at: FieldValue.serverTimestamp(),
};
const newDoc = await firebaseHelper.firestore
.createNewDocument(db, 'friendships', friendship);
const deletedContact = await firebaseHelper.firestore
.deleteDocument(db, 'pending_friendships', friendshipToApprove.id);
console.log('friendship accepted');
}
catch (error) {
console.log('error occured', error);
}
} else {
return null;
}
}

/**

Returns the list of all inactive users.
*/
async function pendingFriendship() {

let date = new Date();
date.setDate(date.getDate() - 1);
console.log('date is ', date)
//for prod
let userIds = ['ZZ2qctmj2kTzBzFu6s1eywO4Lzd2', 'ZH1VLx9PapNwP70AGhG917CnreD3', 'O9o91D05g5ek7DUYPzCfSnenqMC2',
'89vXTJlfWCd1yd9Bm34Daiiwsdv1', '0S8JmDFG2haPFsyI71hmJcAOSsl1', 'wIOWwAUfaSTPonThkN4rU1zy4uq2', 'pqrVRB1zhhdOoBcYvwGSGTmajCH2',
'hjf7nq6dTmP5eseoIiuh1XV4air1', 'vUtLDsL7JvUMP4mxJi1eqiB2HXc2'];
// //for staging
// let userIds = ['n8jjShDDNfaQNOoiuYOp0rwPOG82', '6Y4MCrQQLYNKhy80VlEjQR2HsH33','IcLqDsK15dMYCHYOCTO5901CM2G2',
//   '0W1OjdtmoXN890PRxxygJ14ZboA3', 'fN7X29HafRYryP61DnGSG9Cp2ZA3', 'hAG3ezNb9XeQ7JslKFDcPjirHzI2',
//   'AwhKncRjjyVLOpSw97mB8nlXUaD3', 'AwhKncRjjyVLOpSw97mB8nlXUaD3', '1HUPxAQDd8cnfb7yB61Sd1INO222', 'pljEAbuLmIUm2bmQF9woqwRhJ1l2'];
let p_friendship_array = [];
let pend_friendship_data = await firebaseHelper.firestore.queryData(db, 'pending_friendships', [['created_at', '>=', date]]);
let data = [];
Object.keys(pend_friendship_data).forEach((key, Index) => {
let val = pend_friendship_data[key];
val = {...val, id: key}
data.push(val)
})
p_friendship_array = data.filter(u => {
return userIds.indexOf(u.user2) !== -1;
});
console.log('no of pend friendship request is ', p_friendship_array.length)
return p_friendship_array;
}
`
I am getting error as:
memory limit exceeded. Function invocation was interrupted

@siddhant-mohan If you turn on logging (setLogFunction) we can check how many GRPC clients you are using. Given your usage of PromisePool, this should just be one client (which can handle 100 concurrent requests). The debug logs would help us verify.

@schmidt-sebastian hey can you help me how to turn on logging to check GRPC clients my function is using? Sorry I am quite new to the firebase functions

It should be:
admin.firestore.setLogFunction(console.log);

This is what i am getting as a setLogFunction output


Convergence towards Bash compatible  expansion of variables on nuttx_next (upstream Nuttx)

While this issues is really with nuttx, I have placed it here because it is relevant to how the applications in PX4 Firmware will be affected when running on master vs. nuttx_next,
There needs to be a convergence towards Bash compatible  expansion of variables on nuttx_next for submission to upstream Nuttx
Currently on master
in init.s/rcS
# Start MAVLink on default port: ttyS1
set MAVLINK_F "-r 1000"
mavlink start $MAVLINK_F
unset MAVLINK_F

When parsed
mavlink start -r 1000
when passed by create task to Mavlink::task() becomes
``mavlink_if0 mavlink start -r 1000```
in bash and on master argc is 5 and argv is  0:[mavlink_if] 1:[mavlink ] 2:[start] 3:[ -r] 4:[1000]
on nuttx_next argc is 4 and argv is  0:[mavlink_if] 1:[mavlink ] 2:[start] 3:[ -r 1000]
Nuttx test case:
#include <stdio.h>
#include <string.h>

__EXPORT int narg_main(int argc, char *argv[]);

int narg_main(int argc, char *argv[])
{
            printf("argc:%d\n", argc);
            for (int i=0; i < argc; i++)
              {
               printf("argv[%d]:%s\n", i, argv[i]);

              }
            return 0;
}

Per Greg ( @nuttx ) changes will need to be controlled by a CONFIG_NSH_XXXXX define and
meet the following criteria.
If we are going to make any NSH change, [he does] not want to break CASE 2a-c just for CASE 1a.  CASE 1b and 3 are single quote behavior which is a different topic.  Single quoting should work the same as double quoting, except that nothing is expanded.

bash test
$ cat junk.sh
#!/bin/sh
echo "[$0] [$1] [$2] [$3]"

Test Cases
CASE 1a:
$ NARG="1 2 3"
$ ./junk.sh $NARG
[./junk.sh] [1] [2] [3]

CASE 1b:
Single quotes make no difference:

$ NARG='1 2 3'
$ ./junk.sh $NARG
[./junk.sh] [1] [2] [3]

But compare this to the case to where a true string argument appears on the command line:

CASE 2a:
$ ./junk.sh "1 2 3"
[./junk.sh] [1 2 3] [] []

CASE 2b:
$ ./junk.sh '1 2 3'
[./junk.sh] [1 2 3] [] []

CASE 2c:
$ ./junk.sh "$NARG"
[./junk.sh] [1 2 3] [] []

Now the single quotes do work differently:

CASE 3:
$ ./junk.sh '$NARG'
[./junk.sh] [$NARG] [] []

More test cases to consider

"This is a string that contains spaces"
"-n 100"

Fixed by @Inspirati  as nsh_parse.patch applied at build time


Remove PhasedUnit and CompilationUnit from CeylonFile

Once CeylonParseController is abstracted, we won't need to store those in CeylonFile anymore.

This has been fixed by the introduction of CeylonLocalAnalyzer.


how to change default folder path and folder name?

Hi Vikrant. You cannot change the default directory. However you can change the foldername in package.json as productName.
Based on the message displayed, it seems that the Folder does not exist. Can I see your package.json file? You should indicate and run
"scripts": {
    "start": "electron ."

If I manually created folder, then it runs

You can manually create a folder. But it should also automatically be created using the script as I indicated.

But the folder is not automatically created, I'm manually creating folder then only it works

Did you run npm start?

yes. but still same issue

Hi there. I have checked but it creates the folder for me once I started
npm start

Even with empty index.js or main.js as long as electron-db is installed.

yes..right..i have created new electron project..n tried same thing..it work..
but its not working with above package.json

This time what script did you tried? Does it still have
electron ./app/

"dev": "cross-env START_HOT=1 node -r babel-register ./internals/scripts/CheckPortInUse.js && cross-env START_HOT=1 npm run start-renderer-dev"
npm run dev
i also tried electron ./app/

but facing same issue

Try just
electron .

i m using electron-react-boilerplate
https://github.com/chentsulin/electron-react-boilerplate

You must create the js file indicated in your "main":
{
  "name": "test-electron",
  "productName": "MyProduct",
  "version": "1.0.0",
  "description": "Sample desc...",
  "main": "index.js",
  "dependencies": {
    "electron-db": "^0.7.7"
  },

in this example, it's index.js.

`const platform = os.platform();
let appName = '';
if (JSON.parse(fs.readFileSync('package.json', 'utf-8')).productName) {
  appName = JSON.parse(fs.readFileSync('package.json', 'utf-8')).productName;
} else {
  appName = JSON.parse(fs.readFileSync('package.json', 'utf-8')).name;
}
let userData = '';
console.log('platform', process.env);
if (platform === 'win32') {
  userData = path.join(process.env.APPDATA, appName);
} else if (platform === 'darwin') {
  userData = path.join(process.env.HOME, 'Library', 'Application Support', appName);
} else {
  userData = path.join('var', 'local', appName);
}
const exists = fs.existsSync(userData);
if (!exists) {
  fs.mkdir(userData, (error) => { console.log('something went wrong', error); });
}`

i use this  script in my component for temporary solution


Feedback for “NumberBox”

I met this error, how to solve? thanks!
AttributeError                            Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_3960\1434161747.py in
1 # displays large number in the box with border
----> 2 mr.NumberBox(data=123, title="Large number")
AttributeError: module 'mercury' has no attribute 'NumberBox'

Hi @gravitywave2015,
Have you solved the issue?


Publish CI

Undeprecate npm package;
Publish to GPR and setup workflow to automatic publish to npm.
Next release 1.4.0 is wanted;

https://github.com/ecomplus/i18n/commit/12c32f25604033b0d65bac6a2fa6d256d5bc101a
https://github.com/ecomplus/i18n/commit/b224cee15aba43fe5abc2338f877f6edcc315dee

https://github.com/ecomplus/i18n/commit/449698c18912553100a48aa4221261e4fa4c82d2
https://github.com/ecomplus/i18n/actions/runs/46885627


Allow kicking a user

Use case: Kevin leaves for another meeting and skews all the remaining votes. :)

This is not actually possible since the client will just automatically reconnect (assuming they leave their browser open). Because there is not auth the only way we identify users is via their SignalR connection. However, anyone with the Facilitator role can demote people out of the Team Members list as into an observer role with this button:


Some further improvements to modernization

Make custom backend registration more generic. Add an example of django_snowflake registration to demonstrate config adjustment for non-trivial backends.
Make README.md clearer.
Address some open PRs from upstream repository and add test for them.
Refactor tests, store test cases in yaml file and migrate to pytest to make tests more maintainable.
Show descriptive error messages when url is not properly quoted or custom backend has not been registered.
Drop support for deprecated options. If this project is going to be published under different name, I believe it's safe to drop support for those options. Also it's a small effort to migrate and new names are more sane.
Remove some code which addressed python2/3 urllib.parse differences.

@dcwatson Could you please take a look at this?

This project only exists under a different name because the original project(s) seem to be unmaintained, not because I wanted to make larger sweeping changes necessarily. For instance, I purposely left the legacy options in place, and kept the API the same, and kept the unit tests as close as possible. I think some of this could still work (allowing register to be a decorator for further customization, for instance) but some of it I'm not really interested in changing. For now, I do want this to be a drop-in replacement, at least for the built-in databases.
However, I also see the value in re-thinking and continuing to improve upon the API. Personally, I think (and I believe Jacob has said as much in various forums) that some version of this belongs in Django proper. So I consider this a stop-gap until that happens.


feat: Support http.route attribute for gin and chi

The name of the HTTP span must be low-cardinality.
To achieve this, the http.route attribute can be included in the span's name.
Semantic conventions for HTTP spans

HTTP span names SHOULD be {method} {target} if there is a (low-cardinality) target available.
The {target} SHOULD be one of the following:

http.route for HTTP Server spans

However, most Go-based web frameworks use different methods for parsing routing paths.
It's not feasible to handle all cases with just an eBPF probe for net/http.
Therefore I submit a patch that adds the eBPF probe support for gin and chi frameworks.

@MrAlias Thank you for your prompt feedback. I'm aware of the #780 in question.
However, most frameworks, including Gin, do not use the ServeMux pattern. So I think there are limitations to using only net/http probes.

@MrAlias The e2e test is written like this line. It passed by this line. I ran the e2e test by this workflow.
Below is the backtrace from Delve attached to our nethttp example.
> net/http.(*conn).serve() /opt/homebrew/Cellar/go/1.23.1/libexec/src/net/http/server.go:2093 (PC: 0x10077c16c)
Warning: debugging optimized function
Values returned:

  2088:			// in parallel even if their responses need to be serialized.
  2089:			// But we're not going to implement HTTP pipelining because it
  2090:			// was never deployed in the wild and the answer is HTTP/2.
  2091:			inFlightResponse = w
  2092:			serverHandler{c.server}.ServeHTTP(w, w.req)
=>2093:			inFlightResponse = nil
  2094:			w.cancelCtx()
  2095:			if c.hijacked() {
  2096:				return
  2097:			}
  2098:			w.finishRequest()
(dlv) p req
("*net/http.Request")(0x140000b4000)
*net/http.Request {
	Method: "GET",
	URL: *net/url.URL {
		Scheme: "",
		Opaque: "",
		User: *net/url.Userinfo nil,
		Host: "",
		Path: "/hello/42",
		RawPath: "",
		OmitHost: false,
		ForceQuery: false,
		RawQuery: "query=true",
		Fragment: "",
		RawFragment: "",},
	Proto: "HTTP/1.1",
	ProtoMajor: 1,
	ProtoMinor: 1,
	Header: net/http.Header [
		"User-Agent": [
			"Go-http-client/1.1",
		],
		"Authorization": [
			"Basic dXNlcjo=",
		],
		"Accept-Encoding": ["gzip"],
	],
	Body: io.ReadCloser(net/http.noBody) {},
	GetBody: nil,
	ContentLength: 0,
	TransferEncoding: []string len: 0, cap: 0, nil,
	Close: false,
	Host: "localhost:8080",
	Form: net/url.Values nil,
	PostForm: net/url.Values nil,
	MultipartForm: *mime/multipart.Form nil,
	Trailer: net/http.Header nil,
	RemoteAddr: "[::1]:59968",
	RequestURI: "/hello/42?query=true",
	TLS: *crypto/tls.ConnectionState nil,
	Cancel: <-chan struct {} {},
	Response: *net/http.Response nil,
	Pattern: "/hello/{id}",
	ctx: context.Context(*context.cancelCtx) *{
		Context: context.Context(*context.cancelCtx) ...,
		mu: (*sync.Mutex)(0x1400019c2e0),
		done: (*"sync/atomic.Value")(0x1400019c2e8),
		children: map[context.canceler]struct {} nil,
		err: error nil,
		cause: error nil,},
	pat: *net/http.pattern {                  <--------------------------- /** `pat` is filled **/
		str: "/hello/{id}",
		method: "",
		host: "",
		segments: []net/http.segment len: 2, cap: 2, [
			(*"net/http.segment")(0x1400007ec60),
			(*"net/http.segment")(0x1400007ec78),
		],
		loc: "/Users/manjong/Documents/sources/opentelemetry-go-instrumentatio...+38 more",},
	matches: []string len: 1, cap: 1, ["42"],
	otherValues: map[string]string nil,}

And this is the one for our Gin example.
> net/http.(*conn).serve() /opt/homebrew/Cellar/go/1.23.1/libexec/src/net/http/server.go:2093 (PC: 0x10477abac)
Warning: debugging optimized function
Values returned:

  2088:			// in parallel even if their responses need to be serialized.
  2089:			// But we're not going to implement HTTP pipelining because it
  2090:			// was never deployed in the wild and the answer is HTTP/2.
  2091:			inFlightResponse = w
  2092:			serverHandler{c.server}.ServeHTTP(w, w.req)
=>2093:			inFlightResponse = nil
  2094:			w.cancelCtx()
  2095:			if c.hijacked() {
  2096:				return
  2097:			}
  2098:			w.finishRequest()
(dlv) p req
("*net/http.Request")(0x14000489180)
*net/http.Request {
	Method: "GET",
	URL: *net/url.URL {
		Scheme: "",
		Opaque: "",
		User: *net/url.Userinfo nil,
		Host: "",
		Path: "/hello-gin/1",
		RawPath: "",
		OmitHost: false,
		ForceQuery: false,
		RawQuery: "",
		Fragment: "",
		RawFragment: "",},
	Proto: "HTTP/1.1",
	ProtoMajor: 1,
	ProtoMinor: 1,
	Header: net/http.Header [
		"User-Agent": [
			"Go-http-client/1.1",
		],
		"Accept-Encoding": ["gzip"],
	],
	Body: io.ReadCloser(net/http.noBody) {},
	GetBody: nil,
	ContentLength: 0,
	TransferEncoding: []string len: 0, cap: 0, nil,
	Close: false,
	Host: "localhost:8080",
	Form: net/url.Values nil,
	PostForm: net/url.Values nil,
	MultipartForm: *mime/multipart.Form nil,
	Trailer: net/http.Header nil,
	RemoteAddr: "[::1]:60307",
	RequestURI: "/hello-gin/1",
	TLS: *crypto/tls.ConnectionState nil,
	Cancel: <-chan struct {} {},
	Response: *net/http.Response nil,
	Pattern: "",
	ctx: context.Context(*context.cancelCtx) *{
		Context: context.Context(*context.cancelCtx) ...,
		mu: (*sync.Mutex)(0x1400019a010),
		done: (*"sync/atomic.Value")(0x1400019a018),
		children: map[context.canceler]struct {} nil,
		err: error nil,
		cause: error nil,},
	pat: *net/http.pattern nil,            <--------------------------- /** `pat` is nil **/
	matches: []string len: 0, cap: 0, nil,
	otherValues: map[string]string nil,}

@iBluemind Thank you for the contribution!
I didn't go in-depth to the code yet, just addressing @MrAlias points:

I think the original reason we removed the gin probes is that we didn't collect any gin-specific data and those spans didn't add any extra data. This PR adds the http.route attribute for those frameworks which is something super useful and I saw a few users who would greatly benefit from these changes. So in general, if we can add attributes specific to gin or chi (which goes together with the semconv) - I think this has great value.
large PR - I agree - but a lot of it is offsets and test JSON data.
#1105 is a good point, however, as I said I do see a lot of value in adding more attributes to gin users.

I understand the name is different. But I'm not seeing that as justification in adding redundant telemetry as identified (and fixed by) #780.

I agree that having 3 probes(nethttp, gin, chi) seems like a bit of redundancy. Is there a better way to do this?

Can we update the nethttp instrumentation to detect what name parsing should be performed and signal the eBPF code with that information? I.e. detect if gin, chi, or net/http routing patterns should be used and send an enum as a flag.

Can we update the nethttp instrumentation to detect what name parsing should be performed and signal the eBPF code with that information? I.e. detect if gin, chi, or net/http routing patterns should be used and send an enum as a flag.

That would be better, thank you


Failing test

https://github.com/JoyOfJavaScript/joj/blob/9a3002f39c849973a6304c869a6a1119f20c1e6f/src/blockchain/src/lib/fp/data/validation2/validation.spec.mjs#L43 does not work, as toString() implementation will return _Success perhaps if you change the toString() into
class Validation {
...
  toString() {
      return `${this.constructor.name.substr(1)} (${this.#val})`
  }
}

That's odd it all seems to be working for me. Are you running the test within the Docker container? Here's what I'm seeing:
blockchain_tests_1  |   Validation:
blockchain_tests_1  |     ✔ Constructor passed
blockchain_tests_1  |     ✔ Validation#map passed
blockchain_tests_1  |     ✔ Validation#ap passed
blockchain_tests_1  |     ✔ Validation#equals passed
blockchain_tests_1  |     ✔ Validation#toString passed
I checked in my latest, please update and try again. Let me know if you still see it so I can dig into it further and thanks!

No, I'm running it outside Docker container, I'll dig into it to see if some transitive dependency is the cause. Keep up the hard work, Love what you do.

Thanks for the comment.  Since you're not running the Docker instance, make sure you're the Node.js version you have in your system is at least > v12.0


Multiple spectra analysis is too slow

Based on the Performance seems this code call analyseMultiplet and this should not be the case

@lpatiny
it's related to the function that we use from the ml-spectra-analysis and multiple-analysis package the speed depend on how much wide is, if you zoom in into specific are and select a precise area it will be fast.

This full code should not be executed. It is used for the multiplet analysis but here we only calculate integral.


Typo/awkward sentence

Hi,
The first sentence under the heading 'Log and monitor' says "One way to detect attacks is through activity monitoring activity and logging analytics."
I think one of the activities should be removed to make it easier to read, e.g., One way to detect attacks is through activity monitoring and logging analytics.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 77fc49b6-8652-e740-aab9-b00df8de5ac5
Version Independent ID: 38a06554-d181-4697-9c93-32a97330abb4
Content: Securing Azure Functions
Content Source: articles/azure-functions/security-concepts.md
Service: azure-functions
GitHub Login: @ggailey777
Microsoft Alias: glenga

@Disorganise Thanks for catching that! We have worked up a PR to address this, which once merged should reflect in a couple of hours.

@Disorganise We have a PR merged for this and should reflect post MSFT Build.
We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.


don't requeue if dashboard CR are broken

fixes #1091

@weisdd is it worth doing some kind of status update, or should we just skip the reconcile?


OpenEBS deployment on Digital Ocean.

Create workflow documentation and automation script for deploying OpenEBS on Digital Ocean.

A cluster provisioned with OpenEBS can directly be created using DigitalOcean's Marketplace.
Follow the below-mentioned steps to get started.

Login to your DigitalOcean account, click on Marketplace under Discover as shown in the image.

Scroll down to find OpenEBS, clicking on the icon you get the option to create a cluster.

Provide the necessary details such as Data Center region, number of nodes etc. and click on
create cluster button.

While the cluster is getting created you can connect the cluster to your local machine or remote
management server. Follow the steps mentioned under the Overview section.

5. Once the cluster is created, you can execute the following command:
 $ kubectl get ns
The output will have openebs namesapce in active state.
NAME       STATUS   AGE
default    Active    9 s
opnebs     Active    9 s

Also, execute
$ kubectl get pods -n openebs
All the pods must be in running state.
You can refer to this article to know about provisioning OpenEBS volumes..


Avatar Icons: Deprecated: Required parameter $size

Deprecated: Required parameter $size follows optional parameter $css in .../plugins/Customizer/Helper/DynamicAvatar.php on line 26
Deprecated: Required parameter $size follows optional parameter $css in .../plugins/Customizer/Helper/DynamicAvatar.php on line 47
No avatar image for header or board can be uploaded

I'm not really maintaining this plug in anymore. sorry.
It sounds like your php version may be too new, try php 7.x

it's PHP 8 with the kanboard docker image v1.2.22 and I got the same issues/warnings

hi, I fixed it in this way and it's tested with php 7.4 and 8 and works for me.
diff Customizer/Helper/DynamicAvatar.php Customizer/Helper/DynamicAvatar.php.fixed
26c26
<     public function dynamic($user_id, $username, $name, $email, $avatar_path, $css = '', $size)

public function dynamic($user_id, $username, $name, $email, $avatar_path, $css = 'avatar-left', $size = 48)

47c47
<     public function boardDynamic($user_id, $username, $name, $email, $avatar_path, $css = '', $size)

public function boardDynamic($user_id, $username, $name, $email, $avatar_path, $css = 'avatar-left', $size = 48)

sorry, I'm not familiar with git/github to commit/send you the file...

perfect execution. I'll merge your pull request when I get the chance.

Fixed by 908de2168d4caac47fb4448b46b6430c0c6851c0

Actually pretty sure this was already fixed here: https://github.com/creecros/Customizer/pull/132

im slacking, and haven't updated any releases in probably a year, so there are plenty of new merges that haven't actually been released.

oh ok, i can help with this. I will take a look at the repos soon, might have time tomorrow.

I drafted a new release: https://github.com/creecros/Customizer/releases/tag/1.13.8
and submited a pull request to update the plugin repo: https://github.com/kanboard/website/pull/310

you rock, thanks @Jieiku

no problem, happy to help.


Kotlin Playground Empty & not editable

URL of codelab
https://developer.android.com/training/kotlinplayground
In which task and step of the codelab can this issue be found?

Create a base class, item1

Describe the problem
The Kotlin playground can't be edited. No codes are available on the link.
Steps to reproduce?

Go to Kotlin Basic
Click on Module 2
Click on Create a Base class
See item 1
Click the link directing to Kotlin playground
See error

Versions
Android Studio version:
API version of the emulator:
Additional information
Include screenshots if they would be useful in clarifying the problem.

@ralph-embalsado
Thank you for reaching out to us.
The link given in the codelab is correct: https://developer.android.com/training/kotlinplayground
Try changing the code inside the code block.

There won't be any code in the playground, you need to copy-paste from the codelab.
Alternatively you can use the following Kotlin playground:
https://play.kotlinlang.org/
If you still face issues please send in the screenshots.

The issue is closed as no further clarification of it was provided.


Modal / Action Sheet events are ambiguous

This is a (multiple allowed):

[ ] bug

[x] enhancement

[ ] feature-discussion (RFC)

Framework7 Version: 1.6.0

Platform and Target: Google Chrome, Phonegap

Live Link or JSFiddle/Codepen: https://jsfiddle.net/weaponx86/k8584mdt/3/

What you did
Made multiple calls to myApp.actions with different options, created an actions:closed listener. Trying to distinguish them from one another
Expected Behavior
Need to be able to pass an id to myApp.actions() to be sent along with events or some distinguishing value needs to be passed into the template.
Actual Behavior
Can't bind action sheet events to individual action sheets. Callbacks do not send back any meaningful identifier.

.actions() returns the genrated HTMLElement, actions:closed is triggered with the target of that HTMLElement
Either add an id to the HTMLElement returned by .actions() or store it and compare it to actions:closed target

Adding the id made a lot more sense.
Here is my code for anyone else:
$(document).on('click','#btn-sort-home', onClickSort);
$(document).on('actions:closed','#action-sheet-home-sort',function(e){
	console.log('Sort action sheet closed');
});

function onClickSort(){

	var aoActions = [];

	for (var sField in oGlobals['oSortFields']){
		aoActions.push({
			text: oGlobals['oSortFields'][sField],
			data: JSON.stringify({field: sField}),
			value: sField
		});
	}

	var hModal = myApp.actions(aoActions);

	jqModal = $(hModal);

	jqModal.attr('id','action-sheet-home-sort');
}


Title of the buttons

Title of every button should be at centre it is bottom left aligned.
Assign me this issue i can fix this !

@Debajyoti14 please assign this to me

@VISHALGUPTA100803 No this is the design we have gone with.
With centre align it will not look that good.
So closing this issue


Update CONTRIBUTORS.md

updated my name in the contributors.md file

Added my name to contributors.md