API google.pubsub.v1.Publisher exceeded 5000 milliseconds when running on Cloud Run

When the image is run locally on Docker the application is able to successfully publish messages to Cloud PubSub.
When the same image is deployed to Cloud Run, it won't publish a single message. All attempts fail with the error:
GoogleError: Total timeout of API google.pubsub.v1.Publisher exceeded 5000 milliseconds before any response was received.
    at repeat (/usr/root/node_modules/google-gax/build/src/normalCalls/retries.js:66:31)
    at Timeout._onTimeout (/usr/root/node_modules/google-gax/build/src/normalCalls/retries.js:101:25)
    at listOnTimeout (internal/timers.js:557:17)
    at processTimers (internal/timers.js:500:7)
code 4

Sample Code:
import { mongoose } from '@shared/connections'
import MongoDbModel from '@shared/models'
import { PubSub } from '@google-cloud/pubsub'

import {
  REMINDER_COPY_SMS = '',
  REMINDER_MAX_DAYS_SINCE_LAST_MSG = 0,
  REMINDER_MAX_DAYS_SINCE_LAST_READING = 0,
  REMINDER_MESSAGE_TEMPLATE = 'reading-reminder',
  TOPICS_MESSAGE_DISPATCH,
  KEYFILE_PATH,
} from '../constants.js'

const pubsubClient = new PubSub({
  projectId: process.env.GOOGLE_CLOUD_PROJECT,
  keyFilename: KEYFILE_PATH,
})

const msMaxSinceLastReading = REMINDER_MAX_DAYS_SINCE_LAST_READING * 24 * 60 * 60 * 1000
const msMaxSinceLastMsg = REMINDER_MAX_DAYS_SINCE_LAST_MSG * 24 * 60 * 60 * 1000

const { model: Patient } = new MongoDbModel(mongoose, 'Patient')

const findInactivePatients = () => Patient.find({
  status: 'ACTIVE',
  $and: [{
    $or: [{
      last_reading_at: { $exists: false },
    }, {
      last_reading_at: { $lt: new Date(Date.now() - msMaxSinceLastReading) },
    }],
  }, {
    $or: [{
      last_reading_reminder_at: { $exists: false },
    }, {
      last_reading_reminder_at: { $lt: new Date(Date.now() - msMaxSinceLastMsg) },
    }],
  }, {
    // Test with known patients only
    $or: [{
      first_name: { $regex: /Bruno/i }, last_name: { $regex: /Soares/i },
    }],
  }],
}, {
  first_name: 1,
  last_name: 1,
  gender: 1,
  last_reading_at: 1,
  phones: 1,
}, {
  lean: true,
})

const updateLastReadingReminderAt = (patient) => Patient.findOneAndUpdate({
  _id: patient._id,
}, {
  last_reading_reminder_at: Date.now(),
})

const createMessagePayload = (patient) => ({
  body: REMINDER_COPY_SMS,
  channels: [{
    name: 'sms',
    contacts: patient.phones.map((phone) => phone.E164),
    specifications: {
      template: REMINDER_MESSAGE_TEMPLATE,
    },
  }],
})

const dispatchMessage = async (patient) => {
  if (!patient.phones || !patient.phones.length) {
    return
  }
  const payload = createMessagePayload(patient)
  console.info('Worker reminder-report-vitals', 'event payload', JSON.stringify(payload))
  try {
    await pubsubClient.topic(TOPICS_MESSAGE_DISPATCH).publishMessage({ json: payload })
    console.info('Worker reminder-report-vitals', 'event published to cloudPubSub')
    pubsubClient.close()
    await updateLastReadingReminderAt(patient)
    console.info('Worker reminder-report-vitals', 'last_reading_reminder_at updated for patient', patient._id)
  } catch (err) {
    console.error(err)
  }
}

const handler = async () => {
  console.info('Worker reminder-report-vitals', 'exectution started')
  try {
    const inactiveList = await findInactivePatients()
    inactiveList.forEach(dispatchMessage)
  } catch (err) {
    console.error(err)
  }
}

export default handler

The code hangs after logging this line:
console.info('Worker reminder-report-vitals', 'event payload', JSON.stringify(payload))
Sample Event Payload:
{"body":"This is {{ORG}}. We haven't been receiving your vitals. Reply \"start\" to report your vitals now. Reply \"stop\" at any time to opt-out from automated reminders.","channels":[{"name":"sms","contacts":["+15555555555"],"specifications":{"template":"reading-reminder"}}]}

Environment details

OS: Linux Alpine
Node.js version: 14.16.1
@google-cloud/pubsub version: 2.18.4

Steps to reproduce

Create PubSub topic to receive messages and replace its name in constants.js TOPICS_MESSAGE_DISPATCH.
Create nodejs application image from sample code. For sake of replicating only the issue, database operations may be removed - instead the sample event payload may be used.
Run the container locally with Docker.
Push the image to GCP Container Registry
Deploy a Cloud Run service using the image
See timeout erro in Cloud Logs

FYI I experienced the same behavior running on Cloud Functions. Rolling back to version 2.12.0 unblocked me.

Yeah we are experiencing issues on cloud run.

For what it's worth, I am seeing the same error message when running the emulator locally, which in that case appears to be because the port used by the emulator (random, in the 8000 range) doesn't match the one exported through the PUBSUB_EMULATOR_HOST variable.
I'll try a container-based approach, but I suspect a port mismatch.

I don't believe this is the issue in production. Most likely cloud run now kills the network access when the container is inactive (aka when the request has ended) which happens more often on low load. It used to work though, so I believe it might be a change in cloud run infrastructure. I am making sure that all the pubsub calls are awaited properly on my side. You should also set .topic(topic, { batching: { maxMessages: 1 } }) so that the message is sent right away instead of waiting in a queue.

For context: I was referring to an issue with ports on the emulator. I have deleted the relevant comment because I also believe it's not what's happening here. Apologies for the confusion.

Can confirm, we are facing the same issue on Cloud Functions with @google-cloud/pubsub version 2.18.4.

What's new about this case? I have the same problem

We revered to 2.17. I believe the bug was introduced in 2.18.

@Sytten Just checking — the issue does not reoccur with 2.17, only with 2.18? Which minor version were you using? I do see a change that went out re: publish timeouts, https://github.com/googleapis/nodejs-pubsub/releases/tag/v2.18.0 and then updated in https://github.com/googleapis/nodejs-pubsub/releases/tag/v2.18.3. It looks like @feywind did some investigation in an earlier issue here: https://github.com/googleapis/nodejs-pubsub/issues/1425 and it seems like there was an upstream change we're trying to figure out. Our current believe is that 2.18.3 fixes the issue, but please let me know if this isn't the case.

Getting the same issue in google cloud function trigger.

@meredithslota We had the problem with 2.18.3 too

same here too,  with 2.18.3 and 2.18.3 version

FYI I experienced the same behavior running on Cloud Functions. Rolling back to version 2.12.0 unblocked me.

Tried this solution it did not work for my case, while using a background cloud function.
const { PubSub } = require("@google-cloud/pubsub");
const pubsub = new PubSub();

exports.main = async (message, context) => {
  const data = {
    event: "AccountCreated"
  };

  const pubMessage = Buffer.from(JSON.stringify(data));
  const topic = pubsub.topic(topicName);

  topic.publish(pubMessage);
};

{
  "name": "sample-pubsub",
  "version": "0.0.1",
  "dependencies": {
    "@google-cloud/pubsub": "^2.12.0"
  }
}

@apettiigrew Remove ^... use version only. "@google-cloud/pubsub": "2.17.0"

It occured to me today as well after upgrading from 2.17x, reverted back but why did they introduce 5s timeout?

It occured to me today as well after upgrading from 2.17x, reverted back but why did they introduce 5s timeout?

Same here, too. Running 2.18.4 with dockerized node application.

Same problem here with 2.15.1

Same here using google cloud functions with @google-cloud/pubsub, reverted to "@google-cloud/pubsub": "2.17.0" and it's working again.

I had the same issue using Google Cloud Functions. I was publishing successive messages probably the wrong way :
using "batching" option and publishing them using Promise.all reduced the processing time, it's now under 5000 ms

Inside cloud run, i needed to disable the batching (maxMessages = 1), and make sure the promise was awaited before the http response was returned.

@Sytten Just checking — the issue does not reoccur with 2.17, only with 2.18? Which minor version were you using? I do see a change that went out re: publish timeouts, https://github.com/googleapis/nodejs-pubsub/releases/tag/v2.18.0 and then updated in https://github.com/googleapis/nodejs-pubsub/releases/tag/v2.18.3. It looks like @feywind did some investigation in an earlier issue here: #1425 and it seems like there was an upstream change we're trying to figure out. Our current believe is that 2.18.3 fixes the issue, but please let me know if this isn't the case.

Inside cloud run, i needed to disable the batching (maxMessages = 1), and make sure the promise was awaited before the http response was returned.
Note: By default the cloud run indeed dont allocate cpu after the request is finished (https://cloud.google.com/blog/products/serverless/cloud-run-gets-always-on-cpu-allocation)

I don't believe this is the issue in production. Most likely cloud run now kills the network access when the container is inactive (aka when the request has ended) which happens more often on low load. It used to work though, so I believe it might be a change in cloud run infrastructure. I am making sure that all the pubsub calls are awaited properly on my side. You should also set .topic(topic, { batching: { maxMessages: 1 } }) so that the message is sent right away instead of waiting in a queue.

Inside cloud run, i needed to disable the batching (maxMessages = 1), and make sure the promise was awaited before the http response was returned.
Note: By default the cloud run indeed dont allocate cpu after the request is finished (https://cloud.google.com/blog/products/serverless/cloud-run-gets-always-on-cpu-allocation)

We are facing the same issue as well, using pub sub version 2.18.3, is downgrading the only solution?

We are seeing this error: Received error while publishing: Total timeout of API google.pubsub.v1.Publisher exceeded 600000 milliseconds before any response was received.

We are facing the same issue as well, using pub sub version 2.18.3, is downgrading the only solution?

We only use pubsub in Nodejs for some tests locally. We use the Java sdk on production where it works fine.
I tried deploying it to the gcloud just to see and didn't get the timeout there.I do not know why but I am thinking there is something wrong with the way we have implemented nodejs.
Will do some more tests with some other versions of writing it today or tomorrow.

We are facing the same issue as well, using pub sub version 2.18.3, is downgrading the only solution?

Same here - we are using the version 2.18.1 though. So I guess downgrading needs to go at least below that - if that helps anyway.
We also opened (yet another) P3 support case at google. They will refer to the product team - lets hope this issue resolves quick.

We are seeing the same issue. It doesn't happen on every message, but it seems pretty unreliable.
Can anyone confirm that downgrading to 2.17 will fix this in the meantime?

I think this is kind of related to #847 - there were some PRs (see thread) to solve this issue and i believe that it is now causing this timeout. Previously, pushing e.g. 4k messages of messages resulted in like 30k messages being published instead; now when i try to publish 4k messages around 2k are published then we get this timeout error instead...

I had the same warning in CF log. Rolling back to version 2.12.0 unblocked me.

Same error. 2.12.0 worked for me.
Thks @buffolander 👍🏼

I had to update "@google-cloud/pubsub": "^2.12.0" and  also add "google-gax": "^2.28.1" into package.json to get this to work.

@buffolander Just to double check something in the originally posted code:
  try {
    await pubsubClient.topic(TOPICS_MESSAGE_DISPATCH).publishMessage({ json: payload })
    console.info('Worker reminder-report-vitals', 'event published to cloudPubSub')
    pubsubClient.close()
    await updateLastReadingReminderAt(patient)
    console.info('Worker reminder-report-vitals', 'last_reading_reminder_at updated for patient', patient._id)
  } catch (err) {
    console.error(err)
  }

This probably shouldn't be calling pubsubClient.close() unless you don't expect to execute this function again. CF can hold on to container instances it makes for functions, so Node may remain running, and thus pubsubClient would still be alive between requests. There were some changes not too long ago with how close() handles shutdown, if I recall correctly, so it might be related.

I should also mention, you might want to cache the result of pubsubClient.topic() since that can be reused for future publishes, and it will make better use of message batching.

It happens only emulator mode on my side.
Pubsub Receiver Project
firebase-tools: 9.16.0
@google-cloud/pubsub: 2.16.1
Pubsub Sender Project
@google-cloud/pubsub: 2.18.5
Is this cause of Issue?
@feywind

Funny, the change has been reverted in the latest version in this commit: https://github.com/googleapis/nodejs-pubsub/commit/34a4d4aa50c6f07adcec1dcba1ee0eaf941f71c0 a ticket to update the copyright years.
The ticket that it belonged to: https://github.com/googleapis/nodejs-pubsub/pull/1465

I can confirm that updating to the latest version solved my problem.

Everyone affected by this issue should also review https://github.com/googleapis/nodejs-pubsub/issues/1463

Folks make sure you don't have PUBSUB_EMULATOR_HOST environment variable set when your app is running (obviously if you want to use GCP pubsub, not emulator).
In my case, this was the issue, as I had this environment variable equal to localhost:PORT which was giving timeout errors all the time when I had expected it to make requests to GCP - after .env was excluded from the image we use deploy in GCP the problem is solved.
Cheers

I am getting the same issue of 600000-millisecond timeout when calling pubishMessage locally. I initially had @google-cloud/pubsub: ^2.10.0.
I tried all combinations of upgrading and adding gaxios to my packages.json, nothing helped.
The creation and subscription work fine with the pubsub-client but when pushing the messages it shows the times out error.
I thought it may be an environmental issue, and I changed the code to invoke pure rest-API using JWT client from 'google-auth-library', which uses the same gaxios client internally, and the rest-API works fine.
Not sure, it seems the client is not even trying any HTTP post and gets stuck somewhere.
Thanks

The only reliable fix that worked for us was deploying to CloudRun with --no-cpu-throttling.

@Sytten the endpoint will have quite a bit of traffic later on so it's OK for us.
We have tried using { batching: { maxMessages: 1 } }) and it worked most of the time, but occasionally failed as well.

Wow, that commit message for https://github.com/googleapis/nodejs-pubsub/commit/34a4d4aa50c6f07adcec1dcba1ee0eaf941f71c0 was really not great. It looks like the protos actually changed to set back the timeout value, and OwlBot just happened to pull them in.
Setting the max message queue to 1 is expensive in other ways, but if you're literally sending one message in a Cloud Function, that might honestly be the best way to handle it. The library is currently designed more with bulk send in mind.

For me this was happening in version 2.17 too.
I needed to create a dummy version in cloud run to restart servers.

I have tried both pubsub 2.17 and gax 2.28.1, but doesn't seem to work for this simple use case-
exports.main = async (message, context) => {
  const data = {
    event: "AccountCreated"
  };
  const topicName = 'some valid topic name'
  const pubMessage = Buffer.from(JSON.stringify(data));
  const topic = pubsub.topic(topicName);

  topic.publish(pubMessage);
};

Keep getting the same error when the await on publishMessage hangs for a bit and crashes-
GoogleError: Total timeout of API google.pubsub.v1.Publisher exceeded 60000 milliseconds before any response was received.
    at repeat (/Users/rocktech/LeidenUniv/sorting-as-a-service/server/node_modules/@google-cloud/pubsub/node_modules/google-gax/build/src/normalCalls/retries.js:66:31)
    at Timeout._onTimeout (/Users/rocktech/LeidenUniv/sorting-as-a-service/server/node_modules/@google-cloud/pubsub/node_modules/google-gax/build/src/normalCalls/retries.js:101:25)
    at listOnTimeout (node:internal/timers:564:17)
    at process.processTimers (node:internal/timers:507:7) {
  code: 4
}

I have tried both pubsub 2.17 and gax 2.28.1, but doesn't seem to work for this simple use case-
exports.main = async (message, context) => {
  const data = {
    event: "AccountCreated"
  };
  const topicName = 'some valid topic name'
  const pubMessage = Buffer.from(JSON.stringify(data));
  const topic = pubsub.topic(topicName);

  topic.publish(pubMessage);
};

Keep getting the same error when the await on publishMessage hangs for a bit and crashes-
GoogleError: Total timeout of API google.pubsub.v1.Publisher exceeded 60000 milliseconds before any response was received.
    at repeat (/Users/user/path/app/server/node_modules/@google-cloud/pubsub/node_modules/google-gax/build/src/normalCalls/retries.js:66:31)
    at Timeout._onTimeout (/Users/user/path/app/server/node_modules/@google-cloud/pubsub/node_modules/google-gax/build/src/normalCalls/retries.js:101:25)
    at listOnTimeout (node:internal/timers:564:17)
    at process.processTimers (node:internal/timers:507:7) {
  code: 4
}

Are you calling it concurrently? I had a similar issue to this post when running om GCP and my issue was that i was sending lots of request concurrently without awaiting the promises, causing my service to scale down. Instead you can do multiple publishes in a single request and await the respones (if doing scale-to-zero)

This is actually running on a local machine, not publishing multiple messages concurrently but sequentially within a loop, the error comes on the first iteration itself.
The code looks similar to
const topicName = 'some valid topic name'
const topic = pubsub.topic(topicName);
array.forEach(async (value)=>{
  const pubMessage = Buffer.from(JSON.stringify(value));
  const topic = pubsub.topic(topicName);
  await topic.publish(pubMessage);
}

I face with the same issue while using pubsub emulator. But previously it seemed working:
GoogleError: Total timeout of API google.pubsub.v1.Subscriber exceeded 60000 milliseconds before any response was received.
           |     at repeat (/usr/src/app/node_modules/@google-cloud/pubsub/node_modules/google-gax/src/normalCalls/retries.ts:84:23)
           |     at Timeout._onTimeout (/usr/src/app/node_modules/@google-cloud/pubsub/node_modules/google-gax/src/normalCalls/retries.ts:125:13)
           |     at listOnTimeout (node:internal/timers:559:17)
           |     at processTimers (node:internal/timers:502:7) {
           |   code: 4

Linked to the meta-issue about transport problems: b/242894947

Not to make this even stranger, I've observed this happening when running Cloud Functions (HTTP CF v1) that are triggered by Cloud Tasks HTTP calls. What's odd is I have many Pub/Sub triggered Cloud Functions (Event driven/background functions, also CF v1) that are part of the same flow, and I have not observed the error in them.
I have been running this using ^3.0.1 across all of them.
--
CCing @campbellbrobbel

I get google.pubsub.v1.Publisher exceeded 60000 milliseconds before any response was received. off cloud run. Our use is so trivial that I have a hard time believing its something we did.

I fixed like this:
before:
await publisherClient.publish({
  topic: `projects/${projectId}/topics/${topic}`,
  messages: [
    {
      data: JSON.stringify(body),
    },
  ],
});

after:
await publisherClient
  .topic(`projects/${projectId}/topics/${topic}`)
  .publishMessage({
     data: Buffer.from(JSON.stringify(body)),
  });

@blue-hope Your fix seems like a red-herring. Here is what I'm doing in production and started hitting these errors. Our events/sec on this topic is max 100/s so the use of orderKey seems like non issue.
// Where eventsTopic is created before hand as eventsTopic = publisherClient.topic('thetopic')
eventsTopic.publishMessage({
    json: {
      op: 'event',
      this: 'that',
      thenext: 1
    },
    orderingKey: specificKey
  });

Just ran into this issue today. Is there a known fix? The issue only seems to happen when we deploy to our staging servers. Locally, everything works as expected.

Also ran into this yesterday, but on GKE instead of Cloud Run (I think Cloud Run is basically Knative on top of GKE).
Any tips to debug this if it happens again?

I managed to fix this issue by caching the topic when the HTTP server started and just reusing it. I'm confused as to why this worked though, because even though the old implementation didn't cache the topic, during our tests we were only sending 1 HTTP request to the web server.

Happening here as well.

I get this issue happening quite a bit using Google cloud functions.
Super simple message subscriptions (using it to call function 2 from function 1 so there's a single subscription between function 1 --> function 2, ), low load (handful of messages / hr).
For some function connections (e.g., f1->f2), everything works great, messages seem to always get published correctly.
For other function connections (eg f3->f4) it's super erratic. ~50% of the time, it will fail with this error message (GoogleError: Total timeout of API google.pubsub.v1.Publisher exceeded 60000 milliseconds before any response was received). The other 40% of the time, the messages are published just fine. And then maybe 10% of cases, the message is published twice...?

I'm suspicious this might be related to ECONNRESET. I was hitting this error for other kinds of connections, moved the CPU to "CPU is always allocated", and no longer get these messages.

dont know if this helps anyone but in my case, i wasnt validating the pubsub client properly, as i'm using a non-google service to call the publisher
  const pubsub = new PubSub({
        projectId,
        grpc,
        credentials: {
            client_email: context.values.get("GCP_CLIENT_EMAIL"),
            private_key: context.values.get("GCP_PRIVATE_KEY").replace(/\\n/g, '\n'),
        },
    })

specifically  i didnt add  .replace(/\\n/g, '\n') and for 2 days i've been trying to understand why it was doing the timeout because it doesnt error out with authorisation issues.

I fixed like this in version 3.2.0:
before:
const publisherClient = new PublisherClient({ projectId })
await publisherClient.publish({
  topic: `projects/${projectId}/topics/${topic}`,
  messages: [
    {
      data: JSON.stringify(body),
    },
  ],
});

after:
const publisherClient = new PubSub({ projectId })
await publisherClient
  .topic(`projects/${projectId}/topics/${topic}`)
  .publishMessage({
     data: Buffer.from(JSON.stringify(body)),
  });

I am on version 3.3.0 and I am having the same issue

@bullishgopher are you saying that your fix ("after" code snippet) resolved the issue?

Sorry @JamesTheTerry
I mean I have the same issue

error - unhandledRejection: GoogleError: Total timeout of API google.pubsub.v1.Publisher exceeded 60000 milliseconds before any response was received.

While the symptom of these issues (publisher timeouts) is similar across the cases, it is highly likely the the underlying causes are different and so this GitHub issue now covers a lot of different cases where more user-specific information is needed. If you are still experiencing issues, please create a support case. Thanks!

I got erro "google.pubsub.v1.Publisher exceeded 60000 milliseconds before any response was received.", By increasing gaxOpts timeout value to 100,000 solved our problem.
const publishOptions = {
  gaxOpts: {
    timeout: 100000,
  },
};

const topicChannel = pubsub.topic(TOPIC_ID, publishOptions);

I got error "google.pubsub.v1.Publisher exceeded 60000 milliseconds before any response was received.", By increasing gaxOpts timeout value to 100,000 solved our problem.
const publishOptions = {
  gaxOpts: {
    timeout: 100000,
  },
};

const topicChannel = pubsub.topic(TOPIC_ID, publishOptions);

Thanks @koushil-mankali for your solution. It's working. I am facing a new issue related to google cloud logging. Can you please let me know how to fix the below error. It will be very helpful. Thanks
Error: Total timeout of API google.logging.v2.LoggingServiceV2 exceeded 60000 milliseconds before any response was received.

Does anyone know why it would take over a minute to publish?

Does anyone know why it would take over a minute to publish?

If the queue has a lot of data to process, it will take more time. You can reproduce the issue by load testing with multiple requests. The issue will be fixed by increasing the timeout value to a max value of 540000 i.e. 540 secs.

Something to consider - we noticed this issue arising from an HttpFunction where we were sending a response before publishing to PubSub. Having awaits after sending a response is contraindicated by google, and when we switched the order to publish first, the issue went away.

Some point to consider about this issue, in case your application is not receiving consistent traffic/load, this error may affect the initial request(s)/action(s) that boot(s) up your cloud run instance(s). With subsequent requests/actions in the same timeframe, you'll notice that this error may not re-occur.
I cannot explain why it takes that long to publish to pub/sub but increasing the timeout as in koushil-mankali's solution here will save you some headaches.

I got the error "google.pubsub.v1.Publisher exceeded 60000 milliseconds before any response was received.", By increasing the gaxOpts timeout value to 100,000, we solved our problem.
const publishOptions = {
  gaxOpts: {
    timeout: 100000,
  },
};

const topicChannel = pubsub.topic(TOPIC_ID, publishOptions);

Thanks, @koushil-mankali, for your solution. It's working. I am facing a new issue related to Google Cloud logging. Can you please let me know how to fix the below error? It will be very helpful. Thanks
Error: The total timeout of API google.logging.v2.LoggingServiceV2 exceeded 60000 milliseconds before any response was received.

@bharathjinka09 Did you find any solutions for your last error? I'm facing the same problem, and I have tried several solutions but haven't succeeded.
Solutions I have tried:

Increased timeout from 60 seconds to 600 seconds in pub/sub client.
Increased CPU and RAM for cloud functions.
Used batch messaging for efficiency.
Create a single topic instance and use it across all messages.

Just to reiterate what I said back in June: While the symptom of these issues (publisher timeouts) is similar across the cases, it is highly likely the the underlying causes are different and so this GitHub issue now covers a lot of different cases where more user-specific information is needed. If you are still experiencing issues, please create a support case. Thanks!
Increasing the gax timeout may be a viable solution in some instances, but it is not a general-purpose answer to how to address timeouts in all cases.
We need specific information for each individual case in order to properly diagnose these issues and so a support case is the proper venue for that exchange of information.

I solved this issue for me by having a single instance of the topic so you can change your code to something like this:
 const topics = {}; const dispatchMessage = async (patient) => {   if (!patient.phones || !patient.phones.length) {     return   }   const payload = createMessagePayload(patient)   console.info('Worker reminder-report-vitals', 'event payload', JSON.stringify(payload))   try {     if(!topics[TOPICS_MESSAGE_DISPATCH]){       topics[TOPICS_MESSAGE_DISPATCH] = pubsubClient.topic(TOPICS_MESSAGE_DISPATCH, {         batching: {           maxMessages: 100, // set it to something like 200-300 depending upon how your subscriber is configured.           maxMilliseconds: 100, // max wait before messages are published to the server (in ms)         }       })     }     await topics[TOPICS_MESSAGE_DISPATCH].publishMessage({ json: payload })     console.info('Worker reminder-report-vitals', 'event published to cloudPubSub')     pubsubClient.close()     await updateLastReadingReminderAt(patient)     console.info('Worker reminder-report-vitals', 'last_reading_reminder_at updated for patient', patient._id)   } catch (err) {     console.error(err)   } }
Give it a try.

I have solved this issue. The problem was with environmental variables. Google Cloud functions parse each variable added from the GUI console, and ads escape sequences. So \n in my service account's private will be converted to \\n resulting in wrong secrets.
It's strange that the Pub/Sub SDK does not return proper error messages; it always returns an API timeout, no matter how much time I increase the timeout.
Removing escape sequences from the account's private key solves the problem.

Hi @ts-geek22, were you able to publish messages through the service at all? Or did the error occur after the application had been running for around 1-2 hours?

It's a string parsing issue, so it occurs from the start, as a wrong string is always the wrong string.

A timeout error is a general error in the case of cloud publishing, as it throws the same error for multiple reasons and does not provide any specific contextual information.
This is a link to my Stack Overflow issue, where I have listed a few different solutions I have tried; feel free to give them a try; you might find one that works for you.

Im experiencing a similar issue and a solution that worked for me was to initiate pubsub inside the function and then close the topic. Reason for this was apparently that we kept batching it, resulting in cloud function closing before it had a chance to send it.
    try {
      // WARNING shifting new Pubsub outside of this function and removing close causes pubub time
      const pubSubClient = new PubSub();
      const topic = pubSubClient.topic(topicName, {
        batching: { maxMessages: 1 },
        gaxOpts: {
          timeout: 100000,
        },
      });

      const messageId = await topic.publishMessage({ json }).catch((err) => {
        logger.error(
          `Received error while publishing to ${topicName}: ${err.message}`
        );
        logger.error(err);
        throw new InternalServerError();
      });
      await pubSubClient.close();
      logger.info(
        `Message ${messageId} was published to ${topicName} with message ${JSON.stringify(
          json
        )}`
      );
    } catch (err) {
      logger.error(err);
      throw new InternalServerError();
    }
 }

However, somebody changed it according to documentation a while ago and we started having that issue again just recently, so need to retest that. Just throwing this idea out there, maybe it will help somebody. Will try to give an update after more testing.
In the mentions above, I noticed people talking about importance of awaiting pubsub calls. In our use case we return a response from a server after we call a function that triggers pubsub that is not awaited to pretty much avoid having pubsub fail before we get a chance to respond. However, we await withing the called function. Example
async function example () {
....
      initPubsubMessages(
      message1,
      message2,
      ...
      );
return data;
}

async function initPubsubMessages(message1, message2) {

  await Pusub.publish(topic, message1);
  await Pusub.publish(topic, message2);
}

We use firebase functions gen 2 which are pretty much google cloud functions at this point. To my knowledge. In my head everything here is correct and until the processes initiated in initPubsubMessages, which has been awaited, should be processed without a hitch. But I'm also aware the cloudrun has some quirks so I thought ill ask for somebody else's opinion on that. Would firebase functions/cloudrun stop all processes as soon the server responds, is my question i guess?
I am grasping at straws here but I've experienced weirder things using that platform.

@jdziek I don't know much about firebase functions, but Cloud function will unallocate memory and CPU as soon as you return response in HTTP cloud function. It assumes that your function's execution is complete as you have returned a response.

The same issue about Cloud Run was already mentioned here before: https://github.com/googleapis/nodejs-pubsub/issues/1442#issuecomment-1006978912

Was worried that this might be the case. The thing is that this has been happening in places where we await as well. Will add an await just to be safe in this case, but doubt it will solve the issue fully :/
Now I just have to hope that hanging pubsub wont prevent response alltogether.
Thank you.

Was good for about two weeks. Checked we are awaiting everything. Put in tweaks mentioned above. Just had a pubsub error chain this morning. Really at a loss here. Was really hoping that it will work. Any suggestions at this point welcome. Might try to put pubsub back to how it was setup in documentation, maybe this will work this time with proper awaitig.

@jdziek

Was good for about two weeks. Checked we are awaiting everything. Put in tweaks mentioned above. Just had a pubsub error chain this morning. Really at a loss here. Was really hoping that it will work. Any suggestions at this point welcome. Might try to put pubsub back to how it was setup in documentation, maybe this will work this time with proper awaitig.

Are you referring to your configuration mentioned earlier?
      const topic = pubSubClient.topic(topicName, {
        batching: { maxMessages: 1 },
        gaxOpts: {
          timeout: 100000,
        },
      });

Yeah. Noticed though that its happening only in one service now. So currently reviewing the code there. Reverted pubsub implementation to how its shown in the docs and pushed. So hopefully nothing new will appear in two weeks

Yeah. Noticed though that its happening only in one service now. So currently reviewing the code there. Reverted pubsub implementation to how its shown in the docs and pushed. So hopefully nothing new will appear in two weeks

Hey @jdziek are you using any other library to connect to any other Google API's?

yeah, no. Just google-cloud ones for firestore, pubsub, and storage.

yeah, no. Just google-cloud ones for firestore, pubsub, and storage.

Update those packages to the latest versions; hopefully, that will fix your issue. We faced a similar problem with a few of our services; in our case, it was the Google Secret Manager package. Once we had bumped it to the latest version, that fixed this problem. I believe it's due to the gRPC library that most of the npm packages from Google use for communication with their APIs

Good point. I only updated Pubsub really. Will review it now. Thanks

Hey, I'm subscribing as well to this issue.
I have the same error in all my cloud run services, everything was fine until a few weeks ago when we started having this issue. We batched to 1, we opened and closed the client for each message, but we're still having it. I'm trying to bump all google dependencies to see if it changes anything, let's keep each other posted!

Hi, folks - I am publishing almost 100k messages to a topic somehow after a while of publishing the topic I started getting the "Error: Total timeout of API google. pub-sub.v1.Publisher exceeded 600000 milliseconds before any response was received" and no matter how many timeout values I increased the error will be still there with waiting for that amount of time specified as the timeout value.

@vinay-panwar-04 Same i have the timeout increased on the function and can confirm the timeout on the function is larger than 60000ms and i can also see the acknowledgment deadline for the pubsub subscription is also greater than 60000ms but yet i still see this error
Error: Total timeout of API google.pubsub.v1.Publisher exceeded 60000 milliseconds before any response was received.
    at repeat (/workspace/node_modules/google-gax/build/src/normalCalls/retries.js:66:31)
    at Timeout._onTimeout (/workspace/node_modules/google-gax/build/src/normalCalls/retries.js:102:25)
    at listOnTimeout (node:internal/timers:569:17)
    at process.processTimers (node:internal/timers:512:7)

I don't see a way to set the gaxTimeout for firebase functions

Good point. I only updated Pubsub really. Will review them now. Thanks

Hey @jdziek, did updating the packages fixed the issue for you?

Hey all,
We had this timeout issue happening randomly on our Cloud Run instances, that could be mitigated by deploying a new revision. I haven't investigated the root cause, but instinctively it looks like when the Cloud Run instance runs for too long, the connection to pubsub stops working, maybe for something related to auth or long-term gax connections.
We finally managed to fix the issue. We needed to upgrade all google packages, including @google/<something> packages, google-auth-library, google-gax, googleapis-common and other sub-dependencies.
If (like us) you're using yarn v1, here's the list of commands that can help you:

yarn upgrade-interactive: select all google packages and bump them
yarn outdated: see all packages that have new major versions
https://www.npmjs.com/package/yarn-deduplicate to remove duplicates in your lockfile (that's the issue we had, we had olders versions of google packages that were still in our lockfile).
yarn why googleapis-common to check that you have a single version of a lib and that is the latest one

Hope it helped, good luck!

@Kripu77 So far so good. We havnt had issues. Yet. The problem has been so erratic that i cant say for sure yet, but its seems at least like its improved.

With the new configuration the errors seem to be disappeared completely. There were no occurance in the last 2 weeks:
    const publishOptions = {
         gaxOpts: {
             timeout: 540000
         },
     };
    const target = pubsub.topic(rule.topic, publishOptions);

Previously, I did increase the timeout of the cloud function itself, which did not help.

I had this issue as well when publishing messages from a V2 node cloud function and I found a solution which resolved my problem. Increasing the timeout as @bkovari recommended prevented the timeout error from being repeated, but there are a few other things to keep in mind when firing large volumes of messages from cloud functions.
I found that batching messages in large numbers led to the node PubSub client hanging in the Cloud Function's execution and not delivering anything. I resolved this by limiting the batch size to one. This may not be ideal for your use case, but for the vast majority it will be fine:
  const publishOptions = {     batching: {       maxMessages: 1,        maxMilliseconds: 1,     },     gaxOpts: {       timeout: 540000,     },   };
The next thing I can recommend if you are working with JS or TS is resolving the promise from the message send immediately before moving on to your next message send. Allowing numerous unresolved message send promises to build up seems to result in the PubSub client hanging and quietly failing.
Here is the JS code that worked for me:
`import {PubSub} from "@google-cloud/pubsub";
export const pubsubService = {
publishMessage,
};
// initialize pubsub client with token
const pubSubClient = new PubSub();
interface Attributes {
[k: string]: string;
}
/**

Publishes a pubsub message to a topic.

@param {string} topicNameOrId - The name or id of the topic you would like to publish to.
@param {string} data - The data to publish to the topic
@throws {Error} Will throw an error if the message cannot be published.
*/
async function publishMessage(topicNameOrId: string, data: string) {
const publishOptions = {
batching: {
maxMessages: 1,
maxMilliseconds: 1,
},
gaxOpts: {
timeout: 540000,
},
};

// Add two custom attributes, origin and username, to the message
const customAttributes: Attributes = {
"Content-Type": "application/json",
};
const dataBuffer = Buffer.from(data);
try {
const messageId = await pubSubClient
.topic(topicNameOrId, publishOptions)
.publishMessage({
data: dataBuffer,
attributes: customAttributes,
});
console.log(Message ${messageId} published.);
} catch (error) {
console.error(
Received error while publishing: ${(error as Error).message}
);
process.exitCode = 1;
}
}
`
In your cloud function you use the function like so and you should have no more (or at the very least less) problems with message delivery:
       await pubsubService.publishMessage(pubsubTopic, jsonString);


Fixes broken Gelato links

Changes proposed in this pull request
Fixes 'Search payments' broken links to Gelato (API explorer)

In hindsight you're right; I've amended


Add the version to the main index page when built

This was previously part of the task that pushed to Github pages, now part of the task that builds the pages.
Includes a fix to the README docs.

The only concern that I have: If you're working locally then the generated pages will display 'version 1.0.2'. What you will be seeing in the browser is v1.0.2 plus the effects of your local changes. Undecided as to whether that is an issue.

I think we have to bear it in mind for when we next add a feature and make a call on it then.

LGTM :+1: in that case. Preparing to merge…


[Metadata Manager] Introduce Metadata

This commit introduces metadata.ts, which contains functions that manipulate the metadata of file system.
ONE-vscode-DCO-1.0-Signed-off-by: Changho Shin<EMAIL_ADDRESS>

We found a problem that takes too much time to load if there are large model and circle files in user workspace.

Could you make this draft into a PR if it's ready?

good!

This part seems to be a mistake code.
We have never had a problem because of the size of the circle, and when we read the 96MB model, we had a problem, which seems to have been misrepresented here.
Of course, it is true that it takes a long time to read a model or circle file or its capacity if it is too large, but specifying its size limit seems to require a lot of testing.
So for now, it would be better to remove the restrictions.


add Mist load-balancing tags

This PR suppose to reduce cross-region redirect number. After first DNS routing we should mostly keep users in the same region.
Discord

@emranemran @leszko Could you review?


Improve command box

Context
Improve command box input
Pressing "del" key will clear the text field.
Pressing "up" key will replace the string with the previous input.
Pressing "down" key will replace the string with the next input.
Links
fixes #84

I've updated the JUnit Test cases for command box!

LGTM


make browserSync + sourcemaps work

I had the sam issue long time ago
more infos: http://www.browsersync.io/docs/gulp/#gulp-sass-maps

:beers:


Missing reference in scipy.optimize.line_search

At the end of the docs of scipy.optimize.line_search it states
For the zoom phase it uses an algorithm by [...].

Anyone remembers what the [...] stands for, so that we could complete that missing information?
Probably slightly related to https://github.com/scipy/scipy/issues/4257, which addresses some document issues with line_search. That issue is quite old, so I thought I'd open a new issue.

Great, thanks @mckib2!


DBZ-763 CI for MySQL JDBC Driver 5.x backward compatibility

http://ci.hibernate.org/view/Debezium/job/debezium-mysql-jdbc-5.x-test/

@gunnarmorling It will definitely change into testing against older db. But there is still the crucial question I've already asked before. Should we force move users from JDBC 5.x to JDBC 8 or should be there a transitional period?

I'd regard the driver / JDBC version as an implementation detail of the
connector, i.e. it's not something we'd "support" as to be overridden by
the user. That is, if the new driver version is fully compatible with the
older server version, which it is in my understanding.

@jpechane I'm going to close this PR as per our discussion on the chat. Let's re-evaluate later on if needed. Thanks!


SIGTERM doesn't interrupt setup for Redis queue worker

The worker will exit as expected, but only after setup has completed. For models with long setups this is unnecessarily wasteful. With the new worker internals listening for heartbeat events and checking should_exit should be pretty straightforward.

Is this issue still in progress? I am excited to contribute to this project.

Closing, because as of https://github.com/replicate/cog/pull/989, Cog no longer depends on Redis.


[Q]: Bug Template description change?

Description
A brief description of the question or issue:
The current Bug issue description is not apt and descriptive.

Perhaps we can change it to something like- "Report a bug on this repo"
(Also, perhaps we could replace that BUG template with something like: REPO-Issue:report problems or suggestions for the repo)
Why is this essential?
The support repo is often like the entry gates to our community.
Often when some new member notice bugs/issues on the repo, it's like an opportunity to start contributing, but they might be confused due to the lacking issue description.
community info:
You can also join the discord community here
Feel free to check out other cool repositories of Eddie Jaoude Community here

@Vyvy-vi Do you want to work on this?

Yes, I can 👍

Thank you both, great team work 👍


[question] Edit file button

Hello!
I'm using docsify at JS MythBusters, thanks for the project, it looks amazing 😄.
One of my goals of the project is make easy people add more information.
Because every item of the table of contents is mapped with a markdown file, do you think that could be more or less easy add a "Edit file" button (like Wikipedia) and automatically create a PR associated with the project?
Maybe it could be written as a plugin.

Hi, @Kikobeats
We can get the file path through the built-in method, so it's easy to implement. For example
function(hook, vm) {
   const path = window.Docsify.route.parse().path
   console.log(vm.$getFile(path)) // such as `/README.md`
}

Do you want to submit a PR to implement it? 👻

could be good idea place the html for the button and put the href using js?

We can insert html by hook function. See https://github.com/QingWei-Li/docsify/blob/master/docs/index.html#L56 and https://docsify.js.org/#/write-a-plugin

Hi, @Kikobeats
I have added it to the doc, you can refer to this example.
window.$docsify = {
  plugins: [
    function(hook, vm) {
      hook.beforeEach(function (html) {
        var url = 'https://github.com/QingWei-Li/docsify/blob/master' + vm.router.getFile()
        var editHtml = '[📝 EDIT DOCUMENT](' + url + ')\n'

        return editHtml
          + html
          + '\n----\n'
          + 'Last modified {docsify-updated} '
          + editHtml
      })
    }
  ]
}

amazing, let me try it into mythbusters website, thanks!

@QingWei-Li any idea how I can insert int after the first h1 ?

Is it possible to use regular expression?

@QingWei-Li sounds very weird


Formulas do not appear in Datatable when using a Group-by filter

With Vega 2, this should be less of an issue thanks to facet transforms taking an inner pipeline.


Create Unit test for " “What treatment did you receive..” Page

“What treatment did you receive..” Page
a) None -navigate to the “Thank you..” Page
b) Oxygen and fluids -navigate to the “Thank you..” Page
c) Non-invasive ventilation -navigate to the “Thank you..” Page
d) Invasive ventilation -navigate to the “Thank you..” Page
e) Other -navigate optional text filed page
f) Other optional text field - done button

Thank you for creating this issue.
We are reviewing our approach to testing + will update the guidance around testing once that is completed.


Suggestion: Remove dependency on the requests lib

As far as I can tell, requests is only used in one place, to make a simple HTTP GET request:
https://github.com/pypa/flit/blob/ec6e6b7c8994d5e79d0fd951d9e4c26a7c5d9977/flit/validate.py#L55-L57
I suggest replacing that with slightly more complex code using urllib.request(), to remove the dependency (and its sub-dependencies).

Is there a particular issue with the dependency? For flit_core I try to minimise dependencies because it's needed for bootstrapping a lot of the packaging ecosystem, but the flit command line tool is a normal package, and can have dependencies. requests is a very common package that lots of Python software uses.
Requests is also used in the upload module.

There is no particular issue, just seemed like flit should minimize dependencies. I didn't well enough understand the separation of flit_core, and that does make sense.
Since it's also used for a non-trivial POST request, it's probably a good call to continue using requests.


Switch syscall to x/sys

See https://github.com/opencontainers/runc/issues/1364 and  https://github.com/docker/docker/issues/31925
ppc64le is skipping some docker integration tests b/c this is broken. They weren't a big deal at the time but now that we know the root cause it would be great to address it. There may be other things wrong for other architectures as well, since syscall is being replaced by x/sys.
I'm working on this for runc, and then will try to get to docker/docker, so I figured I would open this as well since we know it's affecting ppc64le there as well.

Since Go 1.4, any third-party Go code should start migrating to using the sys package instead of syscall, per the official Go documentation:

NOTE: This package is locked down. Code outside the standard Go repository should be migrated to use the corresponding package in the golang.org/x/sys repository. That is also where updates required by new systems or versions should be applied. See https://golang.org/s/go1.4-syscall for more information.

https://golang.org/pkg/syscall/
Notably, some Linux archs such as mips64, platforms such as Solaris, and others have large sets of functionality that are missing from the syscall package.  Because of the continued use of the syscall package, any new platforms, or those that were not completely supported at the time of Go 1.4 cannot build components such as this one.
/cc @epilatow

@binarycrusader I'm not sure what you're getting at. Not trying to be snarky -- just genuinely confused. :)
I'm working on getting the legal approval (giant corporations have so many rules) to submit this fix, which I've tested on ppc64le. logrus works fine there.

@clnperez not getting at anything; just supportive of the change since I've encountered the same issue on other platforms.  Thanks!


Seeing the precompiled nodegit binary from S3 fail hard on Travis CI (due to Node 7.0?)

It looks like Travis CI is failing when it needs to fetch the precompiled binary for nodegit from S3, not finding a binary it can install and subsequently failing in the full compile attempt. It's entirely possibly that the release of Node v7.0, with the new version of V8, means that the current compilation instructions don't actually work anymore:
This job is running on container-based infrastructure, which does not allow use of 'sudo', setuid and setguid executables.

If you require sudo, add 'sudo: required' to your .travis.yml

See https://docs.travis-ci.com/user/workers/container-based-infrastructure/ for details.

Updating nvm to v0.32.0

3.32s$ nvm install stable

######################################################################## 100.0%

Computing checksum with sha256sum

Checksums matched!

Now using node v7.0.0 (npm v3.10.8)

$ node --version

v7.0.0

$ npm --version

3.10.8

$ nvm --version

0.32.0

490.12s$ npm install

npm WARN deprecated<EMAIL_ADDRESS>cross-spawn no longer requires a build toolchain, use it instead!

><EMAIL_ADDRESS>preinstall /home/travis/build/mozilla/mofo-ui/node_modules/.staging/typechecker-e604c60e

> node ./cyclic.js

><EMAIL_ADDRESS>preinstall /home/travis/build/mozilla/mofo-ui/node_modules/.staging/nodegit-769e0895

> node lifecycleScripts/preinstall

[nodegit] Running pre-install script

[nodegit] Configuring libssh2.

npm WARN prefer global<EMAIL_ADDRESS>should be installed with -g

><EMAIL_ADDRESS>install /home/travis/build/mozilla/mofo-ui/node_modules/node-sass

> node scripts/install.js

Start downloading binary at https://github.com/sass/node-sass/releases/download/v3.10.1/linux-x64-51_binding.node

Binary downloaded and installed at /home/travis/build/mozilla/mofo-ui/node_modules/node-sass/vendor/linux-x64-51/binding.node

><EMAIL_ADDRESS>install /home/travis/build/mozilla/mofo-ui/node_modules/nodegit

[nodegit] Fetching binary from S3.

[nodegit] Failed to install prebuilt binary:

{ Error: Command failed: "/home/travis/build/mozilla/mofo-bootstrap/node_modules/nodegit/node_modules/.bin/node-pre-gyp" install --fallback-to-build=false

node-pre-gyp ERR! install error

Experiencing the same issue with Node v7.0, can reproduce on my Linux laptop as well.

Just build one from source for now. I believe node 7 bindings are not yet out.

"Just" implies that's easily done. Travis CI breaks when it tries the recompile. There is no "just" here, unfortunately. (There is almost never a "just", in fact. It's really a word to leave on the shelf in almost every instance it's used =)

Further details:
CXX(target) Release/obj.target/nodegit/src/annotated_commit.o
../src/annotated_commit.cc: In member function ‘virtual void GitAnnotatedCommit::FromFetchheadWorker::HandleOKCallback()’:
../src/annotated_commit.cc:266:48: error: ‘class v8::Object’ has no member named ‘GetHiddenValue’
         Local<v8::Value> checkValue = nodeObj->GetHiddenValue(Nan::New("NodeGitPromiseError").ToLocalChecked());
                                                ^
../src/annotated_commit.cc: In member function ‘virtual void GitAnnotatedCommit::FromRefWorker::HandleOKCallback()’:
../src/annotated_commit.cc:430:48: error: ‘class v8::Object’ has no member named ‘GetHiddenValue’
         Local<v8::Value> checkValue = nodeObj->GetHiddenValue(Nan::New("NodeGitPromiseError").ToLocalChecked());
                                                ^
../src/annotated_commit.cc: In member function ‘virtual void GitAnnotatedCommit::FromRevspecWorker::HandleOKCallback()’:
../src/annotated_commit.cc:599:48: error: ‘class v8::Object’ has no member named ‘GetHiddenValue’
         Local<v8::Value> checkValue = nodeObj->GetHiddenValue(Nan::New("NodeGitPromiseError").ToLocalChecked());
                                                ^
../src/annotated_commit.cc: In member function ‘virtual void GitAnnotatedCommit::LookupWorker::HandleOKCallback()’:
../src/annotated_commit.cc:815:48: error: ‘class v8::Object’ has no member named ‘GetHiddenValue’
         Local<v8::Value> checkValue = nodeObj->GetHiddenValue(Nan::New("NodeGitPromiseError").ToLocalChecked());
                                                ^
nodegit.target.mk:235: recipe for target 'Release/obj.target/nodegit/src/annotated_commit.o' failed

@mathieumg It seems that you've run into https://github.com/nodejs/nan/issues/587

Yeah, did a bit more research, seems like this is wrapped here https://github.com/nodegit/nodegit/commit/38bcb340bbac946d3a389d3b13ffe1f62f0958b6#diff-6ac095c73b0cfbf7cd78ee2fd594fdc8 Didn't realize there was a breaking release (0.16) with that change, trying it at the moment.

This should fix #1157 as well.

Node 7 is EOF. Issue can be closed.

Node 7 is not an LTS version. The current LTS versions of Node are 6 and 8, with Node 6 being an in-support LTS version until April 2019.


fix(modelgen-js): use optional fields for FK in manyToMany

Description of changes

Use optional fields for manyToMany foreign key fields. This only has impact on the JS modelgen since in native platforms FK fields are always removed.
Issue #, if available

Description of how you validated changes
Checklist

[x] PR description included
[x] yarn test passes
[x] Tests are changed or added
[x] Relevant documentation is changed or added (and PR referenced)
[ ] Breaking changes to existing customers are released behind a feature flag or major version update
[x] Changes are tested using sample applications for all relevant platforms (iOS/android/flutter/Javascript) that use the feature added/modified

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

Merging #462 (459d5fe) into main (3a4611c) will increase coverage by 0.69%.
The diff coverage is 97.98%.

@@            Coverage Diff             @@
##             main     #462      +/-   ##
==========================================
+ Coverage   85.19%   85.88%   +0.69%
==========================================
  Files         142      143       +1
  Lines        6695     7065     +370
  Branches     1727     1852     +125
==========================================
+ Hits         5704     6068     +364
- Misses        900      906       +6
  Partials       91       91

Impacted Files
Coverage Δ

...sync-modelgen-plugin/src/utils/process-has-many.ts
87.93% <88.23%> (-2.98%)
:arrow_down:

...psync-modelgen-plugin/src/utils/process-has-one.ts
96.00% <94.73%> (+11.38%)
:arrow_up:

...-plugin/src/visitors/appsync-typescript-visitor.ts
80.29% <97.77%> (+4.56%)
:arrow_up:

...nc-modelgen-plugin/src/visitors/appsync-visitor.ts
96.15% <98.13%> (+0.67%)
:arrow_up:

packages/amplify-codegen/src/commands/models.js
84.74% <100.00%> (+0.13%)
:arrow_up:

...appsync-modelgen-plugin/src/configs/java-config.ts
100.00% <100.00%> (ø)

...en-plugin/src/languages/swift-declaration-block.ts
90.44% <100.00%> (ø)

...ges/appsync-modelgen-plugin/src/utils/constants.ts
100.00% <100.00%> (ø)

...es/appsync-modelgen-plugin/src/utils/fieldUtils.ts
100.00% <100.00%> (ø)

...nc-modelgen-plugin/src/utils/process-belongs-to.ts
93.93% <100.00%> (+4.28%)
:arrow_up:

... and 11 more

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more


feat: Updates pamqp to 3.0.1

This PR updates pamqp to the current version, which is 3.0.1, which makes this library compatible with the most current version of aiormq.

Merged PR with more recent main branch https://github.com/nikitanovosibirsk/amqp-mock/pull/25


We don't need this modification anymore as we don't use native parsers

Hey @Robert-Frampton, I assume this superseeds https://github.com/metal/metal-uri/pull/27?

I didn't see that you had sent this already, but yeah, let's go with this one since it matches what we did for metal-drag-drop.

done! 😉


Prepared Release

Hey @Acconut, I thought about making the jump to 1.0.0
I prepared everything and a little explanation in the changelog. Need a final testing and approval from your side.
Production tests and suggestions from your side are welcome :)

@Acconut revised the changelog. How is it about now.


Session Service Deprecation in Symfony 5.3 and removal in 6

SessionInterface got deprecated in 5.3 and removed as public service in 6
RequestStack shoudl be used instead
https://symfony.com/blog/new-in-symfony-5-3-session-service-deprecation

Hi @olia-bn  - would you be able to submit a PR with a proposed change to support this deprecation?

#166 is fixing this issue.

I will have a look at it next week :)

Oh, already merged haha.


Remove hash sign from code

The code sample included an errant hash sign. This is causing the remaining code sample to appear as a remark.
Please remove.

@themichaelbender : Thanks for your contribution! The author(s) have been notified to review your proposed change.

The changes looks good. Please merge. #sign-off


Fix rpm build

I see, you made things much nicer with specfile and venv options that I didn't find!
Just: we now have two copies of the service file. Maybe a symlink is better suited there?

Just: we now have two copies of the service file. Maybe a symlink is better suited there?

Yes, I'll fix that once this branch is merged. I'm still trying to fix the selinux issue. You mentioned you had this problem too, any idea how to fix that in a clean way, i.e. which booleans to enable?

No, I didn't even notice an selinux issue.  I usually disabel it immediatly :(
Merge as you wish.


No Auth Provider found for name "gcp"

Adding the following import should allow for GCP clients:
_ "k8s.io/client-go/plugin/pkg/client/auth/gcp"

I imagine other managed kubernetes services would appreciate having the auth packages included for them too :)

Actually the following import would be better, it should import all providers then:
_ "k8s.io/client-go/plugin/pkg/client/auth"


Update Releasing section of README

What
Update the documentation for creating a release.
Why
With the Github Actions workflow added in dcc1d69, release binaries are now generated automatically when a tag is pushed, so the documentation should reflect this.
@raymondberg

@raymondberg - sure, though I'm realizing that, if for any reason you need to generate the binaries locally, you will still the script. Gonna leave it for now but we can revisit when and if need be.

Looking again and it's actually script/release.sh that we shouldn't need anymore.

Okay, ended up just removing a GNUMakefile dependency so that binaries can still be manually generated and so that the CHANGELOG generation is also preserved as the last step before publishing that release.


updated workflow to test upload to s3 bucket

In this PR I added uploading to S3 as a workflow step. While we're testing, I also limited the number of urls for which to run the analysis

Woohoo


[WIP] add unit tests for Layout

I'm trying to add unit tests for kivy.uix.layout.Layout but the tests fails. Could someone tell me the tests itself is incorrent? or layout_hint_with_bounds() behaves incorrectly?

Eh!? 🤪

@matham thank you. Sorry for the silly mistake. I'll fix that PEP8 violation.

@matham If you are ok with it, I'm going to squash all commits two commits (I mean git push -f), cuz I think some commits aren't worth leaving.

@matham I'm satisfied with the current state of the PR. Can you check if it's ok or not?

Hey, sorry. I have not been having any time to work on kivy stuff beyond managing issues. Please feel free to merge.


Annotate output matrices with salmon effective length

DESeqDataSetFromTximport requires both counts and effective length. While PISCES uses effective length internally for DESeq2, it does not write out the values for use in downstream analysis.

@marrolu1

@lucamarrocco - it looks like I already implemented this feature as part of 9f7ba0ee8d0a3b4faf41f0b3de0389d050e01a49, and I just need to do integration testing and push a new release.


Basic init setup fails to run Storybook [React, Typescript]

Hi!
Problem:
Storybook fails to start when doing the basic example with React and Typescript.
System 1 (I tried in 2 laptops)

Node v16.13.0
npm v8.1.0
macOS Monterey v12.0.1

System :

Node  v14.15.4
npm 7.6.0
macOS Big Sur v11.6

Steps

npm init vite@latest
Choose react and react-ts
cd into the project and run npm install && npm run dev: it all works fine
Now run npx sb@next init --builder storybook-builder-vite (this works fine)
Now run npm run storybook

Error
Storybook crashes, logs:
> npm run storybook

><EMAIL_ADDRESS>storybook
> start-storybook -p 6006

info @storybook/react v6.4.0-beta.26
info
info => Loading presets
info => Using prebuilt manager
Pre-bundling dependencies:
  react
  react-dom
  @base2/pretty-print-object
  @emotion/core
  @emotion/is-prop-valid
  (...and 70 more)
(this will be run only when your dependencies or config have changed)
╭────────────────────────────────────────────────────╮
│                                                    │
│   Storybook 6.4.0-beta.26 for React started        │
│   648 ms for preview                               │
│                                                    │
│    Local:            http://localhost:6006/        │
│    On your network:  http://<IP_ADDRESS>:6006/    │
│                                                    │
╰────────────────────────────────────────────────────╯
Sourcemap for<EMAIL_ADDRESS>points to missing source files
Sourcemap for<EMAIL_ADDRESS>points to missing source files
Sourcemap for<EMAIL_ADDRESS>points to missing source files
[BABEL] Note: The code generator has deoptimised the styling of /Users/amina/Documents/Projects/vite-app/node_modules/.vite/chunk-MHXLKORO.js?v=e8b96788 as it exceeds the max of 500KB.
Sourcemap for "/Users/amina/Documents/Projects/vite-app/node_modules/.vite/chunk-AKDXD5X2.js" points to missing source files
Sourcemap for "/Users/amina/Documents/Projects/vite-app/node_modules/.vite/regenerator-runtime_runtime_js.js" points to missing source files
Sourcemap for "/Users/amina/Documents/Projects/vite-app/node_modules/.vite/react.js" points to missing source files
Sourcemap for<EMAIL_ADDRESS>points to missing source files
Sourcemap for "/Users/amina/Documents/Projects/vite-app/node_modules/.vite/chunk-PHBKWG7E.js" points to missing source files
Sourcemap for "/Users/amina/Documents/Projects/vite-app/node_modules/.vite/chunk-35YZ7F37.js" points to missing source files
Sourcemap for "/Users/amina/Documents/Projects/vite-app/node_modules/.vite/ts-dedent.js" points to missing source files
Sourcemap for<EMAIL_ADDRESS>points to missing source files
Sourcemap for "/Users/amina/Documents/Projects/vite-app/node_modules/.vite/chunk-JQLGGSPV.js" points to missing source files
Sourcemap for "/Users/amina/Documents/Projects/vite-app/node_modules/.vite/util-deprecate.js" points to missing source files
Sourcemap for "/Users/amina/Documents/Projects/vite-app/node_modules/.vite/prop-types.js" points to missing source files
Sourcemap for "/Users/amina/Documents/Projects/vite-app/node_modules/.vite/global.js" points to missing source files
Sourcemap for "/Users/amina/Documents/Projects/vite-app/node_modules/react-element-to-jsx-string/dist/esm/index.js" points to missing source files
Sourcemap for<EMAIL_ADDRESS>points to missing source files
Sourcemap for "/Users/amina/Documents/Projects/vite-app/node_modules/.vite/qs.js" points to missing source files
node:events:368
      throw er; // Unhandled 'error' event
      ^

RangeError: Invalid WebSocket frame: invalid status code 56853
    at Receiver$1.controlMessage (/Users/amina/Documents/Projects/vite-app/node_modules/vite/dist/node/chunks/dep-85dbaaa7.js:50517:18)
    at Receiver$1.getData (/Users/amina/Documents/Projects/vite-app/node_modules/vite/dist/node/chunks/dep-85dbaaa7.js:50385:42)
    at Receiver$1.startLoop (/Users/amina/Documents/Projects/vite-app/node_modules/vite/dist/node/chunks/dep-85dbaaa7.js:50110:22)
    at Receiver$1._write (/Users/amina/Documents/Projects/vite-app/node_modules/vite/dist/node/chunks/dep-85dbaaa7.js:50045:10)
    at writeOrBuffer (node:internal/streams/writable:389:12)
    at _write (node:internal/streams/writable:330:10)
    at Receiver$1.Writable.write (node:internal/streams/writable:334:10)
    at Socket.socketOnData (/Users/amina/Documents/Projects/vite-app/node_modules/vite/dist/node/chunks/dep-85dbaaa7.js:52459:37)
    at Socket.emit (node:events:402:35)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at Socket.Readable.push (node:internal/streams/readable:228:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:199:23)
Emitted 'error' event on WebSocket$2 instance at:
    at Receiver$1.receiverOnError (/Users/amina/Documents/Projects/vite-app/node_modules/vite/dist/node/chunks/dep-85dbaaa7.js:52346:13)
    at Receiver$1.emit (node:events:390:28)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  code: 'WS_ERR_INVALID_CLOSE_CODE',
  [Symbol(status-code)]: 1002

This is a great project btw! A time saver for sure. I can't figure out what's going on, I appreciate any suggestions.

Can you try an rm -rf node_modules && npm install and try again?  I wonder if something got into a funky state.

@IanVS thanks for the reply. I just tried it but no luck.

I was having a similar issue, then I did this:
// .storybook/main.js
async viteFinal(config) {
    config.server.port = 6001;
    config.server.https = false;
    config.server.host = true;
    config.server.hmr =  {
      port: 443,
      protocol: 'ws'
    }

    return config
  }
}

and it solved for me (reference https://github.com/vitejs/vite/pull/1926#issuecomment-774728965)

Hmmmm, that's still a little strange, and not something that I needed to configure thus far...
Which version of vite are you using?  And after reinstalling node modules, do you still see the missing sourcemap messages?

The latest beta has moved from SSE to websockets for conveying Story index changes from node to the browser, so that's probably what's different cc @tmeasday

@fsblemos that worked! Thank you 😄
For the record, these are the deps in my package.json
  "devDependencies": {
    "@babel/core": "^7.16.0",
    "@storybook/addon-actions": "^6.4.0-beta.26",
    "@storybook/addon-essentials": "^6.4.0-beta.26",
    "@storybook/addon-links": "^6.4.0-beta.26",
    "@storybook/react": "^6.4.0-beta.26",
    "@types/react": "^17.0.0",
    "@types/react-dom": "^17.0.0",
    "@vitejs/plugin-react": "^1.0.0",
    "babel-loader": "^8.2.3",
    "storybook-builder-vite": "^0.1.5",
    "typescript": "^4.3.2",
    "vite": "^2.6.4"
  }

vite version installed 2.6.13
storybook-builder-vite version installed 0.1.5
I'll close the thread because my specific problem is solved. Thank you all.

Ok it sounds like there is a conflict between SB's websocket and Vite's @IanVS. Would it just make sense to change the default port that Vite uses like @fsblemos has done here in the builder?

Yes, it sounds like that is what we should do.

I posted a similar bug over on Storybook — https://github.com/storybookjs/storybook/issues/16616 — and got sent over here. Pleased to say the fix in this thread solved my problem too.

Hello! I was getting;
RangeError: Invalid WebSocket frame: invalid status code 65000 thrown by events.js:377, so a slightly different code.
Fix also worked for me, without configType === 'DEVELOPMENT'

Would it be possible to solve this issue without changing the Websocket port?
// pseudocode just to illustrate
var viteWS = new WebSocket("ws://localhost:6006/vite");
var storybookWS = new WebSocket("ws://localhost:6006/storybook");

viteWS.onmessage = function (event) {
  alert("Some Vite HMR stuff going on");
};

storybookWS.onmessage = function (event) {
  alert("Some Storybook story index update");
};

It should be possible to differentiate between the websockets based on their URL path?

We could make that change in Storybook if it helps, but it sounds like the Vite builder needs to make a change anyway 🤷

I've got a PR ready to go, but then I realized that we'll need to change the config to ws only for certain versions of storybook, right?  Is there any kind of flag that we can use to determine this, so we don't need to parse the version string from options.packageJson.version and apply semver rules to it to determine whether or not we should use websockets?
I guess the other alternative would be to make a new minor (breaking) release of storybook-builder-vite that requires storybook 6.4.0-beta.? (when was the change to ws exactly?).

I mentioned it in #144 - is it possible to push back the SSE->Websocket change in Storybook until 6.5 (or put it behind a feature flag) which could buy us some time fo figure out how to solve this?

I've got a PR ready to go, but then I realized that we'll need to change the config to ws only for certain versions of storybook, right? Is there any kind of flag that we can use to determine this, so we don't need to parse the version string from options.packageJson.version and apply semver rules to it to determine whether or not we should use websockets?

Hmm, I don't think I'm quite following. Here's how I understand the situation, please forgive me if I have it wrong:

For HMR, Vite uses a websocket, by default on the same port as the (HTTP) webserver
In 6.4, in v7 mode, Storybook also uses a websocket, on the same port as the (HTTP) webserver
Unsurprisingly that causes a problem as the sockets are conflicting with each other.

The solution as suggested on this thread is just to change Vite's configuration to use a different port for (1).
I'm not really sure why you wouldn't just do that all the time @IanVS? Is there any reason why you'd only want to do it for situations where there'd otherwise be a conflict?

I mentioned it in #144 - is it possible to push back the SSE->Websocket change in Storybook until 6.5 (or put it behind a feature flag) which could buy us some time fo figure out how to solve this?

Hmm, not really, the SSE solution really didn't work and caused SB to hang if you had more than 1 tab open to the same Storybook. This is all behind the v7 store feature flag already, but that probably doesn't help much because that's the ideal mode for Vite.
Maybe ping me on Discord? I can make any changes we need on the SB end pretty easily (ie. only serving the SB websocket on a path, maybe?)

Oh, now I just read the comment from https://github.com/eirslett/storybook-builder-vite/pull/144#issuecomment-964033992

One of the limitations of going this route is that it's hardcoded to port 443, would could be used for something else, there is no port conflict resolution like in Storybook (suggests using port 6007 whenever port 6006 is busy) etc., also you won't be able to run two Storybooks at the same time locally. (Maybe not such a big deal?)

Can someone tell me how to reproduce the problem? I can take a look at what we can do without changing the Vite configuration.

@tmeasday I was able to reproduce this by just creating a new vite project and adding storybook:
npm init vite@latest
npx sb@next init --builder storybook-builder-vite

Fix here: https://github.com/storybookjs/storybook/pull/16665

Fix here: storybookjs/storybook#16665

I haven't tried it, but it looks to me like this should be an acceptable solution! Not sure if we also have to make any changes on the Storybook-builder-vite side, we'll have to investigate that a bit.

From my experiments you should be OK without changes but LMK.

OMG... thank you for fixing this. I just ran 6.4.0-beta.33 and can confirm the hot reload is fixed :)

Storybook 6.4.0-rc.0 has just been released, and I've verified that it works with the vite-builder out of the box with no config changes needed.  So for those in this thread, I suggest you revert the changes you made to your viteFinal, and upgrade to the release candidate.  I'll close this out, but feel free to open a new issue if you continue to have problems with the latest version of storybook.


improve file exclusion

resolves #226

@carlosas Could you prove this is working for you?

Working well 👍


Validation not implemented in Spark

When this will be implemented?

This is not something that is immediate on my plan, but that said it is possible to implement by following the structure layed out in FhirController and FhirService class by using the https://github.com/FirelyTeam/fhir-net-api.
There are examples on how to do validation in the fhir-net-api and also a Validation Demo

Just to make sure, which of the following validation are implemented?
https://hl7.org/fhir/R4/validation.html
Structure: Check that all the content in the resource is described by the specification, and nothing extra is present
Cardinality: Check that the cardinality of all properties is correct (min & max)
Value Domains: Check that the values of all properties conform to the rules for the specified types (including checking that enumerated codes are valid)
Coding/CodeableConcept bindings: Check that codes/displays provided in the Coding/CodeableConcept types are valid
Invariants: Check that the invariants (co-occurrence rules, etc.) have been followed correctly
Profiles: Check that any rules in profiles have been followed (including those listed in the Resource.meta.profile, or in CapabilityStatement, or in an ImplementationGuide, or otherwise required by context)
Questionnaires: Check that a QuestionnaireResponse is valid against its matching Questionnaire
Business Rules: Business rules are made outside the specification, such as checking for duplicates, checking that references resolve, checking that a user is authorized to do what they want to do, etc.

The validation that exists in Spark today is enforced by the xml and json parsers and will validate Structure, Cardinality and Value Domains.


Issue with #2266 and fix within mainline

[Issue]
I explicitly asked for and am responsible for the implementation of #2266.
After some discussion we decided to find the root cause instead of hiding it.
Let's see how to fix it properly. Thanks!
[Content]
more on   https://github.com/ROCmSoftwarePlatform/MIOpen/pull/2266

I think we all agree that the problem is caused by a broken solver

No, after talking to @shurale-nkn I think that it is caused by incorrect usage of our internal APIs from outside of MIOpen.

this change should stay as an ugly but reasonable "hotfix"

This is not a fix for two reasons.

There is no error in GenericSearch. It correctly reports that it's been provided incorrect data for whatever reason. As a method of error reporting, a log line is printed and an exception is thrown. The exception is handled by find and the solver is skipped as invalid for this case.
This doesn't fix error. It propagates error further to GetSolution. Is returning a default-constructed performance config when solver says no config is valid considered a valid practice? I don't think so. The only way to know that such practice is acceptable is to somehow fetch info from the solver itself. As of now it just assumes that this way of handling is correct (which it is likely not).

@junliume My advice is to revert the PR and to find what causes the error. I think that it is incorrect usage by MIGraphX, but I am not sure. Maybe, there is some undocumented UB in our internals (which is not good), but that's kind of why they are internals and not public API. I am 100% sure that the only change that may be made to GenericSearch from the report is to add
if (n_runs_total == 0)
    MIOPEN_THROW(...);

at the begfining of the function. But the only effect it would have is skipping 0 config space size warnings before the error.
Originally posted by @DrizztDoUrden in https://github.com/ROCmSoftwarePlatform/MIOpen/issues/2266#issuecomment-1643903706

@DrizztDoUrden @atamazov @shurale-nkn @dmikushin
Indeed it seems there are consequences if we tolerate n_runs_total == 0 without throwing errors:
Now the run fails with:
[2023-07-23T11:45:17.673Z] MIOpen(HIP): Error [Compile] 'hiprtcCompileProgram(prog.get(), c_options.size(), c_options.data())' static_kernel_gridwise_convolution_forward_implicit_gemm_v4r4_xdlops_nchw_kcyx_nkhw.cpp: HIPRTC_ERROR_COMPILATION (6)
[2023-07-23T11:45:17.673Z] MIOpen(HIP): Error [BuildHip] HIPRTC status = HIPRTC_ERROR_COMPILATION (6), source file: static_kernel_gridwise_convolution_forward_implicit_gemm_v4r4_xdlops_nchw_kcyx_nkhw.cpp
[2023-07-23T11:45:17.673Z] MIOpen(HIP): Warning [BuildHip] In file included from /tmp/comgr-8fd95e/input/static_kernel_gridwise_convolution_forward_implicit_gemm_v4r4_xdlops_nchw_kcyx_nkhw.cpp:2:
[2023-07-23T11:45:17.673Z] In file included from /tmp/comgr-8fd95e/include/static_kernel_gridwise_convolution_forward_implicit_gemm_v4r4_xdlops_nchw_kcyx_nkhw.hpp:8:
[2023-07-23T11:45:17.673Z] In file included from /tmp/comgr-8fd95e/include/static_kernel_gridwise_gemm_xdlops_fp16_bfp16.hpp:10:
[2023-07-23T11:45:17.673Z] In file included from /tmp/comgr-8fd95e/include/static_kernel_blockwise_gemm_xdlops.hpp:6:
[2023-07-23T11:45:17.673Z] /tmp/comgr-8fd95e/include/static_kernel_xdlops_gemm.hpp:1018:43: error: function 'GetXdlopsInfo<_Float16, 4, 4>' with deduced return type cannot be used before it is defined
[2023-07-23T11:45:17.673Z]     static constexpr index_t MRepeats   = GetXdlopsInfo().MRepeats;
[2023-07-23T11:45:17.673Z]                                           ^

We should not have these tuning parameter GetXdlopsInfo<_Float16, 4, 4> to begin with, it could either:

MIOpen host/API did not filter out these invalid parameters before passing it to compiler to compiler?
MIOpen is forced to bypass "inapplicable()" check.

Could you please provide some opinions? Thanks!

@junliume

MIOpen host/API did not filter out these invalid parameters before passing it to compiler to compiler?

MIOpen performs validation, but it uses Solver's functionality for this. If a Solver has issues (for example, IsValidPerformanceConfig is incorrect), then wrong performanceConfig may sneak. Note that the reasons of failing compilation are numerous (e.g. error in the kernel source, error in the compiler, and different errors in the solver).

MIOpen is forced to bypass "inapplicable()" check.

Perhaps this is possible via the Find 2.0 API? @DrizztDoUrden, am I right?

[Informative] Please consider this as a continuation of #2253 (which is not really fixed yet).

🔴 In order to find the root cause we need clear reproduction instructions.

I am working on a minimal reproduce example.

@atamazov the log is attached.
pt_resnet50v1.5_pruned0.4_1.2_M_float.log
However, when I tried to run:
MIOPEN_FIND_ENFORCE=3 ./bin/MIOpenDriver convfp16 -n 1 -c 256 -H 56 -W 56 -k 20 -y 1 -x 1 -p 0 -q 0 -u 1 -v 1 -l 1 -j 1 -m conv -g 1 -F 1 -t 1 -S 0

I do not get similar output as shown in the log.
root@zt-dh170-16:/opt/rocm# MIOPEN_FIND_ENFORCE=3 ./bin/MIOpenDriver convfp16 -n 1 -c 256 -H 56 -W 56 -k 20 -y 1 -x 1 -p 0 -q 0 -u 1 -v 1 -l 1 -j 1 -m conv -g 1 -F 1 -t 1 -S 0
MIOpenDriver convfp16 -n 1 -c 256 -H 56 -W 56 -k 20 -y 1 -x 1 -p 0 -q 0 -u 1 -v 1 -l 1 -j 1 -m conv -g 1 -F 1 -t 1 -S 0
Forward Conv solutions available: 2
- id: 88 algo: 0, time: 11.1111 ms, ws: 0, name: GemmFwd1x1_0_1
- id: 85 algo: 1, time: 1000 ms, ws: 0, name: ConvDirectNaiveConvFwd
MIOpen Forward Conv. Algorithm: 0, Solution: 88/GemmFwd1x1_0_1
GPU Kernel Time Forward Conv. Elapsed: 0.029036 ms (average)
stats: name, n, c, ho, wo, x, y, k, flopCnt, bytesRead, bytesWritten, GFLOPs, GB/s, timeMs
stats: fwd-conv1x1u1, 1, 256, 56, 56, 1, 1, 20,  32112640, 1615872, 125440, 1106, 60, 0.029036
Forward Convolution Verifies OK on CPU reference (0.000258885)

in the attached log:
MIOpen(HIP): Command [LogCmdFindConvolution] ./bin/MIOpenDriver convfp16 -n 1 -c 256 -H 56 -W 56 -k 20 -y 1 -x 1 -p 0 -q 0 -u 1 -v 1 -l 1 -j 1 -m conv -g 1 -F 1 -t 1 -S 0
MIOpen(HIP): Warning [SearchImpl] Searching the best solution in the 4 dim space. Please, be patient...

MIGrpahX uses find 2.0 API. There can be a bug in the API implementation.
I'll switch to this and try to debug it. It may reproduce by test_conv_2d_find2, unless it's either incorrect usage (I haven't studied 100% of MIGraphX source related to find 2.0, so I can't vouch for whether it is good or not) or our test missises some options. Which it kind of does, we only try to cover net configs there. That is not the only info the library gets from user.
Also, the MIGraphX command has resnet50_model.onnx. I am not familiar with these formats and MIGraphX caching, but if solution is loaded, IsApplicable is likely to be skipped. Maybe we should attach MIOpen version to solution and force testing IsApplicable for such cases. It would be better to attach specific solver versions or even IsApplicable version, but it would be hard to enforce developers actually incrementing them on every meaningful change.

@DrizztDoUrden yes I guess "IsApplicable is likely to be skipped" is the problem.
Here is another reproducible example:
./bin/MIOpenDriver convfp16 -n 4 -c 88 -H 20 -W 20 -k 336 -y 1 -x 1 -p 0 -q 0 -u 1 -v 1 -l 1 -j 1 -m conv -g 1 -F 1 -t 1 -S 64

Warning: Solution id (64) is not reported by the library. Trying it anyway...
MIOpen Forward Conv. Algorithm: -1, Solution: 64/ConvHipImplicitGemmForwardV4R4Xdlops
GPU Kernel Time Forward Conv. Elapsed: 0.013387 ms (average)
stats: name, n, c, ho, wo, x, y, k, flopCnt, bytesRead, bytesWritten, GFLOPs, GB/s, timeMs
stats: fwd-conv1x1u1, 4, 88, 20, 20, 1, 1, 336,  94617600, 340736, 1075200, 7068, 106, 0.013387
Forward Convolution FAILED: 0.183908 > 0.0082

Another issue with #2266  (again my mistake :( ) is
when n_runs_tota==0, we should not execute the lines below:
https://github.com/ROCmSoftwarePlatform/MIOpen/blob/7264cb80f5a75d2edc57ea8f3557af10a10ea2ff/src/include/miopen/generic_search.hpp#L556-L565

Is there an env var where we can use FIND 2.0 path? @DrizztDoUrden

I don't understand the question, find 2.0 in develop calls find 1.0 internally. There no find 2.0 specific files or paths.
If you want to execute the config from command line - the only way is to use test_conv*_find2

@DrizztDoUrden yes I guess "IsApplicable is likely to be skipped" is the problem.
Here is another reproducible example: when "ConvHipImplicitGemmForwardV4R4Xdlops" solver is used explicitly by passing "-S 64" to the driver:
./bin/MIOpenDriver convfp16 -n 4 -c 88 -H 20 -W 20 -k 336 -y 1 -x 1 -p 0 -q 0 -u 1 -v 1 -l 1 -j 1 -m conv -g 1 -F 1 -t 1 -S 64

Warning: Solution id (64) is not reported by the library. Trying it anyway...
MIOpen Forward Conv. Algorithm: -1, Solution: 64/ConvHipImplicitGemmForwardV4R4Xdlops
GPU Kernel Time Forward Conv. Elapsed: 0.013387 ms (average)
stats: name, n, c, ho, wo, x, y, k, flopCnt, bytesRead, bytesWritten, GFLOPs, GB/s, timeMs
stats: fwd-conv1x1u1, 4, 88, 20, 20, 1, 1, 336,  94617600, 340736, 1075200, 7068, 106, 0.013387
Forward Convolution FAILED: 0.183908 > 0.0082

This looks like a bug in ConvHipImplicitGemmForwardV4R4Xdlops. It has produced a wrong result via immediate mode.

@junliume The problems in the core tuning algorithm are resolved now, and I don't see what else needs to be done to resolve this issue.  Could you please review this ticket, and close it or denote unresolved sub-issues. Thank you.


Fix tests on main branch

1 import * as Earthstar from<EMAIL_ADDRESS>                             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/server/sync_test.ts:12:2 - error TS2304: Cannot find name 'Deno'.

Maybe I can remove the dependency on Deno?

Hm, looking into Deno a bit it sounds like the main drawback for me at this point is it doesn't handle types for npm  modules well, but it also sounds like this was fixed in v1.27. I'll give it a go!

Installing the Deno extension for VSCode and running Deno:enable like in https://github.com/denoland/vscode_deno fixed the errors


What type of server is used?

It showed acess denied when I tried setting up a db, what db is used, and do you need to create one?
(sorry if this sounds weird im a beginner)

（警告：这是翻译，可能不正确）
固定。


[FEATURE] Ability to more easily exit from continuous detection creation mode

It's not clear that you need to press 'esc' to get out of continuous detection creation mode, needs to say that somewhere predominately or there needs to be another method to do this that is user intuitive

just a quick test-sample of what could potentially be in place to indicate creation modes as well as all the other modes that a user could be in.  This isn't perfect yet (it doesn't update after creation, but I think the idea is there).  This is where an escape button could be used to cancel creation mode or cancel the current editing mode as well.
https://user-images.githubusercontent.com/61746913/152044762-89ec8c10-36e2-4569-b045-9f5d9514e250.mp4


Expose some is_*_available utils in docs

This PR adds the following to the documentation:

is_bf16_available
is_torch_version
is_tpu_available

My motivation for this is while they're mildly small utility functions, they have more usecases than just being used internally. Especially the is_bf16_available, as pytorch's own check isn't shown in their general documentation so many people did not even know it existed!
I'm 50/50 on if we want to show is_torch_version, but bare minimum is_tpu_available and is_bf16_available are a must IMO as they have different underlying checks than just "is this package installed"

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


flit install fails

running flit install on  f2967a57ee78c5ce194e0c02cf9c3806ea56ea20 (https://github.com/matthew-brett/xibabel/pull/6)
results in
Building wheels for collected packages: xibabel
  Building wheel for xibabel (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for xibabel (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [26 lines of output]
      Unexpected names under [project]: home-page
      Traceback (most recent call last):
        File "/home/pi/.pyenv/versions/3.11.0/envs/xib2/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/pi/.pyenv/versions/3.11.0/envs/xib2/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/pi/.pyenv/versions/3.11.0/envs/xib2/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-o20hyr_3/overlay/lib/python3.11/site-packages/flit_core/buildapi.py", line 72, in build_wheel
          info = make_wheel_in(pyproj_toml, Path(wheel_directory))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-o20hyr_3/overlay/lib/python3.11/site-packages/flit_core/wheel.py", line 224, in make_wheel_in
          wb.build(editable)
        File "/tmp/pip-build-env-o20hyr_3/overlay/lib/python3.11/site-packages/flit_core/wheel.py", line 210, in build
          self.copy_module()
        File "/tmp/pip-build-env-o20hyr_3/overlay/lib/python3.11/site-packages/flit_core/wheel.py", line 164, in copy_module
          self._add_file(full_path, rel_path)
        File "/tmp/pip-build-env-o20hyr_3/overlay/lib/python3.11/site-packages/flit_core/wheel.py", line 111, in _add_file
          zinfo = zipfile.ZipInfo.from_file(full_path, rel_path)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/pi/.pyenv/versions/3.11.0/lib/python3.11/zipfile.py", line 528, in from_file
          st = os.stat(filename)
               ^^^^^^^^^^^^^^^^^
      FileNotFoundError: [Errno 2] No such file or directory: 'src/xibabel/testing/ds000009/derivatives/mriqc/aMRIQC.csv'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for xibabel
Failed to build xibabel
ERROR: Could not build wheels for xibabel, which is required to install pyproject.toml-based projects

[notice] A new release of pip is available: 23.3.1 -> 23.3.2
[notice] To update, run: python -m pip install --upgrade pip
Traceback (most recent call last):
  File "/home/pi/.pyenv/versions/xib2/bin/flit", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/pi/.pyenv/versions/3.11.0/envs/xib2/lib/python3.11/site-packages/flit/__init__.py", line 219, in main
    installer.install()
  File "/home/pi/.pyenv/versions/3.11.0/envs/xib2/lib/python3.11/site-packages/flit/install.py", line 432, in install
    self.install_with_pip()
  File "/home/pi/.pyenv/versions/3.11.0/envs/xib2/lib/python3.11/site-packages/flit/install.py", line 370, in install_with_pip
    check_call(cmd, shell=shell)
  File "/home/pi/.pyenv/versions/3.11.0/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/pi/.pyenv/versions/3.11.0/envs/xib2/bin/python', '-m', 'pip', 'install', '.[docs,developer,test]']' returned non-zero exit status 1.

This might be a separate issue, but flit install -s installation worked ok, but does not permit datalad to do its magic
$ python -m xibabel.testing minimal
get(error): /home/pi/.pyenv/versions/xib2/lib/python3.11/site-packages/xibabel/testing/ds000105/sub-1/func/sub-1_task-objectviewing_run-01_bold.nii.gz [path not associated with dataset Dataset(/data/pi/fmri/take2/xibabel)]
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/pi/.pyenv/versions/xib2/lib/python3.11/site-packages/xibabel/testing/__main__.py", line 26, in <module>
    main()
  File "/home/pi/.pyenv/versions/xib2/lib/python3.11/site-packages/xibabel/testing/__main__.py", line 22, in main
    get_set(test_set)
  File "/home/pi/.pyenv/versions/xib2/lib/python3.11/site-packages/xibabel/testing/__init__.py", line 75, in get_set
    out_paths.add(get_file(path_str))
                  ^^^^^^^^^^^^^^^^^^
  File "/home/pi/.pyenv/versions/xib2/lib/python3.11/site-packages/xibabel/testing/__init__.py", line 64, in get_file
    return _SOURCE2FETCHER[source](path_str)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/.pyenv/versions/xib2/lib/python3.11/site-packages/xibabel/testing/__init__.py", line 41, in get_datalad_file
    check_call(['datalad', 'get', str(rel_path)])
  File "/home/pi/.pyenv/versions/3.11.0/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['datalad', 'get', '/home/pi/.pyenv/versions/xib2/lib/python3.11/site-packages/xibabel/testing/ds000105/sub-1/func/sub-1_task-objectviewing_run-01_bold.nii.gz']' returned non-zero exit status 1.

Yeah - at the moment - the tests and data installation only work for in-place install - because an install to your site-packages won't take across the submodules.  I wonder whether we should just refactor this out to let datalad do its own git initialization.

Is this working now?

I think this is fixed, and tested.  Reopen if I'm wrong?


When using the option --date "on the current day", total production is incorrect

When using the --day flag to load the historical data for "today", the total production in kph is incorrect because it is actually the one returned for "yesterday" by the GoodWe API.
In such case, I would recommend instead to just sum up incrementally the production data (because the daily total is computed asynchronously in batch mode by GoodWe server), instead of trying to normalise it based on GoodWe calculations.

I noticed the same problem and found that is was linked to the correction factor that is being calculated in the gw_api.py, getDayReadings function.  If the correction factor worked properly it might make the total production more accurate, but I haven't looked into that.  I found that if I ignore the correction factor it is good enough for me.  So what I changed this line in the gw_api.py, getDayReadings function:
increase = pgrid_w * sample['minutes'] * factor
to this:
increase = pgrid_w * sample['minutes'] / 1000 / 60
This change worked for me and seems pretty accurate for total production.

When looking into issue #23 I noticed that the return from /api/PowerStationMonitor/GetPowerStationPowerAndIncomeByDay is two days of total power value.  It includes the date passed in the --date parameter and the previous day:

'data': [{'i': 1.1573, 'd': '07/11/2019', 'p': 16.3}, {'i': 1.6614, 'd': '07/12/2019', 'p': 23.4}]

The script assigns the power for the day using this code:
eday_kwh = data[0]['p']
I seems to me that the prior day's power is used rather than the current days value.  Assuming the GoodWe return data is consistent the code needs to change to:
eday_kwh = data[1]['p']

I've done some further investigation and found some additional changes that need to be made to correct this problem.

As mentioned above the previous day's power is being used.  Correction as per above.

The units are not correct when determining the correction factor.  The eday_from_power calculation shoud be: eday_from_power += sample['pac'] * sample['minutes'] / 60 / 1000 to keep this value in kWh as per the return of the daily power.

The units are not correct when calculating the eday_kWh increase.  The increase calculation should change to: increase = pgrid_w * sample['minutes'] * factor /60 /1000.

There maybe a better way to do this, but this is what makes sense to me.  I've tested it out on one day of my data and works, although I haven't implemented it in my production script yet.  So with all the changes this section of code would look like this:
`
eday_kwh = data[1]['p']
    payload = {
        'id' : self.system_id,
        'date' : date_s
    }
    data = self.call("PowerStationMonitor/GetPowerStationPacByDayForApp", payload)
    if 'pacs' not in data:
        logging.warning(date_s + " - Received bad data " + str(data))
        return result

    minutes = 0
    eday_from_power = 0
    for sample in data['pacs']:
        parsed_date = datetime.strptime(sample['date'], "%m/%d/%Y %H:%M:%S")
        next_minutes = parsed_date.hour * 60 + parsed_date.minute
        sample['minutes'] = next_minutes - minutes
        minutes = next_minutes
        eday_from_power += sample['pac'] * sample['minutes'] / 60 / 1000
    factor = eday_kwh / eday_from_power if eday_from_power > 0 else 1

    eday_kwh = 0
    for sample in data['pacs']:
        date += timedelta(minutes=sample['minutes'])
        pgrid_w = sample['pac']
        increase = pgrid_w * sample['minutes'] * factor
        if increase > 0:
            eday_kwh += increase
            result['entries'].append({
                'dt' : date,
                'pgrid_w': pgrid_w,
                'eday_kwh': round(eday_kwh, 3))}`

I realized that the first correction above is not a particularly robust solution.  Instead of eday_kwh = data[1]['p'] it would be better to look for the correct day in the data.  So instead I think it should be:
convertedDate = date_s[5:7] + "/" + date_s[-2:] + "/" + date_s[:4]
eday_kwh = 0
for day in data:
    if day['d'] == convertedDate:
        eday_kwh = day['p']

This way in case there days are out of order or they decide to pass more days we can get the correct power.

I believe I have not addressed all the issues with using the --day option.  Note that you need to wait for awhile after a day is over for this option to work.  I'm using it at 4:30 am the next day and have good results.  I will also add a --day yesterday option to make this easier to use on a daily script.  I'm running a crontab daily to get the prior days readings to pvoutput.  I also found that the power curve was starting at the correct time of day and that was fixed in the pull request that is referenced above.

Should be fixed in release 1.3.2.


nimble/controller: Fix legacy scanning

In legacy scanning next_phy is unconfigured so we should not try to
configure channels for next_phy as it will corrupt some random
memory.
Also add assert to make sure cur_phy is always configured, because it
should be.

test this please