Rails in production - getting "Errno::ENOENT: No such file or directory @ rb_sysopen - public/uploads/cache/…"

Hi,
I'm using Shrine to upload Images and Videos to my Rails application and they are being processed in the background via Sidekiq. In Development everything worked really well, but in Production I get this error for the cache folder from my Sidekiq Monitor:
Errno::ENOENT: No such file or directory @ rb_sysopen - public/uploads/cache/message/image/2e021ba0c168ef46e9ff1d8fbde25d6e.png
Normally I'm using jQuery fileupload to upload and post the record, but also with a basic post submit without jQuery this error happens. This error happens most of the time, so sometimes, the file is being processed and everything works out. How can it happen that in production, I get this error?
EDIT: When I ssh into my production server and check the paths mentioned in the errors, the files are actually there.
My shrine initializer:
require "shrine"
require "shrine/storage/file_system"
require "shrine/storage/memory" if Rails.env.test?

if Rails.env.development? || Rails.env.production?

  Shrine.storages = {
    cache: Shrine::Storage::FileSystem.new("public", prefix: "uploads/cache"),
    store: Shrine::Storage::FileSystem.new("public", prefix: "uploads/store"),
  }

elsif Rails.env.test?

  Shrine.storages = {
    cache: Shrine::Storage::Memory.new,
    store: Shrine::Storage::Memory.new,
  }

end

Shrine.plugin :activerecord
Shrine.plugin :logging, logger: Rails.logger
Shrine.plugin :direct_upload
Shrine.plugin :hooks
Shrine.plugin :backgrounding

# execute file promotion / deletion in background
Shrine::Attacher.promote { |data| PromoteJob.perform_async(data) }
Shrine::Attacher.delete { |data| DeleteJob.perform_async(data) }

My uploaders:
require "image_processing/mini_magick"

class BaseUploader < Shrine
  include ImageProcessing::MiniMagick

  plugin :pretty_location
  plugin :delete_raw
  plugin :delete_promoted
  plugin :cached_attachment_data
  plugin :determine_mime_type
  plugin :validation_helpers
  plugin :processing
  plugin :versions

  def preprocess(image)
    convert(image, "jpg") do |cmd|
      cmd.auto_orient
      cmd.quality 82
      cmd.strip
    end
  end

  def blur(image)
    convert(image, "jpg") do |cmd|
      cmd.blur "0x8"
      cmd.resize "5120x2880"
    end
  end
end

class ImageUploader < BaseUploader

  # validate images
  Attacher.validate do
    validate_mime_type_inclusion %w[image/jpeg image/png image/gif],
                            message: "You can only upload image files of the type .jpg, .jpeg, .gif or .png."
    validate_max_size 5.megabyte,
                            message: "You can not upload images bigger than 5MB."
  end

  # process / versions
  if Rails.env.development? || Rails.env.production?

    # processing and versions
    …
  else

    # in test do no processing
    …
  end
end

My model:
class Message < ApplicationRecord
  belongs_to :user
  scope :published, -> { where(processing: false) }
  scope :processing, -> { where(processing: true) }

  # uploaders
  include ImageUploader[:image]
  include VideoUploader[:video]

  …
end

Thanks for any advice!

Could you describe more your production environment (e.g. are you using more than one server, are the Sidekiq jobs executed on the same servers as your app etc)?
I'm wondering if it isn't a filesystem syncing bug. Firstly, when you write to files in Ruby, Ruby keeps an internal buffer so that it doesn't flush to disk on each write. Secondly, the operating system itself usually has its own buffer as well. As far as I know, there are two ways to sync in Ruby; one is setting File#sync to true when opening the file (this will essentially disable Ruby's buffer), and calling File#fsync after writing, which actually tells the operating system to flush its buffer. However, my assumption was that flushing automatically happens once you close the file (at least Ruby's buffer), since you delete the file descriptor.
Could you check whether the cached file exists immediately after uploading?
class Message < ApplicationRecord
  # ...

  def image=(value)
    super
    puts image.exists? if image
  end

  def video=(value)
    super
    puts video.exists? if video
  end
end

If it outputs true, but the Sidekiq jobs still aren't able to find them, then something else is wrong, the files are not found for some other reason. But if they output false, then it's most likely a syncing issue and should definitely be fixed on the Shrine side.

Hi, thanks for the advice so far.
First of all, I checked the functionality without backgrounding the processing via Sidekiq and everything works well. With backgrounding, the image=(value) method outputs true for image.exists?. So the cached file seem to be present immediately after uploading.
I'm running everything one simple server with Ubuntu 16.04, so the Sidekiq jobs are executed on the same server as my app. Sidekiq and Redis are initiated on server boot via systemd. Besides that everything is standard configuration and I made no specific setups.
What surprised me is also that rarely the file passes through, is being processed by Sidekiq and everything works out..

@R4ttlesnake That's really strange. Sidekiq worker classes don't have any specific configuration? One other thing to try is removing the delete_promoted plugin, perhaps it's somehow deleting the cached files when it shouldn't. Could you also post processing code? Finally, could you post a backtrace of the Errno::ENOENT error, so that I can see where it's coming from?

Thanks so far. Without the delete_promoted plugin, I get the same error. Also I tried rescheduling the PromoteJob to perform 2 minutes later than the job was called, again the same error.
Processing from base_uploader.rb:
def preprocess(image)
  convert(image, "jpg") do |cmd|
    cmd.auto_orient
    cmd.quality 82
    cmd.strip
  end
end

def blur(image)
  convert(image, "jpg") do |cmd|
    cmd.blur "0x8"
    cmd.resize "5120x2880"
  end
end

Processing from image_uploader.rb:
if Rails.env.development? || Rails.env.production?

  process(:store) do |io, context|

    base = preprocess(io.download)

    large_2x = resize_to_limit(base, 5120, 2880)
    large = resize_to_limit(large_2x, 2560, 1440)

    medium_2x = resize_to_limit(large_2x, 2880, 1800)
    medium = resize_to_limit(large, 1440, 900)

    small_2x = resize_to_limit(medium_2x, 2048, 1536)
    small = resize_to_limit(medium, 1024, 768)

    mobile_2x = resize_to_limit(small_2x, 1334, 1334)
    mobile = resize_to_limit(small, 667, 667)

    blur = blur(medium)

    { mobile: mobile, mobile_2x: mobile_2x,
      small: small, small_2x: small_2x,
      medium: medium, medium_2x: medium_2x,
      large: large, large_2x: large_2x,
      blur: blur }
  end

else

  def process(io, context)

    if context[:action] == :store

      # "fake" versions
      { mobile: io, mobile_2x: io.download,
        small: io.download, small_2x: io.download,
        medium: io.download, medium_2x: io.download,
        large: io.download, large_2x: io.download,
        blur: io.download }
    end
  end
end

Sidekiq error from syslog:
Mar 29 08:58:47 Almoist sidekiq[1306]: 2017-03-29T08:58:47.803Z 1306 TID-gq8sx6kn0 PromoteJob JID-b101330907a4c82192788fcb INFO: start
Mar 29 08:58:47 Almoist sidekiq[1306]: 2017-03-29T08:58:47.821Z 1306 TID-gq8sx6kn0 PromoteJob JID-b101330907a4c82192788fcb INFO: fail: 0.018 sec
Mar 29 08:58:47 Almoist sidekiq[1306]: 2017-03-29T08:58:47.823Z 1306 TID-gq8sx6kn0 WARN: {"context":"Job raised exception","job":{"class":"PromoteJob","args":[{"attachment":"{\"id\":\"e5eacf3244bd8ffd91f2c876b6102a1e.jpeg\",\"storage\":\"cache\",\"metadata\":{\"filename\":\"font1.jpeg\",\"size\":281094,\"mime_type\":\"image/jpeg\"}}","record":["Message","30"],"name":"image","shrine_class":"ImageUploader","action":"store","phase":"store"}],"retry":true,"queue":"default","jid":"b101330907a4c82192788fcb","created_at":1490777678.5545304,"enqueued_at":1490777927.8021653,"error_message":"No such file or directory @ rb_sysopen - public/uploads/cache/e5eacf3244bd8ffd91f2c876b6102a1e.jpeg","error_class":"Errno::ENOENT","failed_at":1490777817.1959364,"retry_count":2,"retried_at":1490777927.8223574},"jobstr":"{\"class\":\"PromoteJob\",\"args\":[{\"attachment\":\"{\\\"id\\\":\\\"e5eacf3244bd8ffd91f2c876b6102a1e.jpeg\\\",\\\"storage\\\":\\\"cache\\\",\\\"metadata\\\":{\\\"filename\\\":\\\"font1.jpeg\\\",\\\"size\\\":281094,\\\"mime_type\\\":\\\"image/jpeg\\\"}}\",\"record\":[\"Message\",\"30\"],\"name\":\"image\",\"shrine_class\":\"ImageUploader\",\"action\":\"store\",\"phase\":\"store\"}],\"retry\":true,\"queue\":\"default\",\"jid\":\"b101330907a4c82192788fcb\",\"created_at\":1490777678.5545304,\"enqueued_at\":1490777927.8021653,\"error_message\":\"No such file or directory @ rb_sysopen - public/uploads/cache/e5eacf3244bd8ffd91f2c876b6102a1e.jpeg\",\"error_class\":\"Errno::ENOENT\",\"failed_at\":1490777817.1959364,\"retry_count\":1,\"retried_at\":1490777879.4835765}"}
Mar 29 08:58:47 Almoist sidekiq[1306]: 2017-03-29T08:58:47.825Z 1306 TID-gq8sx6kn0 WARN: Errno::ENOENT: No such file or directory @ rb_sysopen - public/uploads/cache/e5eacf3244bd8ffd91f2c876b6102a1e.jpeg
Mar 29 08:58:47 Almoist sidekiq[1306]: 2017-03-29T08:58:47.825Z 1306 TID-gq8sx6kn0 WARN: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/storage/file_system.rb:144:in `initialize'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/storage/file_system.rb:144:in `open'
Mar 29 08:58:47 Almoist sidekiq[1306]: message repeated 2 times: [ /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/storage/file_system.rb:144:in `open']
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine.rb:765:in `open'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine.rb:779:in `download'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/releases/20170325145146/app/uploaders/image_uploader.rb:16:in `block in <class:ImageUploader>'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/plugins/processing.rb:69:in `instance_exec'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/plugins/processing.rb:69:in `block in process'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/plugins/processing.rb:68:in `each'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/plugins/processing.rb:68:in `inject'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/plugins/processing.rb:68:in `process'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine.rb:337:in `processed'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/plugins/hooks.rb:75:in `block in processed'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/plugins/hooks.rb:82:in `around_process'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/plugins/hooks.rb:75:in `processed'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/plugins/logging.rb:96:in `block in processed'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/plugins/logging.rb:101:in `block in log'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/plugins/logging.rb:161:in `block in benchmark'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/.rbenv/versions/2.4.1/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/plugins/logging.rb:161:in `benchmark'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/plugins/logging.rb:101:in `log'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/plugins/logging.rb:96:in `processed'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine.rb:205:in `upload'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/plugins/hooks.rb:56:in `block in upload'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/plugins/hooks.rb:62:in `around_upload'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/plugins/hooks.rb:56:in `upload'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine.rb:611:in `store!'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine.rb:544:in `promote'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/plugins/delete_promoted.rb:12:in `promote'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/shrine-2.5.0/lib/shrine/plugins/backgrounding.rb:133:in `promote'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/releases/20170325145146/app/jobs/promote_job.rb:7:in `perform'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:167:in `execute_job'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:139:in `block (5 levels) in process'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/rails.rb:90:in `block in call'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/activesupport-5.0.2/lib/active_support/execution_wrapper.rb:85:in `wrap'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/rails.rb:89:in `call'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:135:in `block (4 levels) in process'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/server/logging.rb:10:in `call'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:133:in `invoke'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:134:in `block (3 levels) in process'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/logging.rb:32:in `with_context'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:132:in `block (2 levels) in process'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:183:in `stats'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:131:in `block in process'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq.rb:35:in `block in <module:Sidekiq>'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:126:in `process'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:82:in `process_one'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:70:in `run'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/util.rb:17:in `watchdog'
Mar 29 08:58:47 Almoist sidekiq[1306]: /home/deploy/almoist_chat_app/shared/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/util.rb:26:in `block in safe_thread'
Mar 29 09:00:22 Almoist redis-server[1324]: 1324:M 29 Mar 09:00:22.022 * 10 changes in 300 seconds. Saving...
Mar 29 09:00:22 Almoist redis-server[1324]: 1324:M 29 Mar 09:00:22.024 * Background saving started by pid 3987
Mar 29 09:00:22 Almoist redis-server[1324]: 3987:C 29 Mar 09:00:22.030 * DB saved on disk
Mar 29 09:00:22 Almoist redis-server[1324]: 3987:C 29 Mar 09:00:22.033 * RDB: 0 MB of memory used by copy-on-write
Mar 29 09:00:22 Almoist redis-server[1324]: 1324:M 29 Mar 09:00:22.125 * Background saving terminated with success
Mar 29 09:02:05 Almoist kernel: [ 1616.253052] ip_tables: (C) 2000-2006 Netfilter Core Team

Ok, from what I can see, the error is happening on this line, as expected:
if Rails.env.development? || Rails.env.production?
  process(:store) do |io, context|
    base = preprocess(io.download) # <======
    # ...
else
  # ...
end

However, I still have no idea why would Sidekiq workers not be able to find the cached file sometimes. Another thing that comes to mind is to try to setup the FileSystem storage to use full paths instead of relative:
  Shrine.storages = {
    cache: Shrine::Storage::FileSystem.new(Rails.root.join("public"), prefix: "uploads/cache"),
    store: Shrine::Storage::FileSystem.new(Rails.root.join("public"), prefix: "uploads/store"),
  }

Another way to check whether Shrine is at any point deleting the cached file is to override Shrine::Storage::FileSystem#delete and add a puts statement there.
  Shrine.storages = {
    cache: Shrine::Storage::FileSystem.new(Rails.root.join("public"), prefix: "uploads/cache"),
    store: Shrine::Storage::FileSystem.new(Rails.root.join("public"), prefix: "uploads/store"),
  }

  Shrine.storages[:cache].instance_eval do
    def delete(id)
      puts path(id) # print each deleted file
      super
    end
  end

Shrine doesn't delete cached files by default at all, so the only thing that should trigger this deletion is the delete_promoted plugin.
One more thing is if you're using Shrine::Storage::FilesSystem#clear!(older_than: some_expiration_date) in a recurring task, it might be worth disabling it to see if it fixes the error. Now that I'm writing this, I realized that the current implementation (excluding new changes in master) of Shrine::Storage::FilesSystem#clear!(older_than: some_expiration_date) is much more messed up that I thought, because it rm -r's old subdirectories, but I found out now that the "last modified" time of a directory doesn't get updated recursively up the stack. As we can see from the following script, only the "last modified" time of the inner-most directory is changed when a new file is created.
require "pathname"
require "tmpdir"

tmpdir = Pathname.new(Dir.tmpdir)
tmpdir.join("dir1").rmtree

tmpdir.join("dir1/dir2").mkpath # create directories
puts tmpdir.join("dir1").mtime      #=> 2017-03-30 13:31:44 +1100
puts tmpdir.join("dir1/dir2").mtime #=> 2017-03-30 13:31:44 +1100

sleep 1

FileUtils.touch tmpdir.join("dir1/dir2/file")  # create new file
puts tmpdir.join("dir1").mtime      #=> 2017-03-30 13:31:44 +1100
puts tmpdir.join("dir1/dir2").mtime #=> 2017-03-30 13:31:45 +1100

I fixed the implementation recently in https://github.com/janko-m/shrine/commit/eb877f8ec2288eeab1cb967e737364430bd52471, but haven't released a new version yet, though I will really hurry it up now.

I will close this, since it's probably Shrine::Storage::FilesSystem#clear!(older_than: some_expiration_date) being called in production, and there is already a fix on master.

Hi, first of all thanks for all the help, I really appreciate it and Shrine, of course. The problem is solved, though the source was somewhere else.
I was not calling Shrine::Storage::FilesSystem#clear! and I tried all the recommended changes, but finally it turned out that my Sidekiq server was not restarting properly after deploying to production (I guess Sidekiq must have looked in the old release folders, not sure). Now, I'm restarting Sidekiq with a Capistrano task and Shrine's file handling with Sidekiq works well.
I hope this didn't cause to much of an inconvenience and at least helped to fix the issue with Shrine::Storage::FilesSystem#clear!.
Nothing to do with Shrine, but for anybody interested: I'm starting Sidekiq via Systemd based on this example. With a Capistrano task based on this example, I'm restarting Sidekiq after each deployment via sudo service sidekiq restart (to skip sudo's password prompt, the deployment user needs permission to execute this sudo command without password).
# lib/capistrano/tasks/sidekiq.rake

namespace :sidekiq do
  task :quiet do
    on roles(:app) do
      puts capture("pgrep -f 'sidekiq' | xargs kill -USR1")
    end
  end

  task :restart do
    on roles(:app) do
      execute :sudo, :service, :sidekiq, :restart
    end
  end
end

after 'deploy:starting', 'sidekiq:quiet'
after 'deploy:reverted', 'sidekiq:restart'
after 'deploy:published', 'sidekiq:restart'

@R4ttlesnake No worries, I'm glad it's solved 👍


Ledger pairing error in Helium Wallet

Describe the bug
A clear and concise description of what the bug is.
I am trying to transfer my IOT and HNT from my Helium Wallet app on my iPhone to another wallet. When I select my Ledger wallet account and try to do this I have my Ledger paired to my iPhone as shown in the bluetooth settings but I keep getting a pairing failed notice (see attached). Have tried everything but am not able to ever get transfer to work and get the same pairing failed notice. Would appreciate any suggestions to get this working.

To Reproduce
Steps to reproduce the behavior:

Go to Helium Ledger Account
Click on HNT token

Hit Send button

Enter address and amount
Hit pay button
Hit swipe to send HNT
Hit approve button

Expected behavior
A clear and concise description of what you expected to happen.
Expected the HNT token to be sent to desired Solana address
Screenshots
If applicable, add screenshots to help explain your problem.
Resulting pairing failed notice

Smartphone (please complete the following information):

Device: iPhone 15 Pro Max
OS: IOS 17.3.1
App Version: Helium HNT Wallet version 2.6.4

Additional context
Add any other context about the problem here.

I would try re pairing your Ledger with helium wallet. If that doesn't work, you could always try Solflare or Phantom.

I am having this same issue. Please help! I can transfer HNT, SOL, and IOT into my Ledger wallet on the Helium wallet app, but I am unable to to transfer coins from my Ledger wallet back to my Helium wallet or exchange coins in my Ledger wallet. I keep getting this error.

Hi, i have the same issue? Can someone help me?


ELK stack for demo and small POCs

While the ELK stack (elasticsearch, logstash and kiban) is very popular, we're missing a quick and easy way to spin up an ELK stack for demonstration and for small POCs.
In #11 and #12 it would also be neat with such a setup for trying out the technologies.
Goals

simple way to start and ELK stack preferable using containers
one or two example on how to use and create some data visualization

https://elk-docker.readthedocs.io/ seems like it could be of use

Exactly, thanks @drBosse but I actually found another more useful. I'll put that in writing very soon.

@buep @carmosin and I about to make a templated solution to get data out of Jenkins and visualize it using ELK. The plan is to try it out for Volvo and AtlasCopco. I will call for a demo as soon as we have something to show, i.e. in a couple of weeks

Thanks @Andrey9kin - will it be Jenkins specific, or more like a script I can run using Jenkins?

The idea is to have two pieces - one is a groovy script that you can run as a part of pipeline job that would go through the builds, collect data and push them to Elk. And the second part is templated ELK instance with predefined data storage and dashboard. So first part is Jenkins specific, the second one is not. You could push data from any CI server - just make sure that it is formatted for ELK data store

@ewelinawilkosz2  FYI

I'm going to close this issue.
First goal was achieved, didn't get that far that we have any examples, but we have other things in progress in other project, that uses ELK and could contribute with such examples.
For now the result is just an easy way to run an ELK stack.

@buep you mentioned that you found another more useful resource than what
Bosse suggested. Which one was that ?
On Wed, Jan 4, 2017 at 10:20 AM, buep<EMAIL_ADDRESS>wrote:

Closed #18 https://github.com/Praqma/praqma-rnd/issues/18.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/Praqma/praqma-rnd/issues/18#event-910916787, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AIsrPuhDGk2rWK8gGmxxSTQov2FcXJGnks5rO2RugaJpZM4LX0QM
.

--
[image: ____________________________________________________________]
Morten Christensen
Consultant
<EMAIL_ADDRESS>+45 5360 7207 <+45%205360%207207>
Praqma
www.praqma.com
<+4536772762>

It is committed in the readme.

I see it is still in the branch - Are you going to merge it ?  If not, why ?

Done :-)


Hints to circumvent scale out of bounds

Hello everyone :)
I have an application, in which I perform some homomorphic multiplications.
When I perform x2^2, x2x, and x2x3, the result is correct.
But when performing the last multiplication, x2*x7, the application simply breakes with the following error:

terminate called after throwing an instance of 'std::invalid_argument'
what():  scale out of bounds

This is the snippet code of function:
`
Ciphertext g(Ciphertext x, Evaluator* evaluator, RelinKeys relin_keys){
    Ciphertext x1 = x;
    Ciphertext x2 = x;
    (*evaluator).square_inplace(x2);
    (*evaluator).relinearize_inplace(x2, relin_keys);

    Ciphertext x3 = x2;
    (*evaluator).multiply_inplace(x3,x);
    (*evaluator).relinearize_inplace(x3, relin_keys);

    Ciphertext x5 = x2;
    (*evaluator).multiply_inplace(x5,x3);
    (*evaluator).relinearize_inplace(x5, relin_keys);

    Ciphertext x7 = x2;
    (*evaluator).multiply_inplace(x7,x5);
    (*evaluator).relinearize_inplace(x7, relin_keys);

    return x7;

}`
This is the vector I am trying:
vector<double> input{ 2.2 , 0, 1, -3, -0.5 };
And this is my setup:
`EncryptionParameters parms = createParameters();
    auto context = SEALContext::Create(parms);
    print_line(__LINE__);
    cout << "Set encryption parameters and print" << endl;
    print_parameters(context);
    cout << endl;

    KeyGenerator keygen(context);
    auto public_key = keygen.public_key();
    auto secret_key = keygen.secret_key();
    auto relin_keys = keygen.relin_keys();

    Encryptor encryptor(context, public_key);
    Evaluator evaluator(context);
    Decryptor decryptor(context, secret_key);

    CKKSEncoder encoder(context);

    size_t slot_count = encoder.slot_count();
    cout << "Number of slots: " << slot_count << endl;

    double scale = pow(2.0, 30);`

Could anyone point a direction for me to solve this issue?
Actually, I will need to perform lots of multiplications, and I was thinking that the relinearize function would allow me to lighten my polynomial so that it could handle several multiplications.
Best regards :)

`EncryptionParameters createParameters(){
    EncryptionParameters parms(scheme_type::CKKS);

    size_t poly_modulus_degree = 16384;
    parms.set_poly_modulus_degree(poly_modulus_degree);
    parms.set_coeff_modulus(CoeffModulus::Create(poly_modulus_degree, { 40, 40, 40, 40, 40 }));

    return parms;

}
`

The reason is that you didn't "rescale" the ciphertexts after multiplication/relinearization. Then if you add "rescale", you also need to match the scales of two ciphertexts given into a multiplication function. Please compile, execute, and check the native/examples/4_ckks_basics.cpp. Let me know if you get it working or have further questions.

Thanks for the response.
When applying rescaling after multiplication and relinearization it works (i.e., runs without error). However, the first multiplication after rescaling is yielding wrong results.
For instance, I am trying to get x^3.
First, I compute as an example the square of 2.2, and I get 4.84. But when I do the next multiplication to get x^3, the outcome gets unpredictable (completely different from expected).
Is there anything else I am missing?
My new code snippet:
`Ciphertext g(Ciphertext x, Evaluator* evaluator, RelinKeys relin_keys){
Ciphertext x1 = x;
Ciphertext x2 = x;
(*evaluator).square(x1, x2);
(*evaluator).relinearize_inplace(x2, relin_keys);
(*evaluator).rescale_to_next_inplace(x2);

(*evaluator).rescale_to_next_inplace(x);
Ciphertext x3 = x2;
(*evaluator).multiply_inplace(x3,x);
(*evaluator).relinearize_inplace(x3, relin_keys);
(*evaluator).rescale_to_next_inplace(x3);

return x3;

}`
Best regards

Nevermind. I just noticed that I have to match the modulus index of the Ciphertexts for it to work properly. Now I have it working. Thanks ;-)

Although the results of my multiplications are correct, in a given moment get the "not enough relinearization keys" error.
I've already spent a few hours trying to figure out how to increase this value but I couldn't find.
Does anyone have a clue of how to circumvent this problem?
Best regards

Increasing this value is not necessary or recommended. Every multiplication of ciphertext and ciphertext or square of ciphertext should be followed by a relinearization. The error is likely caused by missing one relinearization.

Thanks for the quick reply. It was exactly what you said. I was missing one relinearization. :-)

Hi WeiDaiWD,
First of all, I apologize for flooding here with my doubts :)
I have two more questions:

is there any way of limiting the size of the Ciphertext other than poly_modulus_degree / 2 (I am using CKKSEncoder).
I am trying to perform matrix multiplications. Is there any simplified form to do that? I am trying to implement it from scratch but, in my implementation, I would need to sum the values of all elements of a single Ciphertext. However, it seems that there is no such function that does that. Is there any approach you would recommend?

Best regards

The size of Ciphertext (Ciphertext::size()) should be 2. After ciphertext and ciphertext multiplication, it should be 3. After relinearization, it goes back to 2. This is the same in BFV and CKKS. It has nothing to do with encoder or poly_modulus_degree. poly_modulus_degree / 2 is the maximum number of message slots that can be packed into a plaintext, using CKKSEncoder.
An optimal way to perform matrix multiplication is not trivial. It depends on the dimensions of both matrices. Each ciphertext (and the plaintext encrypted) can pack all elements or some elements of a column, row, or a submatrix. This paper explains some techniques to compute matrix multiplication with the CKKS scheme. Sum all the values of all elements of a single ciphertext is doable. See the (pseudo) code below.

// Try any encryption parameters
EncryptionParameters parms(scheme_type::CKKS);
size_t poly_modulus_degree = 8192;
parms.set_poly_modulus_degree(poly_modulus_degree);
parms.set_coeff_modulus(CoeffModulus::Create(poly_modulus_degree, { 60, 30, 60 }));
auto context = SEALContext::Create(parms);
KeyGenerator keygen(context);
SecretKey secret_key = keygen.secret_key();
PublicKey public_key = keygen.public_key();
GaloisKeys galois_keys = keygen.galois_keys();

CKKSEncoder encoder(context);
auto m = encoder.slot_count();
vector<double> msg(m, 1.0); // m instances of 1.0
Plaintext pt;
double scale = pow(2.0, 30); // just an example, change it to whatever you want
encoder.encode(msg, scale, pt);

Encryptor encryptor(context, public_key);
Ciphertext ct;
encryptor.encrypt(pt, ct);

// sum all m slots of ct, store the results in the first slot of ct
Evaluator evaluator(context);
Ciphertext ct_temp;
for (size_t i = m / 2; i > 0; i >>= 1) {
evaluator.rotate_vector(ct, i, galois_keys, ct_temp);
evaluator.add_inplace(ct, ct_temp);
}

// further set all other slots of ct to zero, which consumes 1 prime
Plaintext pt_mask;
vector<double> mask(m, 0.0);
mask[0] = 1.0; // set the first slot to 1.0
encoder.encode(mask, scale, pt_mask);
evaluator.multiply_plain_inplace(ct, pt_mask);
evaluator.rescale_to_next_inplace(ct);

// examine result
Decryptor decryptor(context, secret_key);
decryptor.decrypt(ct, pt);
vector<double> result;
encoder.decode(pt, result);
// result should be [4096.0, 0.0, ..., 0.0]


[Bug][lake] lake does not handle 404´d Gitlab MR and fails the whole blueprint pipeline

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
I have a blueprint which worked earlier and started failing. It integrates with gitlab and started to fail when fetching "Code review" data entities. The UI showed that message:

The message is not straightforward but I guessed that maybe the MR was not actually existing anymore and was proven right by simply trying to fetch it in the gitlab GUI:

That's not good because the whole job just chokes on that missing MR instead of gracefully ignoring it and continuing. As a temporary workaround I've removed the collection of "Code review" data entities and the pipeline now passes, but that also leads to a lot of valuable information not making it to grafana.
I haven't checked the code of lake but I would imagine a relatively simple exception handling block catching and ignoring those errors or a conditional expression would suffice. I really hope you can prioritize fixing this bug.
What do you expect to happen
The pipeline should just ignore MRs which are 404'd by gitlab and continue its information gathering - instead of completely failing.
How to reproduce
The steps mentioned in "What happened" are precise enough I believe.

project integrating with gitlab
gitlab project with a few MR. you delete one of them.
you run the pipeline.
things will likely fail as I describe

Do note that I haven't reproduced myself but this is likely to be what has to be done to be able to.
Anything else
No response
Version
v0.16.0
Same behavior in v0.15.1-beta5 and beta6.
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Thanks for your report, I agree with you we should skip the missing MR, do you know why the MR is gone between collections? is it normal for PR to be deleted from the data source?
@hezyin @Startrekzky  @mindlesscloud @likyh @keon94 @CamilleTeruel I think we should ignore all 404 error for all secondary-data-collections (collectors that rely on other entity, e.g. changelogs , comments, etcs)

I would not say it's normal if what you mean is "is it the default". I actually don't know why the team owning this repository deleted it and I've asked them out of curiosity but I guess there could be plenty of reasons.
Maybe you pushed secrets that should never have been pushed and want them removed. Maybe you for some reason are not happy with what you coded and don't want anyone to stumble upon it. Maybe the MR was so broken there is no reason to keep it for historical reason. Maybe a team wants to delete very old MR from their repos to clean up and only keep the newest ones.
In any case as long as the possibility is there someone will always use it :)
That's great if you can do something. If you orient me a bit in your code base so I get an idea of where to patch I could probably attempt to propose a PR. And if I feel I'm not going to be able to find time I could let you know. Let me know.

@CamilleTeruel I agree that it is API-specific. What is your solution then?
Should we address the issue on the Gitlab plugin?
How do we report the error without breaking the whole pipeline?

I don't have a solution and I'm not saying that the framework mustn't ignore those 404s. It's just that we don't yet have a generic way to tell whether those errors happen because the parent resource has been deleted or for another reason.
On the top of my head I see two strategies to tell that, but they are not great:

Store the API url to get the item in raw rows, not the url used to fetch the whole collection as it is right now. Then the framework can try to refetch the parent item in case of 404. If it fails again we know we can safely ignore the error. The issue is that all plugins have to find or construct that url for each item.
When we face one of those 404s, rerun collection/extraction/conversion of parent resource from scratch to be sure we are up to date, and then retry the child resource collection. This might slow down collection significantly and might be hard to implement.

Now does it worth it? I don't know how often it can be for another reason in practice, I just have the example of CircleCI API that returned a 404 for getting the jobs of some existing but failed build. But thinking about this one case, the proper handling is also to ignore it, so... maybe we don't care.
BTW, shouldn't we also delete the parent row when we find out it has been deleted from the datasource?

I don’t know the codebase at all and not have gotten the following correctly but allow me one comment:

When we face one of those 404s, rerun collection/extraction/conversion of parent resource from scratch to be sure we are up to date, and then retry the child resource collection. This might slow down collection significantly and might be hard to implement.

Retrying is not a good strategy imho. Wastes resources in my case at lease because if it’s gone, it’s gone, it’s not going to help to retry.
In addition the pipelines are running periodically and i would argue that it is a form of retry.
Thanks for discussing this, good to see that the project is alive :)

I don't have a solution and I'm not saying that the framework mustn't ignore those 404s. It's just that we don't yet have a generic way to tell whether those errors happen because the parent resource has been deleted or for another reason.
On the top of my head I see two strategies to tell that, but they are not great:

Store the API url to get the item in raw rows, not the url used to fetch the whole collection as it is right now. Then the framework can try to refetch the parent item in case of 404. If it fails again we know we can safely ignore the error. The issue is that all plugins have to find or construct that url for each item.
When we face one of those 404s, rerun collection/extraction/conversion of parent resource from scratch to be sure we are up to date, and then retry the child resource collection. This might slow down collection significantly and might be hard to implement.

Now does it worth it? I don't know how often it can be for another reason in practice, I just have the example of CircleCI API that returned a 404 for getting the jobs of some existing but failed build. But thinking about this one case, the proper handling is also to ignore it, so... maybe we don't care.
BTW, shouldn't we also delete the parent row when we find out it has been deleted from the datasource?

I see. Good points. I agree with you that there is no good general solution for this kind of problem. Especially cases like #5464 degrade the collection speed drastically.
Let's deal with them separately unless we come out with a better solution.

@renflo We will fix this particular issue ASAP.


Music does not resume perfectly when unpaused

There's some delay. Also, the notes seem glitched when paused...

This sometimes happen as the audio didn't get unpaused when user leave the viewState of application. It might be that the listener didn't catch the application were in inactive state


Spyder does not run my new code - Error Message(Code did not run through)

Issue Report Checklist

[x] Searched the issues page for similar reports
[x] Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
[ ] Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)
[ ] Could not reproduce inside jupyter qtconsole (if console-related)
[ ] Tried basic troubleshooting (if a bug/error)

[ ] Restarted Spyder
[ ] Reset preferences with spyder --reset
[ ] Reinstalled the latest version of Anaconda
[ ] Tried the other applicable steps from the Troubleshooting Guide

[ ] Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description
My pythpn script read some files (*.csv) and did some transformation and send the results  as files to Google drive .
What steps reproduce the problem?

When I try to run the code I always get this Error-Massage
"Code did not run through" even i've update  spyder to its last version but and restart the kernel and my PC as well but the probelem still.

What is the expected output? What do you see instead?
write some files to Google Drive
Paste Traceback/Error Below (if applicable)

PASTE TRACEBACK HERE

Versions

Name                    Version                   Build  Channel
spyder                    4.1.3                    py37_0

Spyder version:
Python version:
Qt version:
PyQt version:
Operating System name/version:  Windows 10  pro

Dependencies

PASTE DEPENDENCIES HERE

Hi @Ahmed0028, thanks for the report. Is this error appearing on any script you run, or only on that specific script you mention?
Cheers

Hi @goanpeca  , the error appearing only on that specific script.  Thanks!

Could you share the script? Or a similar minimal example that we can run on our side to reproduce the issue?

Any part of my Script reproduce  the  issue, even just import some libraries
For Example:
`import re
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
import os
import datetime
import pygsheets
path  = 'C:/Users/Reports/'
#authorization
os.chdir(path+'credentials/')
gauth = GoogleAuth()
gauth.LoadCredentialsFile("mycreds.txt")
if gauth.credentials is None:
gauth.LocalWebserverAuth()
elif gauth.access_token_expired:
gauth.Refresh()
else:
gauth.Authorize()
Save the current credentials to a file
gauth.SaveCredentialsFile("mycreds.txt")
connect to google sheet
gc = pygsheets.authorize(service_file=path+'credentials/pygsheet_serviceaccount.json')
Create GoogleDrive instance with authenticated GoogleAuth instance.
drive = GoogleDrive(gauth)
def delta_define(value,variable):
'''Adjustments done for specific value of variable as input to google ads script'''
if variable=='zero_days_6':
if value==1.0:
return -0.5
elif value>=0.9:
return -0.35
`

Is the problem only occurring when importing these libraries?
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
import pygsheets

How did you install them?

No, the issue was reproduced  when i import these libs as well:
import os, os.path import pandas as pd import time import datetime as dt import numpy as np
I've installed them using pip on Spyder command line.

Does your script run in the anaconda prompt with "python

NO. Also doesn't run on jupyter or any other Python environment.

If it doesn't run in python or jupyter then this seems like a problem in your code or with pygsheets. Unfortunately we don't have enough experience with this package to help you. Sorry.


alpine downgrade to 3.16

hopefully this fixes issues with pytype and unit tests

problem was not aalpine 3.17.


docker-compose faild and can't login

When I Clone this repository and run docker-compose up.I can success launch default browser,but I can not login and regist my account.I used CentOS.

Is the local-test.me domain name blocked for you?

I do nothing about local-test.me,May be this domain doesn't work. and I only use https://github.com/exceptionless/Exceptionless/blob/master/docker-compose.6x.yml to build my container, but when I create new account,this email input checked always failed.You can try it on CensOS or Cloud like AliCloud

Hello,
You cannot use the 6.x yml as that is not supported (we have it in there
for testing but it needs to be removed). Can you see if you can even view
the localtest.me domain name. I'm wondering if it's blocked all it does is
reroute localhost traffic through their domain name and give you ssl.
Thanks
-Blake Niemyjski
On Tue, Jul 16, 2019 at 6:02 AM 786744873<EMAIL_ADDRESS>wrote:

I do nothing about local-test.me,May be this domain doesn't work. and I
only use
https://github.com/exceptionless/Exceptionless/blob/master/docker-compose.6x.yml
to build my container, but when I create new account,this email input
checked always failed.You can try it on CensOS or Cloud like AliCloud
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/exceptionless/Exceptionless/issues/435?email_source=notifications&email_token=AAHZFI42EA52U6YGIRQEXI3P7WTD7A5CNFSM4IDUKNYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2AP3LY#issuecomment-511770031,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHZFI5GP2S3EPP7WN6MMWLP7WTD7ANCNFSM4IDUKNYA
.

How can I modify some files to make this email check option use my local engining?I seached from official documen but get no result

Set an environment variable or docker variable with Ex_Email_ConnectionString set to an smtp uri. https://github.com/exceptionless/Exceptionless/blob/master/src/Exceptionless.Core/Configuration/EmailOptions.cs#L59-L63
Did it work once you used the non 6.x docker compose file

Can you pull latest from GitHub and run docker-compose again. I filled out default settings for base url and it's working now. It also makes it easier to configure.


Add Attestation Service dev environment file

Add .env.clabs to secrets store based on the .env.development base
Add this new file to .gitignore so it's not accidentally added.

@eelanagaraj adding a note to ask you about this

@alecps this ticket is still relevant as long as we still need to maintain the ASv1, more context: it's really annoying currently to manually look up all of the API keys needed to get the ASv1 running locally, but these access tokens shouldn't be published.


Exec process may cause shim hang

Description
when dozens exec process simultaneously run in a shim, shim will get hang.
Steps to reproduce the issue:

a simple shell script can reproduce the problem

#!/bin/bash

for ((i=0;i<10;i++));do
	(sudo ctr t exec --exec-id "t1-1-$i" c1 echo 1) &
	(sudo ctr t exec --exec-id "t1-2-$i" c1 echo 1) &
	(sudo ctr t exec --exec-id "t1-3-$i" c1 echo 1) &
	(sudo ctr t exec --exec-id "t1-4-$i" c1 echo 1) &
	(sudo ctr t exec --exec-id "t1-5-$i" c1 echo 1) &
	(sudo ctr t exec --exec-id "t1-6-$i" c1 echo 1) &
	(sudo ctr t exec --exec-id "t1-7-$i" c1 echo 1) &
done

shim will get hang, we can check ctr t ls  command will get hang, since shim.State never get shim lock which block at reaper.Wait

$ sudo ctr t ls
^C

After check the code in shim, I found the last running exec process is block at Monitor.Wait, this will caused runtime.Exec() block, so this process is bolcked, shimService.Start() -> execCreatedState.Start() -> execProcess.Start().
// Wait blocks until a process is signal as dead.
// User should rely on the value of the exit status to determine if the
// command was successful or not.
func (m *Monitor) Wait(c *exec.Cmd, ec chan runc.Exit) (int, error) {
    for e := range ec {  --> wait here
        if e.Pid == c.Process.Pid {
            // make sure we flush all IO
            c.Wait()
            m.Unsubscribe(ec)
            return e.Status, nil
        }
    }
    // return no such process if the ec channel is closed and no more exit
    // events will be sent
    return -1, ErrNoSuchProcess
}

enlarge buffersize in runtime/v1/shim/reaper.go can resolve the problem, but it can not solve problems fundamentally.
I think the lock implement in shim may have some problem which cause this problem. I will do more check in code.
@fuweid also reproduce this problem, and get some more detail info about goroutines, please help to post information here.
Describe the results you received:
Describe the results you expected:
Output of containerd --version:
$ containerd -v
containerd github.com/containerd/containerd v1.2.0-rc.1 0c5f8f63c3368856c320ae8a1c125e703b73b51d

The shim.Default does publish all the child process runc.Exit event to the Subscriber. For the service, service's processExits always receives all the runc.Exit events. As the @Ace-Tang mentions, if there are a lot of the exec exit-child processes, the channel will be filled up.
If there is State call which granted the lock and the range Default.subscribers doesn't pick up the State call's channel, the State call will be hang by the shim.Monitor.Wait.
-> checkProcess can't get the lock and the channel will be filled up.
goroutine 5 [semacquire]:
sync.runtime_SemacquireMutex(0xc42013c004, 0x6b7a00)
»▸▸▸▸▸▸▸/usr/local/go/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc42013c000)
»▸▸▸▸▸▸▸/usr/local/go/src/sync/mutex.go:134 +0x108
github.com/containerd/containerd/runtime/v1/shim.(*Service).checkProcesses(0xc42013c000, 0xbee7a70ed2772d2f, 0x6878b7b7d, 0x8c0720, 0x2510, 0x0)
»▸▸▸▸▸▸▸/home/vagrant/go/src/github.com/containerd/containerd/runtime/v1/shim/service.go:513 +0x45
github.com/containerd/containerd/runtime/v1/shim.(*Service).processExits(0xc42013c000)
»▸▸▸▸▸▸▸/home/vagrant/go/src/github.com/containerd/containerd/runtime/v1/shim/service.go:508 +0xd0
created by github.com/containerd/containerd/runtime/v1/shim.NewService
»▸▸▸▸▸▸▸/home/vagrant/go/src/github.com/containerd/containerd/runtime/v1/shim/service.go:91 +0x3e9

--> State has lock but never release it.
goroutine 202 [chan receive]:
github.com/containerd/containerd/runtime/v1/shim.(*Monitor).Wait(0x8bf820, 0xc4202e7600, 0xc4202d1620, 0x0, 0x0, 0x8)
»▸▸▸▸▸▸▸/home/vagrant/go/src/github.com/containerd/containerd/runtime/v1/shim/reaper.go:82 +0x52
github.com/containerd/containerd/vendor/github.com/containerd/go-runc.cmdOutput(0xc4202e7600, 0x742c01, 0x0, 0x0, 0x0, 0x0, 0x0)
»▸▸▸▸▸▸▸/home/vagrant/go/src/github.com/containerd/containerd/vendor/github.com/containerd/go-runc/runc.go:701 +0x110
github.com/containerd/containerd/vendor/github.com/containerd/go-runc.(*Runc).State(0xc42013e300, 0x742cc0, 0xc4200560c0, 0xc420150000, 0x40, 0x8c90d0, 0xc420083080, 0xc4200564c0)
»▸▸▸▸▸▸▸/home/vagrant/go/src/github.com/containerd/containerd/vendor/github.com/containerd/go-runc/runc.go:87 +0xce
github.com/containerd/containerd/runtime/v1/linux/proc.(*Init).Status(0xc420134160, 0x742cc0, 0xc4200560c0, 0x0, 0x0, 0x0, 0x0)
»▸▸▸▸▸▸▸/home/vagrant/go/src/github.com/containerd/containerd/runtime/v1/linux/proc/init.go:229 +0xb2
github.com/containerd/containerd/runtime/v1/linux/proc.(*execProcess).Status(0xc42025cf00, 0x742cc0, 0xc4200560c0, 0x0, 0x0, 0x0, 0x0)
»▸▸▸▸▸▸▸/home/vagrant/go/src/github.com/containerd/containerd/runtime/v1/linux/proc/exec.go:198 +0x57
github.com/containerd/containerd/runtime/v1/shim.(*Service).State(0xc42013c000, 0x742cc0, 0xc4200560c0, 0xc420067f70, 0x0, 0x0, 0x0)
»▸▸▸▸▸▸▸/home/vagrant/go/src/github.com/containerd/containerd/runtime/v1/shim/service.go:307 +0x109
github.com/containerd/containerd/runtime/v1/shim/v1.RegisterShimService.func1(0x742cc0, 0xc4200560c0, 0xc4202cc460, 0xc4202da0a0, 0x5, 0xc42000a0c0, 0x0)
»▸▸▸▸▸▸▸/home/vagrant/go/src/github.com/containerd/containerd/runtime/v1/shim/v1/shim.pb.go:1607 +0xc8
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serviceSet).dispatch(0xc42000c018, 0x742cc0, 0xc4200560c0, 0xc420077710, 0x25, 0xc4202da0a0, 0x5, 0xc4202dc140, 0x42, 0x42, ...)
»▸▸▸▸▸▸▸/home/vagrant/go/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/services.go:87 +0x10e
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serviceSet).call(0xc42000c018, 0x742cc0, 0xc4200560c0, 0xc420077710, 0x25, 0xc4202da0a0, 0x5, 0xc4202dc140, 0x42, 0x42, ...)
»▸▸▸▸▸▸▸/home/vagrant/go/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/services.go:60 +0xb5
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serverConn).run.func2(0xc42010a320, 0x742cc0, 0xc4200560c0, 0xb3, 0xc4202c2300, 0xc420014180, 0xc420014240, 0xc4000000b3)
»▸▸▸▸▸▸▸/home/vagrant/go/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go:417 +0xaa
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serverConn).run
»▸▸▸▸▸▸▸/home/vagrant/go/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go:416 +0x73c

ping @crosbymichael any update for this issue?

I have few things i'm working on right now.  If you two could look into this it would be very helpful.  If you have questions, i could answer.

Thank you. Will try to fix this issue

The problem can be part fixed with #2743 , the other part is located at shim.Monitor.Reap and shim.Monitor.Wait will compete for monitor lock when chan is full.
goroutine for shim.Monitor.Reap
goroutine 1 [chan send, 1 minutes]:
github.com/containerd/containerd/runtime/v2/shim.Reap(0xc4200ca3c0, 0xc420215a40)
    /home/ace/gosrc/src/github.com/containerd/containerd/runtime/v2/shim/reaper_unix.go:44 +0x168

goroutine for shim.Monitor.Wait
goroutine 610 [semacquire, 1 minutes]:
sync.runtime_SemacquireMutex(0x9f5284, 0x0)
    /usr/local/go/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0x9f5280)
    /usr/local/go/src/sync/mutex.go:134 +0x108
github.com/containerd/containerd/runtime/v2/shim.(*Monitor).Unsubscribe(0x9f5280, 0xc4203a49c0)

So I this the implement in shim.Reap may need be changed.

Is this resolved with #2743

@crosbymichael I assumed no from this comment by @Ace-Tang: The problem can be part fixed with #2743 , the other part is located at shim.Monitor.Reap and shim.Monitor.Wait will compete for monitor lock when chan is full.

Ahh, Do we know what chan is filling up?

From the above discussion, pretty sure it is this one: https://github.com/containerd/containerd/blob/master/runtime/v1/shim/reaper.go#L97 ; defaults to 32; both in v1 shim and v2. The PR fix for locking was only on v1, but I see the initial discussion here seems to be about v2, and it looks like the reaper implementation for runc is effectively the same, with this same exit channel used, and the Wait() impl. uses a range over this fixed size channel.

Ok, that buffer can be increased for sure.  Probably 2048 is safe.  We could also add a check on the send to see if 32 is actually just a poor choice or we are failing to drain the channel.  I feel like 32 is a poor choice that I probably made when not considering high number of exec commands.

Thanks @crosbymichael and @estesp !!.
We also increased the buffer to 256 to reduce the odds case internally, because it just showed at several machines. I was thinking to publish the exit event to specific channel indexed by pid. If find it, send the event and close it. It seems that it is more reasonable to close channel by sender, not by receiver. If not, just send it to processExits. Sometimes, ip.KillAll will hold lock for a while and it might increate the channel size.
But, to drop event is good choice with 2048 big sizes!
Hope it doesn't drop the container task's event.😄 Thanks for the help! Again.

Great， also thanks for help, @crosbymichael @estesp .


Social media links in the footer are working now

Fixes: #226
Social media links in the footer are working now , Please take a look .

As the PR is already merged , you haven't assigned me the task ,
So please assign me the task so that the PR gets count in my Leaderboard .


Reduce Diagnostics.ToString() by excluding traces under 1ms.

The Diagnostics.ToString() can be extremely large at times. To improve the performance and reduce the string size information that is not needed should be removed. Looking at past diagnostics anything that completed in less than 1MS can be ignored since it completed so quickly.
Proposal to make the following changes:

Remove any cache call where it responded under 1 MS.
Remove any request handler that took under 1 MS. The request handler completed quickly so there is no need to include it in the logs.
Client info, system info, and client side request stats will always be logged

One idea could be filter based on Type? For example, set all non I/O related ones as Verbose/Debug and ToString() only log Info (I/O)?

If other consumers of the Trace (ie Compute) want to log everything, they still can

The diagnostics would still always collect everything. The ToString() logic would simply do the filtering to remove traces that are to small to have any impact on the latency so they don't help and just create noise.

The number of store responses should be limited to the first 10 and last 5. There are scenarios for 404/1002 that can be retried hundreds of times causing the diagnostics to be massive. All the store responses in the middle are not needed because looking at the first and last gives a good enough understanding of the scenario. Limiting the count will reduce the string size without impacting troubleshooting.

For this issue the only changes that would be made are in the Diagonostics.ToString() logic and no change in actual diagnostics collected.  Client info, system info, and client side request stats will always be logged.
Proposed approaches:
•	Remove any request handler that took under 1 MS. The request handler completed quickly so there is no need to include it in the logs.
Pros: We remove around 287 characters per parent request handler.
Cons: We would be displaying partial information in terms of handler invoked.
•	Number of store responses should be limited. Remove the store responses from the middle. First few and last few can give a good understanding of the scenario.
Pros: We remove around 7000 characters per store response.
Cons: which ones to keep and how confident are we if they will provide good info for all scenarios?
•	Remove id and start time fields from each child. We are already recording duration of each child and we have start time specified in the Summary.
Pros: We remove around 120 characters per child

Further investigation:

Java sdk doesn't include information about any request handler but it displays all store responses. We can take a decision to either remove request handlers or just remove handlers that take less than 1ms. We remove around 287 characters per request handler.
The main reason behind increased string length is retries. Each retry adds approximately 7000 characters.
Small changes that are feasible and wouldn't disrupt any experience:

Remove cache collection less than 1ms. We remove around 240 characters
Remove id and start time fields from each child. We are already record duration of each child and have start time specified in the Summary. We remove around 120 characters per child


Not enough space error

Etcher ver:1.0.0-beta.13:
Operating system and architecture: Ubuntu 16.04 LTS
Do you see any meaningful error information on DevTools?
I didn't know about this until afterwards, so no.

I plugged my Raspberry Pi (Max2Play) 16GB Sandisk SD card into my reader and made a copy using:
sudo dd bs=4M if=/dev/sdb | gzip > rasbian.img.gz
I then used Etcher to write this image to another SD card (identical in capacity and make), but eventually got an error saying there was not enough room and that I should try again with a larger card.
However, the process was successful if I used:
gunzip --stdout rasbian.img.gz | sudo dd bs=4M of=/dev/sdb
from the command line:
0+269514 records in
0+269514 records out
15523119104 bytes (16 GB, 14 GiB) copied, 1081.82 s, 14.3 MB/s
Do SD cards need some preparation before using Etcher? - E.g. deleting all partitions?

Hi @richard-kimber ,
Thanks a lot for reporting this, which is very weird. If the drive is really not large enough, Etcher would report that and prevent you from writing in the first place (unless you're writing a bz2 image, which we can't know the uncompressed size in advance).
Some thought:

We might be failing to get the correct drive size, but somehow the size validation gets bypassed for some reason (worth investigating if its the case). In order to discard this hypothesis, can you show me the output of lsblk -b -d $device --output SIZE, where $device is the device in question? Can you also run this small bash script (when the problematic drive is plugged in) and show me the output: https://github.com/resin-io-modules/drivelist/blob/master/scripts/linux.sh?

Another hypothesis would be that we're interpreting the estimated uncompressed size from the gz image REALLY badly. Can you try decompressing the .img.gz and flashing the .img directly? Do you get the same error? Also, do you have a link to the exact file you're trying to flash?

lsblk (using /dev/sdk) gives:
SIZE
15523119104
the script gives:-
device: /dev/sda
description: "Samsung SSD 850 "
size:<PHONE_NUMBER>60
mountpoint: /boot/efi,/mnt/common,
name: /dev/sda
protected: False
system: True
device: /dev/sdb
description: "Samsung SSD 850 "
size:<PHONE_NUMBER>04
mountpoint: /,/usr/local,/mnt/scratch,/mnt/record,/media/info,
name: /dev/sdb
protected: False
system: True
device: /dev/sdc
description: "WDC WD10EZRX-00L"
size:<PHONE_NUMBER>016
mountpoint: /mnt/backups,/mnt/dat,/media/spare,
name: /dev/sdc
protected: False
system: True
device: /dev/sdd
description: "WDC WD30EFRX-68E"
size:<PHONE_NUMBER>016
mountpoint: /media/audio,
name: /dev/sdd
protected: False
system: True
device: /dev/sde
description: "Zalman SSD0060F1"
size:<PHONE_NUMBER>6
mountpoint: /home,
name: /dev/sde
protected: False
system: True
device: /dev/sdf
description: "2105            "
size:<PHONE_NUMBER>016
mountpoint:
name: /dev/sdf
protected: False
system: True
device: /dev/sdg
description: "EZRX-22D8PB0    "
size:<PHONE_NUMBER>016
mountpoint:
name: /dev/sdg
protected: False
system: False
device: /dev/sdk
description: "Multi-Reader  -3"
size:<PHONE_NUMBER>4
mountpoint: /media/psrwebs/cea0b7ae-2fbc-4f01-8884-3cb5884c8bb7,/media/psrwebs/boot,
name: /dev/sdk
protected: False
system: False
I decompressed the image and got the same 'not enough space' error with the full-sized .img file. Maybe there's something funny about the image.  Initially, when I started using Max2Play, I used Etcher successfully to write the (much smaller) image.  I then used Max2Play's facility to extend the filesystem to the full size of the card so that there was room to install other things. It is this 15.5GB image that is causing the problems. Though, as I said, it works OK from the command line with dd.
I'll post a link to the compressed image in a few hours - my upload speed is very slow and it's 1.4GB in size

Hi @richard-kimber ,
Thanks a lot for the information. Everything looks fine on that side. I'll wait for the image link to check if I can reproduce myself, and if I can detect anything weird by inspecting the file myself.

Here is the link:
https://www.dropbox.com/s/a5712r4mi0tppby/rasbian.img.gz?dl=0

@richard-kimber thanks for the test image, now have some idea what's happening, at least in part (@jviotti)
Apparently the gzip file format cannot accurately return the size of files above 4GB (2^32 bytes), but returns the modulo:
That matches the things seen above.
Gzip's own assessment:
$ gzip -l rasbian.img.gz
         compressed        uncompressed  ratio uncompressed_name
<PHONE_NUMBER><PHONE_NUMBER>  43.2% rasbian.img

which is the same as etcher sees:
({"path":"/home/greg/Downloads/rasbian.img.gz","size":2638217216})

but the correct file size<PHONE_NUMBER>4, and then indeed<PHONE_NUMBER>4 mod 2^32 =<PHONE_NUMBER>.
But this is probably just part of the story, and likely not the source of the failure, if the uncompressed image does not work either.
I could reproduce the issue by taking an SD, dd-ing it off to a file, and then trying to flash it back with etcher. It goes to 100% then fails with a not enough space error.
{"command":"progress","data":{"type":"write","percentage":99,"eta":0,"speed":17603788}}
{"command":"progress","data":{"type":"write","percentage":100,"eta":0,"speed":17611442}}
{"command":"error","data":{"message":"Not enough space on the drive","code":"ENOSPC"}}

Though it looks like the contents are written properly (testin: make a copy of the card with dd, clear the partitions & save, flash it with etcher, get not enough space error, dd off again and compare copy: matches)

Thanks.  So, I should just ignore the error, and unmount the SD and use it normally?

Thanks a lot for the valuable investigation :)

Apparently the gzip file format cannot accurately return the size of
files above 4GB (2^32 bytes), but returns the modulo:

Right, I read this somewhere :/ Do you know if we have any alternatives?

Though it looks like the contents are written properly (testin: make a
copy of the card with dd, clear the partitions & save, flash it with
etcher, get not enough space error, dd off again and compare copy:
matches)

This is weird. First, if the estimated size is smaller than the real
size, we shouldn't have hit 100%, right? The ENOSPC should have been
thrown before. Second, if the image was not written completely, how is
it possible that you can read back with dd and it matches the whole
image?
Can you try filling the drive with /dev/random and then flash with
Etcher? Does it still match?
On Fri, Aug 19, 2016 at 03:07:51AM -0700, Gergely Imreh wrote:

@richard-kimber thanks for the test image, now have some idea what's happening, at least in part (@jviotti)
Apparently the gzip file format cannot accurately return the size of files above 4GB (2^32 bytes), but returns the modulo:
That matches the things seen above.
Gzip's own assessment:
$ gzip -l rasbian.img.gz
         compressed        uncompressed  ratio uncompressed_name
<PHONE_NUMBER><PHONE_NUMBER>  43.2% rasbian.img

which is the same as etcher sees:
({"path":"/home/greg/Downloads/rasbian.img.gz","size":2638217216})

but the correct file size<PHONE_NUMBER>4, and then indeed<PHONE_NUMBER>4 mod 2^32 =<PHONE_NUMBER>.
But this is probably just part of the story, and likely not the source of the failure, if the uncompressed image does not work either.
I could reproduce the issue by taking an SD, dd-ing it off to a file, and then trying to flash it back with etcher. It goes to 100% then fails with a not enough space error.
{"command":"progress","data":{"type":"write","percentage":99,"eta":0,"speed":17603788}}
{"command":"progress","data":{"type":"write","percentage":100,"eta":0,"speed":17611442}}
{"command":"error","data":{"message":"Not enough space on the drive","code":"ENOSPC"}}

Though it looks like the contents are written properly (testin: make a copy of the card with dd, clear the partitions & save, flash it with etcher, get not enough space error, dd off again and compare copy: matches)
--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/resin-io/etcher/issues/629#issuecomment-240980243

--
Juan Cruz Viotti
Software Engineer

@jviotti

gzip size problem
Looks like on the command line people recommended something like zcat file.gz | wc -c or gzip -dc test1.img.gz | wc -c which give the correct value - though then decompresses the the file twice. Might have to do that for gzip in the end, though, since likely >4GB files are common for Etcher's use case. I do think this should be a separate issue, will file it as such.

not enough space
Sorry, looks like I wasn't clear enough, mixing up these two issues.
The "not enough space" error happens regardless of gzip, whenever the flashed image size is close to the drive's capacity (close to or equal).

I had a found a ~512 MB SD card that I could play with quicker, here's what I've found:

Drive size: 507379712 B = 495488 KiB = 483.875 MiB
Images up to 505413632 B = 493568 KiB = 482.0 MiB are burned fine (note, smaller than the drive size)
Images above that size (but smaller than the drive's size) failed in the end (482 < x < 483.975): this is the bad behaviour
Images above the drive's size are not getting selected, correctly.

@richard-kimber looking at this, I think it's not okay to ignore the error, the last 1-2MB on the drive don't seem to be written correctly (checked with zeroing drive out with /dev/zero and burning the image again, the checksum does not match).

Thanks.
I've looked at the manual and
zcat file.gz | wc -c
will give the correct size.

Hi @imrehg

since likely >4GB files are common for Etcher's use case. I do think this should be a separate issue, will file it as such.

Sounds good, let's continue discussion for that particular issue separately.

Sorry, looks like I wasn't clear enough, mixing up these two issues.
The "not enough space" error happens regardless of gzip, whenever the flashed image size is close to the drive's capacity (close to or equal).

Etcher doesn't write the image byte by byte, but block by block. While the image is indeed a bit smaller than the drive's capacity, the block size (which is usually ~1MB) exceeds the capacity of the drive, and thus throws the error.
I guess we can try to handle this edge case (where the last block causes ENOSPC) by splitting the block.

Hey @jviotti I did find that it writes by chunks, that's why I tried make the test image shorter by a whole chunck, but found that that's not enough, I had to be shorter by almost 2 chuncks to be able to write.

Writing 482.0 MiB = 482 chunks writes properly
Above that, e.g. writing 482.1 MiB ~ 483 chunks does runs out of space
Drive capacity: 483.875 MiB  (so even if in your assessment up to 483.0 MiB = 483 chunk it should be okay, that's not appear to be the case)

Thus I think there's something more than an issue with the last chunk.

I see. Looks like we have an off-by-one bug in the conditional that
checks if we can still push one more chunk or not. I'll definitely
investigate and fix.
On Sun, Aug 21, 2016 at 08:03:47PM -0700, Gergely Imreh wrote:

Hey @jviotti I did find that it writes by chunks, that's why I tried make the test image shorter by a whole chunck, but found that that's not enough, I had to be shorter by almost 2 chuncks to be able to write.

Writing 482.0 MiB = 482 chunks writes properly
Above that, e.g. writing 482.1 MiB ~ 483 chunks does runs out of space
Drive capacity: 483.875 MiB  (so even if in your assessment up to 483.0 MiB = 483 chunk it should be okay, that's not appear to be the case)

Thus I think there's something more than an issue with the last chunk.
--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/resin-io/etcher/issues/629#issuecomment-241305451

--
Juan Cruz Viotti
Software Engineer

I've sent a PR fixing the off-by-one issue on the last block. I'm intentionally not covering the edge case where you write, e.g: 483.1 MiB to a 483.875 MiB, since we explicitly pad unaligned blocks to our desired block size (1 MiB) on the last block (expanding 483.1 MiB to 484 MiB), otherwise Windows throws EPERM.

The PR I sent closes this issue. Let's continue discussion about the particular gz modulo uncompressed size problem in the newly created issue.


port changing

when i put a IP(not domain) it chnges to 19132. when i tried to change the port "19132" on the packages, it still shows up 19132 somehow. even if i changed the number 19132 in any packages, anyway it dont work

If you mean createClient is connecting on 19132 while you are putting a different port in the options, then it's probably because the server is advertising the incorrect port in its PONG message. This could be a bug in the server software, and you can check by doing
const { ping } = require('bedrock-protocol')
ping({ host: 'localhost', port: 12345 }).then(console.log)

To checkout the port that the server is returning.

yes but the bot wont join with 19132. works only with the real port

If you mean createClient is connecting on 19132 while you are putting a different port in the options, then it's probably because the server is advertising the incorrect port in its PONG message. This could be a bug in the server software, and you can check by doing
const { ping } = require('bedrock-protocol')
ping({ host: 'localhost', port: 12345 }).then(console.log)

To checkout the port that the server is returning.

If you mean createClient is connecting on 19132 while you are putting a different port in the options, then it's probably because the server is advertising the incorrect port in its PONG message. This could be a bug in the server software, and you can check by doing
const { ping } = require('bedrock-protocol')
ping({ host: 'localhost', port: 12345 }).then(console.log)

To checkout the port that the server is returning.

also its a realm, i just got the ipwith some mod. how do i make the bot join a realm?

how do i make the bot join a realm?

See the README


Add entry for German Drupal Association founding

closes #43

Id like to keep the milestone list to globally important events.

Id like to keep the milestone list to globally important events.


Does not cache ambiguous modules

Description
The linter runs for a very long time if the "import/no-cycle" rule is enabled on a large project with typescript. I think the problem is that eslint does not cache all modules. For example, these files will always be parsed again:

https://github.com/aws/aws-sdk-js/blob/master/clients/all.d.ts
<EMAIL_ADDRESS>
Minimal repository for reproduce: https://github.com/stenin-nikita/eslint-plugin-import-bug
Possible solution
This line https://github.com/import-js/eslint-plugin-import/blob/68cea3e6b6fe5fd61e5cf2e2c7c0be9e8dc597cb/src/ExportMap.js#L349
replace with:
if (exportMap == null) {
  exportCache.set(cacheKey, null);
  return null;
}

Locally, this change reduced the check time from 20 to 4 minutes for me

May be a similar problem here #2076

That definitely seems like a worthy change. Care to make a PR?

ОК. I did a PR and wrote a test

@ljharb When will a version with this fix be released?

Fixed by #2531.
Releases aren't done on a particular schedule, but hopefully soon.


Text queries that have variables are no longer displayed on Stat type panels.

Describe the bug
Since version 4.1.0 of the plugin text mode queries that have variables are no longer displayed on Stat type panels.
Expected behavior
We're expecting that the text mode queries with variables work in the same way as in version 4.0.2 and earlier.
Screenshots
Behavior in version 4.1.4:

When the field "Fields" is manually selected as below, the value appears, but the $Inteface variable no longer works:

Behavior in version 4.0.2:

Software versions

Grafana
Zabbix
Grafana-Zabbix Plugin

7.4.3
5.0.9
4.1.4

Because now panel tries to display numeric fields by default. So you have to change it in the Display menu.

Ok, I'll take a look what's a difference between 4.0.2 and latest.


Catch schedd timeout exception in JobStatusLite

The job tracking may fail when the schedd is quite busy. For instance vocms0310
2016-09-26 20:42:32,364:139703540537088:INFO:BossAirAPI:Running/changing/completing 42966/1242/594 jobs in plugin PyCondorPlugin.

2016-09-26 20:42:32,364:139703540537088:INFO:BossAirAPI:About to change 1242 jobs
2016-09-26 20:42:32,364:139703540537088:INFO:BossAirAPI:About to complete 594 jobs
2016-09-26 20:43:49,705:139703540537088:INFO:StatusPoller:Running job status poller algorithm...
2016-09-26 20:43:52,895:139703540537088:INFO:BossAirAPI:About to start building running jobs
2016-09-26 20:50:46,053:139703540537088:INFO:BossAirAPI:About to look for 43817 loadedJobs.

2016-09-26 20:51:12,300:139703540537088:ERROR:BossAirAPI:Unhandled exception while tracking jobs for plugin PyCondorPlugin!
Timeout when waiting for remote host
2016-09-26 20:51:12,534:139703540537088:ERROR:BaseWorkerThread:Error in worker algorithm (1):
Backtrace:
  <WMCore.BossAir.StatusPoller.StatusPoller instance at 0x7f0f4e1ac320> BossAirException
Message: Unhandled exception while tracking jobs for plugin PyCondorPlugin!
Timeout when waiting for remote host
        ModuleName : WMCore.BossAir.BossAirAPI
        MethodName : track
        ClassInstance : None
        FileName : /data/srv/wmagent/v1.0.19.patch2/sw/slc6_amd64_gcc493/cms/wmagent/1.0.19.patch2/lib/python2.7/site-packages/WMCore/BossAir/BossAirAPI.py
        ClassName : None
        LineNumber : 520
        ErrorNr : 0

Traceback:
  File "/data/srv/wmagent/v1.0.19.patch2/sw/slc6_amd64_gcc493/cms/wmagent/1.0.19.patch2/lib/python2.7/site-packages/WMCore/BossAir/BossAirAPI.py", line 508, in track
    localRunning, localChanges, localCompletes = pluginInst.track(jobs=jobsToTrack[plugin])

  File "/data/srv/wmagent/v1.0.19.patch2/sw/slc6_amd64_gcc493/cms/wmagent/1.0.19.patch2/lib/python2.7/site-packages/WMCore/BossAir/Plugins/PyCondorPlugin.py", line 555, in track
    jobInfo, dummySd = self.getClassAds()

  File "/data/srv/wmagent/v1.0.19.patch2/sw/slc6_amd64_gcc493/cms/wmagent/1.0.19.patch2/lib/python2.7/site-packages/WMCore/BossAir/Plugins/PyCondorPlugin.py", line 996, in getClassAds
    for slicedAds in grouper(itobj, 1000):

  File "/data/srv/wmagent/v1.0.19.patch2/sw/slc6_amd64_gcc493/cms/wmagent/1.0.19.patch2/lib/python2.7/site-packages/Utils/IterTools.py", line 19, in <lambda>
    return iter(lambda: list(islice(iterable, n)), [])

  File "/data/srv/wmagent/v1.0.19.patch2/sw/slc6_amd64_gcc493/cms/wmagent/1.0.19.patch2/lib/python2.7/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 172, in __call__
    self.algorithm(parameters)
  File "/data/srv/wmagent/v1.0.19.patch2/sw/slc6_amd64_gcc493/cms/wmagent/1.0.19.patch2/lib/python2.7/site-packages/WMCore/BossAir/StatusPoller.py", line 68, in algorithm
    self.checkStatus()
  File "/data/srv/wmagent/v1.0.19.patch2/sw/slc6_amd64_gcc493/cms/wmagent/1.0.19.patch2/lib/python2.7/site-packages/WMCore/BossAir/StatusPoller.py", line 94, in checkStatus
    runningJobs = self.bossAir.track()
  File "/data/srv/wmagent/v1.0.19.patch2/sw/slc6_amd64_gcc493/cms/wmagent/1.0.19.patch2/lib/python2.7/site-packages/WMCore/BossAir/BossAirAPI.py", line 520, in track
    raise BossAirException(msg)

thx @amaltaro , can this GH be labelled in a particular way so that we track agent issue ?

Btw, this is addressed in the SimpleCondorPlugin already, all calls to the htcondor python bindings are wrapped in try/except.

Addressed in both PyCondor and SimpleCondor. Closing


Update outdated Issue Templates to reflect our current process

In our early stages of Design System work, I created some issue templates as a way to describe our process.
Now that we have iterated on our process more, and about to do a presentation about this project and show our GitHub and how we are working in the open, I've updated our templates.

Cleaned up the Proposal template based on conversations
Added some basic issue templates
Adjusted the order of the templates (as they appear when you visit the Github Add new issue page.
Included a support template because our form error message needs a fallback, and Github can be our fallback (though it requires a login, last we checked)
Adjusted the language on the templates
Adjusted the headers

At the very least, these template give some visibility into the kinds of reviews we do on our components at different stages - and can be a reference for a PM, or an independent team who wants to understand how to contribute.
@TODO

Document our baselines for accessibility, translations, and performance reviews.
Validate the UX review with Adam
Improve our WIKI with some of the information in the issue templates.
Update, clean up project labels
Decide if we want any assignees - like perhaps a governance team in GitHub with assignees in the team?
Test the templates
Technical review of our checklists
Link to any baseline docs we create (in the wiki or on the website)
Link to our contributing page

Doing a first round of review with ODI engineers and Art/Koji (for DWSN presentation)
Then will share with some others


property type wrong

I have several boolean properties defined with Object.defineProperties on my class prototype returning the value with a get function. Unfortunately, I cannot use them as parameter where a boolean is required:
myfile.js:309: ERROR - actual parameter 2 of Date.prototype.strftime does not match formal parameter
found   : {get: function (this:myCtrl): boolean}
required: (boolean|undefined)
	return this.format ? d.strftime(this.format, this.showUtc) : d.toString();
	                                             ^^^^^^^^^^^^

Casting to boolean also does not work, though coercing the value with !! does work.

You need to post exactly how you are defining these.

Defining properties (reduced to 1) is done as follows:
Object.defineProperties(myCtrl.prototype, /** @lends {myCtrl.prototype} */ {
	showUtc: {
		/** @this {myCtrl} */
		get: function() { return (this._ctrlFlags & CF_SHOWUTC) == CF_SHOWUTC; },
	},
});

Does this work?
Object.defineProperties(myCtrl.prototype, /** @lends {myCtrl.prototype} */ {
	showUtc: {
		/**
                  * @return {boolean}
                  * @this {myCtrl}
                  */
		get: function() { return (this._ctrlFlags & CF_SHOWUTC) == CF_SHOWUTC; },
	},
});

No, I already tried with @return, only changes the return type in the error message.

@dimvar what kind of annotation is needed to make sure the type checker correctly recognizes a case like this?

I tried an example with lends and it works as expected. Can you check what your code does differently from my example?

Take a look at this example.
The warning is an error in my case, since I set almost all warning levels on compilation to error.

The way you wrote the getter and setter is not correct. If you write it like this, then type checking works as expected.

It's nice this syntax works.
But it's not the official syntax, see the MDN Docs.
So if I want this to work, I need to use the shortcut too?

You're right, I was writing the object literal incorrectly. The type checker needs to look for getters/setters differently when the object literal is an argument to Object.defineProperties.

While this did seem fixed for a while, now that I've converted my code to modules and issue #2639 was fixed, this is a problem again.

Created Google internal bug b/119819950

Very strange, how did I unassign @dimvar from this issue?
Don't remember doing that, and I don't think I even could do that.
Is there a way for me to view that "internal google bug"?

@ray007 GitHub updates the assignee when a comment is posted if the assignee for some reason doesn't have privileges to be assigned to the issue (maybe not the part of the GitHub org anymore or something). Technically you did the action because your comment triggered the "check", but, indeed, very confusing,

@ray007 Sorry, we don't currently have a way for you to view the internal Google bug. We're working on improving coordination between GitHub and our internal bug-tracking system.

Interestingly enough, if I change the check from !== to !=, the error in line 1711 goes away.
The warning in line 949 can be avoided by adding a cast to the functions first line:
var r = /** @type {boolean} */ (this._visible && this._enabled);

To summarize: I can make the warnings go away, but there's still something not quite right with the types.


JS error on IceCat

IceCat complains about JS not running,
because it's blocked it,
because it thinks it's closed source,
because it hasn't got a JS Web Labels manifest.
See: https://www.gnu.org/software/librejs/free-your-javascript.html

The goal of this project was to show that a smartphone can turn into a 3D controller with a web app. This goal has been accomplished and the project works best in Chrome. So there are no plans to support other browsers like IceCat. But feel free to fork it!


GA release of @google-cloud/web-security-scanner

Package name: @google-cloud/web-security-scanner
Current release: GA
Proposed release: GA
Instructions
Check the lists below, adding tests / documentation as required. Once all the "required" boxes are ticked, please create a release and close this issue.
Required

[ ] 28 days elapsed since last beta release with new API surface
[ ] Server API is GA
[ ] Package API is stable, and we can commit to backward compatibility
[ ] All dependencies are GA

Optional

[ ] Most common / important scenarios have descriptive samples
[ ] Public manual methods have at least one usage sample each (excluding overloads)
[ ] Per-API README includes a full description of the API
[ ] Per-API README contains at least one “getting started” sample using the most common API scenario
[ ] Manual code has been reviewed by API producer
[ ] Manual code has been reviewed by a DPE responsible for samples
[ ] 'Client Libraries' page is added to the product documentation in 'APIs & Reference' section of the product's documentation on Cloud Site

Released as GA.


Loss Nan

Hi, I load the pretrain s0 model of 'model_s0_final.pth', and train the stage 3 without using BL30K, the loss is Nan, How can I fix this ?

Hi, sorry to reply you so late.
I'm already working and rarely look at the previous repo.
If you use mixed precision operations, you may encounter very large loss scales during training. But don't worry, it usually takes a few iterations to get back to normal.


Rework EmbeddableProjectile Offset and Fix Exploding Pens

About the PR

I played with how the offset was previously and regardless of what I did, I could not get it to function in any way that made sense, so I just reworked how it does it
EmbeddableProjectile's Offset is removed in favor of an EmbedPercent, which allows you to dictate just how "deep" into the target you want the projectile to go. For example, an EmbedPercent of 0.15 will mean the object will clip into the target by 15%.
Why / Balance

Fixes https://github.com/space-wizards/space-station-14/issues/30103
With how offset was done previously, there wasn't any really indication as to what part of the Vector2 would help, and changing the values just sent the object in some insane directions away from your intended target. There wasn't any kind of limit to the value either, so you could set the number to a value that didn't make any sense.
With how I've done it, the projectile's possible positions are strictly limited to the edge of the target's hit box, or directly on the target itself.


[Feature][Datasource]replace DataSourceController API with string JSON and cast to DTO

Purpose of the pull request
replace DataSourceController API with string JSON and cast to DTO
Brief change log
replace DataSourceController API with string JSON and cast to DTO
Verify this pull request
test locally

Codecov Report

Merging #10276 (00a5a8b) into dev (4c1ef0a) will decrease coverage by 0.10%.
The diff coverage is 48.76%.

:exclamation: Current head 00a5a8b differs from pull request most recent head 7b0019f. Consider uploading reports for the commit 7b0019f to get more accurate results

@@             Coverage Diff              @@
##                dev   #10276      +/-   ##
============================================
- Coverage     41.25%   41.15%   -0.11%
- Complexity     4760     4772      +12
============================================
  Files           854      857       +3
  Lines         34579    34659      +80
  Branches       3821     3828       +7
============================================
- Hits          14265    14263       -2
- Misses        18944    19019      +75
- Partials       1370     1377       +7

Impacted Files
Coverage Δ

...che/dolphinscheduler/api/ApiApplicationServer.java
60.00% <ø> (ø)

...scheduler/api/controller/DataSourceController.java
38.70% <0.00%> (-4.15%)
:arrow_down:

...lphinscheduler/common/enums/AuthorizationType.java
0.00% <0.00%> (ø)

...asource/api/datasource/BaseDataSourceParamDTO.java
78.57% <ø> (ø)

.../org/apache/dolphinscheduler/spi/enums/DbType.java
0.00% <0.00%> (ø)

...permission/ResourcePermissionCheckServiceImpl.java
24.00% <24.00%> (ø)

...r/plugin/datasource/api/utils/DataSourceUtils.java
48.00% <25.00%> (-28.32%)
:arrow_down:

...hinscheduler/api/service/impl/BaseServiceImpl.java
90.69% <50.00%> (-4.18%)
:arrow_down:

...urce/clickhouse/ClickHouseDataSourceProcessor.java
5.76% <50.00%> (+3.76%)
:arrow_up:

...rce/api/datasource/db2/Db2DataSourceProcessor.java
50.98% <50.00%> (-0.05%)
:arrow_down:

... and 19 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4c1ef0a...7b0019f. Read the comment docs.


Fix socat invocation in dnsmasq image

Description
https://github.com/appgate/sdp-k8s-injector/commit/57390d6b3e71f9731704d7ab5e2561b5d66fb326#diff-bc9bd89c20c87435874782d8b85ac7b101f17fc7b8814629e9ca2c071625cb7eL4 this PR made the killing mechanism to fail because some changes in how socat exists.
See https://repo.or.cz/socat.git/blob/2da070164d454971d5c970b5278e645051f0d0f7:/CHANGES
In previous Socat versions errors EPIPE and ECONNRESET on read() were handled at warning level, thus not automatically leading to termination with exit code 1.

Checklist

[ ] Bump .version and .appVersion in k8s/crd/Chart.yaml
[ ] Bump .version and .appVersion in k8s/chart/Chart.yaml

Does this require a version bump?

yeah


Tag-styles incorrectly match substrings, like nested tags

Short:
The tag condition matches any note that contains the mentioned tag anywhere in its tag list, disregarding nested tags or substrings.
Sample:

Add styling rule: Note has tag "#Company"

Create two sample notes:

Note 1 with nested tag: #Finances/company/bank_usd
Note 2 with tag: #CompanyBills

Expected:
The above sample should not style the sample notes. They do not contain the tag "#Company" (rather, they contain "#Finances" and "#CompanyBills")
Actual:
The current CSS logic targets all notes that contain the text "company" somewhere in the tags field.
Suggestion

Currently, the styling CSS uses .data-link-icon[data-link-tags*="Company" i]::before to recognize "#Company" tags.
More accurate would be .data-link-icon[data-link-tags*="#Company" i]::before (to skip nested tags)
The most accurate CSS would be:

.data-link-icon[data-link-tags*="#Company " i]::before, /* e.g. "#Company #Client" */
.data-link-icon[data-link-tags*="#Company/" i]::before, /* e.g. "#Company/Client" */
.data-link-icon[data-link-tags$="#Company" i]::before   /* e.g. "#Company" */

Good point. I'll try this out for the next version


examples/mongo-sidecar/backup-actionset.yaml is missing name

I was trying instructions from https://docs.kanister.io/#getting-started . I got this error:
$ kubectl apply -f examples/mongo-sidecar/backup-actionset.yaml
error: error when retrieving current configuration of:
&{0xc4214132c0 0xc422206e70 default  examples/mongo-sidecar/backup-actionset.yaml 0xc421fe6b28 0xc421fe6b28  false}
from server for: "examples/mongo-sidecar/backup-actionset.yaml": resource name may not be empty

@tamalsaha - I believe it's related to the usage of generateName in the yaml spec not being compatible with apply -f. Try kubectl create -f examples/mongo-sidecar/backup-actionset.yaml
https://github.com/kanisterio/kanister/tree/master/examples/mongo-sidecar has the right command - I'll push a PR to update the docs to reflect this.

Addressed by a commit that just went in.


#99 Add support for Commerce and custom content providers

I added content provider key to RedirectRule. It allows for getting content by content id from other than default content provider.

This PR does not fully solves #99 as it looks like events handling is not adjusted to Commerce (ie casting to PageData returns null and redirect rules are not created). But I would suggest merging it anyway and handle issue of events in another PR


PHP7.4 bootup issue.

Hi,
Is this fork version of zf1 is only limited to PHP7.3? because i am trying to set it up on PHP7.4 and it's not booting up and it's also not giving any kind of error.
So my question is it possible to run this fork version on PHP7.4 by any means.

Hi,
waiting for your reply.

Hi, thanks for the reminder. Well I think it should work also on PHP 7.4 but I don't have any machine to test it on. I am currently running ZF future on PHP 7.0.30 and PHP 7.1.2 and it works fine.
Can you post here phpinfo() and output of index page of your application? Also php_errors log would be nice, if it's available. There can be problem with configuration or something else on the server but it should give some output or errors when you start the application.

phpinfo details.
In index file i am just loading my bootstrap file with all application configuration.
And i am not getting any kind off error its just showing a white page.

ZF1-Future running fine on my PHP7.4 server.
Run your app and check the apache log file for errors.

ZF1-Future running fine on my PHP7.4 server.
Run your app and check the apache log file for errors.

Hi is there anyway through which i can check if the framework is loading up or not. Because i am not getting any kind of error.

ZF1-Future running fine on my PHP7.4 server.
Run your app and check the apache log file for errors.

Hi is there anyway through which i can check if the framework is loading up or not. Because i am not getting any kind of error.

Can you check with Developer Tools? There can be something in source code or you should get at least some HTTP headers like this:

ZF1-Future running fine on my PHP7.4 server.
Run your app and check the apache log file for errors.

Hi is there anyway through which i can check if the framework is loading up or not. Because i am not getting any kind of error.

Can you check with Developer Tools? There can be some errors in html code or you should get at least some HTTP headers like this:

In developer console it giving an error

THIS REQUEST HAS NO RESPONSE DATA

Check your apache error log - as suggested by @gavinsbtm

Hi,
This is what i got
[04-Feb-2020 08:44:55 UTC] PHP   1. {main}() D:\wamp64\www\orchard\public\zf_index.php:0
[04-Feb-2020 08:44:55 UTC] PHP   2. Zend_Application->run() D:\wamp64\www\orchard\public\zf_index.php:11
[04-Feb-2020 08:44:55 UTC] PHP   3. Bootstrap->run() D:\wamp64\www\orchard\vendor\shardj\zf1-future\library\Zend\Application.php:384
[04-Feb-2020 08:44:55 UTC] PHP   4. Zend_Controller_Front->dispatch() D:\wamp64\www\orchard\vendor\shardj\zf1-future\library\Zend\Application\Bootstrap\Bootstrap.php:105
[04-Feb-2020 08:44:55 UTC] PHP   5. Zend_Controller_Plugin_Broker->preDispatch() D:\wamp64\www\orchard\vendor\shardj\zf1-future\library\Zend\Controller\Front.php:941
[04-Feb-2020 08:44:55 UTC] PHP   6. Web_Controller_Plugin_OrchAuth->preDispatch() D:\wamp64\www\orchard\vendor\shardj\zf1-future\library\Zend\Controller\Plugin\Broker.php:309
[04-Feb-2020 08:44:55 UTC] PHP   7. Service_UIManage->createUIRestrictions() D:\wamp64\www\orchard\library\Web\Controller\Plugin\OrchAuth.php:162
[04-Feb-2020 08:44:55 UTC] PHP   8. Model_UiRestrictions::getAllUiRestrictionByAccountType() D:\wamp64\www\orchard\src\Service\UIManage.php:31
[04-Feb-2020 08:44:55 UTC] PHP   9. Doctrine_Query::create() D:\wamp64\www\orchard\application\models\UiRestrictions.php:21
[04-Feb-2020 08:44:55 UTC] PHP  10. Doctrine_Query->__construct() D:\wamp64\www\orchard\vendor\siturra\doctrine1-php7\lib\Doctrine\Query.php:192
[04-Feb-2020 08:44:55 UTC] PHP  11. spl_autoload_call() D:\wamp64\www\orchard\vendor\siturra\doctrine1-php7\lib\Doctrine\Query\Abstract.php:292
[04-Feb-2020 08:44:55 UTC] PHP  12. Composer\Autoload\ClassLoader->loadClass() D:\wamp64\www\orchard\vendor\siturra\doctrine1-php7\lib\Doctrine\Query\Abstract.php:292
[04-Feb-2020 08:44:55 UTC] PHP  13. Composer\Autoload\includeFile() D:\wamp64\www\orchard\vendor\composer\ClassLoader.php:322

Hi mate, I'm currently still running everything on PHP 7.2 and likely won't change to 7.4 for another half a year. However when I tested swapping over to 7.4 inside my docker container a little while ago I don't remember having any issues.
I notice in your stack trace that the issue seems to be happening in your OrchAuth plugin though so that's likely where you need to look.
I'd recommend you set up xdebug and step through so you can try to figure out what's working differently for you when compared to the previous version you were using.

Hi mate, I'm currently still running everything on PHP 7.2 and likely won't change to 7.4 for another half a year. However when I tested swapping over to 7.4 inside my docker container a little while ago I don't remember having any issues.
I notice in your stack trace that the issue seems to be happening in your OrchAuth plugin though so that's likely where you need to look.
I'd recommend you set up xdebug and step through so you can try to figure out what's working differently for you when compared to the previous version you were using.

Hi mate,
Its working fine now actually had some application level misconfiguration. Fixed those and now its runing. So closing this issue.


Use full screen for PR comparison

Follow-up #22844
close #28142
Before

After

Should we backport this? I can consider this as an enhancement.