Webpack 4.10.0 destroys production build

Bug report
What is the current behavior?
Uncaught TypeError: Cannot read property 'call' of undefined
in different projects, only in production build, only after update from 4.9.2 to 4.10.0
If the current behavior is a bug, please provide the steps to reproduce.
What is the expected behavior?
not error

Other relevant information:
webpack version:  4.10.0
Node.js version: 10.1.0
Operating System: Windows and Linux
Additional tools: -
in webpack.build.js:
const MiniCssExtractPlugin = require('mini-css-extract-plugin');
const merge = require('webpack-merge');
const Webpack = require('webpack');
const CompressionPlugin = require('compression-webpack-plugin');

const config = require('./config');

const publicPath = process.env.npm_package_config_public_path;

module.exports = merge(config, {
  module: {
    rules: [
      {
        test: /\.js$/,
        use: ['babel-loader'],
        exclude: /node_modules/,
      },
      {
        test: /\.s?css$/,
        use: [
          MiniCssExtractPlugin.loader,
          'css-loader',
          'postcss-loader',
          'sass-loader',
        ]
      }
    ],
  },
  mode: 'production',
  output: {
    filename: '[name].[chunkhash].js',
    chunkFilename: '[name].[chunkhash].chunk.js',
    publicPath: publicPath,
  },
  optimization: {
    splitChunks: {
      cacheGroups: {
        vendor: {
          test: /node_modules/,
          name: 'vendor',
          chunks: 'initial',
          enforce: true
        }
      }
    },
    minimize: true,
    runtimeChunk: {
      name: 'vendor'
    }
  },
  plugins: [
    new MiniCssExtractPlugin({
      filename: '[name].[chunkhash].css',
      chunkFilename: '[name].[chunkhash].css'
    })
  ],
});

Thanks for your report. It would be great if you reduce your issue to a small reproducible example. Best put this example into a github repository together with instructions how to get to the problem.
Or please provide a stack trace with copies of the lines in the stacktrace.

I have problem with production build, too.
4.10.0 and 4.10.1 fails with the following:
ERROR in chunk about-us
assets/276.2732141f9c704103bdef.css
/home/milosz/ivyspace/ivyspace-frontend/node_modules/babel-loader/lib/index.js??ref--4!/home/milosz/ivyspace/ivyspace-frontend/frontend/scripts/features/public/aboutUs/components/Details.js
RuntimeTemplate.moduleId(): Module /home/milosz/ivyspace/ivyspace-frontend/node_modules/lodash-es/camelCase.js has no id. This should not happen.

ERROR in chunk app [entry]
assets/app.2732141f9c704103bdef.css
/home/milosz/ivyspace/ivyspace-frontend/node_modules/babel-loader/lib/index.js??ref--4!/home/milosz/ivyspace/ivyspace-frontend/frontend/scripts/features/common/components/ScreenSizeManager.js
RuntimeTemplate.moduleId(): Module /home/milosz/ivyspace/ivyspace-frontend/node_modules/lodash-es/debounce.js has no id. This should not happen.

4.9.2 builds fine.

we're also experiencing this issue. webpack 4.10.1
ERROR in chunk client [entry]
    [name].[chunkhash:8].js
    /.../node_modules/babel-loader/lib/index.js??ref--4!/Volumes/HD1/SITES/UNWRAPT/unwrapt-web-app/src/modules/users.js
    RuntimeTemplate.moduleId(): Module /.../node_modules/redux-form/es/SubmissionError.js has no id. This should not happen.

changing mode to 'development' in webpack.config.js resolves the issue, so is definitely related to production mode.
Any steps we can take to help debug, please let me know, and I can contribute updates to this issue.

@evilebottnawi interesting. I can't reproduce with simple repo. so may be more to it.  Endeavouring to reproduce outside current bloated project.

these are my errors
public/js/[name][chunkhash].js
<EMAIL_ADDRESS>RuntimeTemplate.moduleId(): Module /home/pipeline/63551784-fde3-4803-96e3-0610fccde3c5/node_modules/strip-loader/lib/index.js?strip[]=console.info,strip[]=console.log!/home/pipeline/63551784-fde3-4803-96e3-0610fccde3c5/node_modules/lodash-es/forEach.js has no id. This should not happen.

ERROR in chunk app [entry]
public/js/bundle[chunkhash].js
<EMAIL_ADDRESS>RuntimeTemplate.moduleId(): Module /home/pipeline/63551784-fde3-4803-96e3-0610fccde3c5/node_modules/strip-loader/lib/index.js?strip[]=console.info,strip[]=console.log!/home/pipeline/63551784-fde3-4803-96e3-0610fccde3c5/node_modules/rxjs/_esm5/internal/Observable.js has no id. This should not happen.

@tomasztomys could you please try 4.10.2?

@sokra Appears to be fixed (was having what @mieszko4 was having previously). Thanks!

This was my original issue:
ERROR in chunk main [entry]
main-240b36fe47f735bd8184.css
/Users/swerner/Workspace/init/display/node_modules/source-map-loader/index.js??ref--4!/Users/swerner/Workspace/init/display/node_modules/edge-core/lib/browser.es5.esm.js
RuntimeTemplate.moduleId(): Module /Users/swerner/Workspace/init/display/node_modules/apollo-client/ApolloClient.js has no id. This should not happen.

Seems like the update to 4.10.2 fixed it! Thanks @sokra.

I confirm 4.10.2 works for me! Thanx!

I am running into this in my app also in production. I'm still seeing this in 4.10.2.
Based on the comments in this thread it looks like there might be two issues:

The originally reported Uncaught TypeError: Cannot read property 'call' of undefined.
The has no id. This should not happen.

For us, we're only experiencing the 1st issue.

It works now

I can confirm that an issue still exists in 4.10.2. I'm getting Unhandled promise rejection TypeError: Cannot read property 'call' of undefined when building in production with 4.10.2. 4.9.2 works fine.

@dagstuan can you create minimum reproducible test repo?

@evilebottnawi I cant seem to be able to reproduce it in a minimal repo, but the full stack trace is as follows:
    at o (bootstrap:81)
    at Object.288 (immutable.js:37)
    at o (bootstrap:81)
    at Object.313 (formWarning.js:41)
    at o (bootstrap:81)
    at Object.522 (formActions.js:44)
    at o (bootstrap:81)
    at Module.<anonymous> (cc78b3ce504028caff47.js:1)
    at Module.296 (rollout.js:13)
    at o (bootstrap:81)
    at .*\.js$ namespace object:168
    at a (es6.promise.js:75)
    at es6.promise.js:92
    at MutationObserver.c (_microtask.js:18)```

It seems to crash when trying to import something that is re-exported from redux-form. The relevant source file in redux form is here: https://github.com/erikras/redux-form/blob/master/src/immutable.js

it's happening in v4.10.2 with redux-form for me too.
The missing moduleId is 1022, which I can trace to:
var _SubmissionError__WEBPACK_IMPORTED_MODULE_2__ = __webpack_require__(1022);
@dagstuan i notice it's borking with your promise rejection, can you confirm that removing the SubmissionError import fixes it?
@sokra it's hardly minimal, but if you need a repo to test it with, you can use https://github.com/ParabolInc/action/tree/27bc61b3d66ef1dfde026d09e96bb588fb609b45 and run yarn bs to build and start the server on localhost:3000. you'll see the error right away in the browser console.
Full stack trace
bootstrap:81 Uncaught TypeError: Cannot read property 'call' of undefined
    at __webpack_require__ (bootstrap:81)
    at Object.149 (_nodeUtil.js:30)
    at __webpack_require__ (bootstrap:81)
    at Object.160 (app_a4a32c429d4825e19a8b.js:1354)
    at __webpack_require__ (bootstrap:81)
    at Module.241 (app_a4a32c429d4825e19a8b.js:2357)
    at __webpack_require__ (bootstrap:81)
    at checkDeferredModules (bootstrap:43)
    at bootstrap:216
    at bootstrap:216

@mattkrick removing imports to SubmissionError didn't fix the issue for me. So I must be importing something else somewhere that makes it crash :)

I don't want to speculate, but maybe the library is exporting something that has some side effects? And by doing that it breaks the tree shaking?

Stil got the same error bootstrap:2 Uncaught TypeError: Cannot read property 'call' of undefined with webpack 4.11.0

This issue got a bit confusing, some reporting it fixed, some reporting broken. I closed this now as the original poster reports it fixed. If you still have the issue, please open a new issue with stack trace and repro repo.

A fix that worked for me with webpack 4.11.1:
I use rxjs 6.2 with the rxjs-compat module. If I change the add operator imports from:
import 'rxjs/add/operator/delay';
import 'rxjs/add/operator/skip';
/* ... */

to:
import 'rxjs-compat/add/operator/delay';
import 'rxjs-compat/add/operator/skip';
/* ... */

The issue goes away.

I encounter this issue only in production build.
I try the below version
4.1.0  ---> build success
4.9.2  ---> build success
4.10.0 ---> build fail
4.10.1 ---> build fail
4.10.2 ---> build fail
I have 2 type of build fail message:
Sometimes is this(in terminal):
        ERROR in chunk Auth
        js/[name].[hash].js
        /Users/My_Project/node_modules/babel-loader/lib/index.js!/Users/My_Project/src/features/Auth/LoginForm.js
        RuntimeTemplate.moduleId(): Module /Users/My_Project/node_modules/redux-form/es/Field.js has no id. This should not happen.

        ERROR in chunk bundle [entry]
        js/[name].[hash].js
        /Users/My_Project/node_modules/babel-loader/lib/index.js!/Users/My_Project/src/features/TestImgAndCss/TestImgAndCssEpics.js
        RuntimeTemplate.moduleId(): Module /Users/My_Project/node_modules/redux-observable/lib/esm/operators.js has no id. This should not happen.

Sometimes is this(build success, but open file in browser has error):
        Uncaught TypeError: Cannot read property 'call' of undefined

And I found it can be fixed  after removing the vendor in my entry setting
from
      const config = {
        mode: 'production',
        entry: {
          bundle: ['babel-polyfill', './src/index.js'],
          vendor: [
            'lodash', 'react', 'react-dom',
            'react-redux', 'react-router', 'redux', 'redux-form', 'redux-thunk',
          ],
        },
        output: {
          filename: 'js/[name].[hash].js'
        },
      };

to
      const config = {
        mode: 'production',
        entry: {
          bundle: ['babel-polyfill', './src/index.js'],
        },
        output: {
          filename: 'js/[name].[hash].js'
        },
      };

Got the same error with webpack >= 4.10.0 and redux-form. However, I cannot create an example to reproduce the bug.

For the bug related to redux-form I've reproduced it in a minimal case explained here: https://github.com/webpack/webpack/issues/7499#issuecomment-396831278

@henrist

This issue got a bit confusing, some reporting it fixed, some reporting broken. I closed this now as the original poster reports it fixed. If you still have the issue, please open a new issue with stack trace and repro repo.

entry: {home: '/webpack-demo/src/js/home.js'}

chunkFilename："js/[name]-chunk-[chunkhash].js"

What is the name of the output JS file？
My js file name is 'home-chunk-70aa09e8630fee6b4bb2.js'

I am seeing this issue also using webpack 4.14.0 and nodejs v8.
Building my application in two identical environment (GAE Docker) once with production deployment and the other with development. Dev works fine whilst production creates the following error:
demo:1 TypeError: Cannot read property 'call' of undefined
    at o (demo:1)
    at Object.349 (ui.chunk.bundle.js:1)
    at o (demo:1)
    at o.t (demo:1)

It does seem this is do with code splitting functionality.
Any update on this issue, if it has been found or is the best thing to do just revert to webpack 4.9.2 for now?

Still happening in webpack 4.16.2 development works fine but production fails for some reason.
"webpack": "4.16.2",
"dll-link-webpack-plugin": "^3.0.5",

Also a weird thing is that it says something about bootstrap. I don't use bootstrap anywhere.
TypeError: Cannot read property 'call' of undefined

@developer239 bootstrap means "setting up" not bootstrap library

I think that this is the problem:
/******/ 	deferredModules.push([null, "assets","default","vendor"]);

because the first index is null it cannot find a module with that name in the modules array, I changed it to the name of the entry point js file, and it worked,
/******/ 	deferredModules.push(["./main.js", "assets","default","vendor"]);

I also tied setting it to zero, but that didn't work.

Still seeing this issue in webpack 4.19.1. We are using dll, code splitting.

Adding sideEffects:false fixed it for me. Whew!
optimization: { sideEffects: false }

the year is 2023-02-20:
i can confirm that "webpack": "^5.75.0" still removes code that should not have been removed.
package.json:
"scripts": {
"start": "webpack-dev-server --mode development --config webpack.config.cjs",
"build": "webpack --mode production --config webpack.config.cjs"
},
removed important code which caused me days of debugging.
this fixed it webpack.config.js:
optimization: {
minimize: false, //if you leave this out webpack strips out code in production build...
},


Several modifications to make the Jolokia JVM Agent start and work on…

Hi Roland,
currently at a customer that really wants to use the IBM JVM for some of their Docker-images and thus I had to modify the Jolokia JVM Agent that I want to use for my projects to work with that JVM as well. Hope it's useful. In short, Jolokia seems to work for me now and the tests go through on both OpenJDK 8 and IBM J9 v8.
/D

Oops, just saw that "All checks have failed" ( what an exaggeration ;-) ), it seems the Oracle JDK7 checks fail due to:
Error: JAVA_HOME is not defined correctly.
  We cannot execute /usr/lib/jvm/java-7-oracle/bin/java

Forgot to add the contribution sign-off. Sending a new pull request.


fix(build): stringify config.define

When config.define is set like below (vite.config.js)
module.exports = {
  define: {
    __VAR__: 'string'
  }
}

and main.js is like below.
console.log(__VAR__)

On dev server it becomes like
console.log('string')

though built script becomes like
console.log(string)

This PR will fix the build to be same as dev server.

This is the expected behavior. Users should wrap it with JSON.stringify explicitly if they expected it is a string. Or you may need to write something like console.log("__VAR__") in your source code. We should update the docs to proper documentation though. Thanks for your contribution but sorry as I am closing this for now.

So you mean the dev server's output(console.log('string')) is a bug and it should be console.log(string)?
Or is it expected that the output differs between dev server and build?

Sorry, it's my bad, I misread the issue, could you reopen it? Thanks 😅

And yes, the bug is on the dev side.

I have restored the branch and made a change to fix it.
I am not sure whether this works properly, but it worked when I wrote __VAR__ inside .ts.

I am sorry I forgot checking the test.
I will resolve conflicts and fix it.

I have fixed it and added a test! 😃

It's better if move replace logic into serverReplacePlugin, because it is not related of module rewrite.

I have moved it into serverPluginReplace.ts and changed it not to replace inside <template> for consistency.
https://github.com/vitejs/vite/blob/47eb448709fa09f8bd02e1d4159d68e467ed68a7/src/node/build/index.ts#L352-L361

I rethink the orignal issues and we should fix build.

Why did you think so? (I just want to know the reason.)
Is it because replacing to a raw source code is a bit curious?

It is unnecessary and has  some overhead of transfromed file can be replaced.It can be apply correct value by window, we should follow the pervious implement.

I see. Thanks for answering.


Decouple OSGi service and api to standalone bundles

I am not sure why the greeter api now is packaged with the base app, but any service should have it's own api bundle.
Please decouple the Greetservice, see:
https://github.com/Maurice-Betzel/base-starter-flow-karaf/tree/update-19.0.1

You are right and we actually discussed it a while a back in the team that that would be the correct way to do it. But then again getting rid of that when starting to develop against your own service is bit more complicated than just deleting that one file, so we left it there at that point.
If you want to make a PR for that, you are welcome, otherwise we'll look into this later when working again with the example.

This is now prepared in this branch:
https://github.com/vaadin/base-starter-flow-karaf/tree/feature/separate-api-module-for-greetservice
Based on v19 branch because master don't work for me ATM.


CI Fixes

Install node types
Pass secrets to deploy script

@Piccirello I added ts-node and @types/node to the main package lock and it seems to work in my testing 👍 Good for your review


Function as password

Would you accept a PR that allows the password to be a zero-arity function? There are a few use cases for it. One example is pulling the password from a secrets store. Another is using token that is regenerated after a period of time, like AWS IAM authentication in Postgres RDS

Hey, sorry I hadn't replied to this yet! I keep going back and forth on whether this is ok. The latter case certainly can't be solved by other options like an environment variable populated from the secrets store... But would the password then have to be re-evaluated after every disconnect? Not necessarily bad if that is what the requirement for getting these tokens is.
And should it really be zero arity and not pass the database name, uri, etc?

If it weren't for the latter case I'd suggest starting the pool manually instead of through pgo application startup, but that doesn't help if you need it to get a new password when it tries to reconnect.

I don’t think the fun to retrieve the credential would necessarily need the DB URI, etc. I figured it’s better to use a zero arity fun because you can wrap whatever values you need in a zero arity fun. In the case of IAM, the common case is pulling the credential from your EC2 instance. I haven’t dug too deep, but I don’t think that credential is tied to the DB. It’s tied to the EC2 instance. You wouldn’t use the DB URI.
You’ll definitely eat the cost of the function for every new connection. But, it’s a pool so we amortize the credential retrieval cost over the life of the connection in the pool.


typo fix

Sorry for being pedantic, came across the typo so thought of fixing with a PR

great, thank you!


Agnostic button, how to ?

Hi, I'm playing with node-red + node-red-contrib-homekit-bridged with great fun, so first of all I want to thank you for this beautiful project.
I've read about the differences between switch and statelessProgrammableSwitch but none of them works in my case, I suppose I'm missing something because my need is very basic.
I have some remote controlled outlets with two remotes. I would like to use one of them with Siri, telling to turn ON/OFF the lights.

I would like HomeKit to be absolutely agnostic about the light states.
As you can see remotes are two, so even if the lightbulb or switch state are ON inside HomeKit, that could be false (someone could have triggered the other switch). And beside it, the fact to be, for instance ON in HomeKit means that a switch can't be turned ON again without turning it Off before, HomeKit won't let you do it.
statelessProgrammableSwitch could have been the solution, but:
1. it needs a Hub
2. it changes the states of switches, which brings me back to the first problem, the switch state isn't stateless.
Any ideas ?

You need to decide what you want expose to HomeKit:

the stateful outlets (as Outlet or Lightbulb); or
the stateless remote switches (as Stateless Programmable Switch).

With 1 you could tell Siri to switch on the lights, with 2, you could use the remotes to control HomeKit accessories, if you have a HomeKit hub.
In both cases, you’d need some hardware that talks the same protocol as the remotes and outlets, and you’d need a homebridge plugin that transalates this protocol to HomeKit.

Thanks @ebaauw, I think I need something like a push button. I need to better grasp on how it works, I'm able to setup one in node-red and to show it into my hub (iPad) but then I'm stuck. A switch state can be modified in HomeKit app (ON/Off), a SPS can't (or, I'm unable to). Any example is appreciated.

A push button (stateless programmable switch) is an input-only device, just like the physical remote.  It’s used to control other HomeKit devices, it cannot be controlled itself from HomeKit (nor by Siri).
If you want to provide a virtual push button to a HomeKit app, you’d need to create a write-only custom characteristic, like Identify.  Unfortunately, neither Apple’s Home app nor Siri support custom characteristics.  The only thing you can do is create a regular Switch service, and automatically reset the On chacteristic to false, say 500ms after it’s been pressed.

The only thing you can do is create a regular Switch service, and automatically reset the On chacteristic to false, say 500ms after it’s been pressed.

Unfortunately that doesn't solve the problem, defining it back to Off means that I can only turn it on, which is not what I want as I could turn the same switch off (for instance) multiple times in a row.
Thanks for the SPS explanation, it makes sense now.

Just to add a bit more to this, Siri doesn’t check the switch state before sending a command, so you can ask Siri to turn on a lamp this is already on etc.

On May 23, 2018, at 1:58 PM, Daniele Tabanella<EMAIL_ADDRESS>wrote:
Closed #587 https://github.com/KhaosT/HAP-NodeJS/issues/587.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/KhaosT/HAP-NodeJS/issues/587#event-1642455913, or mute the thread https://github.com/notifications/unsubscribe-auth/AS5CmMFQAaiM7QC-wtKXnoHtwozoslsuks5t1aM2gaJpZM4UI6F8.

Unfortunately that doesn't solve the problem, defining it back to Off means that I can only turn it on, which is not what I want as I could turn the same switch off (for instance) multiple times in a row.

You’d create two of these: one for On and one for Off.

One year and a half later I've found an easy way to use both remote control and HomeKit, I've published a repository with some instructions.

@leRenart maybe you could do it with latest node-red-contrib-homekit-bridged?
I would suggest it if you are using node-red anyway.

@Shaquu yeah I'm using it, maybe not the latest version, I'll take a look and update the doc


imapnotify connecting but not triggering notifications

Hi, I've been using the latest version of imapnotify for a while now, installed from the AUR (0.1.0-6). However, it seems to have stopped working in the last few days. It appears to connect fine, but doesn't notify on new emails. I'm using the systemd service.
$ systemctl status --user<EMAIL_ADDRESS>●<EMAIL_ADDRESS>- Execute scripts on new messages using IDLE imap command
   Loaded: loaded<EMAIL_ADDRESS>enabled; vendor preset: enabled)
   Active: active (running) since Sat 2016-07-16 11:29:26 AEST; 5min ago
 Main PID: 11945 (node)
   CGroup: /user.slice/user-1000.slice/user<EMAIL_ADDRESS>           └─11945 node /usr/bin/imapnotify -c /home/protist/.config/imapnotify/main.js

Jul 16 11:29:26 protist-XPS-17 systemd[596]: Started Execute scripts on new messages using IDLE imap command.
Jul 16 11:29:27 protist-XPS-17 imapnotify[11945]: {"name":"imap_inotify","hostname":"protist-XPS-17","pid":11945,"level":30,"box":"INBOX","msg":"Connected to server","time":"2016-07-16T01:29:27.026Z","v":0}
Jul 16 11:29:27 protist-XPS-17 imapnotify[11945]: {"name":"imap_inotify","hostname":"protist-XPS-17","pid":11945,"level":30,"box":"INBOX","msg":"Selecting box","time":"2016-07-16T01:29:27.028Z","v":0}

After a new email arrives on the server, nothing happens, and nothing changes in the status output.
OTOH, IMAP IDLE on my Android phone's K-9 app appears to be working fine with the same account. Is there a way to troubleshoot imapnotify further?

Hi, can you start imapnotify process with DEBUG environment variable set, e.g:
DEBUG=1 /usr/bin/imapnotify -c /home/protist/.config/imapnotify/main.js

May be there will be something interesting in the debug output.

Nothing seems to happen. I disabled the systemd service, and started it manually as recommended.
$ DEBUG=1 /usr/bin/imapnotify -c ~/.config/imapnotify/main.js
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":6958,"level":30,"box":"INBOX","msg":"Connected to server","time":"2016-07-17T23:42:04.667Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":6958,"level":30,"box":"INBOX","msg":"Selecting box","time":"2016-07-17T23:42:04.668Z","v":0}

I sent myself an email. It was received with K-9's IDLE, but nothing was triggered with imapnotify.

I did a bit more testing. I tested imapnotify with another email account from a different provider, and that worked fine.
However, I just tested the original "broken" email account with Thunderbird, and that worked fine with IDLE too. I'm confused as to what might have caused this problem, as imapnotify hasn't been updated in quite a while.
It looks like the AUR version of imapnotify has a single dependency on nodejs. I checked my pacman logs, and I upgraded this from 6.2.2-1 -> 6.3.0-1 on 2016-07-07. I tried reverting to the previous version of nodejs, then manually starting imapnotify again, but this didn't help. Prior to this, I upgraded nodejs from 6.2.1-1 -> 6.2.2-1 on 2016-06-18. I'm pretty sure I didn't notice the problem from this long ago, but I tried reverting anyway. This also made no difference.

I just emailed my email provider, and they suggested that perhaps "Dovecot has been updated on the server side and the mail protocols have been updated". Is it possible that imapnotify only works with an older version of dovecot?

It is possible.
Can you ask your provider to tell what version of Dovecot they are running now so I will be able to test it.

They just replied. It's dovecot-2.2.25-1.cp1158.x86_64. They were unable to tell me the previous version.

Thanks, I will try to reproduce the bug as soon as I have free time.

Thank you, I appreciate it. Also, I'm happy to make you a temporary account on the email server I use, if you like. However, I'm not sure how useful this is, since I don't think I have access to any debugging logs from their end. I guess this will let you reproduce this issue at least.

Oh, that will be great, thanks.

I've emailed you the details. Thanks again for looking at this.

Hello, I was just wondering if you have had time to look into this. Otherwise, I'll probably look into other alternatives. (I don't mean to sound rude; I just want my email to work again.) Cheers.

Sorry, I haven't had a chance to look into this issue.
I do not use imapnotify myself anymore (I used email only for my work, and we switched to other tools).
I welcome you to look for alternatives, on my part, I will try to look into the issue in a week or two (it's interesting after all, why it does not work)

No worries at all. I understand that it's no longer a high priority for you.
However, I could only find two other programs that do similar things to imapnotify, and they will require me learning some basic perl or python. Since transitioning to another system will take a bit of work for me personally, I'm happy to wait a week or two and see what happens with imapnotify. Thanks again.

Could you please email me temporary account details one more time, it seems like I lost it.

No worries; done. Thank you again for looking at this.

I tested imapnotify with the config you gave me and it seems to work.
Can you test it once again on the temporary account and confirm it does not work.
{"name":"imap_inotify","hostname":"hostname","pid":56217,"level":30,"box":"INBOX","msg":"Connected to server","time":"2016-09-05T10:55:31.906Z","v":0}
{"name":"imap_inotify","hostname":"hostname","pid":56217,"level":30,"box":"INBOX","msg":"Selecting box","time":"2016-09-05T10:55:31.908Z","v":0}

{"name":"imap_inotify","hostname":"hostname","pid":56217,"level":30,"box":"INBOX","msg":"New mail","time":"2016-09-05T10:56:38.157Z","v":0}
{"name":"imap_inotify","hostname":"hostname","pid":56217,"level":20,"msg":"Found command: ls","time":"2016-09-05T10:56:38.161Z","v":0}

This is very odd! I tested again, but it still doesn't work for me. I just get the same two initial lines.
$ DEBUG=1 /usr/bin/imapnotify -c ~/.config/imapnotify/test.js
{"name":"imap_inotify","hostname":"my_hostname","pid":22308,"level":30,"box":"INBOX","msg":"Connected to server","time":"2016-09-06T02:39:50.517Z","v":0}
{"name":"imap_inotify","hostname":"my_hostname","pid":22308,"level":30,"box":"INBOX","msg":"Selecting box","time":"2016-09-06T02:39:50.519Z","v":0}

I was trying to work out how our systems might differ, which might explain why it works for you and fails for me. I wasn't sure if you were using the same password manager, so I also tried hardcoding the password into the config file, with exports.password = 'foo';. This still failed.
I'm not sure what else might be different. Is nodejs the only dependency? I'm on version 6.5.0-1 now (in Arch Linux), although as above, I did try reverting to previous versions.

Can you try to test from a different IP?

I move my laptop between home and work, so I have already tested for my normal email accounts. I just tested again for this new test account, but it still doesn't work unfortunately.

BTW what distro do you use? It might be another way to troubleshoot?

I'm testing from mac os. I'll try to use docker to test from ubuntu.

I tried from archlinux and node docker images, both times imapnotify worked.
Can you try to test it in node docker image.

Sorry, I'm not familiar at all with docker. Could you please explain in more detail or show me a tutorial? I couldn't find anything clear on the web.

Huh… that's odd. I installed imapnotify on my wife's computer, which also runs an up-to-date Arch Linux system. It works fine there! All relevant package versions would be the same, but the computer itself is different. I really don't know where else to find a difference between the two.

FWIW I also set up a new user on my computer, that imapnotify fails for that user too.

That's interesting, maybe you set up firewall rules, that somehow prevent impanotify to work correctly. Maybe try to disable it and test.

Good idea, but unfortunately disabling my firewall made no difference.

Ok, let's try to test in docker.

install docker package
run node image docker run -it node bash
inside the container, clone node-imapnotify repo git clone https://github.com/a-sk/node-imapnotify
install npm deps: cd node-imapnotify && npm i
put imapnotify config file somewhere
run it: node-imapnotify/bin/imapnotify -c absolute path to the config file

Just for posterity, in case I need to repeat it, here are the precise commands I had to use. And imapnotify didn't work, unfortunately. (See the last block for details.)
$ sudo pacman -S docker
$ sudo gpasswd -a ${USER} docker

Log in and out again.
$ sudo systemctl start docker
$ docker run -it node bash
root@11ba2a33fc1a:/# git clone https://github.com/a-sk/node-imapnotify
root@11ba2a33fc1a:/# cd node-imapnotify && npm i && cd ..

In the host system,
$ docker cp ~/.config/imapnotify/test.js 11ba2a33fc1a:/test.js

In the container,
root@11ba2a33fc1a:/# node-imapnotify/bin/imapnotify -c /test.js
{"name":"imap_inotify","hostname":"11ba2a33fc1a","pid":46,"level":30,"box":"INBOX","msg":"Connected to server","time":"2016-09-07T10:39:37.349Z","v":0}
{"name":"imap_inotify","hostname":"11ba2a33fc1a","pid":46,"level":30,"box":"INBOX","msg":"Selecting box","time":"2016-09-07T10:39:37.354Z","v":0}

Send email to account and nothing changes.

So, this is not your IP, not your environment, not your firewall, but it's definitely something on your computer.
Can your boot from a usb drive and try to test there.
I'm almost out of ideas ;(

Ugh, I've spent an hour or so on this, but I'm having trouble, possibly due to corrupt USB sticks. This might be more complicated that envisaged.
I did try dual booting into my (K)ubuntu install, and imapnotify works perfectly well there. You did have it working in Arch though right? Was that just a "normal" install? I could also try installing Arch in virtualbox and testing there…

This is really, really odd… I tested with Arch in Virtualbox (within my Arch host), and it works fine. However, when testing and previously in Kubuntu, I logged into the webmail of the test account, and sent an email to itself.
I tested again from the original Arch system that doesn't work, sending mail via webmail. Now it works! I tested again using my "normal" testing routine… mutt + msmtp. This time, imapnotify didn't work again. So for some bizarre reason, on my normal system, imapnotify works if I send email through the same web interface, but not if I send it through mutt + msmtp + a different smtp server. I don't know how that can make a difference, since I've been confirming that the email is actually received in the box in all cases. Both situations should be the same.
(As an aside, when imapnotify worked on my wife's computer, it was using the "normal" test, i.e. mutt + msmtp. I also retested Arch-in-Arch Virtualbox, and this works with the "normal" test, too.)
Aaaaaaanndd… just to be thorough, I tested three more times with the original Arch system and webmail, and now it fails. But, it did work at some stage on this system, so that shows it is possible.

Wow this is odd.
Can you test once again and try to sniff traffic using tcpdump,
something like tcpdump -vv -l -A  'host $your_host'
And for the test, change port to 143 and tls to false to see the text and not encrypted garbage.
If this fails and you don't see any traffic you can try to add iptables rule to log imap traffic.

I made those changes in the imapnotify config, but it doesn't seem to want to connect now. These are the changes I made:
--- /home/protist/.config/imapnotify/test.js	2016-09-06 12:39:48.921828612 +1000
+++ /home/protist/.config/imapnotify/test.nocrypt.js	2016-09-09 21:40:35.325162334 +1000
@@ -6,8 +6,8 @@
 }

 exports.host = "FOO"
-exports.port = 993;
-exports.tls = true;
+exports.port = 143;
+exports.tls = false;
 exports.tlsOptions = { "rejectUnauthorized": true };
 exports.username =<EMAIL_ADDRESS> exports.password = 'baz';

And now it won't run.
$ imapnotify -c ~/.config/imapnotify/test.nocrypt.js
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":23430,"level":50,"box":"INBOX","msg":"Error registered","time":"2016-09-09T11:40:47.230Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":23430,"level":50,"box":"INBOX","msg":"{ Error:<PHONE_NUMBER>58528:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:794:\n\n    at Error (native)\n    at Socket.ondata (_stream_wrap.js:58:20)\n    at emitOne (events.js:96:13)\n    at Socket.emit (events.js:188:7)\n    at readableAddChunk (_stream_readable.js:176:18)\n    at Socket.Readable.push (_stream_readable.js:134:10)\n    at TCP.onread (net.js:543:20) source: 'socket' }","time":"2016-09-09T11:40:47.246Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":23430,"level":30,"box":"INBOX","msg":"Restarting immediately","time":"2016-09-09T11:40:47.247Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":23430,"level":50,"box":"INBOX","msg":"Error registered","time":"2016-09-09T11:40:47.418Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":23430,"level":50,"box":"INBOX","msg":"{ Error:<PHONE_NUMBER>58528:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:794:\n\n    at Error (native)\n    at Socket.ondata (_stream_wrap.js:58:20)\n    at emitOne (events.js:96:13)\n    at Socket.emit (events.js:188:7)\n    at readableAddChunk (_stream_readable.js:176:18)\n    at Socket.Readable.push (_stream_readable.js:134:10)\n    at TCP.onread (net.js:543:20) source: 'socket' }","time":"2016-09-09T11:40:47.420Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":23430,"level":30,"box":"INBOX","msg":"Scheduling restart in 3000","time":"2016-09-09T11:40:47.420Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":23430,"level":50,"box":"INBOX","msg":"Error registered","time":"2016-09-09T11:40:50.548Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":23430,"level":50,"box":"INBOX","msg":"{ Error:<PHONE_NUMBER>58528:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:794:\n\n    at Error (native)\n    at Socket.ondata (_stream_wrap.js:58:20)\n    at emitOne (events.js:96:13)\n    at Socket.emit (events.js:188:7)\n    at readableAddChunk (_stream_readable.js:176:18)\n    at Socket.Readable.push (_stream_readable.js:134:10)\n    at TCP.onread (net.js:543:20) source: 'socket' }","time":"2016-09-09T11:40:50.548Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":23430,"level":30,"box":"INBOX","msg":"Scheduling restart in 6000","time":"2016-09-09T11:40:50.548Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":23430,"level":50,"box":"INBOX","msg":"Error registered","time":"2016-09-09T11:40:56.676Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":23430,"level":50,"box":"INBOX","msg":"{ Error:<PHONE_NUMBER>58528:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:794:\n\n    at Error (native)\n    at Socket.ondata (_stream_wrap.js:58:20)\n    at emitOne (events.js:96:13)\n    at Socket.emit (events.js:188:7)\n    at readableAddChunk (_stream_readable.js:176:18)\n    at Socket.Readable.push (_stream_readable.js:134:10)\n    at TCP.onread (net.js:543:20) source: 'socket' }","time":"2016-09-09T11:40:56.676Z","v":0}
/usr/lib/node_modules/imapnotify/bin/imapnotify:83
        throw Error('Max retry limit reached')
        ^

Error: Max retry limit reached
    at Error (native)
    at Connection.<anonymous> (/usr/lib/node_modules/imapnotify/bin/imapnotify:83:15)
    at Connection.g (events.js:286:16)
    at emitOne (events.js:96:13)
    at Connection.emit (events.js:188:7)
    at TLSSocket._onError (/usr/lib/node_modules/imapnotify/node_modules/imap/lib/Connection.js:150:10)
    at emitOne (events.js:96:13)
    at TLSSocket.emit (events.js:188:7)
    at emitErrorNT (net.js:1272:8)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)

I answered to the email address you sent me test account details from.

That new config still won't connect for me. I wonder if that is informative?
Just to make sure that there was no copy-paste error, I modified your new test.json config, reverting the port back to 993 and enabling tls. This connects fine (although I didn't test the notify part), so presumably we're using the same file. Using the unmodified config file results in the following errors, similar to before.
$ imapnotify -c ~/.config/imapnotify/test.nocrypt.json
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":5204,"level":50,"box":"INBOX","msg":"Error registered","time":"2016-09-09T23:18:27.809Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":5204,"level":50,"box":"INBOX","msg":"{ Error:<PHONE_NUMBER>96480:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:794:\n\n    at Error (native)\n    at Socket.ondata (_stream_wrap.js:58:20)\n    at emitOne (events.js:96:13)\n    at Socket.emit (events.js:188:7)\n    at readableAddChunk (_stream_readable.js:176:18)\n    at Socket.Readable.push (_stream_readable.js:134:10)\n    at TCP.onread (net.js:543:20) source: 'socket' }","time":"2016-09-09T23:18:27.826Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":5204,"level":30,"box":"INBOX","msg":"Restarting immediately","time":"2016-09-09T23:18:27.827Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":5204,"level":50,"box":"INBOX","msg":"Error registered","time":"2016-09-09T23:18:27.945Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":5204,"level":50,"box":"INBOX","msg":"{ Error:<PHONE_NUMBER>96480:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:794:\n\n    at Error (native)\n    at Socket.ondata (_stream_wrap.js:58:20)\n    at emitOne (events.js:96:13)\n    at Socket.emit (events.js:188:7)\n    at readableAddChunk (_stream_readable.js:176:18)\n    at Socket.Readable.push (_stream_readable.js:134:10)\n    at TCP.onread (net.js:543:20) source: 'socket' }","time":"2016-09-09T23:18:27.946Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":5204,"level":30,"box":"INBOX","msg":"Scheduling restart in 3000","time":"2016-09-09T23:18:27.946Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":5204,"level":50,"box":"INBOX","msg":"Error registered","time":"2016-09-09T23:18:31.066Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":5204,"level":50,"box":"INBOX","msg":"{ Error:<PHONE_NUMBER>96480:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:794:\n\n    at Error (native)\n    at Socket.ondata (_stream_wrap.js:58:20)\n    at emitOne (events.js:96:13)\n    at Socket.emit (events.js:188:7)\n    at readableAddChunk (_stream_readable.js:176:18)\n    at Socket.Readable.push (_stream_readable.js:134:10)\n    at TCP.onread (net.js:543:20) source: 'socket' }","time":"2016-09-09T23:18:31.067Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":5204,"level":30,"box":"INBOX","msg":"Scheduling restart in 6000","time":"2016-09-09T23:18:31.067Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":5204,"level":30,"box":"INBOX","msg":"Connection ended","time":"2016-09-09T23:18:31.071Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":5204,"level":50,"box":"INBOX","msg":"Error registered","time":"2016-09-09T23:18:31.191Z","v":0}
{"name":"imap_inotify","hostname":"protist-XPS-17","pid":5204,"level":50,"box":"INBOX","msg":"{ Error:<PHONE_NUMBER>96480:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:794:\n\n    at Error (native)\n    at Socket.ondata (_stream_wrap.js:58:20)\n    at emitOne (events.js:96:13)\n    at Socket.emit (events.js:188:7)\n    at readableAddChunk (_stream_readable.js:176:18)\n    at Socket.Readable.push (_stream_readable.js:134:10)\n    at TCP.onread (net.js:543:20) source: 'socket' }","time":"2016-09-09T23:18:31.192Z","v":0}
/usr/lib/node_modules/imapnotify/bin/imapnotify:83
        throw Error('Max retry limit reached')
        ^

Error: Max retry limit reached
    at Error (native)
    at Connection.<anonymous> (/usr/lib/node_modules/imapnotify/bin/imapnotify:83:15)
    at Connection.g (events.js:286:16)
    at emitOne (events.js:96:13)
    at Connection.emit (events.js:188:7)
    at TLSSocket._onError (/usr/lib/node_modules/imapnotify/node_modules/imap/lib/Connection.js:150:10)
    at emitOne (events.js:96:13)
    at TLSSocket.emit (events.js:188:7)
    at emitErrorNT (net.js:1272:8)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)

Oh, please, try the latest version from git.
It seems like you could not turn off tls in the older versions: #6

Well, the latest version from git works perfectly! I've tested half a dozen times, and it's extremely reliable. I guess I should have tested the git version earlier. Having said that, while testing, I installed the stable release on my wife's computer, which worked. The stable release had also worked fine for me for quite a while, before breaking for some (still unknown) reason. Thanks for all the help, and sorry for all the trouble!
(I tried to work out which commit the stable AUR release referred to, which was 0.1.0, but it wasn't clear to me. Perhaps you could also push the latest git version as the stable release?)

Yaaahoo! I'm glad that it's finally working for you.
I'm no longer the maintainer of the AUR package, but I marked it as out of date package and hope it will be updated soon.


Set minimum nori and rspec versions.  Updated for HTTPS

Specs were failing due to https endpoints returned from https://rxnav.nlm.nih.gov/REST/json
NDFRT lists endpoints as http, but they respond to https as well: http://rxnav.nlm.nih.gov/REST/Ndfrt/json

Hey there @joeadcock, thanks for the contribution! Had a couple of comments on this, then happy to merge it in (and look at the second one that's presumably based off this)

@jbender, glad I could help...  thanks for creating it!


Shortening the Kubernetes name to k8s

The name is used as part of the names of k8s assets. This can cause
length issues and the Kubernetes name was too long. Shortening.

Cross-referencing https://github.com/helm/hub/issues/362


ssh issue: Warning: Remote connection disconnect. Retrying...

HI,
I'm using host with centos8 and virtualbox. I upgarde versions in install command.
When doing Vagrant up, I'm getting connection error to the first vm ( repo):
repo: Warning: Remote connection disconnect. Retrying...
repo: Warning: Remote connection disconnect. Retrying...
repo: Warning: Remote connection disconnect. Retrying...
repo: Warning: Remote connection disconnect. Retrying...
repo: Warning: Remote connection disconnect. Retrying...
repo: Warning: Remote connection disconnect. Retrying...
repo: Warning: Remote connection disconnect. Retrying...
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.
If you look above, you should be able to see the error(s) that
Vagrant had when attempting to connect to the machine. These errors
are usually good hints as to what may be wrong.
If you're using a custom box, make sure that networking is properly
working and you're able to connect to the machine. It is a common
problem that networking isn't setup properly in these boxes.
Verify that authentication configurations are also setup properly,
as well.
If the box appears to be booting properly, you may want to increase
the timeout ("config.vm.boot_timeout") value.
I tried also with user/password and get the same error.
However, repo vm is stared and can be reached from virtualbox directory but it's configuration is not complete ( network pour adapter 2)
Regards

Using root user for vagrant operations fix the problem.
Why this is not working with a regular user?
Regards,


How does one interpret /v2/stats/store JSON data?

Below is some example store JSON data from one of our nodes (i.e. URL /v2/stats/store). Can someone help me interpret exactly what the values mean?
For example, how does one interpret a getsSuccess versus a getsFail? It would appear getsSuccess are a count of the number of key gets. But what does gtesFail mean? Isn't it only a client who knows if a key get failed or is this metric the result of an internal etcd failure?
I think watchers is self-explanatory and does correspond to the number of watchers on this node.
{
"getsSuccess": 2538969,
"getsFail": 7249604,
"setsSuccess": 905188,
"setsFail": 853,
"deleteSuccess": 4752,
"deleteFail": 4885,
"updateSuccess": 643812,
"updateFail": 340,
"createSuccess": 3461,
"createFail": 43555,
"compareAndSwapSuccess": 0,
"compareAndSwapFail": 0,
"compareAndDeleteSuccess": 1948,
"compareAndDeleteFail": 26,
"expireCount": 0,
"watchers": 2
}

May I post additional question?
I am having quite high number of "getsFail": 7249604, and its rising rapidly. I would like to find out which key is not found.
I've ran etcd with -vv flags but it didn't tell me which keys have failed. Is there some way to find out.
I am runnig etcd 0.4.7

@veverjak
There is no easy way to find the missed key in server side. The clients
should know more about the failure IMO. In etcd 2.1, it supports to print
out all HTTP requests when running in debug mode, which may be helpful for
your case.
On Mon, Jun 15, 2015 at 6:34 AM, veverjak<EMAIL_ADDRESS>wrote:

May I post additional question?
I am having quite high number of "getsFail": 7249604, and its rising
rapidly. I would like to find out which key is not found.
I've ran etcd with -vv flags but it didn't tell me which keys have failed.
Is there some way to find out.
I am runnig etcd 0.4.7
—
Reply to this email directly or view it on GitHub
https://github.com/coreos/etcd/issues/878#issuecomment-112071602.

ok, thanks,
the question is which client it is :)
Hopefully we will get to using etcd 2.1 soon too... Thanks for info

We should improve the docs, I was confused by getFails too. Can someone take a swing at this?


[ReflectionException] Class TCG\Voyager\FormFields\imageHandler does not exist

Laravel Version: 5.3.31
Voyager Version: 0.11.10
PHP Version: 7.0.15
Database Driver & Version:

Hi,
I got
ReflectionException
Class TCG\Voyager\FormFields\imageHandler does not exist
error when i try voyager:install or anything related with artisan.
Already composer dump-autoload && php artisan cache:clear && php artisan optimize command tried...
its a clean laravel install and everything seems work well.
hope to use ur package soon.

Same issue is here. I can't found imageHandler class(with case sensitive) anywhere. Could it be a composer script bug?

It should exist at vendor/tcg/voyager/src/FormFields/ImageHandler.php.  Does it not?  If not, please delete the vendor/tcg/voyager directory and run composer update.
If it does exist, please run composer dump-autoload and try again.

Hello again,
I tried ur suggestions before and tried again now but no luck.
Now i changed uppercase i's with lowercases in "image" word as;
TCG\Voyager\FormFields\ImageHandler -> TCG\Voyager\FormFields\imageHandler
TCG\Voyager\FormFields\MultipleImagesHandler -> TCG\Voyager\FormFields\MultipleimagesHandler
and now it works. Is there any other solution?

Are you on windows?  I'm guessing yes.
Windows doesn't care about case.  I'm willing to bet that by switching to lowercase, you somehow tricked php into finding the class(es).  If this is the case, then a composer dump-autoload should work as well.

I'm on Ubuntu 16.04.
One thing i see now is muratbsts has same issue and we both Turkish. We have 4 "i" letter in turkish keyboard as "i", "İ", "ı" and "I". Maybe our system language cause this error?

Hmm... that's possible.  The files are named with the capitalized roman "i" (ASCII character #73, or 0x49, for reference).  You should make sure you're using that character.

Hi @karpuzkan @fletch3555 . I've using two different OS as macOs and Fedora. My project was made on macOS, and I'm developing on Fedora currently. I think there is a bug related to composer dump-auto and case sensitivity. macOs and Fedora has different file systems.

 No problem or error! As u mentioned maybe its composer maybe not... Anyway happy coding.

@muratbsts, if it's a composer problem, it's not a Voyager problem.  Also, it's likely unrelated to this issue, but you'd need to provide much more detail than that for help.  If you still have an issue, please either create your own issue or join our Slack group.

@fletch3555 yep. It's not a Voyager problem. You can close this issue if you want. Best regards...

It's not an issue with Voyager, it's caused by Turkish locale where there is i, I and İ and ı. Can be fixed simply by renaming the files ImageHandler.php and MultipleImageHandler.phptoimageHandler.phpandMultipleimageHandler.phpinvendor/tcg/voyager/src/FormFields/```

in linux change system lang to english is easy for future problems


tag new release

The current release is currently outdated, hence having a new one would be nice. Plus, CartoDB would work with Tyler.

Yeah there are quite some unreleased commits: https://github.com/JuliaGeo/TileProviders.jl/compare/v0.1.2...main
Project.toml version is v0.1.4, but should probably be set back to v0.1.3, since that release, triggered in https://github.com/JuliaGeo/TileProviders.jl/commit/b66e195055ef323f52392ca23148ec1bc32cd2ab, somehow was never merged in General.
Also needs to add
[compat]
Dates = "1"


OpenCL is 3x slower

So I tried running this on the provided mnist.py. My main use case is that I'm on a laptop and I want it to train faster and use less battery. What I did not expect is that it was 3x slower using the GPU than using the CPU. I am on Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz with intel-opencl-icd. Furthermore, the GPU version caused the laptop fans to start spinning, but the CPU version didn't.
 How many compute units the GPU has? Show clinfo output?
run dlprim_flops on this device to see general performance.
Also mnist is really tiny network that does benefit much of even high performance GPU. On the other hand pytorch comes with highly optimized CPU implementation from Intel. So it is hard to beat CPU in this case.
In any case, using channel first memory order pytorch is using the performance of my implementation is actually is good and sometimes even better than GPU routines implemented by Intel.

See this issue: https://github.com/oneapi-src/oneDNN/issues/1194 regarding the Intel's implementation of GPU operations performance.

First of all, thanks for the project! It's really cool that pytorch can utilize opencl now (I'm not fan of nvidia, and very unlikely will have their video card).
I have very similar setup and results - opencl on intel video card is more than 3 times slower than when executing on 4 cores of cpu. But I also have an older laptop which has a discrete amd card with mesa clover opencl. And there I have ~1.5 performance boost comparing to cpu (which is still slower than cpu on newer laptop). But probably it all will depend on a task that is executed.

How many compute units the GPU has? Show clinfo output? It isn't even clear if GPU has more gflops than CPU.

Here what I have for the intel card:
  Max compute units                               24
  Max clock frequency                             1150MHz
  Device Partition                                (core)
    Max number of sub-devices                     0
    Supported partition types                     None
    Supported affinity domains                    (n/a)
  Max work item dimensions                        3
  Max work item sizes                             256x256x256
  Max work group size                             256
  Preferred work group size multiple (device)     32
  Preferred work group size multiple (kernel)     32
  Max sub-groups per work group                   32
  Sub-group sizes (Intel)                         8, 16, 32

I also have a old desktop with an AMD card. It's a shame that it's stuck on OpenCL 1 with Clover, but Rusticl has OpenCL 3. Mesa 22.3 should be out by December.

OpenCL 1 with Clover,

AFAIR it is OpenCL 1.1. I tested rx560 with Clover driver and it worked fine - was somewhat slower than ROCm but yet worked.

Dear Artyom-beilis,
I am a huge fan of your fantastic work from the beginning. I am a volunteer to help tuning CLBlast for many devices (https://github.com/CNugteren/CLBlast/issues/1).
I know you also mentioned CLBlast before on Winograd or something related to it.
 I am just wondering if it is properly tunned for Intel GPUs. I know Intel is good at tuning CPU gemm (MKL), however, their pervious opensource implementation on OpenCL with OpenCV were good for textbook. Therefore, to speed up its GPU, I guess CLBlast is good.
After tuning with CLBlast, 3x performance boost were achieved with my iGPU (before tuning, GPU is worse than CPU). My testing was here: https://www.researchgate.net/figure/i7-1165g7-tuned-clblast_fig9_366394124.
So, may I suggest adapting CLBlast to DLPrimitives to see if it gets more speedup on fully connected layers/Conv on im2col.
In addition, it has been used in llamma.cpp to achieve comparative performance as cuBLAS.
Please forgive me if I am wrong. I am just very interested in your fantastic work.
Best wishes,
Jinchuan

1st of all I'm not sure that clblast has much advantage over dlprimitives. For example GEMM m=n=k=512 both give ~130-140 GFlops.
For example my gemm: Intel(R) UHD Graphics 630 [0x3e9b] on Intel(R) OpenCL HD Graphics
When this GPU top GLOPS is around 401.413
dlprimitives sgemm
GEMM
  NN  0:  512,  512,  512      136.9 GFlops (34.09%)      1.6 GB/s (11.07%) limited by gflops 34.09%
  NT  0:  512,  512,  512      104.4 GFlops (26.01%)      1.2 GB/s ( 8.44%) limited by gflops 26.01%
  TN  0:  512,  512,  512      190.9 GFlops (47.56%)      2.2 GB/s (15.44%) limited by gflops 47.56%
  TT  0:  512,  512,  512      162.0 GFlops (40.35%)      1.9 GB/s (13.10%) limited by gflops 40.35%

clblast sgemm
NN: 143.772 GFLOPS
TN: 154.547
NT: 129.747
TT: 143.407

So it not necessary clear cut that clblast has better performance. Also note that when running convolition the im2col is actually integrated to the gemm itself so switching to clblast isn't trivial at all.
On the same note I need to mention that Intel's own deep learning library does not run well for channel first convolution since they just don't optimise for one. https://github.com/oneapi-src/oneDNN/issues/1194
Actually dlprimitives performs better on channel 1st order.

Hi,
the problem is size, such as MINIST has 60000 test samples, each with a size of 2828 pixels. Hence, I am very interested in how this (60000 742 * 742742 as an example to imitate the input and the weights to the hidden layer) performs compared to 512512. Based on my experience, GPU gemm normally performs close to CPU when size is small.
Best wishes

GPU gemm normally performs close to CPU when the size is small.

Which GPU do you have?
Typical max Intel Skylake GPU has ~400GFPLOPS which is quite a few. But typical multi core cpu has much more flops
If I run torch.matmul on same computer where I have 400GFLOPs GPU on CPU I get ~500GFLOPs performance (for mat mul i.e. way-way more than I can get from GPU.
Intel GPUs unfortunately aren't as powerful as modern data CPUs - they are good for graphics - less so for computation.
So I'm not surprised that you have better performance for CPU rather than on GPU.

Dear Artyom,
I have several Intel iGPU and Arc GPUs in my lab. Take i7-1165G7 as an example, what I got is here: i7-1165g7.zip
I admit that the code I implemented did not consider zero-copy for iGPU.
Best wishes,
Jinchuan

I have several Intel iGPU and Arc GPUs in my lab.

Nice. Can you post benchmark numbers for Arc GPUs? i.e run ./dlprim_flops I have not managed to test it or tune for it.

CPU cores (CPU used Openblas).

OpenBLAS is somewhat smaller.  The 500GFPLOPs was with torch.matmul. OpenBLAS's sgemm gives only 250 GFLOPS

Dear Artyom,
Please let me interest you with something more interesting as on an AMD 7900XTX.
I freshly build a windows dlprim_flops and receive the following result:
7900XTX.txt
I will try to make a test on ARC 770 16G when I find a free machine to plug off the CUDA/AMD card and then plug in the ARC card.
Best wishes,
Jinchuan

Dear Artyom,
I am also supervising an undergraduate student to do a comparative study on the performance of different GPU random number generators. May I ask if there is any paper/material that is related to the random kernel? (https://github.com/artyom-beilis/dlprimitives/blob/master/src/kernels/random.cl)
Thank you very much!
Best wishes,
Jinchuan

I am also supervising an undergraduate student to do a comparative study on the performance of different GPU random number generators. May I ask if there is any paper/material that is related to the random kernel?

It is fairly standard Philox-4x32, see here the paper: https://www.thesalmons.org/john/random123/papers/random123sc11.pdf
I mentioned it in API: https://github.com/artyom-beilis/dlprimitives/blob/master/include/dlprim/core/common.hpp#L47

Dear Artyom, Please let me interest you with something more interesting as on an AMD 7900XTX. I freshly build a windows dlprim_flops and receive the following result: 7900XTX.txt
Best wishes, Jinchuan

Cool. It looks like my measurement of FLOPS are way incorrect it should be much more capable GPU according to wikipedia. That is why I get more than 100% flops for some GEMM ops.

Apple M2 MAX.txt
HDF5_cpp is not build by default according to its website. Thus, disabled HDF5 lib in the makefile to get there.

I am also supervising an undergraduate student to do a comparative study on the performance of different GPU random number generators. May I ask if there is any paper/material that is related to the random kernel?

It is fairly standard Philox-4x32, see here the paper: https://www.thesalmons.org/john/random123/papers/random123sc11.pdf
I mentioned it in API: https://github.com/artyom-beilis/dlprimitives/blob/master/include/dlprim/core/common.hpp#L47

Thank you very much!

Dear Artyom,
https://github.com/artyom-beilis/dlprimitives/blob/46b9d17b76c40d05b323a1a0ea484d61ac5f17b2/src/core/pointwise.cpp#L77C24-L77C25
would this be quite expensive to handling \n to \\n using for loop, or is this necessary? I did not find similar lines to specifically handle this in CLBlast which also used R" for kernels.
https://github.com/CNugteren/CLBlast/blob/master/src/utilities/compile.cpp#L24
Best wishes,
Jinchuan

The pointwise/broadcast is a special kind of kernels that are shortcuts for common simple operation: activations, various reductions needed for normalization, loss functions etc.
Here you get the actual line of code and embed it into a kernel, like here: https://github.com/artyom-beilis/pytorch_dlprim/blob/master/src/pointwise_ops.cpp#L155
Now, I agree that this code can be optimised, but it is also important to remember that since the GPU and CPU executions are asynchronous, as long as you can push kernels to execution queue faster than GPU can run them you don't bottleneck your system.

And what is more important it allows to guys who aren't familiar with GPU coding write some operations.

The independent kernels are mostly for some computationally intensive/non-standard tasks like pooling, convolutions etc.

There is another way which uses a fixed-length kernel. All the broadcasting size info regarding dimensions needs to be aligned and stored in a host buffer and then uploaded to the device as global mem to the kernel. The kernel is in charge of using those info to index the correct location of Array A and B. As a result, this kernel becomes synchronous due to the fact a blocking memory write needs to be issued to the command queue. This is fine for scientific computing tasks, but for AI especially with multiple in-platform GPUs, I guess your way of asynchronous is more welcome.

Intel Arc 770.txt
as promised. This one really take time for it can freeze while executing some cases.

Thanks. Impressive GPU.
Can you give clinfo or it?
I wonder if it is possible to enable Winograd on one. It depends on shared memory configuration.

clinfo770.txt
You are welcome.

In the meantime, there is a website https://compubench.com/device.jsp?benchmark=compu20d&os=Windows&api=cl&D=Intel(R)+Arc(TM)+A770+Graphics&testgroup=info
which could provide more results for new devices.


New demographics.py

Here we've replace the old demographics.py with a new demographics.py, which is more thoroughly commented, more effectively modularized, and intended to more accurately make calculations necessary to OG-USA (e.g. immigration rates).

@chrisrytting, Changes look good.  Before merging, let's workout issues with Travis CI built fail.  Current issue is an import of demog in execute_small.runner().
FYI - if you click on "Details" next to the Travis CI build failed message, you will probably see enough detail to work out any other issues.

I can't judge the merit of code itself, but it does seem good to only have one demographics module, instead of demog.py and demographics.py. The only thing is that we still have code around that imports both. That is why the one test is failing. If you could remove the imports of demog from these two files, I think that will fix the failing test.
https://github.com/open-source-economics/OG-USA/blob/master/Python/execute.py
https://github.com/open-source-economics/OG-USA/blob/master/Python/execute_small.py

@jdebacker @talumbau Alright I've fixed the issue in execute_small.py by changing every instance of demog to demographics and it seems as if the checks have passed, so I would imagine we're good to go now.

we don't have a great mechanism for this yet, but execute_small.py is meant to "mirror" the behavior of execute.py, so changes in one should happen in the other. The point of execute_small.py is to make it so that we can quickly test a code path that is in use for the web application. So, if you change one, you should really change the other to make sure they stay in sync. Until we come up with a better solution, the only alternative to this arrangement was to have the test take 2.5 hours to run, which seemed bad. So would you mind making a similar change in execute.py so they stay in sync. Apologies for the confusion.


Fixing license headers in all Markdown files.

Some files had partially translated or overflowing and malformed license headers. This commit fixes all of them to use the text of the Apache license, and formats them all in the same way.

Wow, thanks!
Merge away.

GitHub has fallen out of sync with ASF Git. ):

It lives!


Speed up evselect_spectrum preparation stage

I've sped up source declaration by shifting regions to being generated on demand, but this has dramatically slowed down the pre-run stage of evselect_spectrum for larger samples. See if anything can be done about this.

Ran a profiler - seems like (as I sort of expected) a lot of the problems come from the getting sas region stage, where figuring out which sources are within which other sources is a big time sink. As is converting to pixel coordinates etc.


HTML Output rendering does not have all information

Hello,
When I rendering my results to HTML the title of columns say object Object (I am using the custom provider). Is there a way I can get the provider instead, maybe the file name that I use could be better instead. Moreover, there is no information about what assertion type was executed against the test case as well. It would be nice to add that information as well.

Thanks for the report.  This is fixed in 0.35.1


New Sets Of Races Identified

1
WARNING: DATA RACE
Read at 0x00c00014a048 by goroutine 12:
  github.com/slackhq/nebula.(*udpAddr).Copy()
      github.com/slackhq/nebula/udp_linux.go:96 +0x18d
  github.com/slackhq/nebula.(*HostMap).PunchList()
      github.com/slackhq/nebula/hostmap.go:403 +0x135
  github.com/slackhq/nebula.(*HostMap).Punchy()
      github.com/slackhq/nebula/hostmap.go:425 +0x137

Previous write at 0x00c00014a048 by goroutine 19:
  [failed to restore the stack]

Goroutine 12 (running) created at:
  github.com/slackhq/nebula.Main()
      github.com/slackhq/nebula/main.go:195 +0x497e
  main.main()
      github.com/slackhq/nebula/cmd/nebula/main.go:52 +0x6e4

Goroutine 19 (running) created at:
  github.com/slackhq/nebula.(*Interface).run()
      github.com/slackhq/nebula/interface.go:137 +0x957
  github.com/slackhq/nebula.(*Control).Start()
      github.com/slackhq/nebula/control.go:34 +0x858
  main.main()
      github.com/slackhq/nebula/cmd/nebula/main.go:64 +0x83b

2
WARNING: DATA RACE
Read at 0x00c000594048 by goroutine 12:
  github.com/slackhq/nebula.(*HostMap).PunchList()
      github.com/slackhq/nebula/hostmap.go:403 +0x168
  github.com/slackhq/nebula.(*HostMap).Punchy()
      github.com/slackhq/nebula/hostmap.go:425 +0x137

Previous write at 0x00c000594048 by goroutine 19:
  [failed to restore the stack]

Goroutine 12 (running) created at:
  github.com/slackhq/nebula.Main()
      github.com/slackhq/nebula/main.go:195 +0x497e
  main.main()
      github.com/slackhq/nebula/cmd/nebula/main.go:52 +0x6e4

Goroutine 19 (running) created at:
  github.com/slackhq/nebula.(*Interface).run()
      github.com/slackhq/nebula/interface.go:137 +0x957
  github.com/slackhq/nebula.(*Control).Start()
      github.com/slackhq/nebula/control.go:34 +0x858
  main.main()
      github.com/slackhq/nebula/cmd/nebula/main.go:64 +0x83b

3
WARNING: DATA RACE
Read at 0x00c00044c008 by goroutine 12:
  github.com/slackhq/nebula.(*HostMap).PunchList()
      github.com/slackhq/nebula/hostmap.go:402 +0x11b
  github.com/slackhq/nebula.(*HostMap).Punchy()
      github.com/slackhq/nebula/hostmap.go:425 +0x137

Previous write at 0x00c00044c008 by goroutine 19:
  [failed to restore the stack]

Goroutine 12 (running) created at:
  github.com/slackhq/nebula.Main()
      github.com/slackhq/nebula/main.go:195 +0x497e
  main.main()
      github.com/slackhq/nebula/cmd/nebula/main.go:52 +0x6e4

Goroutine 19 (running) created at:
  github.com/slackhq/nebula.(*Interface).run()
      github.com/slackhq/nebula/interface.go:137 +0x957
  github.com/slackhq/nebula.(*Control).Start()
      github.com/slackhq/nebula/control.go:34 +0x858
  main.main()
      github.com/slackhq/nebula/cmd/nebula/main.go:64 +0x83b

4
WARNING: DATA RACE
Read at 0x00c0003e4640 by goroutine 12:
  github.com/slackhq/nebula.(*HostMap).PunchList()
      github.com/slackhq/nebula/hostmap.go:402 +0x33f
  github.com/slackhq/nebula.(*HostMap).Punchy()
      github.com/slackhq/nebula/hostmap.go:425 +0x137

Previous write at 0x00c0003e4640 by goroutine 19:
  [failed to restore the stack]

Goroutine 12 (running) created at:
  github.com/slackhq/nebula.Main()
      github.com/slackhq/nebula/main.go:195 +0x497e
  main.main()
      github.com/slackhq/nebula/cmd/nebula/main.go:52 +0x6e4

Goroutine 19 (running) created at:
  github.com/slackhq/nebula.(*Interface).run()
      github.com/slackhq/nebula/interface.go:137 +0x957
  github.com/slackhq/nebula.(*Control).Start()
      github.com/slackhq/nebula/control.go:34 +0x858
  main.main()
      github.com/slackhq/nebula/cmd/nebula/main.go:64 +0x83b

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Dear Editor: Gay City News
The ONLY issue of your paper I have ever read, was the one entitled: “Impeach Rumsfeld”, where there were photos of the Iraqi prisoners being tortured (Private Lindy EnglandPin auf Disturbing Things , in which all of the senior officers of the US Military: Don’t Ask, Don’t Tell were listed by name.
Pin auf Disturbing Things
10.08.2012 - Lynndie England holding a leash attached to a prisoner at Abu Ghraib prison in Iraq were many prisoners were tortured and abused.
The last time I saw Donald Rumsfeld, was at the Brooklyn Public Library, Grand Army Plaza:
Each ACCOUNT, was allowed to use the computers for an hour at a time. There are camera strategically located throughout the building.
That day, Lindsay strolled in casually, and slipped into the seat next to mine. I immediately did a search for “Lindsay Lohan” after which images of her filled the screen.
Wolé Fayemi v Kings County, Mustafa, Jenardinan et al
Arrived un a private ambulette ordered by Jedediah Burack. Picked up at 280 E21st Street, 1B:
Dr Jinardinan. What should I call you?
DINO: Dino
Dr J: Why do you want to be called that?
DINO: are you willing to sign a Non Disclosure Agreement?
Dr. J: ...Very well.
You will spend the next 72 hours sleeping on the floor of the overcrowded Emergency Room, which is infested with cockroaches and mice, during which time Sukanye of WB11 news, will watch you as you’re crying (laughing), and then you will he visited by Sarah, who will ask you to sign a deposition re: Class Action Lawsuit. You will have a brief meeting with a #PsychobabblebullshitArtist, who will ask you to sign the admission papers, while your brothers high school classmate, Joel Abramson, watches silently.
Dr J: I will assign then assign you to Dr. Colin Powell, III’s son: Director of the FCC, and you will spend the next 7 months there against your will, force injected Haloperidol 3 times a day. You’ll probably pass out and drool, when you’re not screaming at the top of your lungs and telling crazy stories about your girlfriend in San Francisco before your ex-wife: bit.ly/jailbait_juaperez.
Ever since I relayed, on Google Plus, that OBVIOUSLY, the Iraq War was just like Cosovo.
As The CEO of Hard Hat Bid explained it to me, approximately 6 months before the 9/11 bombings, “war is business”. We tear things down, so that we can profit from the contracts to rebuild”
Curiously, after the bombings, the US was hesitant about sending troops in during the summer (it was too hot they said)...
Wolé M. Fayemi
Motorious Automotive, llc
347-499-2253

On Oct 23, 2020, at 3:57 AM, CLAassistant<EMAIL_ADDRESS>wrote:

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.


Add type properties to hooks

AttributeHook has a type property available on it's prototype but the other hooks don't. Providing the type property helps other libraries parse vdom, such as this discussion in vdom-to-html. At the moment there is no easy way to convert a hook to html. Using instanceof requires the same npm package which isn't always possible.
This PR adds a type property to ev-hook, focus-hook and soft-set-hook.
If it makes sense it might also be nice to add corresponding is-ev-hook, is-focus-hook and is-soft-set-hook files in vnode.
Thoughts?

Coverage increased (+0.01%) to 97.5% when pulling<PHONE_NUMBER>04878e9fe2c94dbf99db4be29c82d6 on odojs:hook-to-html into 88534bbd34c5befbd1689fc71e2373c9145a9807 on Matt-Esch:master.

See also #248

@tcoats what do you think about having .toString on hook's prototype?

I think @DeTeam has the better proposal here.
That way html conversion behavior is defined by whoever defines a hook and elegantly solved by polymorphism instead of starting to switch on some kind of type identifier.
This would solve the issue completely for the future.

@DeTeam I think it's great!
The only drawback is that each component will need to include a stringify library - but the freedom string gives over vdom is excellent.

The only drawback is that each component will need to include a stringify library

@tcoats, not really :) it's all about being able to represent Hook as a string. vdom-to-html lib can't guess on how to do it. If we go with just using hook.value it doesn't fix user hooks, while having something like toString, render as part of Hook interface will allow custom hooks to be automatically handled by vdom-to-html.

@DeTeam In my case I'll probably need to use a stringify lib more if toString is added to widgets as well, which I think is a great idea. Should toString be added to widgets?
toString is an excellent and useful addition!


⚠ Bofrost.fr has degraded performance

In 3670bed, Bofrost.fr (https://www.bofrost.fr) experienced degraded performance:

HTTP code: 200
Response time: 1651 ms

Resolved: Bofrost.fr performance has improved in 85a611f.


Segger RTT peripheral

I am not able to build the project. I am getting the below error.
...........\sdk\platform\arch\arch.h(135): error: Not supported TRNG configuration"
#warning "Config error: Not supported TRNG configuration"
..\segger_rtt\RTT\SEGGER_RTT_printf.c: 1 warning, 7 errors
".\out_DA14531\Objects\user_segger_rtt_periph_531.axf" - 1155 Error(s), 0 Warning(s).
Target not created.
Build Time Elapsed:  00:00:21
can you please look into this issue?
I did not change anything in the project.

i resolved everything
..\src\user_segger_rtt_periph.c(31): error:  #5: cannot open source input file "SEGGER_SWD_printf.h": No such file or directory
it throws the new error even though that file is included


Title broken in German version

Hello,
the title of the article is broken in German version. Please fix it and check article for other issues.
Thank you.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e1b845f2-dab8-a8c5-52ce-88f78e2e1885
Version Independent ID: a599ae52-a9c3-815c-87b7-59d2711a8f36
Content: Responding to Data Subject Rights (DSR) requests for Power Apps customer data - Power Platform
Content Source: power-platform/admin/powerapps-gdpr-dsr-guide.md
Service: power-platform
GitHub Login: @jimholtz
Microsoft Alias: jimholtz

@GiantCrocodile
Thanks for the notice. We'll fix the title.


Move BatchCollection to new AutoRaman

This will allow us to remove all the auto-laser stuff from WasatchDevice.acquire_raman_standard.
Add a checkbox to BatchCollection GUI to "optimize acquisition parameters". If checked, set AutoRamanRequest min/max integration time and gain from the current enlighten.scope.AutoRaman GUI settings. If unchecked, set min/max/start integ and gain to current integ/gain, which would prevent any optimization from occurring. This will allow both "current behavior" (determininistic, fixed-timing BatchCollections showing rise and fall of Raman signal over time) and new "Auto-Raman collections" (optimized Raman measurements over time, a new feature).
While refactoring the laser-control code out of acquire_raman_standard, also refactor-out the "Reading metadata" code (which reads temperatures, battery etc and adds them into the current Reading) into a new function (add_reading_metadata). Then call add_reading_metadata at the end of WasatchDevice.acquire_spectrum, so that the metadata is updated regardless of whether AutoRaman or "standard" acquisition is chosen.

Resolved in 4.1.13


fix: do not ignore screen size on NewContext creation

This needs to be rebased when we merge #1542!


Standard coder tests

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[ ] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[ ] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[ ] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

R: @lukecwik

Rebased, fixed lint, and addressed comments. @lukecwik PTAL

Added license blurb (and a couple of lint fixes).

Changes Unknown when pulling 0c5874e5835c532a02e770a66bc4ccd4e17cd29e on robertwb:standard-coders into ** on apache:python-sdk**.

LGTM

Thanks!


toolbox resets terminal size to 80x24

I'm not quite sure when toolbox changed this behavior, but it worked as expected. but now toolbox resets the terminal size to 80x24 even though it is not on host.

Could this be the cause of #14 ?

No idea. this could be fixed by settings COLUMNS and LINES from host at least.

I confirm I am too facing this issue and patch given in #243 fixes it.

Could this be the cause of #14 ?

That's something else, I think. It was diagnosed in https://github.com/containers/libpod/issues/3179

I'm not quite sure when toolbox changed this behavior, but it
worked as expected. but now toolbox resets the terminal size to
80x24 even though it is not on host.

Indeed. I noticed it yesterday while playing with @HarryPhoon 's patches in https://github.com/debarshiray/toolbox/pull/200 Things like less(1) were not using the full window size.
It's something in Podman because it creates a nested pseudo-terminal pair inside the one created by the terminal emulation application. Possibly in runc because I think that's where the pseudo-terminal pair is created for podman exec.

Filed: https://github.com/containers/libpod/issues/3946

Interestingly, only the initial size is wrong. If the terminal emulator window is resized afterwards, then it picks up the changes correctly.


Error parsing F16 - Core(Mismatch { name: "f16", found: 60 })

I'm using calling deserialize_any with a custom visitor and I noticed that I get the Core(Mismatch { name: "f16", found: 60 }) error whenever I have a 16 bit float in the encoded string (in this case 1.0, which is encoded as 3C 00).
I have narrowed this down to a line in serde/de.rs:93:
marker::F16 => {
  de.reader.advance(1);
  let v = half::f16::decode(&mut de.reader)?;
  visitor.visit_f32(v.into())
},

I think the de.reader.advance(1); should not be there, since the decode method checks the major type again and errors out if it doesn't match. When I comment that line out, the parser works as expected.

You're right, would you like to open a PR?

Created pull request #31.

0.3.3 has been released

done


Nominate Duy Nguyen as a collaborator

Nomination
TSC Project Roles: Collaborator
GitHub Permission Level: Write
GitHub User
Provide the following information about the GitHub user:

Full Name Duy Nguyen
GitHub username duynguyenxa
Organization (optional): Renesas

Supporting Documents
https://github.com/zephyrproject-rtos/zephyr/commits?author=duynguyenxa
https://github.com/zephyrproject-rtos/zephyr/pull/75498/files

done


Add new feature resolver.

This adds a new resolver which handles feature unification independently of the main resolver. This can be enabled with the -Zfeatures flag which takes a comma-separated list of options to enable new behaviors. See unstable.md docs for details.
There are two significant behavior changes:

Ignore targets that are not enabled.
Do not unify features between build_deps, dev_deps, and normal deps.

The "forks" in the unit graph are handled by adding DepKind to UnitFor. The feature resolver tracks features independently for the different dependency kinds.
Unfortunately this currently does not support decoupling proc_macro dependencies. This is because at resolve time it does not know which dependencies are proc_macros. Moving feature resolution to after the packages are downloaded would require massive changes, and would make the unit computation much more complex. Nobody to my knowledge has requested this capability, presumably because proc_macros are relatively new, and they tend not to have very many dependencies, and those dependencies tend to be proc-macro specific (like syn/quote). I'd like to lean towards adding proc-macro to the index so that it can be known during resolve time, which would be much easier to implement, but with the downside of needing to add a new field to the index.
I did not update cargo metadata, yet. It's not really clear how it should behave. I think I'll need to investigate how people are currently using the feature information and figure out how it should work. Perhaps adding features to "dep_kinds" will be the solution, but I'm not sure.
The goal is to try to gather feedback about how well this new resolver works. There are two important things to check: whether it breaks a project, and how much does it increases compile time (since packages can be built multiple times with different features). I'd like to stabilize it one piece at a time assuming the disruption is not too great. If a project breaks or builds slower, the user can implement a backwards-compatible workaround of sprinkling additional features into Cargo.toml dependencies. I think itarget is a good candidate to try to stabilize first, since it is less likely to break things or change how things are built. If it does cause too much disruption, then I think we'll need to consider making it optional, enabled somehow.
There is an environment variable that can be set which forces Cargo to use the new feature resolver. This can be used in Cargo's own testsuite to explore which tests behave differently with the different features set.

I did a fairly major rewrite to try to simplify things. Now that dev-dependency unification is a global setting, it all really boils down to whether or not something is a build dependency, so I just made that a bool.  It's not as flexible, but I can always switch it back to something more complex if needed.
I also add -Zfeatures=all to try all options at once.
BTW, I'd like to squash this before merging, so don't merge as-is.  This also now includes #7857 to fix some issues, but that PR should probably go first to simplify this.

Does this PR also address #2589 and #2524 ? If not I'd try to look deeper into the 2589 myself as it would be quite convenient to have something like
[target.'cfg(not(target_os = "macos"))'.dependencies.a]
version = "0.5"

[target.'cfg(target_os = "macos")'.dependencies.a]
git = "https://github.com/foo/a_patched.git"

@shamatar yes, it should close those issues (and many others).  2589 relates to features, not multiple sources.  Multiple sources is #7753, and I believe that will be very difficult to fix.

@ehuss
Ok, thanks for an answer. Another short question: if paths are not easy to fix, may be it would be able to introduce a syntax like
[target.'cfg(not(target_os = "macos"))'.features]
default = []

[target.'cfg(target_os = "macos")'.features]
default = ["macos_patch"]

or something similar. If it ever makes sense. At least it's a short-term fix for different paths problem.

That is discussed in #1197.

:umbrella: The latest upstream changes (presumably #7855) made this pull request unmergeable. Please resolve the merge conflicts.

Ok this is taking me longer to get to reviewing than I thought it might. I'll be gone most of next week as well. From my previous review and given the changes you've described implementing, I'd be fine landing this in the meantime and I can comment on this when I get a chance to. Or if you're ok waiting a week or so that's ok too!

I did not update cargo metadata, yet. It's not really clear how it should behave. I think I'll need to investigate how people are currently using the feature information and figure out how it should work.

I guess I can say a thing or two about IntelliJ and rust-analyzer, who are perhaps the two biggest users of cargo metadata at the moment. In short, the current Resolve in cargo metadata doesn't really make sense; it works good enough, but it is on the wrong level of abstraction.
Specifically, Resolve is on the level of packages, but what rust-analyzer & IntelliJ want is more close to units. For example, we would like to see "crate ./test/foo.rs depends on crate ./src/lib.rs with these cfg flags set".
I think we can make two steps to make sure that things "make sense".

Recast existing Resolve as an exact reflection of Cargo.lock. Ie, make it fully independent of the set of feature flags and the target, such that it is basically the flat list of packages from a lockfile + a deps which explain why those packages are in lockfile. I am not sure if this would be actually useful, but at least that would be a clear defined things

Add a separate thing (perhaps a new command even?) which outputs a format close to the UnitMap, which speaks not about Cargo packages, but rather crates in rust-reference sense, and dependencies between them. This would depend on the precise set of flags passed to Cargo (we'll get different outptus for cargo build vs cargo test) and would actually resemble a sort-of static build plan.

@matklad I've been thinking the exact same thing.  I've also been wondering if we could just extend cargo metadata or need some other command or flag (like how --build-plan works).
I'd really like a tool to visualize the unit graph, but that is difficult for any non-trivial project.

FYI cargo-guppy relies on metadata as well: https://github.com/calibra/cargo-guppy

Added some comments, and fixed a critical bug in how shared optional dependencies were handled.
@Eh2406 you mentioned in the meeting wanting a high level overview, but I wasn't sure if you meant beyond what is already written.  Is there additional information you'd like to see in this PR?  I think if you start by reading unstable.md to get a sense of what it does and how to use it, you can then go to resolve/features.rs which contains an overview of how it fits.  I've tried to include plenty of comments, but let me know if there's anything you'd like me to add.

Alright I'm definitely happy with everything here, so I'm going to r+
I think it's safe to say this has languished long enough (sorry!) and we can continue to iterate in-tree. Thanks again so much for pushing on this @ehuss!
@bors: r+

:pushpin: Commit 4d0fda7cd9efb14055915e755b09ee3cf64586ff has been approved by alexcrichton

:hourglass: Testing commit 4d0fda7cd9efb14055915e755b09ee3cf64586ff with merge d6fa260d1aa74521c7d8c03d3d3c09067a4147d0...

:sunny: Test successful - checks-azure
Approved by: alexcrichton
Pushing d6fa260d1aa74521c7d8c03d3d3c09067a4147d0 to master...

Thank you so so much!!!


Optimizations to Test and Cookbook Generation

New Contributors:
please review CONTRIBUTING.md
Purpose:
This is a PR to make running tests faster. My test results are below:

Cookbook Tests
Avg Time (s)

Before
7.5

After
5.2

In theory, these optimizations should affect other tests but in reality I couldn't notice any meaningful difference.
I know optimizing tests is a pretty low priority (especially as the whole suite takes under 20 seconds to run) but it's a pretty straightforward set of optimizations: all of them are simply caching the results of pure functions. This can be especially fruitful with reflection as it can be quite slow. Turning linear searches into dictionary lookups is also quite easy.
I haven't looked at optimizing other tests but most that take up a lot of time are slow because they are benchmarks, i.e. they're supposed to be slow.
New Functionality:
N/A

I ran dotnet-format locally which caused two files to be autoformatted incorrectly. Not sure where this issue came from :/

Saved ~1 more second by using Parallel.ForEach instead of normal foreach. I actually resorted to it after getting no benefit from making a bunch of tasks and using Task.WhenAll -_-

I'm a little curious about this parameter: https://github.com/ScottPlot/ScottPlot/blob/master/src/tests/Benchmark/Scatter.cs#L15
I'm not convinced that rendering speed is variable enough that 1000 iterations makes sense, and I also think that if it were the sliding scale would make the larger datasets (where performance matters more anyways) less accurate.

Currently, the performance improvements for the entire test suite look like this:

Full Test Suite
Avg Time (s)

Before
18.2

After
11.6

I did not change what any of the tests were testing except Test_scatter_10kPoints, which now always runs 10 iterations for each size of dataset. That saved ~3 seconds.

Thanks for optimizing this @bclehmann and for your input @StendProg!
I'm happy with the reduction in benchmarking replicates too. It may be useful to create a benchmarking module in the future we can tactically enable only when we are interested in actually inspecting the output.


[BIOMAGE-2286] Added geometric sketching controls to data integration

Description
The new config when downsampling is passed is:
   dataIntegration: {
      downsampling: {
         method: 'geosketch',
         percentageToKeep: 5
      },
      dataIntegration: {
        method: 'seuratv4',
        methodSettings: {
          fastmnn: { numGenes: 2000, normalisation: 'logNormalize' },
          harmony: { numGenes: 2000, normalisation: 'logNormalize' },
          seuratv4: { numGenes: 2000, normalisation: 'logNormalize' },
          unisample: { numGenes: 2000, normalisation: 'logNormalize' }
        }
      },

When disabled (by default) the key downsamplingwon't exist. When specifically disabled after this is released its value will be downsamlping: { method: 'none', percentageToKeep: 100}.  If the value is none no downsampling should be done. The value 100 just indicates that when disabled all cells will be kept.
Details
URL to issue
https://biomage.atlassian.net/browse/BIOMAGE-2286
Link to staging deployment URL (or set N/A)
N/A
Links to any PRs or resources related to this PR
https://github.com/biomage-org/api/pull/73
https://github.com/biomage-org/pipeline/pull/78
https://github.com/biomage-org/ui/pull/105
Integration test branch
master

Merge checklist
Your changes will be ready for merging after all of the steps below have been completed.

Code updates
Have best practices and ongoing refactors being observed in this PR

[ ] Migrated any selector / reducer used to the new format.

Manual/unit testing

[ ] Tested changes using InfraMock locally or no tests required for change, e.g. Kubernetes chart updates.
[ ] Validated that current unit tests for code work as expected and are sufficient for code coverage or no unit tests required for change, e.g. documentation update.
[ ] Unit tests written or no unit tests required for change, e.g. documentation update.

Integration testing
You must check the box below to run integration tests on the latest commit on your PR branch.
Integration tests have to pass before the PR can be merged. Without checking the box, your PR
will not pass the required status checks for merging.

[ ] Started end-to-end tests on the latest commit.

Documentation updates

[ ] Relevant Github READMEs updated or no GitHub README updates required.
[ ] Relevant Wiki pages created/updated or no Wiki updates required.

Optional

[ ] Staging environment is unstaged before merging.
[ ] Photo of a cute animal attached to this PR.

@kafkasl instead of setting downsampling to none in the ui and not always passing in downsampling in the schema under dataIntegration, have you considered adding downsampling to the processingConfig during runs of gem2s, similarly to what we do for other processing settings?

@kafkasl instead of setting downsampling to none in the ui and not always passing in downsampling in the schema under dataIntegration, have you considered adding downsampling to the processingConfig during runs of gem2s, similarly to what we do for other processing settings?

in this case we would have to either:

Migrate all the old experiments processing config
Do not migrate them so still keep this code for old experiments.

In my opinion migrating the processing config should be avoided as much as possible. Considering that it's a json, using schema-on-read (like I do here and in the pipeline) it's a much better solution in my opinion and less error-prone.
Regarding the initial values, having them in the UI is not a great choice. As we move into having more integrations everything required that is set exclusively in the UI and will cause the pipeline to fail is a bad idea. Coding like I did, means that if someone triggers a pipeline using an integration and does not add the downsampling, it will work because we just do schema on read. The other solution will break and require integration changes every time we add a parameter.

@kafkasl instead of setting downsampling to none in the ui and not always passing in downsampling in the schema under dataIntegration, have you considered adding downsampling to the processingConfig during runs of gem2s, similarly to what we do for other processing settings?

in this case we would have to either:

Migrate all the old experiments processing config
Do not migrate them so still keep this code for old experiments.

In my opinion migrating the processing config should be avoided as much as possible. Considering that it's a json, using schema-on-read (like I do here and in the pipeline) it's a much better solution in my opinion and less error-prone.
Regarding the initial values, having them in the UI is not a great choice. As we move into having more integrations everything required that is set exclusively in the UI and will cause the pipeline to fail is a bad idea. Coding like I did, means that if someone triggers a pipeline using an integration and does not add the downsampling, it will work because we just do schema on read. The other solution will break and require integration changes every time we add a parameter.

About this part:

Do not migrate them so still keep this code for old experiments.
The code you have added is also handling specifically cases for old experiments, this is what:
downsampling === undefined || downsampling?.method === 'none' is doing. So I don't see why using this structure vs the other one would change anything, in both cases we need special code to handle old experiments.

In any case, I think trying to avoid migrations in processingConfig is ok, but in that case we should begin having a function that takes the processingConfigs and transforms them into the "migrated" version before using it, because this approach of handling the old and new case every time we access a value is very likely to get out of hand and cause a lot of problems (we'll need ot hadle the same edge cases everywhere everytime we access processingConfig).


Panic case

Hi!
Please look at this simple example:
func didPanic(fn func()) (result bool) {
	result = true // Handle `panic(nil)` case.
	defer func() { recover() }()
	fn()
	return false
}

assigned to result, but reassigned without using the value (wastedassign)
        result = true
        ^

Not bug. Your case should be written like below to avoid the warn from wastedassign.
func didPanic(fn func()) (result bool) {
	defer func() {
		result = true
		recover()
	}()
	fn()
	return false
}

@sanposhiho it's absolutely different code :) it doesn't work properly for not panic case:
https://goplay.tools/snippet/mrRqJbl_pTP

@Antonboom recover() returns the value passed to panic(), so you can check that to conditionally set result to true.
https://go.dev/blog/defer-panic-and-recover
	defer func() {
		if r := recover(); r != nil {
			result = true
		}
	}()

@nickajacks1 thank you for your attention, but your code is not valid for Go < 1.21, because I can do panic(nil).
Anyway the question is not "how to rewrite the code", but "how to improve the linter".

I think I understand now. If fn panics, the assignment in the original code is technically used.  I'm not really familiar with Go's AST, but this sounds complex to statically determine.  Some thoughts:

since fn is not first nil-checked, calling it can panic, so we can statically determine that it could panic.

Otherwise, I assume that it is impossible to know if fn panics without looking at every single usage of didPanic

wastedassign could theoretically check for calls to recover within a defer function and assume that any called function object can panic.  This could lead to false-negatives.

I noticed that ineffassign has opted for this route, though it disclaims in the readme that it opts for false-negatives over false-positives.

You could tell wastedassign that a function could panic with some annotation comment, but that's no better than using a comment to disable the lint warning on the variable assignment.
panic(nil) is probably dangerous anyway (sorry to pick on the example again...)

So I'm not really sure what the best move here is.  I've noticed other times where wastedassign seems to opt for false-positives over false-negatives:
// wastedassign gives a "false positive" here (or maybe it's a bug, I haven't really looked deeper)
thing := 0
switch arg {
case 1:
	thing = 3
case 2:
	thing = 39
default:
	thing = 100
}