splitChunks can create initial chunks that are empty after CSS extraction

Bug report
What is the current behavior?
The splitChunks feature (using chunks: 'all') doesn't take into account CSS extraction (via mini-css-extract-plugin) when deciding whether to create a new inital chunk. A new chunk can end up being created that is empty (other than the webpack functions) and under the default 30KB splitChunks.minSize threshold so shouldn't have been created.
eg:
dist/vendors~pageA~pageB.js
For webpack build log output see:
https://github.com/edmorley/testcase-webpack-splitchunks-css#actual
If the current behavior is a bug, please provide the steps to reproduce.

git clone https://github.com/edmorley/testcase-webpack-splitchunks-css.git
yarn install
yarn build

What is the expected behavior?
There is no dist/vendors~pageA~pageB.js chunk generated, since after the CSS is extracted, there should be no common JS code between the two pages.
Other relevant information:
webpack version:  4.8.3
Node.js version: 10.1.0
Operating System: Windows 10
Additional tools<EMAIL_ADDRESS>

@edmorley Maybe related https://github.com/webpack-contrib/mini-css-extract-plugin/issues/85?

Yep that's true

@sokra Is there any workaround for this issue you can think of? I'm simply trying to split out all vendor css, but this is creating an "empty" ((window.webpackJsonp=window.webpackJsonp||[]).push([[0],[]]);) js file. Any suggestions?

@jpdesigndev try prepending those files' contents into you main entry manually.

Any news on this? I think this is the final blocker for our upgrade to Webpack 4.

This is blocking me to upgrade to webpack 4, but it is planned for webpack 5. I guess I will stop trying to upgrade until 5 comes out.

Yes, it is breaking change

hmm... maybe there is a way we could add it for webpack 4 too

If this needs help, I'm happy to help out

Seems like related to https://github.com/webpack-contrib/mini-css-extract-plugin/issues/151

/cc @sokra will be great solve this for webpack@4, because a lot of people have this problem

Right now I'm using this "workaround":
class MiniCssExtractPluginCleanup {
    apply(compiler) {
        compiler.hooks.emit.tapAsync("MiniCssExtractPluginCleanup", (compilation, callback) => {
            Object.keys(compilation.assets)
                .filter(asset => {
                    return ["*/scss/**/*.js", "*/scss/**/*.js.map"].some(pattern => {
                        return minimatch(asset, pattern);
                    });
                })
                .forEach(asset => {
                    delete compilation.assets[asset];
                });

            callback();
        });
    }
}

It's very specific for my use case and has things hardcoded and I even have just put it directly in the webpack.config.js file (so not published on npm).
So it would be awesome if this could be fixed at webpack level.

@sokra will this still be fixed in 4? We are migrating to 4 currently and everything works except this "empty" file.
Or wait for 5 with https://webpack.js.org/plugins/css-webpack-plugin/? Awesome work by the way, i only had problems with the optimise part!

@danechitoaie 's workaround worked, for me, but it does unnecessary looping, incurs a dependency in minimatch, and isn't very flexible as mentioned. I wrote this which passes in a RegExp to be reused for what to delete, and doesn't do extra loops via higher-order functions:
class MiniCssExtractPluginCleanup {
  constructor(deleteWhere = /\.js(\.map)?$/) {
    this.shouldDelete = new RegExp(deleteWhere)
  }
  apply(compiler) {
    compiler.hooks.emit.tapAsync("MiniCssExtractPluginCleanup", (compilation, callback) => {
      Object.keys(compilation.assets).forEach((asset) => {
        if (this.shouldDelete.test(asset)) {
          delete compilation.assets[asset]
        }
      })
      callback()
    })
  }
}

Which then in their case could be used by passing in the RegExp
module.exports = {
   ...,
   plugins: [
     new MiniCssExtractPluginCleanUp(/scss\/.*\.js(\.map)?$/),
   ],
  ...,
}

Also remember you can just run console command to delete this files after webpack done his job.
// package.json
"scripts": {
    "clean-css": "rm ./build/normalize.bundle.js",
    "build": "webpack --config webpack.dev.js && npm run clean-css",
}

Can be useful: Pre and Post hooks for npm scripting

@Kajeczko in some cases this file can have code that run webpack. Example:
⇒  cat build/_js/route.styles.js
(window["webpackJsonp"] = window["webpackJsonp"] || []).push([["styles"],[]]);

@Dwood15 without this code, webpack modules are not started on the client

@Dwood15 @usebaz
I thought the code in styles.js exports all the names generated by css-modules.
import styles from 'some.css'
...
<div className={styles.myClass} />

If you don't use auto generated css class names (e.g using css-modules), then probably you don't need this file.

Any news on this? Removing the file is not an option, without including it the entry js doesn't run and fails silently.

And my solution happened to be to drop mini-css-extract-plugin and revert to extract-text-webpack-plugin but just upgrade it to "^4.0.0-beta.0". Now there are no empty files and no strange "conflicts" ( webpack-contrib/mini-css-extract-plugin#151 ) - all back to normal!

@sokra :wave: Has there been a decision about fixing this for webpack 4? Thanks!

Is #9040 the fix for this?

@rowanoulton, thanks for mentioning me. Does someone know if this will/can be backported to v4?

@rowanoulton, thanks for mentioning me. Does someone know if this will/can be backported to v4?

agreeeeeeee!! that would ne awesome!

It is breaking change, maybe we can implement this behavior under option
/cc @sokra

Looking for a solution to this. I have documented my case here https://github.com/unstoppablecarl/vue-webpack-css-bug

For me the temp solution was to html-webpack-inline-source-plugin to inline these empty chunks, still better than having 2 extra requests for nothing.
I have files generated like this:
chunk0
(window.webpackJsonp=window.webpackJsonp||[]).push([[0],{464:function(n,o,c){},465:function(n,o,c){},466:function(n,o,c){},467:function(n,o,c){},553:function(n,o,c){},554:function(n,o,c){},558:function(n,o,c){},565:function(n,o,c){},566:function(n,o,c){},570:function(n,o,c){},571:function(n,o,c){},581:function(n,o,c){},582:function(n,o,c){},601:function(n,o,c){},605:function(n,o,c){},606:function(n,o,c){},651:function(n,o,c){},657:function(n,o,c){},658:function(n,o,c){},659:function(n,o,c){},660:function(n,o,c){},664:function(n,o,c){},665:function(n,o,c){},669:function(n,o,c){},673:function(n,o,c){},677:function(n,o,c){},681:function(n,o,c){},685:function(n,o,c){},689:function(n,o,c){},693:function(n,o,c){},697:function(n,o,c){},701:function(n,o,c){},702:function(n,o,c){},706:function(n,o,c){},710:function(n,o,c){},714:function(n,o,c){},715:function(n,o,c){},718:function(n,o,c){},722:function(n,o,c){},729:function(n,o,c){},742:function(n,o,c){},743:function(n,o,c){},744:function(n,o,c){},745:function(n,o,c){},746:function(n,o,c){},748:function(n,o,c){},749:function(n,o,c){},750:function(n,o,c){},751:function(n,o,c){},752:function(n,o,c){},753:function(n,o,c){},754:function(n,o,c){},755:function(n,o,c){},756:function(n,o,c){},757:function(n,o,c){},758:function(n,o,c){},759:function(n,o,c){},760:function(n,o,c){},761:function(n,o,c){},762:function(n,o,c){},763:function(n,o,c){},764:function(n,o,c){},765:function(n,o,c){},766:function(n,o,c){},767:function(n,o,c){},768:function(n,o,c){},769:function(n,o,c){},770:function(n,o,c){},771:function(n,o,c){},772:function(n,o,c){},773:function(n,o,c){},774:function(n,o,c){},775:function(n,o,c){},776:function(n,o,c){},777:function(n,o,c){},778:function(n,o,c){},779:function(n,o,c){},780:function(n,o,c){},781:function(n,o,c){},782:function(n,o,c){},789:function(n,o,c){},790:function(n,o,c){},791:function(n,o,c){}}]);

chunk1
(window.webpackJsonp=window.webpackJsonp||[]).push([[1],[],[[797,3,0,6,4,7,9,8,5,2]]]);

Partial solved in webpack@5:
{
  optimization: {
    splitChunks: {
      cacheGroups: {
        styles: {
          type: 'css/mini-extract',
          chunks: 'all',
          // If you need this uncomment
          // enforce: true,
        },
      },
    },
  },
}

https://github.com/webpack/webpack/pull/11585 should fix it, I will put couple tests in mini-css-extract-plugin to ensure all works fine

Done https://github.com/webpack-contrib/mini-css-extract-plugin/pull/599 :star:

@evilebottnawi I've tried with the following packages
webpack: 5.0.0
mini-css-extract-plugin: 1.0.0
css-loader: 4.3.0
and this config
const MiniCssExtractPlugin = require('mini-css-extract-plugin');

module.exports = () => {
  return {
    entry: {
      test: './src/test.css',
    },
    mode: 'production',
    optimization: {
      removeEmptyChunks: true,
      splitChunks: {
        cacheGroups: {
          styles: {
            type: 'css/mini-extract',
            chunks: 'all',
          },
        },
      },
    },
    module: {
      rules: [
        {
          test: /\.css$/i,
          use: [
            {
              loader: MiniCssExtractPlugin.loader,
              options: {
                esModules: false,
              },
            },
            {
              loader: 'css-loader',
            },
          ],
        },
      ],
    },
    plugins: [new MiniCssExtractPlugin()],
  };
};

but it still outputs a dist/test.css and dist/test.js with the latter being empty, any hint what might be wrong here?

@martinherweg Please create reproducible test repo, I think you still on old webpack

@evilebottnawi https://github.com/martinherweg/webpack-css-chunks here I've created a very small test repo.

@martinherweg to be honestly it is not related to this issue, we create empty js because you need js entrypoint, but because you don't have js it is just empty, it's a bit like this problem, but it's a different problem, can you open a new issue about this?

@evilebottnawi just to confirm my understanding. For which cases is your fix where it doesn't generate an empty JS file?
I've added an additional entry point with a JS file, still it generates an empty JS file for the CSS only entry.
I understand it's not related to that issue, I just read various threads about this and read your comment here that it's fixed.
Do you think it's webpack or mini-css-extract-plugin related?
Thanks.

@martinherweg

@evilebottnawi just to confirm my understanding. For which cases is your fix where it doesn't generate an empty JS file?

For all cases, except non js entry points

Do you think it's webpack or mini-css-extract-plugin related?

webpack

So there still isn't a fix for this?

So there still isn't a fix for this?

Partial fixed

Partial fixed

I created a plugin to fix this for us and hold us over until webpack 5. Works fine after the webpack 5 upgrade, but this plugin did the job until then: https://github.com/michaeldurfey/fix-style-emits-webpack-plugin Just concats the empty js files onto the main entrypoint js.

Still not fixed for me when using webpack 5:
cacheGroups: {
                    styles: {
                        name: 'styles',
                        test: /\.(sa|sc|c)ss$/,
                        //type: 'css/mini-extract',
                        chunks: 'all',
                        enforce: true,
                        minSize: 0,
                        minChunks: 1,
                        priority: 20
                    },
                    vendor: {
                        test: /node_modules/,
                        chunks: "initial",
                        name: "vendor",
                        priority: 10,
                        enforce: true
                    },
                    commons: {
                        chunks: "initial",
                        name: "commons",
                        minChunks: 1,
                        enforce: true,
                        maxInitialRequests: 4,
                        minSize: 0
                    },
                }
}

Produces the following js file:

@meglio Not all chunks can be dropped, please create reproducible test repo, I will look

@alexander-akait , I've been playing with various options and made the following discoverty:
Using test: /\.(sa|sc|c)ss$/ in the chunk configuration does emit the js file besides the css file.
Removing that and using type: 'css/mini-extract' instead stops emitting the js file and only emits the css file.
Does this make any sense, and is there an explanation of this behaviour?

@meglio

Does this make any sense, and is there an explanation of this behaviour?

type was implement for this, test doesn't know which modules should be dropped, it is right solution, can you help update docs?

Is there any way to contribute to docs without running a full local developer copy of the webpack website?

You can send improve here https://github.com/webpack-contrib/mini-css-extract-plugin

Do you suggest adding a note in this section?

Hi, I am still seeing an empty js file in my setup. The issue can be seen in https://github.com/rvramesh/empty-js-webpack-repo
Steps

Clone the repo
execute npm run build
In the dist folder we see main.js which is empty and I don't expect the file to be there.

Please let me know if I am missing something.

Thanks @toastal for your minimal plugin, it does the job!

@alexander-akait , I've been playing with various options and made the following discovery:
Using test: /\.(sa|sc|c)ss$/ in the chunk configuration does emit the js file besides the css file.
Removing that and using type: 'css/mini-extract' instead stops emitting the js file and only emits the css file.

In my case I'm running out of memory if I remove the test entry:
[116765:0x613fc70]    18299 ms: Mark-sweep (reduce) 4086.4 (4109.8) -> 4086.2 (4103.0) MB, 5.5 / 0.0 ms  (+ 39.2 ms in 1 steps since start of marking, biggest step 39.2 ms, walltime since start of marking 57 ms) (average mu = 0.947, current mu = 0.814) al[116765:0x613fc70]    18482 ms: Mark-sweep (reduce) 4099.7 (4115.8) -> 4099.0 (4115.8) MB, 49.8 / 0.0 ms  (average mu = 0.901, current mu = 0.728) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb95be0 node::Abort() [webpack]
 2: 0xa9a7f8  [webpack]
 3: 0xd6f5b0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [webpack]
 4: 0xd6f957 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [webpack]
 5: 0xf4ceb5  [webpack]
 6: 0xf5f38d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [webpack]
 7: 0xf39a7e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [webpack]
 8: 0xf3ae47 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [webpack]
 9: 0xf1c04a v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [webpack]
10: 0x12e139f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [webpack]
11: 0x170e079  [webpack]
error Command failed with signal "SIGABRT".

Can you profile your memory and look where is a problem?


New logo for Downloader

I wanted to contribute to Downloader. I designed a logo for it. I hope you like it :)
Please do not hesitate to write, especially if you have something you want.
Here all the editable files.
https://drive.google.com/open?id=14m7mccIMfw233Y0N7puc9esOSmO0F0R6

Maybe you can add your favicon to the main page ?
https://github.com/Strikesoft/Downloader/blob/master/views/base/base.twig

I'm not sure where to add Favicon. I can design a favicon size and format you want. Could you add that?

Just a quick change, can you move your files in web/img/logo please ?

Sure, I sent new pr.
https://github.com/Strikesoft/Downloader/pull/42
Btw, Also i added favicon version.

Thanks 👍 I close this one and merge the other one :wink


bindy: do not embed useless resources #2413

[ ] An issue should be filed for the change unless this is a trivial change (fixing a typo or similar). One issue should ideally be fixed by not more than one commit and the other way round, each commit should fix just one issue, without pulling in other changes.
[ ] Each commit in the pull request should have a meaningful and properly spelled subject line and body. Copying the title of the associated issue is typically enough. Please include the issue number in the commit message prefixed by #.
[ ] The pull request description should explain what the pull request does, how, and why. If the info is available in the associated issue or some other external document, a link is enough.
[ ] Phrases like Fix # or Fixes # will auto-close the named issue upon merging the pull request. Using them is typically a good idea.
[ ] Please run mvn process-resources -Pformat (and amend the changes if necessary) before sending the pull request.
[ ] Contributor guide is your good friend: https://camel.apache.org/camel-quarkus/latest/contributor-guide.html

The build issue seems unrelated, so merging. Thanks for review :+1:


Ruuvi Sensors Not Updating

I am not sure if this is the same problem as Issue #46 but my Ruuvi temp sensors do not update. They do update correctly when HA restarts but then they just hold at their initial value. I have checked VRM and they do update there. I am using a Cerbo.

Hello and thank you for opening this issue,
Since #46 was resolved last year I'm guessing your issues have also been resolved as well.
Therefore, I'm closing your issue as complete (This will help me cleanup stale issues).
Please feel free to re-open if the issue persists.


"Could not map native type 'a' to GDAL type!"

Hello,
I was eager to try out v1.0.0, but after successfully compiling I ran into an issue that I don't know how to troubleshoot.
OS: Ubuntu 18.04 (in WSL)
Source gdb file: https://coast.noaa.gov/htdata/Inundation/SLR/FloodFreqData/Flood_Frequency_data_dist.zip
Desired layer number: 22
Error as follows:
Projection: GEOGCS["GCS_North_American_1983",DATUM["D_North_American_1983",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]]
Could not map native type 'a' to GDAL type! (Use `c++filt -t` to decode.)
terminate called after throwing an instance of 'std::runtime_error'
  what():  Could not map native data type to GDAL type!
Aborted

Next time please include the exact command you ran.

./build/arc_raster_rescue.exe FINAL_HTF_Minor.gdb 22 ./NC_2_HTF_MINOR.tif

@jwilson8767 : I've just pushed a fix that might work.

Do you have any idea what the data should look like?

Wow crazy fast turn around! Thanks!
I realized I said layer 22, but later in the day I had found that later 25 (NC_5_HTF_MINOR) is what I'm really after for at this time.
Here's the result of manually exporting the data via ArcMap -> Copy Raster: https://drive.google.com/file/d/1K294VXRE8Kli_H5Aeo-Kx1vVCjHKjW7m/view?usp=sharing
And here's what I got from the latest change you made:  https://drive.google.com/file/d/1E4HYuPSwvsHyDcwTaBXZjFReNGcdAKBv/view?usp=sharing
Notably, the NODATA value isn't being detected properly, and it looks like it's malprojected.
Here's a screenshot of the source info from QGIS from the version exported via ArcMap:

And here's the same source info from the rescued version:

Is it maybe just mis-encoding some of the CRS info and the NODATA value?

Are you sure the Arc exported data is in the unsigned byte type? It was encoded as a signed byte, which is what I've tried to export.

It looks as though the number of cells differ between Arc's export and the rescued version. That's probably an effect of no-data blocks.

I think the CRS is coming out right, at least on my end. QGIS might have reprojected yours? I'm getting:
GEOGCS["GCS_North_American_1983",DATUM["D_North_American_1983",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]]

The NODATA value is tricky. If you'd like to help with that, what I need is perhaps five separate small raster GDBs which are all the same except for the NODATA value. I also need to know what the value is. Having these for several different data types would probably also be useful.

Ok! This is by no means exhaustive, but I used ArcMap to draw a (vector) circle in a specific projection (EPSG 3631), then used "Copy Raster" to generate a series of raster versions of the circle based on several types (uint8, int8, and float32) and a couple NODATA values. The circle received the value 1 (based on its OBJECTID) in all rasters.
The circle: https://drive.google.com/file/d/1ec-YQ2Pg1l-NJ8EypbCypRJ40Qusbr24/view?usp=sharing
The GDB: https://drive.google.com/file/d/1gjtsRM59LoC_LbB8O-22YAvtnb0w_V1M/view?usp=sharing
Using ArcRasterRescue on each of the raster layers produced differing results, some of which QGIS correctly detected (such as when NODATA is 0), but I'm not sure if the nodata value was actually being set properly on the raster or if QGIS just guessed.
Anyway, hopefully this is enough to figure out what's going on. I'm happy to generate other samples, if needed. ArcMap only crashed 5 times on me for this process, so I figure it's my lucky day :P


Allow more than 1 active subtitle/multi-line subs, and formatting

Currently if more than one sub title is active it won't render:
                // TODO: Render all active subs, not just the first one
                current_subs = subs[0]->Text().c_str();
                if (subs.size() > 1)
                {
                    LOG_WARNING("Too many active subtitles " << subs.size());
                }

And if a subtitle spans more than 1 line its draw as 1 line - i.e \n is ignored.
Additionally basic formatting such as [i]test[/i] should work. Probably at least i,b and u.

This is half done now, [i]test[/i] [b]test[/b] works. But displaying more than one sub does not.


how to load mmlspark-lgb model with java?

how to load mmlspark-lgb model with java?

Take a look at the SparkML docs. It should be the same API:
https://databricks.com/blog/2016/05/31/apache-spark-2-0-preview-machine-learning-model-persistence.html


Extend support for data breakpoints in VS Code API

What it does
The breakpoint API in VS Code supports to add

Allow the user to manage data breakpoints through vscode.debug API
Add methods to gain information about a potential data breakpoint
Allow data breakpoints to have different sources

Keep use case where data id is already known (current)
Add infrastructure for already existing address resolution
Extend for dynamic variables resolved for a session

Ensure dynamic variables are resolved in the debug model

Communication:

Adapt DataBreakpoint with source between extension host and main
Expose DataBreakpoint in VS Code API, previously not exposed (should not cause any backwards compatibility issues)

Minor:

Make bytes optional in data bytes info, as it is the same in the DAP

Fixes https://github.com/microsoft/vscode/issues/195151
How to test
To test this change, you need to make sure you have a C++ file that you can debug as the Typescript debugger does not support data breakpoints as far as I can see.

Install VS Code C++ Tools
Create a C++ file, e.g.,

#include <iostream>

int main() {
    int counter = 0;

    for (int i = 0; i < 10; i++) {
        counter++;
        std::cout << "Counter: " << counter << std::endl;
    }

    return 0;
}

Create and start an extension (or modify an existing one that is already started with VS Code) with the code similar to this one:

vscode.debug.onDidChangeBreakpoints(evt => {
    vscode.window.showInformationMessage(JSON.stringify(evt, undefined, 2));
});

vscode.debug.onDidChangeActiveDebugSession(session => {
    setTimeout(async () => {
        try {
            const info = await session?.getDataBreakpointInfo('i', 1000);
            vscode.window.showInformationMessage(JSON.stringify(info, undefined, 2));
        } catch (error) {
            vscode.window.showErrorMessage(error.message);
        }
        try {
            const info = await session?.getDataBytesBreakpointInfo('0x7fffffffd7e8', 4);
            vscode.window.showInformationMessage(JSON.stringify(info, undefined, 2));
        } catch (error) {
            vscode.window.showErrorMessage(error.message);
        }
    }, 2000);
});

vscode.debug.addBreakpoints([
    new vscode.DataBreakpoint({ type: 'variable', dataId: '0x7fffffffd7e8,counter,4' }, 'readWrite', false, '', false, 'condition', 'hitCondition', 'logMessage'),
    new vscode.DataBreakpoint({ type: 'address', address: '0x7fffffffd7e8', bytes: 4 }, 'readWrite', false, '', false, 'condition', 'hitCondition', 'logMessage'),
    new vscode.DataBreakpoint({ type: 'dynamicVariable', name: 'i', variablesReference: 1000 }, 'readWrite', false, '', false, 'condition', 'hitCondition', 'logMessage')
]);

When starting the extension you should already see the three data breakpoints that you added:

Enable the data breakpoints (or already produce them enabled in your extension code).
Debug the program with the Debug C/C++ File command.
Depending on your other setup not all breakpoints may resolve correctly but you can see that they are properly evaluated.

@microsoft-github-policy-service agree company="EclipseSource"

@thegecko FYI, I believe you have also shown interest in this issue at some point.

@roblourens Thank you very much for your quick feedback, I really appreciate it!
Regarding the retrieval of the variable information, please see my inline comment.
As for the use cases: The original ticket describes a scenario where a data breakpoint feature is needed for a remote debugger. However, I do not have any details on that. The particular use case we are working on has to do with a dedicated memory view where we wanted to support setting data breakpoints through a context menu in our custom UI. The VS Code Debug UIs already have all the necessary access and many bits and pieces were already in place for most of the other breakpoint types. However, we hit a wall particularly with data breakpoints as they are not exposed in the extension API even though most of the functionality was already in place on the "main" side and only a few if-branches were missing on the extension host side. So the core of this change is really rather small and trying to align the support of data breakpoints with the support of other breakpoints. If you have any further questions, please let me know.

I just wanted to check in and see if there's been any progress on reviewing the PR. Let me know if there's anything I can do to help move it forward. Thank you!

@connor4312 Thank you very much for your feedback, I'll have a more detailed look this week!

@connor4312 Thank you very much for your very valuable feedback! I pushed another commit to move the data breakpoint resolution to an earlier point in time and properly propagate the resolution result throughout VS Code. I'm sure there are still areas to improve and I'd be happy to get more feedback but I want to make sure I'm on the right track here.


[MacOS] java.net.SocketException: Broken pipe (Write failed)

Description
We are using Azure DevOps pipelines with the macos-latest agent to build our Android projects. Randomly, a build will error out when downloading dependencies with the following SSL error. Re-running the failed job on a new agent usually results in success.
Starting: gradle build for release
==============================================================================
Task         : Gradle
Description  : Build using a Gradle wrapper script
Version      : 2.186.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/gradle
==============================================================================
/Users/runner/work/1/s/gradlew assembleRelease
Downloading https://services.gradle.org/distributions/gradle-7.0-bin.zip

Exception in thread "main" java.lang.RuntimeException: javax.net.ssl.SSLException: readHandshakeRecord
	at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAccessManager.java:78)
	at org.gradle.wrapper.Install.createDist(Install.java:47)
	at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:129)
	at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:48)
Caused by: javax.net.ssl.SSLException: readHandshakeRecord
	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1300)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:435)
	at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268)
	at org.gradle.wrapper.Download.downloadInternal(Download.java:59)
	at org.gradle.wrapper.Download.download(Download.java:45)
	at org.gradle.wrapper.Install$1.call(Install.java:60)
	at org.gradle.wrapper.Install$1.call(Install.java:47)
	at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAccessManager.java:65)
	... 3 more
	Suppressed: java.net.SocketException: Broken pipe (Write failed)
		at java.net.SocketOutputStream.socketWrite0(Native Method)
		at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
		at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
		at sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81)
		at sun.security.ssl.TransportContext.fatal(TransportContext.java:355)
		at sun.security.ssl.TransportContext.fatal(TransportContext.java:267)
		at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:438)
		... 13 more
Caused by: java.net.SocketException: Broken pipe (Write failed)

Area for Triage:
Android
Question, Bug, or Feature?:
Bug
Virtual environments affected

[ ] Ubuntu 16.04
[ ] Ubuntu 18.04
[ ] Ubuntu 20.04
[x] macOS 10.15
[ ] macOS 11
[ ] Windows Server 2016 R2
[ ] Windows Server 2019

Image version
20210607.1
Expected behavior
Gradle-7.0-bin.zip is downloaded reliably
Actual behavior
Occasionally, this file will be unable to download with the above SSL error.
Repro steps

Clone a git repository holding a basic Android project that is using Gradle 7.0 (The gradle version might not matter)
Run "/Users/runner/work/1/s/gradlew assemble" which will attempt to download the project dependencies, including gradle 7.0
Run these steps on a new macos-latest build agent a bunch of times. Usually this will be successful, but maybe 5% of the time it will fail with an SSL error

Hello, @andrewclaus
Thank you for your report. We will take a look.

Hello @andrewclaus,
Could you please provide minimal steps to reproduce the issue? Is it enough to download gradle binary?
At this time, the issue looks like server-side SSL problems and should be checked from both ends.
We're looking forward to your reply.

The minimal steps are:

Create a new android project that uses gradle 7.0
Upload to git repo
Run these steps on a MacOS 10.15 build agent:
clone git repo
gradlew clean assemble
it will attempt to download the gradle binary and might fail with an SSL error

@andrewclaus Could you please provide us a sample repo with minimal steps to reproduce the issue? Steps by steps instruction with sample repo would be also much appreciated.
We're looking forward to your reply.

@andrewclaus I tried to build a sample application locally and was not able to reproduce the issue. However, tomorrow I got SSL error locally as well. So, I guess that issue somehow related to Gradle downloading and not agent-specific. Did you face the same issue locally?
We're looking forward to your reply.

I have not heard complaints from my teammates about gradle failing to download locally, but we do not have to do that very often, unlike the build agents which have to download it every time. So it could very well just be an issue with the gradle download servers.

@andrewclaus Have you encountered this problem in recent days on our agents?

I'm going to close the issue until further clarifications from your side.
Feel free to contact us at any time.


Activity State not getting restored when the activity is killed and restored

Hi
I've find a bug in the library when using ActivityStarter.save(this) in the onSaveInstanceState that causes the state to not be saved and instead restoring the App with the original Intent when setting the debug option Don't keep activities, this setting only forces the system to kill the activity, but it's a behaviour that happens in the wild.
You can see the code reproducing the bug here https://gist.github.com/danieldisu/692c11c1e9eb26a1f27a895c57209a75
Ways to reproduce:
Set "Don't keep activities" in the emulator
Init activity with 2 arguments.
Modify one argument.
Navigate to another app and go back to the app
Expected behaviour
The changed argument is changed and the other one is untouched.
Real behaviour
The arguments are the original ones

The Intent obtained in the getIntent is the original one, it ignores all the changes made during the ActivityStarter.fill().
One way to fix this would be to save the state in the Bundle "outState" that is provided in the onSaveInstanceState. And use the bundle provided in the onCreate method to restore the state if present.

You are right. I made this change to allow saving state on property delegates (Kotlin). What do you think about the following generation proposition?
Original:
public class MainActivity extends Activity {
  @Arg(optional = true) String name;
}

Generated:
public final class MainActivityStarter {
  private static final String NAME_KEY = "com.example.activitystarter.nameStarterKey";

  public static void fill(MainActivity activity) {
    Intent intent = activity.getIntent();
    if(intent.hasExtra(NAME_KEY))
         activity.name = intent.getStringExtra(NAME_KEY);
  }

  public static void save(MainActivity activity, Bundle bundle) {
      bundle.putString("com.example.activitystarter.nameStarterKey", activity.name);
  }

  //...
}

@danieldisu Can you check version issue_24-SNAPSHOT
compile 'com.github.MarcinMoskala:ActivityStarter:issue_24-SNAPSHOT'

I think it should be ok in current SNAPSHOT

The generic method ActivityStarter.fill() still doesn't have the possibility to pass the bundle.
But using the ActivityNameStarter.fill(activity, bundle) method works like a charm!
Do you think it would be possible to add an overloaded method ActivityStarter.fill() that receives the Bundle?

In the published version, there is ActivityStarter.fill(this, savedInstanceState); Sorry for the delay, but in the moment I am really overloaded with projects. Although I tested it and here you can see example usage.

Everything is working now!
Thanks!!

But this is not the solution because it does not allow Kotlin property delegates to read state from saved bundle. I am still working on this issue.

I see the problem, but it's going to be a hard one to crack because there is no way to access the savedInstanceState Bundle outside the lifecycle methods.

I see now that there is no way to do it. I will fill values if savedInstanceState is not null and if the property is read-write.

Today I finally had more time to fix this problem. From new version, Kotlin needs also fill called for loading saved state (if we need it). I just distributed it in beta version 1.10


Upgrade/Provide Docker Images based on Ubuntu 20.04 LTS

Upgrade/Provide Docker Images based on Ubuntu 20.04 LTS

@tarmath Please provide details about what platform you are using (operating system, architecture). Also include your TensorFlow version. Also, did you compile from source or install a binary?
Make sure you also include the exact command if possible to produce the output included in your test case. If you are unclear what to include see the issue template displayed in the Github new issue template.
We ask for this in the issue submission template, because it is really difficult to help without that information. Thanks!

I am running the official docker image with gpu support on kubernetes
(upgraded to 2.2.0-gpu as of yesterday)
Currently, those docker images are based on the 2 years old Ubuntu 18.04 LTS (Long Term Support) Release. A new Ubuntu LTS Release (20.04, for April 2020) came out a couple weeks ago, and as such, it hopefully follows that tensorflow would naturally start migrating towards it.
Amongst other things, this latest official release comes with python 3.7 (Released on June 27, 2018), which itself provides a ton of improvements, as announced here:
https://docs.python.org/3.7/whatsnew/3.7.html
It also may be pertinent to mention that the nvidia cuda drivers work perfectly under this new release.
Let me know if you would like anymore background on this issue!

/cc @angerson @perfinion

actually need to amend my previous comment slightly:
Ubuntu 20.04 comes with python 3.8 rather than python 3.7 (even better...!)

@tarmath
Is this still an issue

Yes, this work is as close as can be to being inevitable.
As pointed out in previous comments, there seems to be no blocker for this to occur.
As I understand there may be competing priorities, it would be great to minimally have a sense on when this could potentially happen, if nothing else.
Appreciate you taking time for this! 🙏

the install docs say "Ubuntu 16.04 or later".  please update the docker images to 20.04!  thanks.

This isn't likely to be prioritized any time soon -- all our Dockerfiles are defined here and you're welcome to submit a PR.

@angerson @perfinion Any updates on this? I think many people would like to upgrade to the new LTS, including myself. Currently, this is the only issue holding me back...

why does this issue have the tf2.2 label?

I think there are some dockerfiles based on 20.04 already here.
However, I guess the ci release step doesn't docker push such dockerfiles on docker hub yet.
Also, I think something wrong with using this file. It makes all dockerfiles with Ubuntu 20.04 unintentionally as this file
If you guys have a little time, Could you explain the file or docs which explains "how pushing dockerfiles process works in CI step"?
I'd like to handle this issue, but I couldn't get such process in this project by just searching "docker push" in the source code.

@angerson you have a volunteer to resolve this issue in @anencore94 .  could you please help him per his request ^

@anencore94 thanks for the links.  as a temporary workaround i've simply pre-pended my dockerfile that used to FROM tensorflow/tensorflow:latest-gpu with
ARG UBUNTU_VERSION=20.04

FROM ubuntu:${UBUNTU_VERSION} as base

RUN apt-get update && apt-get install -y curl

# See http://bugs.python.org/issue19846
ENV LANG C.UTF-8

RUN apt-get update && apt-get install -y \
    python3 \
    python3-pip

RUN python3 -m pip --no-cache-dir install --upgrade \
    pip \
    setuptools

RUN ln -s $(which python3) /usr/local/bin/python

ARG TF_PACKAGE=tensorflow
ARG TF_PACKAGE_VERSION=
RUN python3 -m pip install --no-cache-dir ${TF_PACKAGE}${TF_PACKAGE_VERSION:+==${TF_PACKAGE_VERSION}}

that is, i replaced FROM with the above.

@bjarthur just want to point out that since you're not installing the cuda drivers, you will not get any gpu acceleration with this solution.

Still no 20.04 images as of 2.4.1, quite sad. I hope once they are released (if ever), it will be communicated and tracked somewhere as having to literally run a container to find out what ubuntu version it has is an overhead.

Is there any traction on this? We're mostly deciding whether to wait for an official Tensorflow release or attempt it ourselves. If it's in the works then we're happy to wait.

Any news on this?

There's been some progress, but nothing new available yet: see
https://github.com/tensorflow/tensorflow/pull/48371 and
https://github.com/tensorflow/build/pull/21.
On Tue, Apr 13, 2021 at 6:02 AM Alexander Kurakin @.***>
wrote:

Any news on this?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/39232#issuecomment-818717940,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHXWEQBTCANCE3EJTRJYSS3TIQ6IDANCNFSM4M2U3Z4A
.

@tarmath,
The PR to update docker files to Ubuntu 20.04 is merged already and can you confirm if that helps resolving your issue? Thanks!

Hi @sanatmpa1!
Thanks for the nudge!
I have just tried downloading the latest tensorflow/tensorflow:latest-gpu image, and it is still on 18.04:
root@d3bdbdcc7fa2:/# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"

My guess is that we'll be seeing images based on the newer stack (besides the nightly ones) next time tensorflow is released...
I'll be looking forward to that!

@tarmath Can you also try out tensorflow/tensorflow:nightly-gpu? Those have picked up the changes, while the other main ones will get updated once the next TF release happens.


grep issue on macOS 10.14.6 (Mojave)

Tools is installed but error when running on grep command.
grep --version
grep (BSD grep) 2.5.1-FreeBSD

This is the error when i running enumeration.

Try update your grep version
brew install grep
export PATH="/usr/local/opt/grep/libexec/gnubin:$PATH"


MH-13369, Delete Capture Agents

Pull request #384 broke the mechanism to show the icon for deleting
capture agents which are offline from within the admin interface since
the JavaScript code does no longer see the actual status but the
translation key used to represent this status.

Works as advertised


Handling the init parameter

Hi,
I tried experimenting with booting system with init=/usr/lib/systemd/systemd parameter. I uninstalled systemd-sysvcompat to be sure, and I use systemd-boot. With mkinitcpio it's working, but with booster don't. On reddit user rickycoolkid point me to this information:

Booster has init binary hardcoded to /sbin/init
https://github.com/anatol/booster/blob/0.7/init/main.go#L26

I was doing it for teaching myself only, and it is nothing that I very need, but I'm curious why it's hardcoded? If someone wanted to change init parameter, he need to change booster code?

Indeed it is currently hardcoded. It is not that difficult to add support for init parameter. But before doing it I would like to understand the usecase better. Why do you need to use non-sysv init binary?

I did it out of curiosity and education (to teach myself). When installing the system, I saw a message that this parameter can be added and I was curious if in practice it would significantly change the management of the system.

Why do you need to use non-sysv init binary?

init=/bin/bash is used in some case. https://wiki.archlinux.org/title/Reset_lost_root_password#Using_bash_as_init

Pushed this feature to wip branch. Please try and let me know if it works for you as expected.

@anatol I confirm, that commit in WIP branch works ideally.
cat /boot/loader/entries/arch.conf
title Arch Linux
linux /vmlinuz-linux
initrd /amd-ucode.img
initrd /booster-linux.img
options root=UUID=a52a687e-bb55-49b9-96ba-638a28b3d86d init=/usr/lib/systemd/systemd rw rootflags=relatime,compress-force=zstd:2,subvol=arch

Boots without problem. Thank you so much for you hard work!


Ritic Announcement Modal

Looks like this: https://www.dropbox.com/s/ec4r841owtresle/May-31-2017 14-28-04.gif?dl=0

Actually, I also noticed that the has-tooltip event listeners need to be unset on destroy, too. Do those need to be done via afterInsert or can they be normal events in the class?


Face encoding using a lot of ram GPU

face_recognition version: 1.3.0
Python version: 3.6.9
Operating System:   Ubuntu 20.04.1 LTS

Description
Hello
I am using face_recognition for my task.
The general description of my system is a time attendance system that uses facial recognition.
For each person, there will be a range of 100-200 photos with their face, the image size is not fixed. I tested on about 20 people with more than 2000 images.
I used the Colad GPU to encoding that set of images, but the problem happened was that when I was encoding around 190 images, the Colad's memory was full.
Is there any way to optimize this encoding?
Thank you
What I Did
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--dataset", required=True, help="path to input directory of faces and images")
ap.add_argument("-e", "--encodings", required=True, help="path to serialize db of facial encodings")
ap.add_argument("-d", "--detection-method", type=str, default="cnn", help="face detection model to use 'hog' or 'cnn")
args = vars(ap.parse_args())
print("[INFO] quantifying faces...")
imagePaths = list(paths.list_images(args['dataset']))
knownEncodings = []
knownNames = []
for i, imagePath in enumerate(imagePaths):
    print("[INFO] processing image {}/{}".format(i + 1, len(imagePaths)))
    name = imagePath.split(os.path.sep)[-2]
    image = cv2.imread(imagePath)
    rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    boxes = face_recognition.face_locations(rgb, model=args["detection_method"])
    encodings = face_recognition.face_encodings(rgb, boxes)
    for encoding in encodings:
        knownEncodings.append(encoding)
        knownNames.append(name)

print("[INFO] serializing encodings...")
data = {"encoding": knownEncodings, "names": knownNames}
with open(args["encodings"], "wb") as f:
    f.write(pickle.dumps(data))

hello, I want to contribute to this issue in order to validate the university project, can you explain to me how it goes please


Show vtable function calls as normal function call

What is the feature you'd like to have?
VTable function calls currently show up as the vtable ptr with a offset:

Which looks rather dirty and prevents me from jumping into the function by clicking on it, or renaming the function globally.
Implementing a feature to make these show up as actual function calls would fix all the issues explained above.
Is your feature request related to a problem?
No
Are any alternative solutions acceptable?
Finding the target function manually and creating a cross-reference to it may work, but is tedious and not a long-term solution.
Additional Information:
None

There is a workaround for this in the meantime:
You can define a structure manually for the vtable, with function pointers to all relevant functions:
struct Foo_vtable
{
    void (*foo_func1)(Foo* this, int etc);
};
struct Foo
{
    Foo_vtable* vtable;
};

Then retype this to Foo* and HLIL should show this->vtable->foo_func1(this, etc).
It's not perfect and it's not very automated, but you could probably write a script to create structures from RTTI information.

Hey @lexika979, the binary for this one is the one you had me run the plugin for you on, right? Do you mind if I attach it to this issue?

Yep, you can attach it, no problem ^^

There is a workaround for this in the meantime:
You can define a structure manually for the vtable, with function pointers to all relevant functions:
struct Foo_vtable
{
    void (*foo_func1)(Foo* this, int etc);
};
struct Foo
{
    Foo_vtable* vtable;
};

Then retype this to Foo* and HLIL should show this->vtable->foo_func1(this, etc).
It's not perfect and it's not very automated, but you could probably write a script to create structures from RTTI information.

Would this enable me to jump into the function aswell?

Yep, you can attach it, no problem ^^

Thanks!
This is the binary in question: engine.zip

Would this enable me to jump into the function by clicking on it?

Double clicking the vtable functions on that would most likely bring you to the vtable type, since the actual function pointers are not known statically (that's the whole point of a vtable). If you annotated the various vtables in memory as struct Foo_vtable then you should be able to see all the data cross references to see the possible values for that function.

I'd be very grateful to see better handling of VTables and RTTI information in general.

Instead of commenting, just upvote the original issue. That's how we often sort for priority.

With the addition of __data_var_refs (see: C++ Types user docs) this is effectively solved.

Yeah, I think I forgot to close this back when we got the new vtable stuff in that you linked to the docs for. Thanks for the reminder. 🙂
Agreed entirely that we need to handle RTTI better, though, thanks for linking the issues. Your plugin was helpful the last time I needed to use it, but you're right that we should be handling this stuff in the core.


[BUG]

Library name and version
Azure.Data.Table 12.8.0
Describe the bug
When trying to Query or Write to a table library throws an exception "Could not load file or assembly 'System.Text.Json, Version=<IP_ADDRESS>" Binding redirects have no effect, and I have also tried in the fresh core 6 project which just crashes with an access violation message.
The table creates fine but I'm not able to write to it or query it.
Expected behavior
Able to write an entity to table and query an entity
Actual behavior
exception "Could not load file or assembly 'System.Text.Json, Version=<IP_ADDRESS>" in a fresh core 6 project fails with an access violation message.
Reproduction Steps
var cloudStorageAccount = new TableServiceClient(...);
var tableClient = new TableClient("...", "...");
tableClient.CreateIfNotExists();
tableClient.AddEntity(new SuburbMapping("...", "..."));
Environment
.Net 4.8 and Core 6.0

Thank you for your feedback.  Tagging and routing to the team member best able to assist.


fix replace update/insert table not match

Coverage increased (+0.005%) to 98.221% when pulling be35c8d3b92fa9edc57f26494da8ca722ccb85b0 on MOON-CLJ:fix_replace_table into ac26a63cdfe2121ca371d4cc6578395484ed366f on kayak:master.


Create bromite-adblock list

Hi,
this repo should only contain a small set of filter lists that I think are relevant to most people. I will close this PR without merging, but of course feel free to keep your repo up to date :)


ARROW-6131: [C++] Optimize the Arrow Non-Ascii-string-validation

Algorithm and code come from: https://github.com/cyb70289/utf8 (MIT LICENSE)
The patch would introduce a fast non-ASCII validation method with this new algorithm into Apache Arrow and give give a option to non-ascii case validation.
Arm64 platform
Benchmark
Origin:
----------------------------------------------------------------
Benchmark                         Time           CPU Iterations
----------------------------------------------------------------
ValidateTinyAscii                 7 ns          7 ns  107435339   1.42978GB/s
ValidateTinyNonAscii             16 ns         16 ns   42655054   639.503MB/s
ValidateSmallAscii               29 ns         29 ns   24516945    4.4671GB/s
ValidateSmallAlmostAscii         91 ns         91 ns    7677848   1.51182GB/s
ValidateSmallNonAscii           175 ns        175 ns    4009837    731.98MB/s
ValidateLargeAscii            18821 ns      18814 ns      37194   4.95077GB/s
ValidateLargeAlmostAscii      64056 ns      64025 ns      10929   1.45533GB/s
ValidateLargeNonAscii        130321 ns     130249 ns       5375   732.909MB/s

The new algorithm:
----------------------------------------------------------------
Benchmark                         Time           CPU Iterations
----------------------------------------------------------------
ValidateTinyAscii                 6 ns          6 ns  116427650   1.59527GB/s
ValidateTinyNonAscii             17 ns         17 ns   41897276   628.046MB/s
ValidateSmallAscii              117 ns        117 ns    5964896   1113.14MB/s
ValidateSmallAlmostAscii        145 ns        145 ns    4819232    971.76MB/s
ValidateSmallNonAscii           118 ns        118 ns    5947924   1085.68MB/s
ValidateLargeAscii            82297 ns      82247 ns       8511   1.13246GB/s
ValidateLargeAlmostAscii      81145 ns      81138 ns       8627   1.14838GB/s
ValidateLargeNonAscii         81221 ns      81202 ns       8621   1.14805GB/s

x86 platform
Benchmark
Origin:
----------------------------------------------------------------
Benchmark                         Time           CPU Iterations
----------------------------------------------------------------
ValidateTinyAscii                 3 ns          3 ns  180870752   3.02228GB/s
ValidateTinyNonAscii             10 ns         10 ns   73066734    1061.9MB/s
ValidateSmallAscii               11 ns         11 ns   61844060   11.2604GB/s
ValidateSmallAlmostAscii         53 ns         53 ns   10000000   2.60296GB/s
ValidateSmallNonAscii            88 ns         88 ns    7948330   1.41807GB/s
ValidateLargeAscii             4887 ns       4886 ns     140027   19.0622GB/s
ValidateLargeAlmostAscii      31479 ns      31474 ns      22286    2.9604GB/s
ValidateLargeNonAscii         64243 ns      64234 ns      10771   1.45132GB/s

The new algorithm:
----------------------------------------------------------------
Benchmark                         Time           CPU Iterations
----------------------------------------------------------------
ValidateTinyAscii                 4 ns          4 ns  137163202   2.29073GB/s
ValidateTinyNonAscii             10 ns         10 ns   71646804   1068.54MB/s
ValidateSmallAscii               46 ns         46 ns   16725701   2.80417GB/s
ValidateSmallAlmostAscii         63 ns         63 ns   11153069   2.17771GB/s
ValidateSmallNonAscii            55 ns         55 ns   12845765   2.25957GB/s
ValidateLargeAscii            28394 ns      28390 ns      24765   3.28084GB/s
ValidateLargeAlmostAscii      27530 ns      27526 ns      21613    3.3851GB/s
ValidateLargeNonAscii         27492 ns      27488 ns      24760   3.39145GB/s

What's the point of including this in Arrow if it's not used in the codebase?

I had been thinking that this would be exposed in some way to be invoked easily on BinaryArray.
If this patch is accepted we need to add something to LICENSE.txt

This also doesn't have any unit tests

This also doesn't have any unit tests

I'd like to add the unit test case for ValidateNonAscii.

Add ValidateNonAscii to unittest case:  arrow-utility-test.
Arm64 Unit tests results:
Test project /home/builder/arrow/cpp/bld
      Start  2: arrow-array-test
      Start 52: arrow-io-compressed-test
      Start 35: arrow-ipc-read-write-test
      Start 26: arrow-compute-aggregate-test
 1/55 Test #26: arrow-compute-aggregate-test .......   Passed    0.90 sec
      Start 63: arrow-logging-test
 2/55 Test #35: arrow-ipc-read-write-test ..........   Passed    1.51 sec
 3/55 Test #63: arrow-logging-test .................   Passed    0.43 sec
      Start 31: arrow-compute-filter-test
      Start 66: arrow-thread-pool-test
 4/55 Test #66: arrow-thread-pool-test .............   Passed    0.47 sec
 5/55 Test #31: arrow-compute-filter-test ..........   Passed    0.47 sec
 6/55 Test #52: arrow-io-compressed-test ...........   Passed    2.22 sec
      Start 65: arrow-task-group-test
      Start 28: arrow-compute-compare-test
      Start 64: arrow-rle-encoding-test
 7/55 Test #65: arrow-task-group-test ..............   Passed    0.28 sec
      Start 55: arrow-io-memory-test
 8/55 Test #64: arrow-rle-encoding-test ............   Passed    0.38 sec
 9/55 Test #28: arrow-compute-compare-test .........   Passed    0.38 sec
      Start 24: arrow-compute-hash-test
      Start 30: arrow-compute-take-test
10/55 Test #24: arrow-compute-hash-test ............   Passed    0.18 sec
11/55 Test #55: arrow-io-memory-test ...............   Passed    0.38 sec
      Start 53: arrow-io-file-test
      Start 44: arrow-csv-parser-test
12/55 Test #30: arrow-compute-take-test ............   Passed    0.28 sec
13/55 Test #44: arrow-csv-parser-test ..............   Passed    0.20 sec
      Start 61: arrow-compression-test
      Start 56: arrow-io-readahead-test
14/55 Test #53: arrow-io-file-test .................   Passed    0.31 sec
15/55 Test #61: arrow-compression-test .............   Passed    0.20 sec
      Start  4: arrow-extension_type-test
      Start 43: arrow-csv-converter-test
16/55 Test #56: arrow-io-readahead-test ............   Passed    0.21 sec
17/55 Test #43: arrow-csv-converter-test ...........   Passed    0.07 sec
18/55 Test  #4: arrow-extension_type-test ..........   Passed    0.18 sec
      Start  1: arrow-allocator-test
      Start  3: arrow-buffer-test
      Start  8: arrow-result-test
19/55 Test  #8: arrow-result-test ..................   Passed    0.08 sec
20/55 Test  #3: arrow-buffer-test ..................   Passed    0.08 sec
21/55 Test  #1: arrow-allocator-test ...............   Passed    0.18 sec
      Start 41: arrow-csv-chunker-test
      Start 36: arrow-ipc-json-simple-test
      Start 38: arrow-json-integration-test
22/55 Test #38: arrow-json-integration-test ........   Passed    0.08 sec
23/55 Test #36: arrow-ipc-json-simple-test .........   Passed    0.08 sec
24/55 Test #41: arrow-csv-chunker-test .............   Passed    0.18 sec
      Start 18: arrow-compute-test
      Start 25: arrow-compute-util-internal-test
      Start 34: arrow-feather-test
25/55 Test #34: arrow-feather-test .................   Passed    0.09 sec
26/55 Test #25: arrow-compute-util-internal-test ...   Passed    0.09 sec
27/55 Test #18: arrow-compute-test .................   Passed    0.19 sec
      Start 22: arrow-compute-boolean-test
      Start 60: arrow-bit-util-test
      Start  7: arrow-public-api-test
28/55 Test  #7: arrow-public-api-test ..............   Passed    0.08 sec
29/55 Test #60: arrow-bit-util-test ................   Passed    0.08 sec
30/55 Test #22: arrow-compute-boolean-test .........   Passed    0.18 sec
      Start 11: arrow-stl-test
      Start 49: arrow-json-test
      Start 10: arrow-status-test
31/55 Test #10: arrow-status-test ..................   Passed    0.08 sec
32/55 Test #49: arrow-json-test ....................   Passed    0.10 sec
33/55 Test #11: arrow-stl-test .....................   Passed    0.20 sec
      Start 20: arrow-compute-operations-test
      Start 48: arrow-localfs-test
      Start 59: arrow-utility-test
34/55 Test #48: arrow-localfs-test .................   Passed    0.10 sec
35/55 Test #20: arrow-compute-operations-test ......   Passed    0.20 sec
      Start 13: arrow-table-test
      Start 16: arrow-sparse_tensor-test
36/55 Test #59: arrow-utility-test .................   Passed    0.21 sec
37/55 Test #16: arrow-sparse_tensor-test ...........   Passed    0.08 sec
38/55 Test #13: arrow-table-test ...................   Passed    0.18 sec
      Start 14: arrow-table_builder-test
      Start  6: arrow-pretty_print-test
      Start 47: arrow-filesystem-test
39/55 Test #47: arrow-filesystem-test ..............   Passed    0.08 sec
40/55 Test  #6: arrow-pretty_print-test ............   Passed    0.08 sec
41/55 Test #14: arrow-table_builder-test ...........   Passed    0.18 sec
      Start 62: arrow-decimal-test
      Start 78: arrow-dataset-file_test
      Start  5: arrow-memory_pool-test
42/55 Test  #5: arrow-memory_pool-test .............   Passed    0.08 sec
43/55 Test #78: arrow-dataset-file_test ............   Passed    0.08 sec
44/55 Test #62: arrow-decimal-test .................   Passed    0.18 sec
      Start 23: arrow-compute-cast-test
      Start 42: arrow-csv-column-builder-test
      Start 37: arrow-ipc-json-test
45/55 Test #37: arrow-ipc-json-test ................   Passed    0.08 sec
46/55 Test #42: arrow-csv-column-builder-test ......   Passed    0.09 sec
47/55 Test #23: arrow-compute-cast-test ............   Passed    0.19 sec
      Start 54: arrow-io-hdfs-test
      Start 40: arrow-concatenate-test
      Start 12: arrow-type-test
48/55 Test #12: arrow-type-test ....................   Passed    0.08 sec
49/55 Test #40: arrow-concatenate-test .............   Passed    0.08 sec
50/55 Test #54: arrow-io-hdfs-test .................   Passed    0.18 sec
      Start 51: arrow-io-buffered-test
      Start  9: arrow-scalar-test
      Start 15: arrow-tensor-test
51/55 Test #15: arrow-tensor-test ..................   Passed    0.08 sec
52/55 Test  #9: arrow-scalar-test ..................   Passed    0.08 sec
53/55 Test #51: arrow-io-buffered-test .............   Passed    0.19 sec
      Start 19: arrow-compute-expression-test
54/55 Test #19: arrow-compute-expression-test ......   Passed    0.10 sec
55/55 Test  #2: arrow-array-test ...................   Passed    6.17 sec

100% tests passed, 0 tests failed out of 55

Label Time Summary:
arrow-tests      =  19.49 sec*proc (54 tests)
arrow_dataset    =   0.08 sec*proc (1 test)
unittest         =  19.57 sec*proc (55 tests)

Total Test time (real) =   6.18 sec
[100%] Built target unittest

@hrw is trying to add Arm support to Arrow CI matrix on PR: #5024

I'm working on adding a more capable Aarch64 machine to the Ursabot cluster (48-core ThunderX) so that could help with providing more timely CI

Cool. But I still don't think we should hamper the full-ASCII case (which will be very common in e.g. CSV or JSON files).

I agree

Cool. But I still don't think we should hamper the full-ASCII case (which will be very common in e.g. CSV or JSON files).

I'd like to propose a full-ASCII optimization for Arm64 later.

Add Ascii validation:
Before:
ValidateTinyAscii 6.04 ns 6.03 ns 112754472 bytes_per_second=1.54331G/s
ValidateSmallAscii 22.8 ns 22.7 ns 29583112 bytes_per_second=5.60858G/s
ValidateLargeAscii 15166 ns 15164 ns 46164 bytes_per_second=6.14226G/s

After:
ValidateTinyAscii 8.06 ns 8.06 ns 87046764 bytes_per_second=1.15497G/s
ValidateSmallAscii 14.5 ns 14.5 ns 48115118 bytes_per_second=8.77193G/s
ValidateLargeAscii 7840 ns 7839 ns 89218 bytes_per_second=11.8821G/s

Get ~ 1.9x speedup for Large Ascii

Rebase th code for file confliction and also have added Afull-ASCII validation optimization here.

@guyuqi this fails to build on windows due to type truncation issues: https://github.com/apache/arrow/pull/5038/checks?check_run_id=380918799#step:4:510

It would be useful to have an ASCII validation function (versus UTF-8 / non-UTF-8), would you be able to break out the ASCII validation changes into a new PR?

It would be useful to have an ASCII validation function (versus UTF-8 / non-UTF-8), would you be able to break out the ASCII validation changes into a new PR?
Yes, of course.


about the path

Problem encountered on https://dotnet.microsoft.com/learn/aspnet/hello-world-tutorial/create
Operating System: windows
Provide details about the problem you're experiencing. Include your operating system version, exact error message, code sample, and anything else that is relevant.
The system cannot find the path specified.

It's not clear what's the issue here. Can you provide more details?

It's not clear what's the issue here. Can you provide more details?


No data under windows server 2016 / citrix

Hi
we have a device which is sending repeatingly a textstring on COM-Port. We wanted to switch to purejavacomm as an easy replacement for original sun comm-api which doesn't support 64bit JVM on windows.
It works on Win7 x64, it works on Win10 x64 but we don't get it to work with the following environment: Windows Server 2016, served to the client-device by a citrix session having the COM-Port from the device mapped to citrix-session - I call this just "citrix" further on.
In this citrix-session the old configuration (32bit JVM with sun comm api) works fine. The only thing which doesn't work is the combination of "citrix" with x64 JVM and purejavacomm.
following is how we use the COMM-API:
serialPort = (SerialPort) portId.open(appname, timeoutForPortOpen);
serialPort.setSerialPortParams(9600, SerialPort.DATABITS_7, SerialPort.STOPBITS_1, SerialPort.PARITY_EVEN);
serialPort.addEventListener(this);
serialPort.notifyOnDataAvailable(true);
BufferedReader in = new BufferedReader(new InputStreamReader(serialPort.getInputStream()));

and then we work with the serialportevents:
   public void serialEvent(SerialPortEvent event) {
      if (event.getEventType() == SerialPortEvent.DATA_AVAILABLE) {
          String line = in.readLine();
          // work with the String read..
      }
   }

I've tried to debug the problem but didn't came to a final conclusion... The "run()"-method defined in the constructor of PureJavaSerialPort is getting executed regularly but the value for "boolean read" evaluates to false as m_DataAvailableNotified is true (line 1182).
When I debug I've seen that the read()-method inside PureJavaSerialPort.getInputStream() is getting executed several times until the first line is read - but never afterwards. This meens when debugging I am able to receive the value from com-port once but then never again. Without debugging even the first time there is no value read - weired....
This beeing said I guess this might have to do with a timing issue?!? But I have no clue how to solve this.
Can anybody help or advice how to dig down further?
regards
Ralf

My first idea is to eliminate Citrix from the equation, try to run this without Citrix so that you know if the problem persists or is related to Citrix. In my limited experience and based on rumours/hearsay from co-workers Citrix can be a lot of problems when it comes to peripherals.

thank you for the fast response. This is what we already tryed to do (but sorry, I didn't point this out clearly):

works on windows 10 (which should be the same codebase as windows server 2016 according to our admins - no chance to connect the COM device directly to server 2016).
the COM-Port itself works within citrix when using the old 32bit JVM and the original sun implementation of the COM API. The problem is we need to migrate to 64Bit JVM and therefore have to switch the COM-API...

So PJC works on Windows 10 with 64bit JVM?
"which should be the same codebase as windows server 2016 according to our admins - no chance to connect the COM device directly to server 2016)."
Well, 'should' is not same as 'is' and codebase is a loose description, either the code is the same or not so saying 'codebase should be the same' sounds like some critical detail can be different. I infer that you cannot physically access the server, but what you could do is that you could install the same Windows server in local machine and try it there.
Just a thought.

according to my first test: yes, windows 10 with 64bit JVM works. In a second test there was something a little strange.
Finally I didn't see a chance to get this working with a current project-deadline coming up. I therefore switched to a RXTX-fork which provided a 64bit dll and voila - it worked out of the box like a charm.
I'd liked the fact to not having to deal with extra native code but that isn't a real problem in our usecase. None the less, thank you for your efforts in providing PJC and in trying to help me with this problem.
As a conclusion this means there might be a bug in PJC regarding this context but so far I can't tell you where exactly the problem is. I therefore leave it to you if you want to close this issue or keep it open.


(Feature Request) Handling missing data in 1D interpolation of a multi-dimensional vector

Hi,
I have a request for the following case.  Let's say I have a N-Dimensional vector of features for each timestamp in a time-series. So a N+1 dimensional data cube. I want to interpolate each feature, side-by-side in the direction of the time-series (1D interpolation).
Currently the scipy.interpolate.interp1d allows for the side-by-side interpolation of a N-dimensional array in one specific direction, which is great.
However, in case you have missing (NaN) data randomly distributed over the N+1-dimensional time-series, this then doesn't apply anymore but is often the case.
One option is to add the time-series also as a N+1-dimensional array, corresponding to the data values. Currently, this would then be like interpolation with grid data, but that is the multi-dimensional interpolation, It would be good to have this option also for 1D interpolation (single direction)
Another option would be by adding some sort of a mask, or boolean weights to the interp1d function, which would be the same shape as the data array, specifying which data points to actually take into account.
Option number 3 would be to add a value which should be ignored and not taken into account in the interpolation, this seems the easiest.
I would greatly appreciate this, coming as someone in the field of earth-observation, where missing data is very common.
Regards,
Matic

Regardless of the feature request, wouldn't this be more suitable for pandas-like containers? Their existence is due to such requirements in which one has to deal with sporadic and unpredictable NaNs.

While I don't understand all details of what you're after, this sounds similar to gh-8781.
For 1D y, you can do the masking manually, like (untested)
>>> mask = np.isnan(y)
>>> interp1d(x[~mask], y[~mask], ...)

In more complicated scenarios, this quickly gets messier, but is still doable.
However, I agree with Ilhan that the full-fledged handling of missing data is likely out of scope for scipy.interpolate.

Hi @ilayn, @ev-br,
yes, this is doable in pandas, but when I compared it seemed quite slow. Maybe it executed it in a sequence?
I am aware of the 1D case for y, but I'm specifically interested in n > 1 where each row of y with respect to x has different positions of missing data.
A more concrete example:
y = [
[0,1,NaN,3,4,5,NaN,7,8,9],
[0,2,4,6,8,NaN,3,5,7,NaN],
[1,3,NaN,NaN,9,0,2,NaN,6,8]
]

here I would like to do a one-directional interpolation (axis can be defined) and  x cannot be a 1D array.

Ok, this example is I'd say well in the pandas-like land. (Note that you can do linear interpolation easily with interp1d).
Maybe this being slow can be fixed in pandas itself.

I think this link can provide a good example of @mlubej 's proposal. I believe that it would be great if scipy can develop interp1d tools ignoring nan values like np.nansum(). Pandas wrapper is good but too slow for large size data array. @ev-br @ilayn

A basic goal misalignment here is what is a nan exactly: is it 'not-a-number', per IEEE754 semantics, or is it 'missing' as is used in the pandas-land. While I appreciate the need to represent missing data, and I'm aware of the widespread usage and utility, the use of nans for that is ultimately a hack.
scipy.interpolate truly cannot grow a fully fledged set of nan-aware functionality, neither as np.nansum-like functions, nor a nan_policy kwarg which is used by scipy.stats. In the end of the day, users need to do some imputations and mask or remove nans from the inputs to scipy.interpolate functions (e.g. (e.g. https://github.com/scipy/scipy/issues/9315#issuecomment-426214110)). Exact means of doing it strongly depend on a problem, which is one reason why it is on a user.
If I'm to dream and handwave for a minute, the only way we could gracefully handle missing data is that numpy were to grow a dtype with missing data. Then linear algebra needs to  support this dtype --- and then scipy.interpolate can happily follow suit. Note that this is pure speculations, I'm not aware of any plans of this sort in numpy.
So realistically, in the foreseeable future, the only thing interp1d can do is to try being a little more selective in returning an array full of nans. Consider a slightly tweaked version of the example from https://github.com/scipy/scipy/issues/9315#issuecomment-426218715, where the second row in nan-free:
  y = [
  [0,1,NaN,3,4,5,NaN,7,8,9],
  [0,2,4,6,8, 52,3,5,7,11],
  [1,3,NaN,NaN,9,0,2,NaN,6,8]
  ]

Currently, the result of interp1d with this y array is an array full of nans. The code which is responsible for this is
https://github.com/scipy/scipy/blob/main/scipy/interpolate/_interpolate.py#L576 and https://github.com/scipy/scipy/blob/main/scipy/interpolate/_interpolate.py#L704
So what's possible is to teach interp1d to detect that the second row is nan-free and interpolate over this row. The first and the third rows will keep producing nans, as they are today.  How to generalize to ndim > 2 I honestly don't know but maybe it's possible. Or maybe it's enough to only consider 2D y arrays.
Concluding, if this partial nan filtering is useful to you, and you are willing to implement it, go ahead and send a PR. We will review it. Otherwise I'm afraid you'll need to keep having to remove nans from data which feeds into scipy.interpolate functions.


Use __traits(identifier) instead of stringof + startsWith in std.typecons.Proxy

Trivial. It's more lightweight.

Auto-merge toggled on


Migrating with existing persistence data fails

If you have a collection of existing akka persistence data and try to migrate over to pekko using this plugin, you'll get this error:
com.mongodb.MongoCommandException: Command failed with error 85 (IndexOptionsConflict): 'Index with name: pekko_persistence_metadata_pid already exists with a different name'
where in the old version of this library it seems the index was named akka_persistence_metadata_pid and you'll have to wipe your instance of mongo so the index can be recreated with the new name

Thanks @bdoyle0182 for reporting.
I will look into it after my vacation.
My first thought on this is that we would probably have to document migrating from akka persistence, eg by configuring journal, snapshot collection names and index names to the previous akka names.

Okay so being able to update the metadata index name helped. Now failing with this error:
[info] org.apache.pekko.protobufv3.internal.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
The only other thing to note is that the old data is using reactive mongo driver and migrating to this uses scala mongo driver. Hopefully that isn't the issue though.

@bdoyle0182 do you persist protobuf serialized payload in akka persistence?
This persistence plugin does nothing with Protobuf, so I don't think this error message is related to this plugin.

I'll look into that further.
Here is the full stack though:
[info]  at org.apache.pekko.protobufv3.internal.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:129)
[info]  at org.apache.pekko.protobufv3.internal.CodedInputStream$ArrayDecoder.checkLastTagWas(CodedInputStream.java:636)
[info]  at org.apache.pekko.protobufv3.internal.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:890)
[info]  at org.apache.pekko.persistence.serialization.MessageFormats$PersistentMessage.<init>(MessageFormats.java:252)
[info]  at org.apache.pekko.persistence.serialization.MessageFormats$PersistentMessage.<init>(MessageFormats.java:200)
[info]  at org.apache.pekko.persistence.serialization.MessageFormats$PersistentMessage$1.parsePartialFrom(MessageFormats.java:1998)
[info]  at org.apache.pekko.persistence.serialization.MessageFormats$PersistentMessage$1.parsePartialFrom(MessageFormats.java:1992)
[info]  at org.apache.pekko.protobufv3.internal.AbstractParser.parsePartialFrom(AbstractParser.java:158)
[info]  at org.apache.pekko.protobufv3.internal.AbstractParser.parseFrom(AbstractParser.java:191)
[info]  at org.apache.pekko.protobufv3.internal.AbstractParser.parseFrom(AbstractParser.java:203)
[info]  at org.apache.pekko.protobufv3.internal.AbstractParser.parseFrom(AbstractParser.java:208)
[info]  at org.apache.pekko.protobufv3.internal.AbstractParser.parseFrom(AbstractParser.java:48)
[info]  at org.apache.pekko.persistence.serialization.MessageFormats$PersistentMessage.parseFrom(MessageFormats.java:887)
[info]  at org.apache.pekko.persistence.serialization.MessageSerializer.fromBinary(MessageSerializer.scala:75)
[info]  at org.apache.pekko.persistence.serialization.MessageSerializer.fromBinary(MessageSerializer.scala:43)
[info]  at org.apache.pekko.serialization.Serialization.$anonfun$deserialize$4(Serialization.scala:199)
[info]  at org.apache.pekko.serialization.Serialization.withTransportInformation(Serialization.scala:167)
[info]  at org.apache.pekko.serialization.Serialization.$anonfun$deserialize$1(Serialization.scala:198)
[info]  at scala.util.Try$.apply(Try.scala:210)
[info]  at org.apache.pekko.serialization.Serialization.deserialize(Serialization.scala:188)
[info]  at pekko.contrib.persistence.mongodb.Serialized.content$lzycompute(MongoDataModel.scala:71)
[info]  at pekko.contrib.persistence.mongodb.Serialized.content(MongoDataModel.scala:50)
[info]  at pekko.contrib.persistence.mongodb.Event.toRepr(MongoDataModel.scala:207)
[info]  at pekko.contrib.persistence.mongodb.driver.ScalaDriverPersistenceJournaller.$anonfun$replayJournal$1(ScalaDriverPersistenceJournaller.scala:235)
[info]  at org.apache.pekko.stream.impl.fusing.Map$$anon$1.onPush(Ops.scala:64)

@bdoyle0182 to me that looks like the configured Pekko deserializer is used to deserialize persisted payload from the journal - and fails doing so.
What kind of payload did you persist in the journal (which _t field is part of the persisted events)?
There are:

b (boolean)
bson (Mongo BSON)
bin (binary)
l (fixedpoint)
d (floatingpoint)
repr (legacy)
ser (serialized), having:

_h persisted as "Hint"
_sm persisted as "serializer manifest"
_si persisted as "serializer id"

s (string)

From the source, the comment beforehand is pretty telling 😦.  Is it possible the class name is in akka namespace?  If so, some translation would need to be done?
    val tried = (serializedManifest,serializerId,clazz.flatMap(c => Try(ser.serializerFor(c)))) match {
      // Manifest was serialized, class exists ~ prefer read-time configuration
      case (Some(manifest), _, Success(clazzSer)) =>
        ser.deserialize(bytes, clazzSer.identifier, manifest)

      // No manifest id serialized, prefer read-time configuration
      case (None, _, Success(clazzSer)) =>
        ser.deserialize[X forSome { type X <: AnyRef }](bytes, clazzSer.identifier, clazz.toOption)

      // Manifest, id were serialized, class doesn't exist - use write-time configuration
      case (Some(manifest), Some(id), Failure(_)) =>
        ser.deserialize(bytes, id, manifest)

      // Below cases very unlikely to succeed

      // No manifest id serialized, class doesn't exist - use write-time configuration
      case (None, Some(id),Failure(_)) =>
        ser.deserialize[X forSome { type X <: AnyRef }](bytes, id, clazz.toOption) // <<<<< line 71

      // fall back
      case (_,None, Failure(_)) =>
        ser.deserialize(bytes, clazz.get)
    }

I believe I found the issue and it's with using the PersistentFSM.
 "_h" : "akka.persistence.fsm.PersistentFSM$StateChangeEvent" is in entry run on akka and
"_h" : "org.apache.pekko.persistence.fsm.PersistentFSM$StateChangeEvent" is in entry run on pekko

@pjfanning This could be just an issue with how this plugin stored persistence data but I'm not positive and thought you would want to know about this. The PersistentFSM was technically deprecated with Akka 2.6 in favor of typed, though it's possible the typed event sourced behavior has the same issue I haven't checked yet. Still, I think PersistentFSM was a widely used feature of classic akka so a migration note might be needed if not specific to just this plugin or if it is specific to the plugin for other plugins that are migrating to pekko they would need to account for this

I raised https://github.com/apache/incubator-pekko/issues/604

cullxbones

@scullxbones where is this code? It could in theory be changed to handle this case.

the code he posted is in this mongo plugin. I'm putting a string replace at the beginning of that function to convert the classname to org.apache.pekko to see if that fixes it

here is a fix that at least works and first attempts the serialization as is and then retries it replacing the legacy class name with the new class name https://github.com/scullxbones/pekko-persistence-mongo/pull/14

@thjaeckle @scullxbones Okay here are two PR's since I've realized we need the capability to rollback to the old plugin as well.
Pekko Plugin PR: https://github.com/scullxbones/pekko-persistence-mongo/pull/14
Akka Plugin PR: https://github.com/scullxbones/akka-persistence-mongo/pull/590
also can we reopen this issue until we have this sorted, it seems I don't have the capability

Now that the pekko version is released, can we make sure the akka version gets released too so that rollbacks become safe. Does anyone still have access to release the akka version of the plugin?

😄 checks passed on the akka pr now

Will take me a bit to recover my environment, the legacy plugin has a bespoke deploy process 🤦.  I'm wondering now whether this "rollback" should be done here in the tools submodule, since it's only related to having upgraded to pekko.

Well the "rollback" doesn't actually mutate the stored data so I don't think it will work as a part of a migration script. I personally prefer having this so that you can live roll forward or backwards between pekko and akka freely. Less work for users means more likely to adopt. We could add a migration script to actually update the underlying data as well, but still think this is necessary functionality as it's needed to live rollback without downtime.

Released 3.1.0 of the legacy plugin with the change, thanks @bdoyle0182!

Thanks so much for getting that released @scullxbones!
@thjaeckle I think when I made the mr I thought it could be major 4.0.0 so I noted that in the readme here that you need to be on 4.0 to support rollbacks so you may want to update that to 3.1

Ah - I've historically tied majors to akka releases for that plugin, so felt a 3.x series was more suitable.  It's also a backward compatible change, so that bar seemed cleared as well.

Yea it's not big a deal, I just picked major to make it a clear indicator of the version to be on to migrate to pekko, but it can fit as minor

I just went ahead and cleaned up the README.md.  Should be good to go.


#370 - Update and clean Alert component

@pournasserian Alert is ready to review

@TheHadiAhmadi in our component dismiss button is not a and it has a border when it is active.

But, in tabler, it is a and it doesn't have border.

Also as you can see, the alignment of the icon, dismiss, and title is not well (it may be for the dismiss button)

I fixed close button and Color of body, but I didn't understand alignment problem


Add DFix option

add option to run DFix on a file https://github.com/Hackerpilot/dfix

dfix will edit your files in-place. Do not use dfix on files that have no backup copies. Source control solves this problem for you. Double-check the results before checking in the modified code.

IntelliJ probably takes care of this automatically, but make sure there is an easy way to undo the DFix operation.

done and will be in next release


Fix typo in the cmake coverage config

I don't understand. Why is this a typo? gcov and gcovr are two different things. We have Find*.cmake script for both.


RuntimeError: Failed to import transformers.optimization

Describe the bug
I've downloaded the corpus and the model weights, I ran the command bash training/finetune_GPT-NeoXT-Chat-Base-20B.sh and I got the following:
https://gist.github.com/riatzukiza/0930307fc90bf940103364be2d3db5c1
To Reproduce
Steps to reproduce the behavior:

Download weights
download corpus
run bash training/finetune_GPT-NeoXT-Chat-Base-20B.sh
Bam error

Expected behavior
To fine tune the model, or get an out of memory error
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: Pop os

Additional context
Add any other context about the problem here.

Looking at your error log, this seems to be important:
RuntimeError: Failed to import transformers.optimization because of the following error (look up to see its traceback):
Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

What version of protoc do you have?
Can you just try importing transformers.optimization in a Python script and see if it gives you any errors?
Can you try running this line in a Python script and see if you get any errors: from transformers import get_linear_schedule_with_warmup

Honestly, it looks like it may be a package or env thing. Check to make sure your conda env is set up properly.
The error comes from line 12 of training/pipeline_parallel/dist_gpipe_pipeline_async.py. Can't proceed without more info.

I'm not sure, but I agree with @orangetin that the environment might not be set up properly. Did you install packages using environment.yml at the root of the repo?

I used the environment.yml like the readme said, it did take a while. I'm taking a look at it again today/tomarrow

@riatzukiza Try the method from this comment if the installation takes too long. Mamba can be a lot faster.

had the same issue. Try running this pip3 install --upgrade protobuf==3.20.0 to solve it


[Docker] HTTP_PORT ignored, changing it in app.ini fails the container

Gitea version (or commit ref): gitea/gitea:1
Git version: ~
Operating system: RHEL 7.6
Database (use [x]):

[X] PostgreSQL
[ ] MySQL
[ ] MSSQL
[ ] SQLite

Can you reproduce the bug at https://try.gitea.io:

[ ] Yes (provide example URL)
[ ] No
[X] Not relevant

Log gist:

2019/09/03 02:42:15 routers/init.go:106:GlobalInit() [I] SQLite3 Supported
2019/09/03 02:42:15 routers/init.go:37:checkRunMode() [I] Run Mode: Production
2019/09/03 02:42:15 ...xorm/session_find.go:199:noCacheFind() [I] [SQL] SELECT "id", "repo_id", "hook_id", "uuid", "type", "url", "signature", "payload_content", "http_method", "content_type", "event_type", "is_ssl", "is_delivered", "delivered", "is_succeed", "request_content", "response_content" FROM "hook_task" WHERE (is_delivered=$1) []interface {}{false}
2019/09/03 02:42:15 ...xorm/session_find.go:199:noCacheFind() [I] [SQL] SELECT "id", "type", "status", "conflicted_files", "issue_id", "index", "head_repo_id", "base_repo_id", "head_user_name", "head_branch", "base_branch", "merge_base", "has_merged", "merged_commit_id", "merger_id", "merged_unix" FROM "pull_request" WHERE (status = $1) []interface {}{1}
2019/09/03 02:42:15 cmd/web.go:151:runWeb() [I] Listen: http://<IP_ADDRESS>:80
2019/09/03 02:42:15 cmd/web.go:154:runWeb() [I] LFS server enabled
2019/09/03 02:42:15 cmd/web.go:210:runWeb() [F] Failed to start server: listen tcp <IP_ADDRESS>:80: bind: permission denied
2019/09/03 02:42:15 ...dules/setting/log.go:269:newLogService() [I] Gitea Log Mode: File(File:info)
2019/09/03 02:42:15 ...les/setting/cache.go:42:newCacheService() [I] Cache Service Enabled
2019/09/03 02:42:15 ...s/setting/session.go:45:newSessionService() [I] Session Service Enabled
2019/09/03 02:42:15 routers/init.go:49:initDBEngine() [I] Beginning ORM engine initialization.
2019/09/03 02:42:15 routers/init.go:51:initDBEngine() [I] ORM engine initialization attempt #1/10...
2019/09/03 02:42:15 ...-xorm/xorm/engine.go:329:Ping() [I] PING DATABASE postgres
2019/09/03 02:42:15 .../xorm/session_raw.go:143:queryBytes() [I] [SQL] SELECT tablename FROM pg_tables WHERE schemaname = $1 AND tablename = $2 []interface {}{"public", "version"}
2019/09/03 02:42:15 .../dialect_postgres.go:946:IsColumnExist() [I] [SQL] SELECT column_name FROM INFORMATION_SCHEMA.COLUMNS WHERE table_schema = $1 AND table_name = $2 AND column_name = $3 [public version id]
2019/09/03 02:42:15 .../dialect_postgres.go:946:IsColumnExist() [I] [SQL] SELECT column_name FROM INFORMATION_SCHEMA.COLUMNS WHERE table_schema = $1 AND table_name = $2 AND column_name = $3 [public version version]
2019/09/03 02:42:15 .../xorm/session_get.go:99:nocacheGet() [I] [SQL] SELECT "id", "version" FROM "version" WHERE "id"=$1 LIMIT 1 []interface {}{1}
2019/09/03 02:42:15 .../dialect_postgres.go:1074:GetTables() [I] [SQL] SELECT tablename FROM pg_tables WHERE schemaname = $1 [public]
2019/09/03 02:42:15 .../dialect_postgres.go:978:GetColumns() [I] [SQL] SELECT column_name, column_default, is_nullable, data_type, character_maximum_length, numeric_precision, numeric_precision_radix ,
	    CASE WHEN p.contype = 'p' THEN true ELSE false END AS primarykey,
	    CASE WHEN p.contype = 'u' THEN true ELSE false END AS uniquekey
...

Description
Using Docker, Gitea fails to bind to a the 80 port.
2019/09/03 02:42:15 cmd/web.go:210:runWeb() [F] Failed to start server: listen tcp <IP_ADDRESS>:80: bind: permission denied

When it fails, the docker container is brought down.
This (using non-standard port for HTTP/HTTPS on a Docker Container) makes no sense at all, and trying to bind it to 80 also doesn't work. I can't think about a need of using the 3000 port, unless there is a problem on permissions (ports < 1024 are restricted to root and I assume gitea runs on the git user).

Using HTTP_PORT environment variable is ignored.
Trying to use port 80 in the app.ini gives the aforementioned log.
Combining with manually exposing the port (80) also gives no results.

Rather than changing the port inside the container, you can use Docker's host mapping. So if you want to put Gitea on port 80, you would put -p 80:3000 in your docker run command.

Rather than changing the port inside the container, you can use Docker's host mapping. So if you want to put Gitea on port 80, you would put -p 80:3000 in your docker run command.
Documentation: https://docs.docker.com/config/containers/container-networking/

Using that with other CICD containers cause them to fail because they can't reach Gitea on port 80, like when using Drone. If these change to 3000, the public request through the browser doesn't point out Gitea because is running in port 3000. This behind load balancers makes them unusable, which is probably 100% of production environments.

drone.gitea.com, and drone.gitea.io are both behind LB's and are configured in a similar way to how @gary-kim has suggested.
You can keep HTTP_PORT as 3000, but update ROOT_URL (for example ROOT_URL=http://git.development.services.cencosud.cl ) to be a URL that doesn't have a port have so that browser requests go through a LB.

That's a weird setup.
Anyway, this can be fixed by allowing /app/gitea/gitea to bind a port below 1024 using setcap. Since the installation allows to bind any number to the port, the user may find a problem when it changes the port to another one when restarting the Gitea container.
To avoid this, libcap should be installed in the container (since it's Alpine) and let gitea the permissions to do so, without running it as root.
I'll make a PR later. For the time being, this can be fixed by using a Dockerfile:
###########################################################################
# Gitea configuration

FROM gitea/gitea:1

# Add libcap and let the gitea binary to bind < 1024 ports
RUN apk --no-cache add libcap \
 && echo "setcap 'cap_net_bind_service=+ep' /app/gitea/gitea" >> /etc/s6/gitea/setup

For the docker container, it shouldn't matter what the internal binding is. I'm not sure why Gitea needs the ability to bind to any port inside the container.

Using that with other CICD containers cause them to fail because they can't reach Gitea on port 80, like when using Drone

Do you have Gitea and Drone set up in a single docker-compose setup? While you could contact another container through the docker provided hostname, it generally should be the same URL used for public access.

Using that with other CICD containers cause them to fail because they can't reach Gitea on port 80, like when using Drone

Do you have Gitea and Drone set up in a single docker-compose setup? While you could contact another container through the docker provided hostname, it generally should be the same URL used for public access.

Exactly.
The problem I'm having is that Drone, when in the same network, will only find Gitea by its name (for example, http://gitea:3000. When it reroutes the browser to OAuth, that hostname doesn't exists (but it does as http://git.development.services.cencosud.cl). If I do an alias, it does work but I have to change the port number in the browser.

The way I have mine setup is with Gitea running with -p 3000:3000 and using Apache as a reverse proxy from https://git.garykim.dev to port 3000 internally. Could the LB be set up in a similar way? The idea would be that Drone also accesses through your LB.

I Understand, but the internal firewall doesn't allow to use any other port. I'm trying to workaround this, but the only solution I have, and that works, is to just push a setcap.

Okay, IMHO, I don't think it's worth it to add libcap to the Docker container because this seems like a relatively rare situation.

That's a weird setup.
Anyway, this can be fixed by allowing /app/gitea/gitea to bind a port below 1024 using setcap. Since the installation allows to bind any number to the port, the user may find a problem when it changes the port to another one when restarting the Gitea container.
To avoid this, libcap should be installed in the container (since it's Alpine) and let gitea the permissions to do so, without running it as root.
I'll make a PR later. For the time being, this can be fixed by using a Dockerfile:
###########################################################################
# Gitea configuration

FROM gitea/gitea:1

# Add libcap and let the gitea binary to bind < 1024 ports
RUN apk --no-cache add libcap \
 && echo "setcap 'cap_net_bind_service=+ep' /app/gitea/gitea" >> /etc/s6/gitea/setup

Thanks, thats what I needed to get gitea working in docker with a macvlan to get a dhcp ip from my fritz.box router... :-)