Blob ContentHash appears not to be set/read  for large files

Describe the bug
Some files in my container appear to have an empty BlobItem.Properties.ContentHash.  I assume the problem occurs on upload but it's possible the issue is with reading files.  Small files do not exhibit the problem. Large files do.  The threshold appears to be around 50K.  It's possible the problem is actual a side-effect of specifying an InitialTransferLength and  MaximumTransferSize of 64K for upload/download in the StorageTransferOptions structure.
Note there is a similar issue at https://github.com/Azure/azure-sdk-for-net/issues/14037 but this describes a different mechanism.
Expected behavior
I would expect the MD5 of a blob always to be available (why would you ever NOT want it set?)  Even if not set by the client on upload it should be calculated by the server when storing the blob.
Actual behavior (include Exception or Stack Trace)
When reading back the blob properties using BlobContainer.GetBlobsAsync, the returned BlobItem entries contain an empty array for ContentHash when the file is larger than some threshold (64K? MaximumTransferSize? )
To Reproduce
Blobs are stored using this code...
        public async Task StoreBlob(string containerName, string blobName, ImmutableArray<byte> bytes)
        {
            BlockBlobClient blobReference = GetBlobAsync(containerName, blobName);

            var stream = new MemoryStream(bytes.ToArray());
            var options = new BlobUploadOptions
            {
                TransferOptions = TransferOptions
            };
            await blobReference.UploadAsync(stream, options).ConfigureAwait(false);
        }

Blobs are listed using this code ...
        public async Task<BlobItem[]> ListBlobs(string containerName)
        {
            BlobContainerClient client = FindContainer(containerName);

            var blobList = client.GetBlobsAsync();
            var blobs= new List<BlobItem>();

            var tasks = blobList.GetAsyncEnumerator();
            while (await tasks.MoveNextAsync().ConfigureAwait(false))
                blobs.Add(tasks.Current);

            // note that the issue exhibits itself as empty Properties.ContentHash fields for BlobItems describing large blobs

            return blobs.ToArray();
        }

where
  private static readonly StorageTransferOptions TransferOptions = new
            StorageTransferOptions
            {
                MaximumConcurrency = 20,
                InitialTransferLength = 0x10000,
                MaximumTransferSize = 0x10000
            };

Environment:
Azure.Storage.Blobs 12.7.0
Windows 10
Net Core 3.0

Thank you for your feedback.  Tagging and routing to the team best able to assist.  Please be aware that due to the holidays, responses may be delayed.

Hi @NeilMacMullen, this is an artifact of how the SDK updates different sizes of data.
There are two ways to upload blob data into the storage service: Put Blob and Put Block / Put Blob List.  When a blob is created with Put Blob, the service calculates the Content MD5, and the blob is essentially immutable. With Put Block and Put Blob List, the service doesn't calculate MD5, since new blocks and be added, and existing blocks can be re-arranged, the service would have to stream the entire content of the blob each time it was modified to calculate MD5.
The SDK uses Put Blob to upload blobs of less than 256MB.  If the blob is data is > 256 MB or InitialTransferLength is specified, Put Block and Put Block List is used instead.
This behavior is unlikely to change in the future.  Please re-open if you have further questions.
-Sean

@seanmcc-msft Thanks for the explanation - if I have understood correctly then the workaround is simply to leave InitialTransferLength at its default value?
I appreciate the SDK is a work in progress but the design here appears "less than optimal".    As far as I'm concerned as a user, I've just called "Upload" and in some circumstances I get a hashcode and in some cases I don't even when using exactly the same client code! I can't see how anyone would think this lack of consistency is a defensible API design.  (This also shows in the linked issue where you either get or don't get a checksum depending on undocumented characteristics of the source stream.)

The SDK uses Put Blob to upload blobs of less than 256MB. If the blob is data is > 256 MB or InitialTransferLength is specified, Put Block and Put Block List is used instead.

On the face of it this seems..... not good  Based on some opaque and undiscoverable criteria (to the user), the sdk is not only deciding whether or not a checksum will be generated but also changing the semantics of the blob I'm storing.  One minute I think I'm storing an immutable blob and then simply by setting a parameter in a structure that I thought was tuning upload rate I've made it modifiable in the future!
My strong suggestion would be to make the behaviour less flexible and more predictable. If the checksum can't be reliably supplied by the server then make it explicit in the API that the client has to supply it.
FWIW the reason that I care about the checksum is that we treat a container as a virtual file-store.  We have code for synchronising subsets of that file-store to a local machine or another file-store.  Being able to rely on the presence of a checksum is obviously a huge win when doing this since we can avoid transferring blobs which are already present in the target.

@NeilMacMullen, the work around is to calculate the content MD5 of your file, and then set it in BlobUploadOptions.HttpHeaders.ContentHash, and then call BlockBlobClient.Upload(stream, options).

I appreciate the SDK is a work in progress but the design here appears "less than optimal". As far as I'm concerned as a user, I've just called "Upload" and in some circumstances I get a hashcode and in some cases I don't even when using exactly the same client code! I can't see how anyone would think this lack of consistency is a defensible API design.

Our primary concern is not the service-generated checksum, it is making sure that most customer's Upload requests don't timeout.  Many customers are on a slower internet connection, and large Put Blob requests will fail.  In addition, it is faster and more efficient to do a multi-part upload.
@tg-msft  @kasobol-msft

work around is to calculate the content MD5 of your file, and then set it in BlobUploadOptions.HttpHeaders.ContentHash, and then call BlockBlobClient.Upload(stream, options)

Thanks @seanmcc-msft - that's useful to know.  👍

Our primary concern is not the service-generated checksum....

Understood  :-)  My point though is that the fact that the current implementation usually generates the checksum for the user is a "bad thing (TM)", especially for users of the earlier Storage library who (like me) have assumed that the checksum is a standard part of a blob and always available. In my case I had to hit to the particular combination of using TransferOptions (for performance reasons) and to generate files larger than 64K.  The poster in the linked issue got tripped up by using "the wrong kind of stream".
As I said, it would be far preferable IMO to either never set it automatically or always set it.  If that's not feasible then at least a large intellisense/documentation warning on the BlobItem.ContentHash property warning that the user can't rely on it being set automatically would seem to be called for.

I have also experienced the same inconsistency where sometimes (for smaller files) I get ContentHash and for larger files I don't get ContentHash.
I am glad that I found this post.
I upvote @NeilMacMullen suggestion to make it absolutely clear on documentation for BlobItem.ContentHash

I have also experienced the same inconsistency where sometimes (for smaller files) I get ContentHash and for larger files I don't get ContentHash.
I am glad that I found this post.
I upvote @NeilMacMullen suggestion to make it absolutely clear on documentation for BlobItem.ContentHash

To add to my previous comment.

work around is to calculate the content MD5 of your file, and then set it in BlobUploadOptions.HttpHeaders.ContentHash, and then call BlockBlobClient.Upload(stream, options)

I tried above for 2.9GB file. I intentionally supplied reversed md5Checksum in BlobUploadOptions.HttpHeaders.ContentHash
.
.
.
            Array.Reverse(mD5Checksum);

            var blobContentInfo = await blobClient.UploadAsync(new FileStream(localFilePath, FileMode.Open, FileAccess.Read), new BlobUploadOptions()
            {
                HttpHeaders = new BlobHttpHeaders() { ContentHash = mD5Checksum }
            });

            return (blobContentInfo.Value.VersionId, blobContentInfo.Value.ContentHash);

I was surprised to see 2 things:

Upload was successful despite wrong md5Cheksum supplied
ContentHash was not returned in the response

I am interested in ContentHash for the upload integrity. If ContentHash is not reliable, what is the best way to check the upload integrity?
@seankane-msft

To add to my previous comment.

work around is to calculate the content MD5 of your file, and then set it in BlobUploadOptions.HttpHeaders.ContentHash, and then call BlockBlobClient.Upload(stream, options)

I tried above for 2.9GB file. I intentionally supplied reversed md5Checksum in BlobUploadOptions.HttpHeaders.ContentHash
.
.
.
            Array.Reverse(mD5Checksum);

            var blobContentInfo = await blobClient.UploadAsync(new FileStream(localFilePath, FileMode.Open, FileAccess.Read), new BlobUploadOptions()
            {
                HttpHeaders = new BlobHttpHeaders() { ContentHash = mD5Checksum }
            });

            return (blobContentInfo.Value.VersionId, blobContentInfo.Value.ContentHash);

I was surprised to see 2 things:

Upload was successful despite wrong md5Cheksum supplied
ContentHash was not returned in the response

I am interested in ContentHash for the upload integrity. If ContentHash is not reliable, what is the best way to check the upload integrity?
@seankane-msft


Performance downgrade after update to v.6.0.0

Do you want to request a feature or report a bug?
Bug
What is the current behavior?
In ver.6.0.0 current state passing into context provider, it leads to update on every consumer (every connected component (on context.consumers)). Wrapped component isn't updates but React has work to be done and its influence on performance is greatly negative.
Our case when there is connected input to store and there are a lot of connected components on page and user starts typing fast (like long keypress), in this case we see lags (like input freezes and throws updates by portions).
What is the expected behavior?
As it was in v5.1.1.
Which versions of React, ReactDOM/React Native, Redux, and React Redux are you using?  Which browser and OS are affected by this issue? Did this work in previous versions of React Redux?
  "react-dom": "16.7.0",
  "react-redux": "6.0.0",
  "redux": "4.0.1",

As an example, you can check this fiddle
https://x3yqx3ov1q.codesandbox.io/ - (upper component with greetings mustn't be updated when new ToDoItem added)
Source code of fiddle
https://codesandbox.io/s/x3yqx3ov1q
UPD it can be related to this issue in react repo https://github.com/facebook/react/issues/13739
If so, what is the workaround (only one we came up with - downgrade of react-redux to v5.1.1)?

This issue has been partially superseded by the roadmap I just laid out in #1177 .
Key point: we're going to switch connect back to using direct subscriptions internally.  We're also going to work on expanding our benchmark harness, and hopefully ensure that the next meaningful release is at comparable in perf to v5.
I'll leave this open for now, but basically this will get resolved when we actually figure out what the right internal re-mplementation of subscription logic is.

https://github.com/facebook/react/issues/13739

6.0.1 to 6.2.1 - refactor - fix Performance
https://github.com/reduxjs/react-redux/pull/1201

Resolved!  see https://github.com/reduxjs/react-redux/releases/tag/v7.0.1


Sync eng/common directory with azure-sdk-tools repository

Sync eng/common directory with azure-sdk-tools repository

All the work for this repo is happening in dev right now. Is it possible to update the sync process to hit that branch. Same for the iOS repo.

We currently don't have that supported in our tooling but we could probably add it, please file an issue in azure-sdk-tools repo if you really want that support. However I have to ask why not work in master?


microk8s high availability cluster on VPS hosting

Hello, I know this will sound a bit fullish, but I would like to know is it possible and how to set up a high availability cluster on VPS hosting where externally each VPS has its own IP address if, for example, I use some load balancer or on one VPS NS1 and on Second VPS NS2 records or add 2 A records with different IP will it work?
Thanks.

A loadbalancer infront of the VPS would be preferable. you may however be able to use nodeport services [1] assuming your servers are reliable.
[1] https://kubernetes.io/docs/concepts/services-networking/service/


EOPA - Batch Query API

[!WARNING]
:construction: This PR is a WIP, and should not be merged until upstream changes to the OpenAPI spec merge in https://github.com/StyraInc/enterprise-opa/pull/74

What changed?
This PR includes support for the Batch Query API in Enterprise OPA, and includes basic tests for the API. Enterprise OPA is spun up in a container as a test fixture, similar to how we've been running OPA in a container test fixture.
TODOs

[x] Add EOPA_LICENSE_KEY for the CI workflows. (Used to spin up the EOPA test container.)
Test cases:

[x] All-Success
[ ] No input case
[ ] Mixed
[ ] All-Failure

:information_source: Trying to get the test suite to run. :sweat_smile:

:information_source: It looks like resolving the merge conflict in .speakeasy/gen.lock was sufficient to get the normal Pull Request jobs to kick off. :slightly_smiling_face:

(Used to spin up the EOPA test container.)

I think if we're passing --no-license-fallback, we should be OK to run the tests without a (valid) license. Am I mistaken?

:information_source: Added some checks in the testcases for when we're in OPA fallback mode versus EOPA mode (Since that's definitely not intended in CI here). The error messages have the same intent, but they have very different message text when that happens.
OPA Fallback mode:
testdata/condfail.rego:5: eval_conflict_error: object keys must be unique

versus EOPA mode:
object insert conflict


Phash dont work

my code:
for image_dir in image_dir_list:
# Überprüfe, ob das image_dir existiert
if not image_dir.exists() or not image_dir.is_dir():
print(f"Das Verzeichnis {image_dir} existiert nicht oder ist kein Verzeichnis.")
continue
    # Erstelle den "remove"-Unterordner
    remove_dir = image_dir / "remove"
    # Überprüfe, ob der Unterordner "remove" existiert, sonst erstelle ihn
    if not remove_dir.exists():
        remove_dir.mkdir()

    phasher = PHash()

    # Find duplicates using the generated encodings
    duplicates = phasher.find_duplicates(image_dir=image_dir)

error:
2024-03-13 11:04:10,376: INFO Start: Calculating hashes...
0%|          | 0/2 [00:00<?, ?it/s]2024-03-13 11:04:17,839: INFO Start: Calculating hashes...
2024-03-13 11:04:17,844: INFO Start: Calculating hashes...
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 289, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\rr004\eclipse-workspace-2023\ParseHTML\duplicate_phash.py", line 59, in
move_duplicates_to_remove(image_dir_list)
File "C:\Users\rr004\eclipse-workspace-2023\ParseHTML\duplicate_phash.py", line 22, in move_duplicates_to_remove
duplicates = phasher.find_duplicates(image_dir=image_dir)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 303, in find_duplicates
result = self._find_duplicates_dir(
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 363, in _find_duplicates_dir
encoding_map = self.encode_images(image_dir, recursive=recursive, num_enc_workers=num_enc_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 161, in encode_images
hashes = parallelise(function=self.encode_image, data=files, verbose=self.verbose, num_workers=num_enc_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\utils\general_utils.py", line 65, in parallelise
pool = Pool(processes=num_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 215, in init
self._repopulate_pool()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 306, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 329, in _repopulate_pool_static
2024-03-13 11:04:17,852: INFO Start: Calculating hashes...
w.start()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 336, in _Popen
return Popen(process_obj)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\popen_spawn_win32.py", line 45, in init
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
Traceback (most recent call last):
File "", line 1, in
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.  File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 116, in spawn_main

exitcode = _main(fd, parent_sentinel)

File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 289, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\rr004\eclipse-workspace-2023\ParseHTML\duplicate_phash.py", line 59, in
move_duplicates_to_remove(image_dir_list)
File "C:\Users\rr004\eclipse-workspace-2023\ParseHTML\duplicate_phash.py", line 22, in move_duplicates_to_remove
duplicates = phasher.find_duplicates(image_dir=image_dir)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 303, in find_duplicates
result = self._find_duplicates_dir(
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 363, in _find_duplicates_dir
encoding_map = self.encode_images(image_dir, recursive=recursive, num_enc_workers=num_enc_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 161, in encode_images
hashes = parallelise(function=self.encode_image, data=files, verbose=self.verbose, num_workers=num_enc_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\utils\general_utils.py", line 65, in parallelise
pool = Pool(processes=num_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 215, in init
self._repopulate_pool()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 306, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 329, in _repopulate_pool_static
w.start()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 336, in _Popen
return Popen(process_obj)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\popen_spawn_win32.py", line 45, in init
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

Traceback (most recent call last):
File "", line 1, in
2024-03-13 11:04:17,864: INFO Start: Calculating hashes...
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 289, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\rr004\eclipse-workspace-2023\ParseHTML\duplicate_phash.py", line 59, in
move_duplicates_to_remove(image_dir_list)
File "C:\Users\rr004\eclipse-workspace-2023\ParseHTML\duplicate_phash.py", line 22, in move_duplicates_to_remove
duplicates = phasher.find_duplicates(image_dir=image_dir)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 303, in find_duplicates
Traceback (most recent call last):
File "", line 1, in
result = self._find_duplicates_dir(
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 363, in _find_duplicates_dir
2024-03-13 11:04:17,914: INFO Start: Calculating hashes...
encoding_map = self.encode_images(image_dir, recursive=recursive, num_enc_workers=num_enc_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 161, in encode_images
hashes = parallelise(function=self.encode_image, data=files, verbose=self.verbose, num_workers=num_enc_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\utils\general_utils.py", line 65, in parallelise
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 116, in spawn_main
pool = Pool(processes=num_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 119, in Pool
exitcode = _main(fd, parent_sentinel)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 125, in _main
return Pool(processes, initializer, initargs, maxtasksperchild,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 215, in init
prepare(preparation_data)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 236, in prepare
self._repopulate_pool()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 306, in _repopulate_pool
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
return self._repopulate_pool_static(self._ctx, self.Process,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 329, in _repopulate_pool_static
main_content = runpy.run_path(main_path,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 289, in run_path
w.start()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 121, in start
return _run_module_code(code, init_globals, run_name,
self._popen = self._Popen(self)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 336, in _Popen
return Popen(process_obj)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\popen_spawn_win32.py", line 45, in init
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\rr004\eclipse-workspace-2023\ParseHTML\duplicate_phash.py", line 59, in
move_duplicates_to_remove(image_dir_list)
File "C:\Users\rr004\eclipse-workspace-2023\ParseHTML\duplicate_phash.py", line 22, in move_duplicates_to_remove
duplicates = phasher.find_duplicates(image_dir=image_dir)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 303, in find_duplicates
result = self._find_duplicates_dir(
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 363, in _find_duplicates_dir
encoding_map = self.encode_images(image_dir, recursive=recursive, num_enc_workers=num_enc_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 161, in encode_images
hashes = parallelise(function=self.encode_image, data=files, verbose=self.verbose, num_workers=num_enc_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\utils\general_utils.py", line 65, in parallelise
pool = Pool(processes=num_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 215, in init
self._repopulate_pool()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 306, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 329, in _repopulate_pool_static
w.start()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 336, in _Popen
return Popen(process_obj)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\popen_spawn_win32.py", line 45, in init
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

Traceback (most recent call last):
File "", line 1, in
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 289, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\rr004\eclipse-workspace-2023\ParseHTML\duplicate_phash.py", line 59, in
move_duplicates_to_remove(image_dir_list)
File "C:\Users\rr004\eclipse-workspace-2023\ParseHTML\duplicate_phash.py", line 22, in move_duplicates_to_remove
duplicates = phasher.find_duplicates(image_dir=image_dir)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 303, in find_duplicates
result = self._find_duplicates_dir(
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 363, in _find_duplicates_dir
encoding_map = self.encode_images(image_dir, recursive=recursive, num_enc_workers=num_enc_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 161, in encode_images
hashes = parallelise(function=self.encode_image, data=files, verbose=self.verbose, num_workers=num_enc_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\utils\general_utils.py", line 65, in parallelise
pool = Pool(processes=num_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 215, in init
self._repopulate_pool()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 306, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 329, in _repopulate_pool_static
w.start()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 336, in _Popen
return Popen(process_obj)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\popen_spawn_win32.py", line 45, in init
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

2024-03-13 11:04:18,412: INFO Start: Calculating hashes...
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 289, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\rr004\eclipse-workspace-2023\ParseHTML\duplicate_phash.py", line 59, in
move_duplicates_to_remove(image_dir_list)
File "C:\Users\rr004\eclipse-workspace-2023\ParseHTML\duplicate_phash.py", line 22, in move_duplicates_to_remove
duplicates = phasher.find_duplicates(image_dir=image_dir)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 303, in find_duplicates
result = self._find_duplicates_dir(
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 363, in _find_duplicates_dir
encoding_map = self.encode_images(image_dir, recursive=recursive, num_enc_workers=num_enc_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 161, in encode_images
hashes = parallelise(function=self.encode_image, data=files, verbose=self.verbose, num_workers=num_enc_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\utils\general_utils.py", line 65, in parallelise
pool = Pool(processes=num_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 215, in init
self._repopulate_pool()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 306, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 329, in _repopulate_pool_static
w.start()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 336, in _Popen
return Popen(process_obj)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\popen_spawn_win32.py", line 45, in init
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

2024-03-13 11:04:18,520: INFO Start: Calculating hashes...
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 289, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\rr004\eclipse-workspace-2023\ParseHTML\duplicate_phash.py", line 59, in
move_duplicates_to_remove(image_dir_list)
File "C:\Users\rr004\eclipse-workspace-2023\ParseHTML\duplicate_phash.py", line 22, in move_duplicates_to_remove
duplicates = phasher.find_duplicates(image_dir=image_dir)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 303, in find_duplicates
result = self._find_duplicates_dir(
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 363, in _find_duplicates_dir
encoding_map = self.encode_images(image_dir, recursive=recursive, num_enc_workers=num_enc_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 161, in encode_images
hashes = parallelise(function=self.encode_image, data=files, verbose=self.verbose, num_workers=num_enc_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\utils\general_utils.py", line 65, in parallelise
pool = Pool(processes=num_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 215, in init
self._repopulate_pool()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 306, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 329, in _repopulate_pool_static
w.start()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 336, in _Popen
return Popen(process_obj)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\popen_spawn_win32.py", line 45, in init
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

2024-03-13 11:04:18,589: INFO Start: Calculating hashes...
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 289, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\rr004\eclipse-workspace-2023\ParseHTML\duplicate_phash.py", line 59, in
move_duplicates_to_remove(image_dir_list)
File "C:\Users\rr004\eclipse-workspace-2023\ParseHTML\duplicate_phash.py", line 22, in move_duplicates_to_remove
duplicates = phasher.find_duplicates(image_dir=image_dir)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 303, in find_duplicates
result = self._find_duplicates_dir(
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 363, in _find_duplicates_dir
encoding_map = self.encode_images(image_dir, recursive=recursive, num_enc_workers=num_enc_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\methods\hashing.py", line 161, in encode_images
hashes = parallelise(function=self.encode_image, data=files, verbose=self.verbose, num_workers=num_enc_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\site-packages\imagededup\utils\general_utils.py", line 65, in parallelise
pool = Pool(processes=num_workers)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 215, in init
self._repopulate_pool()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 306, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 329, in _repopulate_pool_static
w.start()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 336, in _Popen
return Popen(process_obj)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\popen_spawn_win32.py", line 45, in init
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\rr004\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

I  met the same problem.
Python: 3.10.13
imagededup: 0.3.1
OS: macOS

The phash matched the same types of subjects but the images were very different.

The solution for me:
from imagededup.methods import PHash

if __name__ == '__main__':
    phasher = PHash()
    encodings = phasher.encode_images(
        image_dir='path/to/image/directory',
        num_enc_workers=0  # https://github.com/idealo/imagededup/blob/master/imagededup/methods/hashing.py#L141C171-L142C1
    )
    ...

It has to do with the multiprocessing happening under the hood


using mockNuxtImport  with sidebase/nuxt-auth

Hello there - I've just been trying out the new version - thanks again for the work gone into this!
I was trying to get a POC with https://github.com/sidebase/nuxt-auth and mockNuxtImport but the actual composable seems to override my mock.
I can't pinpoint whether it's an issue with nust-vitest or sidebase/nuxt-auth - if with them lemme know and I'll raise it!
Here's a minimum repro https://stackblitz.com/edit/nuxt-vitest-wjddxq?file=test%2Fapp.nuxt.spec.ts
Thanks!

I'm trying to mock the latest sidebase/nuxt-auth with version 0.6, but I get an error:
TypeError: useAuth is not a function

here is the repo:
https://stackblitz.com/edit/nuxt-vitest-pvfdx1
Is ther any known example I can find a working example of mocking useAuth?


Italian translation review

Very good, thank @nicolo-ribaudo for correct my errors!

If you merge this, adobe/brackets#11318 can be merged

I think @nicolo-ribaudo  can close this PR becouse @MarcelGerber merge this PR (https://github.com/adobe/brackets/pull/11117)

No, this PR is still relevant and you should still merge it.
adobe#11117 has been merged mid-May, so it included other strings.


not sure if this is an issue in your code but just in case....

I just posted this bug https://github.com/flutter/flutter/issues/53911 on flutter relating to this package and the latest dev build.

hi, this issue has been fixed in latest codes, please have a try


Kafka index task is working, how can not found consumer groups

druid version: apache-druid-0.20.0
kafka version: kafka_2.12-2.6.0
druid index task is working and can ingest data, but can not found group id by command below
./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

Druid implements exactly-once consumption by using low level api(such as assign, seek) of kafka consumer, which means there are no consumer groups created at the kafka broker side to keep the consumer offset.  If you want to check the offset, you could check the status of a supervisor, where 'current offset' and 'lag' are listed.

How do I set the consumer group name？
"spec": {
"ioConfig": {
"type": "kafka",
"consumerProperties": {
"bootstrap.servers": "<IP_ADDRESS>:9092",
"group.id": "cg_test"
},
"topic": "CLICK_LOG",
"inputFormat": {
"type": "json"
},
"useEarliestOffset": false
}
group.id     Not effective, Not effective!!!
can you help me~

How do I set the consumer group name？
"spec": {
"ioConfig": {
"type": "kafka",
"consumerProperties": {
"bootstrap.servers": "<IP_ADDRESS>:9092",
"group.id": "cg_test"
},
"topic": "CLICK_LOG",
"inputFormat": {
"type": "json"
},
"useEarliestOffset": false
}
group.id     Not effective, Not effective!!!
can you help me~

How do I set the consumer group name？
"spec": {
"ioConfig": {
"type": "kafka",
"consumerProperties": {
"bootstrap.servers": "<IP_ADDRESS>:9092",
"group.id": "cg_test"
},
"topic": "CLICK_LOG",
"inputFormat": {
"type": "json"
},
"useEarliestOffset": false
}
group.id Not effective, Not effective!!!
can you help me~

consumer group can not be set

How do I set the consumer group name？
"spec": {
"ioConfig": {
"type": "kafka",
"consumerProperties": {
"bootstrap.servers": "<IP_ADDRESS>:9092",
"group.id": "cg_test"
},
"topic": "CLICK_LOG",
"inputFormat": {
"type": "json"
},
"useEarliestOffset": false
}
group.id Not effective, Not effective!!!
can you help me~

consumer group can not be set


add "run in collab badge" to the PyTorch and TF examples

That was a fun experiment but doesn't seem we will be moving in that direction 🙂 Let me close this


remote coverage

add feature for local server:

The server source code should be same as the extension source
Remote coverage will be cached in /tmp/doyo/dir

fixed in pull #8


Rebuild hlsl shaders for deferred example

https://github.com/gfx-rs/gfx/commit/31ea3cce6b288f714a6146712449cd1ca017c474 changed the hlsl shader, requiring to rebuild the fx shaders.
Closes #1065

Thanks! (btw, I hope you don't mind me slightly fixing the original message)
@homu r+

:pushpin: Commit acf4720 has been approved by kvark

:zap: Test exempted - status


Update to use @satvariable syntax, which behaves like JuMP @variable

Codecov Report
Patch coverage: 89.28% and project coverage change: -0.74 :warning:

Comparison is base (43beec5) 83.91% compared to head (99f136f) 83.17%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main       #6      +/-   ##
==========================================
- Coverage   83.91%   83.17%   -0.74%
==========================================
  Files           8        8
  Lines         516      541      +25
==========================================
+ Hits          433      450      +17
- Misses         83       91       +8

Impacted Files
Coverage Δ

src/BooleanOperations.jl
86.04% <ø> (ø)

src/BooleanSatisfiability.jl
100.00% <ø> (ø)

src/IntExpr.jl
73.91% <ø> (-2.18%)
:arrow_down:

src/smt_representation.jl
80.00% <ø> (ø)

src/sat.jl
95.34% <75.00%> (-2.38%)
:arrow_down:

src/BoolExpr.jl
77.77% <88.23%> (-0.80%)
:arrow_down:

src/call_solver.jl
82.69% <91.42%> (+1.29%)
:arrow_up:

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


[CICD] Pull required version-n-go CICD scripts from template-specific repo versus Cobalt

Description

As an application developer using the version-n-go process to setup my application repository, I'd like to pull the dependent scripts from my template repository versus the Cobalt repository in order to allow for template-specific modification and adjustment as necessary.*

Acceptance Criteria
Reference: [Done-Done Checklist] (https://github.com/Microsoft/code-with-engineering-playbook/blob/master/Engineering/BestPractices/DoneDone.md)

[ ] Should execute CICD pipeline with devops/providers/azure-devops/templates/infrastructure/scripts/* pulled from the organizational/template-specific repository versus Microsoft/Cobalt.
[ ] Should pin pull to a specific tagged version of the repository.

Also, here are a few points that need to be addressed:

Preference to utilize the application.yml repository resource without having to add additional variables, but not sure if they are accessible in a pre-defined way;
Need to handle Git repo credentials in the case that the organizational/template-specific repository is private.

Resources
Driven by comments from PR #280
Tasks
Stories are intended to be completed in a single sprint; if task breakdown creates addition work then team should discuss promoting the Story to an Epic.
Reference: [Minimal Valuable Slices] (https://github.com/Microsoft/code-with-engineering-playbook/blob/master/Engineering/BestPractices/MinimalSlices.md)
Reference: [How to Write Better Tasks] (http://agilebutpragmatic.blogspot.com/2012/04/splitting-story-into-tasks-how-to-write.html)
Assignee should break down work into tasks here

#298 is a better solution for the version-n-go process that would not require an additional script download step.


Update ubuntu version and node version in ci-cd.yml

The version of ubuntu is deprecated. Use 20.04 instead.
Also upgrade node version.
Signed-off-by: Bernd Hufmann<EMAIL_ADDRESS>

It fixes previous build due to deprecated Ubuntu 18.04. see #245.

It fixes previous build due to deprecated Ubuntu 18.04. see #245.

Seems like the latter's failing build isn't accessible anymore, but that (this PR) should be ok indeed.


[Feature Request] saveAll for Input

When calling KEntities#saveAll with Collection<Input>, it throws
java.lang.IllegalArgumentException: the collection cannot contains input, please call another overloaded function whose parameter is input collection

But there's no overloaded function for input collection.

Resolved, saveInputs


Add dependent and persistent options into vm_add_disk method

Hello Team. One more pull request related to disks on VMware.
Current addDisk method creates new disk as VirtualDiskMode::Independent_persistent, and it's not always good.
The major issue is that independent disks can not be shapshoted live. So, if one attaches such disk to VM, this VM can not be cloned anymore without powering it off. VMware error looks like this:

Virtual machine is configured to use a device that prevents the operation:
com.vmware.vim.binding.vmodl.fault.SystemError: reason = This operation is not supported with virtual disks
in independent mode or raw disk mappings in physical compatibility mode inherited from
com.vmware.vim.binding.vmodl.fault.SystemError: A general system error occurred: This operation is not supported
with virtual disks in independent mode or raw disk mappings in physical compatibility mode

New parameters allow to choose disk provisioning mode in custom scripts. Default behaviour left unchanged - without new params disk is added as VirtualDiskMode::Independent_persistent.

@gmcculloug Please review

Looks good.  @roliveri If you are ok with these changes please merge.

New commit added to address the issues pointed out by @Fryguy and @roliveri

Found one more minor thing, but otherwise looks good to me.

It's worth doing because MiqVimVm is part of the generic Vim interface and not specific to its clients in MIQ. The current MIQ clients can still call the method with a long list of arguments. My comment was just about the implementation of the addDisk method.
So, the method should be changed as I've described. The clients only need to be changed to pass the args by name to the method (as illustrated in my examples).

Ok. Got it. My bad - I didn't consider Vim and MIQ as really separate things. I'll complete and push it asap.

Done. I'd offer these changes as well - https://github.com/ITDSystems/manageiq/commit/e65ee9485b6ee553aa5c7f126ed2b2918e9739df There are two reasons for them:
A. Similarly with MiqVimVm.addDisk it makes calls more readable:
    vm = VmOrTemplate.find_by_id(vmid)
    vm.add_disk(nil, self.size * 1024, thin_provisioned: true, dependent: true)

vs
    vm = VmOrTemplate.find_by_id(vmid)
    vm.add_disk(nil, self.size * 1024, true, true, true)

B. VmOrTemplate is a generic thing. From my point of view, it's not that good to place VMware specific features into its signature (my bad that they were there in initial pull request). As far as I know, thin/thick provisioning is provided by all major virt platforms, but dependent / independent and persistent / nonpersistent disks are not so common.
If I missed something, and this commit is a bad idea, it can be reverted easily.

@roliveri @gmcculloug any thoughts on updated request? Is the last commit ok, or should it be reverted?

Any update on this? If any corrections are required, I'm ready to implement them asap.

Sorry for the delay. Those changes look good to me.

Thanks!


import boto3  or import sagemaker  not working

I started the the sagemaker-notebook:python3-pytorch-p36
in the notebook and use  the kernel  conda_pytorch_p36,
when I try to do  import sagemaker, I get error:

ModuleNotFoundError: No module named 'sagemaker'

I tried to install sagemaker-python-sdk, but run into error:

Solving environment: failed

InvalidVersionSpecError: Invalid version spec: =2.7

Can someone give some suggestion how O can make it work? Thanks in advance

Sorry, unfortunately I'm not able to find time to look into these issues.
Maybe you can check these out as alternatives:
https://github.com/aws/sagemaker-training-toolkit
https://github.com/aws/deep-learning-containers


Async validators not running before all required fields are filled

Description
When there are required fields in the form, then async validator does not run until all required fields are filled. This is probably not correct (you want to validate the field at the time the person is filling it).
Expectations: Fill a wrong password (example: 54321) and get an error.
Current behavior: Fill a wrong password and not get any error until you fill age as well.
Minimal Reproduction

https://stackblitz.com/edit/ngx-formly-ui-bootstrap-pwzmze?file=src/app/app.component.ts
Your Environment

Angular version: 10
Formly version: 5.10.32 (does not work on latest either)

Additional context
Adding a key to the asynchronously validated form group solves the issue (and will use that as a work-around for the time being).
Is that expected behavior? It feels counter-intuitive because it works differently for synchronous validators.

您好，邮件已经收到，查阅后会尽快给您回复！

That's on purpose, Async validators are called only when form state (invalid -> valid) is changed to avoid an unnecessary calls.
In the above example since the initial form state is invalid (password & age marked required), the call will be done once both fields are filled.

The thing is, that the "password" field is embedded in a group, and my expectation was that it would check only for the validity of the group, as it has no reason to wait for other fields when all fields it depends on are already filled and valid. This is actually what happens when i add a key to the group (like key: 'passwordGroup'), so it struck me as an inconsistency, since i didn't expect this to make a difference.
Thanks for the answer though, i can work around this by transforming the model.

when fieldGroup doesn't have a key we attach its children to the parent Form instance.

i can work around this by transforming the model.

Better :+1:


maks parsing generate a Maximum call stack size exceeded

Hi,
I am getting the following error when define a mask
Uncaught RangeError: Maximum call stack size exceeded
handleMatch @ jquery.inputmask.bundle.js:290
handleMatch @ jquery.inputmask.bundle.js:299
handleMatch @ jquery.inputmask.bundle.js:299
...

The mask definition is like this :
$.extend($.inputmask.defaults.definitions, {
            '?': {
                validator: ".",
                cardinality: 1
            },
            'A': {
                validator: ".",
                cardinality: 1,
                casing: "upper"
            }
        });

$("#myid").inputmask("A{0,1}?{0,49}", { "placeholder": "" });

when using the same pattern with a mask like "?{0,49}" the problem does not happens
it's seems that the error is linked with the usage of "?" as pattern because if I choose an other letter it works as expected.

@aetna-softwares ,
There was a mistake in the tokenizer regex!
Thx for reporting.


Unable to connect

This error message is very generic. I find myself seeing this way to often and have to dig in the log file to find the reason. Is it possible to show a more informative error message? I encounter "Mono.AndroidTools.InsufficientSpaceException" a lot because of the default storage settings being very low when creating new virtual devices. This isn't really a connection problem either

Thanks for filing this, I thought I had an issue for it already but it appears not.
Yes, we should show much better error messages for the various problems that can occur connecting to (especially Android) devices. I believe right now it would show up in a tooltip on the status bar, which is not discoverable at all.


[DOCSTRING]missing function parameters updated

@FrozenGene please check and help to merge. TIA
Thanks for contributing to TVM!   Please refer to guideline https://tvm.apache.org/docs/contribute/ for useful information and tips. After the pull request is submitted, please request code reviews from Reviewers by @ them in the pull request thread.

Thanks @siju-samuel


set language to english in init.lua

NOTE
Please verify that the base repository above has the intended destination!
Github by default opens Pull Requests against the parent of a forked repository.
If this is your personal fork and you didn't intend to open a PR for contribution
to the original project then adjust the base repository accordingly.

what is the styling problem?

Hi.
Thanks for your contribution but this change would break everything for any non English speakers who use kickstart.
I don't think this is appropriate for this project at this time. Closing.

Its is a funny statement.  To me everything breaks for me when neo insist i want the lang to be the os :-) i nearly departed until i found this setting. It is better than deleting the lang sub folders

Glad you found what you needed.
Feel free to modify your PR to add a comment letting people know this setting exists if you feel it's important for all users.


AttributeError: 'CoxPHFitter' object has no attribute 'predict_hazard'

I can't seem to call fitter.predict_hazard()
I am using version 0.24.9

Hi @jlim13, this is by design. The hazard for the cox model is very noisy, due to it being non-parametric. Instead, we encourage use of either predict_cumulative_hazard, predict_partial_hazard  or predict_log_partial_hazard.

gotcha, thanks! I was actually reading through more of the comments in the source code and predict_log_partial_hazard is what I was looking for. Thanks for the prompt response


minotaur-13b results

according to git status there was no updated leaderboard file.
              win_rate  standard_error  n_total
minotaur-13b     67.64            1.64      805

Did you run with the gpt4 evaluator? src/alpaca_eval/leaderboards/data_AlpacaEval/alpaca_eval_gpt4_leaderboard.csv  should be updated
Also, could you remove results/minotaur-13b/reference_outputs.json from the PR? reference outputs are the same on the same dataset so no need to keep track of that

Following up on discord discussion, we require >= python3.10 for this package.
Please try again on python>=3.10 and reinstall package via pip install -U alpaca_eval.
Then run again and add the updated leaderboard to the PR :)

@rtaori let me know if the naming is confusing. I noticed that I had a bug and was missing datasets from the model, so I re-published it as openaccess-ai-collective/minotaur-13b-fixed. It's going to be confusing no matter how I do it I guess. additionally, that's why the win_rate dropped since the PR was initially submitted.


added a new list for chaos engineering with a tech talk

This PR is for adding a new list on Chaos Engineering. Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production.
With this list, I have also included a tech talk done by me, on Chaos Engineering and a demonstration of it.

Hi @Keetmalin,
Thanks a lot for the PR.
I maintain another list for Chaos Engineering: https://github.com/dastergon/awesome-chaos-engineering
Feel free to send a PR there.


[WIP] some small work on the docs

Comments:

The top-most patch 8b9044f4f is optional (well, all of them are at your discretion, of course).  That's why it's on top to drop it most easily if necessary.
not sure if the <tt>-tags in cbdee6e696 are correct or if one should use apostrophes instead (in case you follow a certain style)

Aeesome mate, thanks! :+1:

ready to merge?

On 14.02.2017 06:46, Gawain Lynch wrote:

Could you be so awesome as to use 'B' for Bolt in cases where is is
used for branding 😄

Absolutely. Thanks for the review.

Thank you! Merging in!


feat: Side Infinite Scroll

Description
Added a side scroller that people can use to place custom brand images inside
Related Issue
Fixes #130
Proposed Changes

Added SideInfiniteScroll to showcase: src/showcase/animations/SideInfiniteScroll.tsx
Added card to animation UI Group: src/showcase/ui-group/AnimationCards.tsx
Added docs page: src/app/docs/animations/side-infinite-scroll/page.mdx
Added navigation link: src/components/Navigation.tsx
Added credit: credits.md

Screenshots
https://github.com/Ansub/SyntaxUI/assets/16091805/131add15-d86a-4357-a467-2f4994935ba4
Checklist
Please check the boxes that apply:

[x] I have tested the changes locally
[x] I ran npm run build and build is successful
[x] I have added necessary documentation (if applicable)
[x] I have updated the credits.md file (if applicable)

Additional Context
Please provide any additional context or information about the pull request here.

I feel like marquee slider would be a better name, since thats what this is usually called.


Is IE10 supported by this library?

When I use this library in IE10, most of the styles definitions don't appear in the generated HTML.
An example of the same web page generated in IE10 and IE11.

I'm afraid IE10 is not supported.


[CircleCI] Integration and Codecov posting error

List of errors noticed:
Note: check the check box when the issue has been resolved

[x] Error in codecov.yaml file causing the error as per Action logs. @Maverick099

CircleCI config update.

[x]  Adding command to create venv inside docker
[x]  Adding lines to install local modules.
[x]  Installing requirements.txt modules before running tests.

CodeCov integrations issue.

[x]  Making sure codecov posts the file generated by coverage.py

All above Tasks completed but sill running into errors.

python -m unittest discover -s folder/topleveldirectory/tests
updating the config with this line to tackle  this line error in CircleCi console

this error is solved by optimizing import for files inside the package.
from .response_handlers import  ResponseHandlers, Response
from .auth_handler import generate_jwt_token


Add some DEBUG loggers for connection strategies

I spent some time figuring out why I got an SSLException when trying to make my test container to work.
It appears to be a simple library version conflict on bouncycaste, but the exception cause is not logger at all.
It should be possible to log the exception causes when log level is debug.

Good idea - this would be useful. As we're suggesting in the docs that people set root log level to INFO, this shouldn't have an effect on log noise for normal usage.
Thanks for this!


Consider Ngons when creating initial shapes for PRT

Faces (tris/quads) not part of Ngons are passed through.

Let's consider our built-in conversion from Surface/Brep/Curve/Polyline to Mesh just as a quick starting point for the user. For advanced cases, the user wants to control the conversion in more detail using the GH "Mesh Brep" tool for even the "Ngon" plugin. I've documented these cases accordingly in the user manual section.


cjs / es conflict "cannot import outside of a module"[Bug]:

Expected Behavior
Compilation and startup of front end would be completed without error
Actual Behavior
Startup of project results in error ""cannot import outside of a module"
This is caused by the fact that the storefront-ui package is NOT built and given that cjs cannot utilize the import statement from legacy EMCA
Possible Solution
Add the module "storefront-ui" to the transpile option under the build configuration in the nuxt.js file
Resulting code will look like this
    transpile: ['vee-validate/dist/rules', 'storefront-ui'],
    plugins: [
      new webpack.DefinePlugin({
        'process.VERSION': JSON.stringify({
          // eslint-disable-next-line global-require
          version: version0.version,
          lastCommit: process.env.LAST_COMMIT || ''
        })
      })
    ],

Alternative Solution####
Attempt to utilize the import as a nuxt module approach, which was recently published. Have not tried this yet personally, but is a viable option
Steps to reproduce
Attempt to run command yarn dev:theme and note vue strart up issue
What version of shopify integration are you using?
1.06
What version of Node.js are you using?
16.13.0
What browser (and version) are you using?
Chrome
What operating system (and version) are you using?
MacOS
Relevant log output
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Hi, @jaydubb12 thanks for the report. But could we follow where are you experiencing these bugs as it is deemed not replicable to my machine? Thanks


scanner: Create and store file listings for each resolved provenance

Create and store the file listings (path, sha1) for each resolved provenance.
These can be consumed in future iterations (see #6945) for improving the generated SBOMs.
Resolves: #6943.
I've scanned the HEAD of the main branch of npm mime-types which resolved to 335 provenances. The total size of all file listings in the Postgres database is 5605372 bytes. So, about 16.4kb per file listing in average.
BREAKING CHANGE: One does not have to, but should configure a storage backend in ~/.ort/config/config.yml.
If one doesn't do so, ORT falls back to storing the file listings under ~/.ort which may not be
desired. See the modified reference.yml for how the new configuration feature looks like.

Looking at above discussion I have a hunch it may be more efficient to come to a conclusion in a short call. @sschuberth @mnonnenmacher  would you be in for that?

@mnonnenmacher @sschuberth I've incorporated the changes we've agreed upon in our call yesterday.

What is the rationale for this? The default might change, but this would still be an error.

@mnonnenmacher the removal of the explicit Severity.ERROR happened on my request to align with the code base. If we cannot rely on the value of default arguments, it makes little sense to use default values at all...

Then I would argue for removing the default value, the issue severity is something I would like to see on the call side.

Then I would argue for removing the default value, the issue severity is something I would like to see on the call side.

Are you fine with de-scoping this from this PR then. I don't like changing this back and forth. In my view this is now a different topic. Agreed?


Replacements are not supported

Replacements are supported in Sequalize:
sequalizeDb.query(       'SELECT name FROM users WHERE id = :id',       { replacements: { id: userId } } );
However this throws an error using sequelize-redis-cache
syntax error at or near ":"

This would be a great improve


Upgrade maven-enforcer-plugin 3.4.1 -> 3.5.0

This PR contains the following updates:

Package
Type
Update
Change

maven-enforcer-plugin
build
minor
3.4.1 -> 3.5.0

[ ] If you want to rebase/retry this PR, check this box

Suggested commit message:
Upgrade maven-enforcer-plugin 3.4.1 -> 3.5.0

See:
- https://github.com/apache/maven-enforcer/releases/tag/enforcer-3.5.0
- https://github.com/apache/maven-enforcer/compare/enforcer-3.4.1...enforcer-3.5.0

Edited/Blocked Notification
Renovate will not automatically rebase this PR, because it does not recognize the last commit author and assumes somebody else may have edited the PR.
You can manually request rebase by checking the rebase/retry box above.
⚠ Warning: custom changes will be lost.


Journeyman/model import test

pytest -s tests\test_model_import.py --brain_name testmoab --custom_assess_name my_custom_1 --log_analy_workspace <YOUR_LOG_ANALYTICS_WORKSPACE>

Bonsai does not automatically start logging of custom assessment using CLI. I'll have to think about how to address this. There's a thread going where this was going to be changed.

Please use bonsai-cli from brain/src/Services/bonsaicli2 because logs don't automatically get stored without it. Waiting on release to Pypi
git checkout arkadiya/custom-assessment-log-all-and-limit-20
cd brain/src/Services/bonsaicli2
pip install -e .

Also episode lengths are incorrect
https://bizair.visualstudio.com/Machine Teaching/_workitems/edit/24479/


With multiple results it is hard to access the cause of the failure

On normal calls, the fail method has the exception as the parameter. When syncing multiple calls, the fail method returns a OneReject instance. But it is not easy (if even possible) to access the exception causing the failure through that instance.
https://github.com/jdeferred/jdeferred/blob/master/subprojects/jdeferred-core/src/main/java/org/jdeferred/multiple/OneReject.java

Ah, it feels like OneReject should have the rejection type as a generic.
Would that help w better type safety?
On Wed, Jun 7, 2017 at 12:46 PM Tom Eugelink<EMAIL_ADDRESS>wrote:

On normal calls, the fail method has the exception as the parameter. When
syncing multiple calls, the fail method returns a OneReject instance. But
it is not easy (if even possible) to access the exception causing the
failure through that instance.
https://github.com/jdeferred/jdeferred/blob/master/subprojects/jdeferred-core/src/main/java/org/jdeferred/multiple/OneReject.java
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jdeferred/jdeferred/issues/113, or mute the thread
https://github.com/notifications/unsubscribe-auth/AB6AI65JQKXTKfy7bMw5J4luUHThkOiFks5sBnFvgaJpZM4Nyddz
.

This is not about type safety; I just want to know what caused the fail to be called. The 'normal' fail gets the causing exception as a parameter, where is it hidden here?

Iirc, It should be in the OneReject.getReject()
But you do need to an instanceof and then casting it
On Thu, Jun 8, 2017 at 10:30 AM Tom Eugelink<EMAIL_ADDRESS>wrote:

This is not about type safety; I just want to know what caused the fail to
be called. The 'normal' fail gets the causing exception as a parameter,
where is it hidden here?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/jdeferred/jdeferred/issues/113#issuecomment-307023156,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB6AIwWGeQ_REl1ZSPlfcq6uB4OJ87g5ks5sB6MjgaJpZM4Nyddz
.

Ok. I see. And I see that the other fail method also has a parameter of type Object. Isn't that always a Throwable or Exception?

not quite, you can reject with a non-Exception. E.g., deferred.reject("failed")
However, if you let DeferredManager to create a promise from a Callable, then the rejection is always an Exception.

Ok! With the javascript roots I get that. I would have wrapped that reject in an exception so the API would be identical. Butthat explains it. Thanks.

Sounds like we were good. Closing for now :) Thanks!


Bump does not work properly

Describe the bug
Bump does not reliably bump the version e.g. for a minor change ( feat: ) .In addition sometimes it produces "null" as next version (but i have not found a way to reproduce it).
BTW Just an idea: git cliff --bump --unreleased should offer to create the tag "v0.0.1" if no tag is set before.
To reproduce

Expected behavior
It should be bumped to 0.1.0.
Screenshots / Logs

Software information

Operating system: Win10 as well as WSL2
Project version: git-cliff 1.4.0

Hello, thanks for reporting this!

git cliff --bump --unreleased should offer to create the tag "v0.0.1" if no tag is set before.

Created #344

It should be bumped to 0.1.0.

Not sure what happened there, maybe ping @marcoIeni who is the author of next_version.

It should be bumped to 0.1.0.

This is not how the next_version library works. I clarified it in https://github.com/MarcoIeni/release-plz/issues/1064

I think this behavior is fine as long as it is documented in git-cliff as well, does anyone volunteer to add that?

The behavior has changed to support this use case in 4eef684c568ad16357e5d256180a51b1a46dd0cb


ENH: Implements nan_policy for uniform_filter

Earlier commit (18713491f) implemented ignoring of NaNs in the uniform_filter and uniform_filter1d functions. This commit moves to using the nan_policy keyword argument for determining how NaNs are to be handled so that API calls are consistent across scipy.
Have created a new C-level function that checks/ignores NaN values when computing boxcar averages for the uniform_filter function. This new function is almost identical to the original, but now checks/ignores NaN values when summing and adjusts the number of values in the average based on finite values; i.e., a numpy.nanmean()-like average is taken over the window.
Tests have been added to the testing suite to check that NaN ignoring is working correctly.
A new benchmark has been added to the benchmarking suite; tests indicate very minimal impacts to performance when using ignore_nan=True as opposted to ignore_nan=False. More robust benchmarking should be performed to fully understand any preformance impacts; however, getting the 'right' answer (i.e., not propogating NaNs through entire dataset) is typically more important than being blazing fast.
Closes #7818

Reference issue

What does this implement/fix?

Additional information

Ooo!  I somehow only just noticed this.  I haven't tested it, but having this would be lovely.


Regression on hot_mode_dev_cycle__benchmark

Regression on hotReloadMillisecondsToFrameAfterChange since 20 Dec 2019. It seems to have started consistently going over the baseline after an engine roll.

Possibly caused by: https://github.com/flutter/flutter/commit/0db029990707cf98d79d66c45802207dce768891

Closing as duplicate of https://github.com/flutter/flutter/issues/47842


Request for Enhanced Documentation

Hi there,
Firstly, this project is interesting! I just love the idea of developing our own video editor and recently I'm also trying to work on this.
And I've tried to build your TitanVideoEditor on Ubuntu 22.04 and 23.04. The build went fine, and I could browse files, add tracks onto the timeline, drag clips to the tracks, but couldn't figure out how to preview the video and do something further. Maybe I missed something?
The building instructions was clear in your README, but it'd be awesome to see some screenshots or more details on how we can interact with this app(or the available features), to let user knows the correct behaviors of the application. BTW, I'm also struggling with real-time previews in my video editor project(with ffmpeg)🥲.
I'm pretty interested in this project and although I'm not a coding whiz, I'd love to help if I can.
Looking forward to seeing the project evolve!
Thanks for considering these suggestions.

Thanks for showing interest in this silly little project of mine!
By the way, the video preview/rendering is not implemented yet, still figuring that out.
I took a break to work on another projects (and because of school too), maybe I can finally come back to this.


[WIP] Windows dev environment configuration, update install instructions from source in the docs

Description
Taking over #17206 with fix to #17635, update toolchains to vs2019 and cuda 10.2
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[ ] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[ ] To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

Let me know when this is ready, and I'll test it out.

@vexilligera what is the status of this PR?

@vexilligera what is the status of this PR?

I'm trying to update the AMI to VS2019 and CUDA 10.2 and there's a bug with dmlc-core, probably caused by different MSVC standards. I'm working on that.

Making it public is redistribution and thus license infringement.

I may also need to upload the VS2019 web installer as in install_vs, since the original download page needs redirection to the file url and I couldn't find any short url as for VS2017.

Updated links for cudnn & VS2019

https://windows-post-install.s3-us-west-2.amazonaws.com/vs_community__1246179388.1585201415.exe
https://windows-post-install.s3-us-west-2.amazonaws.com/cudnn-10.2-windows10-x64-v<IP_ADDRESS>.zip

@mxnet-bot run ci [windows-cpu, windows-gpu]

retriggering.. windows

It's failing only for windows-cpu MKLDNN MKL part with the error
"can't find specified path"
[2020-03-27T18:52:40.264Z] "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsx86_amd64.bat" && cmake -G Ninja -DCMAKE_C_COMPILER=cl -DCMAKE_CXX_COMPILER=cl -DUSE_CUDA=OFF
[2020-03-27T18:52:40.264Z] -DUSE_CUDNN=OFF -DENABLE_CUDA_RTC=OFF -DUSE_OPENCV=ON -DOpenCV_RUNTIME=vc15 -DOpenCV_ARCH=x64 -DUSE_OPENMP=ON -DUSE_BLAS=mkl -DUSE_LAPACK=ON -DUSE_DIST_KVSTORE=OFF -DUSE_MKL_IF_AVAILABLE=ON
[2020-03-27T18:52:40.264Z] -DUSE_MKLDNN=ON -DCMAKE_BUILD_TYPE=Release C:\jenkins_slave\workspace\build-cpu-mkldnn-mkl

@larroy is this related to path not being on oneline? or the path itself is incorrect?

Turns out the path mentioned in ci/build_windows.py is correct
VS 2019': r'C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsx86_amd64.bat

That is exactly where the bat file is.
[verified this by launching an instance using the updated windows AMI]

WIN_GPU_MKLDNN is having an issue similar to https://github.com/pytorch/pytorch/issues/25393

@mxnet-bot run ci [windows-gpu]

@mxnet-bot run ci [unix-gpu]

@mxnet-bot run ci [windows-gpu]

@marcoabreu Even in this case, @vexilligera is unable to reproduce the error locally. And Windows GPU + MKLDNN is flaky. How do you suggest we proceed trying to resolve the thrust issue?
Fail : http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-gpu/detail/PR-17808/13/pipeline
Pass : http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-gpu/detail/PR-17808/12/pipeline

@vexilligera as discussed offline, lets try testing locally for WIN_GPU and WIN_GPU_MKLDNN build 10 times each (since 1 run takes 20-30mins) to come up with some basis... (ideally would have tried 100 times but given the resource & time constraints)

@vexilligera as discussed offline, lets try testing locally for WIN_GPU and WIN_GPU_MKLDNN build 10 times each (since 1 run takes 20-30mins) to come up with some basis... (ideally would have tried 100 times but given the resource & time constraints)

On my local test, the WIN_GPU_MKLDNN is much more flaky than WIN_GPU, as all WIN_GPU builds passed while about 1/3 of WIN_GPU_MKLDNN builds failed.
@haojin2 suggests us removing the WIN_GPU_MKLDNN test entirely since MKLDNN doesn't make much sense as we are running on GPU, and GPU_MKLDNN case is covered on other platforms.

We can get rid of WIN_GPU_MKLDNN tests altogether but that still leaves us with the flakiness of WIN_GPU as can be seen in these builds
For roughly same code of this PR & same windows AMI, below are the results so far

WIN_GPU
WIN_GPU_MKLDNN
Build Number
Link

✖
✔
15
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-gpu/detail/PR-17808/15/pipeline

✖
✔
14
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-gpu/detail/PR-17808/14/pipeline

✔
✔
12
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-gpu/detail/PR-17808/12/pipeline

✔
✖
13
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-gpu/detail/PR-17808/13/pipeline

Ofcourse your tests on local have a different story to tell...

@larroy do you have any idea on how to solve Win 126 on CI? Basically the binaries aren't in the search path and dlopen fails due to that

I'll have a look at the logs when I'm at my computer
Just to be clear about two things:

Retries are fine for now, but must be removed before merging.
It is not an option to remove GPU mkldnn. It does make sense for cases where an operator is not supported by the GPU or the user wants to run some parts on CPU. Just because it breaks, it doesn't mean that we can remove it to ease our situation.

Sorry but I don't believe that you are testing the development on a VM when this kind of commit is necessary.

Was testing on another instance and made commits on another...manually copied the code and missed some parts...

1. Retries are fine for now, but must be removed before merging.

With the CI blocked on windows-gpu for last 4 days and this PR trying to unblock the CI with an updated AMI, I guess it would be a good way to unblock the CI by allowing retries for now.
I have created issue : https://github.com/apache/incubator-mxnet/issues/17935 for tracking this problem.
But for unblocking CI do you think it makes sense to add the retries as a temporary hotfix?
@vexilligera tested the retries 10 times to see that adding retries safely ensures successful compilation all the time.

I printed the path to the missing dll and it was:
C:\jenkins_slave\workspace\ut-python-gpu\windows_package\lib\libmxnet.dll
However, the build log shows this is packed into windows_package as you can find here:
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-gpu/detail/PR-17808/19/
@larroy suggests packing the opencv binary as well but since opencv is already in the environment path of the AMI I guess the real problem should be something else (if opencv is missing and libmxnet.dll present you'll get segmentation fault instead, WinError 126 is triggered from Python due to wrong path)
@marcoabreu do you have any suggestion on this?

OSError: [WinError 126] The specified module could not be found
This one means the libmxnet.dll is not in the DLL Search path [acc to Stackoverflow]
Fix would be to add it into the environment
https://stackoverflow.com/questions/43987081/openslide-python-import-error
What's surprising is : libmxnet.dll & mxnet_70.dll are being packed into windows_package.7z and unpacked during test phase correctly.
While WIN_CPU tests don't give this error, WIN_GPU fails

@marcoabreu Also, after adding the retries, from build #18 to #22 all 5 have passed on WIN_GPU and WIN_GPU_MKLDNN compilation stage. That's a compeling reason for using retries to temporarily resolve the intermittent failure I feel.

Test in WIN_GPU fail at
OSError: [WinError 126] The specified module could not be found

This one means the libmxnet.dll is not in the DLL Search path [acc to Stackoverflow]
Fix would be to add it into the environment
https://stackoverflow.com/questions/43987081/openslide-python-import-error
What's surprising is : libmxnet.dll & mxnet_70.dll are being packed into windows_package.7z and unpacked during test phase correctly.
While WIN_CPU tests don't give this error, WIN_GPU fails

@larroy says he used to run into the problem and it turned out that it was solved by packing opencv binary. In this case the opencv is already in the environment path and it should be found. Although WinError 126 could also occur if a dependent dll is missing, in my local tests when I deliberately created that situation, instead, segmentation fault (access violation at 0x0000000000000000) is thrown.

The only dependent dll of libmxnet.dll is cudart64_102.dll (not including the win32 dlls). For mxnet_70.dll, the libraries of opencv and openblas are already in the ami's environment path.

There should be more dependent dlls? For example, if built with fusion support, the cuda driver should also be a dependent library? https://github.com/apache/incubator-mxnet/issues/17858
https://github.com/apache/incubator-mxnet/blob/56e79853ad5cf98baf84454eb595c7658bef6ee6/CMakeLists.txt#L613-L621

The only dependent dll of libmxnet.dll is cudart64_102.dll (not including the win32 dlls). For mxnet_70.dll, the libraries of opencv and openblas are already in the ami's environment path.

There should be more dependent dlls? For example, if built with fusion support, the cuda driver should also be a dependent library? #17858
https://github.com/apache/incubator-mxnet/blob/56e79853ad5cf98baf84454eb595c7658bef6ee6/CMakeLists.txt#L613-L621

They have fixed WinError 126 and now we are looking at a segfault

@leezu Thank you for checking the instance type and changing the cuda arch to 5.2, the ci is running now. @ChaiBapchya previously the segfault was caused by specifying 7.0 on the g3 instance.

@mxnet-bot run ci [windows-gpu]

@ChaiBapchya I added instance type auto-detection and found out c5 was running the GPU ci, which caused the segfault.

A missing dll might not mean that the dll itself is missing but that I can not be loaded to a miss in the dependency chain. I'd recommend to get the artifact and inspect it with some resources manager (similar to ldd). That should tell you which dependencies are expected and you can verify that they are available.
Maybe also starting with a debugger could help to see a more detailed exception which explains which dependency of the dll is missing.

@leezu seems like adding both 5.2 and 7.0 and then drop 7.0 would trigger a normal run...

@mxnet-bot run ci [windows-gpu]

@mxnet-bot run ci [windows-gpu]

Connectivity errors appear fixed now by switching to g4 (with faster cpu and network).
Removed the debug statements and rebased on latest master.

@marcoabreu Do you still stand by your veto?

"Retries are fine for now, but must be removed before merging."

I suggest to go ahead and merge this with the retries. We don't have control over the thrust + VS code incompatibility and may need to wait for a fix on nvidia's or microsoft's side. Doesn't seem worth to block all MXNet development until the bug in "thrust + VS code" is fixed, given it's an open issue affecting other projects as well: https://github.com/pytorch/pytorch/issues/25393

Connectivity errors were never an issue with Windows slaves. It sounds like you've changed something as part of generating the AMIs. Thus, I'm hesitant to just accept a bunch of more changes (which have not been made transparent) as a result to fix a previously created issue.
Please just revert stuff back to what it was and stop putting duckttape over things. The build used to work and the slaves used to be stable. But I have only seen attempts to change things but no rootcausing (I know about the 2GB limit). Some action resulted in our build breaking and the only sane choice is to revert whatever was done and then do these efforts again.
I will not accept multiple layers of fixes which could have been avoided if the changes were properly developed and tested in a separate environment. At the moment, it seems like open heart surgery on a productive system.

Discussion continued at https://github.com/apache/incubator-mxnet/pull/17962#issuecomment-608066411

Closing since this is now fixed in #17962

@vexilligera it's not fixed. We need a script to generate the working AMI. https://github.com/apache/incubator-mxnet/pull/17962 is only a hotfix.
Will you be working on the automated setup or is someone else taking over the work? Please clarify. Thank you


Virus?

It says that there's a virus when I try to download "RDPWInst-v1.6.2.msi". Any ideas why?

It says that there's a virus when I try to download "RDPWInst-v1.6.2.msi". Any ideas why?

What says that?

It says that there's a virus when I try to download "RDPWInst-v1.6.2.msi". Any ideas why?

What says that?

My browser, also a few virus protectors I scanned it with at different times (Malwarebytes/Windows Defender, etc) and I'm wondering why.

This is not a virus.
https://www.microsoft.com/en-us/wdsi/threats/malware-encyclopedia-description?Name=HackTool:Win32/RDPWrap&ThreatID=2147726445
This is "HackTool" (A tool for bypassing software limitations of the OS or third-party software.)

P. S.: DO NOT USE MSI, use ZIP and autoupdate or rdpwrup

So, it's good to know there aren't any malicious strings in the code, but how do I go about getting my Windows Defender to shut up about it?

how do I go about getting my Windows Defender to shut up about it?

Read defender documentation: https://support.microsoft.com/en-us/windows/add-an-exclusion-to-windows-security-811816c0-4dfd-af4a-47e4-c301afe13b26

how do I go about getting my Windows Defender to shut up about it?

Read defender documentation: https://support.microsoft.com/en-us/windows/add-an-exclusion-to-windows-security-811816c0-4dfd-af4a-47e4-c301afe13b26

Thank you! :)


Add support for the a110x Hall Effect Sensor

Add support for the a110x Hall Effect Sensor.  This was tested on the Grove Hall Effect Sensor.
Signed-off-by: Jon Trulson<EMAIL_ADDRESS>

Please ignore, I will redo.


[Cleanup] Use MessageBuilder's makeSystemMessage(...) in /debug-env

In https://github.com/Chatterino/chatterino2/blob/5ba3ef721f2a70cf12ad5937d523243126ca4e62/src/controllers/commands/CommandController.cpp#L458-L462 we can use makeSystemMessage(str) instead of building our own custom message
Originally added in #3727

makeSystemMessage wasn't used because it parses links while this has to not parse links, so it displays actual values

makeSystemMessage wasn't used because it parses links while this has to not parse links, so it displays actual values

Seems good :+1:


[TeamCity] Different settings for pull requests

When someone opens a PR I'd like to trigger different builds than when someone pushes to, lets say, master.

@ruudk unfortunately we are no longer accepting feature requests for github-services. I'm going to close out this issue.