Build fails on Windows 10

Description
Build fails on Windows 10 with error subprocess.CalledProcessError: Command [...] returned non-zero exit status 1. Similar to #582. Issue was supposedly solved in #475.
Console output and logs
Full console output available here.
Traceback (most recent call last):
  File "ungoogled_packaging\build.py", line 187, in <module>
    main()
  File "ungoogled_packaging\build.py", line 164, in main
    buildkit.patches.patch_paths_by_bundle(bundle), source_tree, patch_bin_path=None)
  File "C:\Users\marshmallow\Documents\GitHub\ungoogled-chromium\build\src\ungoogled_packaging\buildkit\patches.py", line 95, in apply_patches
    subprocess.run(cmd, check=True)
  File "C:\Users\marshmallow\Python 3.7.1\lib\subprocess.py", line 481, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['C:\\Users\\marshmallow\\Documents\\GitHub\\ungoogled-chromium\\build\\src\\third_party\\git\\usr\\bin\\patch.exe', '-p1', '--ignore-whitespace', '-i', 'C:\\Users\\marshmallow\\Documents\\GitHub\\ungoogled-chromium\\build\\src\\ungoogled_packaging\\patches\\ungoogled-chromium\\windows\\windows-fix-building-without-safebrowsing.patch', '-d', 'C:\\Users\\marshmallow\\Documents\\GitHub\\ungoogled-chromium\\build\\src', '--no-backup-if-mismatch', '--forward']' returned non-zero exit status 1.

Environment details

ungoogled-chromium version: 71.0.3578.98-2
Configuration bundle: windows
Packaging type: windows
OS and version for building: Windows 10 (64-bit)
Compiler toolchain: Visual Studio Community 2017 C++ tools

Installed sub-components:

Visual C++ core desktop features
Just-In-Time debugger
VC++ 2017 version 15.9 v14.16 latest v141 tools
C++ profiling tools
Windows 10 SDK (10.0.17763.0)
Visual C++ tools for CMake
Visual C++ ATL for x86 and x64
Test Adapter for Boost.Test
Test Adapter for Google Test
Visual C++ MFC for x86 and x64

Building and packaging steps

Install Visual Studio Community 2017 with the above sub-components

Enable Debugging Tools For Windows for Windows 10 SDK

Install Python 2.7.15, Python 3.7.1 and 7-Zip

Run the command pip install pypiwin32

Set PYTHON user environment variable pointing to Python 2.7.15

Run the commands below in ungoogled-chromium Git repository in cmd.exe as administrator:
mkdir build\src
py get_package.py windows build\src\ungoogled_packaging
cd build\src
py ungoogled_packaging\build.py

can someone create an OVA for the windows 64bit build of latest version 71.x and upload it?

@hydrogenpi It's failing

Update:  70.0.3538.110-3 fails differently:
>call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat" >nul

C:\Users\TCB13\Desktop\ungoogled-chromium-70.0.3538.110-3\build\src>"out\Default\gn.exe" "gen" "out\Default" "--fail-on-unused-args"
Traceback (most recent call last):
  File "C:/Users/TCB13/Desktop/ungoogled-chromium-70.0.3538.110-3/build/src/build/vs_toolchain.py", line 471, in <module>
    sys.exit(main())
  File "C:/Users/TCB13/Desktop/ungoogled-chromium-70.0.3538.110-3/build/src/build/vs_toolchain.py", line 467, in main
    return commands[sys.argv[1]](*sys.argv[2:])
  File "C:/Users/TCB13/Desktop/ungoogled-chromium-70.0.3538.110-3/build/src/build/vs_toolchain.py", line 442, in GetToolchainDir
    runtime_dll_dirs = SetEnvironmentAndGetRuntimeDllDirs()
  File "C:/Users/TCB13/Desktop/ungoogled-chromium-70.0.3538.110-3/build/src/build/vs_toolchain.py", line 45, in SetEnvironmentAndGetRuntimeDllDirs
    update_result = Update()
  File "C:/Users/TCB13/Desktop/ungoogled-chromium-70.0.3538.110-3/build/src/build/vs_toolchain.py", line 413, in Update
    subprocess.check_call(get_toolchain_args)
  File "C:\Python27\lib\subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:\\Python27\\python.exe', 'C:\\Users\\TCB13\\Desktop\\ungoogled-chromium-70.0.3538.110-3\\build\\src\\third_party\\depot_tools\\win_toolchain\\get_toolchain_if_necessary.py', '--output-json', 'C:\\Users\\TCB13\\Desktop\\ungoogled-chromium-70.0.3538.110-3\\build\\src\\build\\win_toolchain.json', '3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c']' returned non-zero exit status 1
ERROR at //build/config/win/visual_studio_version.gni:27:7: Script returned non-zero exit code.
      exec_script("../../vs_toolchain.py", [ "get_toolchain_dir" ], "scope")
      ^----------
Current dir: C:/Users/TCB13/Desktop/ungoogled-chromium-70.0.3538.110-3/build/src/out/Default/
Command: C:/Python27/python.exe C:/Users/TCB13/Desktop/ungoogled-chromium-70.0.3538.110-3/build/src/build/vs_toolchain.py get_toolchain_dir
Returned 1 and printed out:

Please follow the instructions at https://chromium.9oo91esource.qjz9zk/chromium/src/+/master/docs/windows_build_instructions.md

See //third_party/angle/gni/angle.gni:79:3: whence it was imported.
  import("//build/config/win/visual_studio_version.gni")
  ^----------------------------------------------------
See //third_party/angle/src/tests/BUILD.gn:5:1: whence it was imported.
import("../../gni/angle.gni")
^---------------------------
See //BUILD.gn:203:7: which caused the file to be included.
      "//third_party/angle/src/tests:angle_unittests",
      ^----------------------------------------------

C:\Users\TCB13\Desktop\ungoogled-chromium-70.0.3538.110-3\build\src>exit
Traceback (most recent call last):
  File "ungoogled_packaging\build.py", line 186, in <module>
    main()
  File "ungoogled_packaging\build.py", line 178, in main
    _run_build_process('out\\Default\\gn.exe', 'gen', 'out\\Default', '--fail-on-unused-args')
  File "ungoogled_packaging\build.py", line 65, in _run_build_process
    **kwargs)
  File "C:\Program Files\Python36\lib\subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '('cmd.exe', '/k')' returned non-zero exit status 1.

Apparently disabling the Safe Browsing patch seems to make the build work, according to this.

@arutr I'll try that solution, however if take a look at my log there are numerous patches that fail way before the one related to safe browsing.

@arutr Disabling that patches seem to work, but I get the following toolchain error:
C:\Users\TCB13\Desktop\ungoogled-chromium-71.0.3578.98-2\build\src>"out\Default\gn.exe" "gen" "out\Default" "--fail-on-unused-args"
Traceback (most recent call last):
  File "C:/Users/TCB13/Desktop/ungoogled-chromium-71.0.3578.98-2/build/src/build/vs_toolchain.py", line 468, in <module>    sys.exit(main())
  File "C:/Users/TCB13/Desktop/ungoogled-chromium-71.0.3578.98-2/build/src/build/vs_toolchain.py", line 464, in main
    return commands[sys.argv[1]](*sys.argv[2:])
  File "C:/Users/TCB13/Desktop/ungoogled-chromium-71.0.3578.98-2/build/src/build/vs_toolchain.py", line 439, in GetToolchainDir
    runtime_dll_dirs = SetEnvironmentAndGetRuntimeDllDirs()
  File "C:/Users/TCB13/Desktop/ungoogled-chromium-71.0.3578.98-2/build/src/build/vs_toolchain.py", line 42, in SetEnvironmentAndGetRuntimeDllDirs
    update_result = Update()
  File "C:/Users/TCB13/Desktop/ungoogled-chromium-71.0.3578.98-2/build/src/build/vs_toolchain.py", line 410, in Update
    subprocess.check_call(get_toolchain_args)
  File "C:\Python27\lib\subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:\\Python27\\python.exe', 'C:\\Users\\TCB13\\Desktop\\ungoogled-chromium-71.0.3578.98-2\\build\\src\\third_party\\depot_tools\\win_toolchain\\get_toolchain_if_necessary.py', '--output-json', 'C:\\Users\\TCB13\\Desktop\\ungoogled-chromium-71.0.3578.98-2\\build\\src\\build\\win_toolchain.json', '3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c']' returned non-zero exit status 1
ERROR at //build/config/win/visual_studio_version.gni:27:7: Script returned non-zero exit code.
      exec_script("../../vs_toolchain.py", [ "get_toolchain_dir" ], "scope")
      ^----------
Current dir: C:/Users/TCB13/Desktop/ungoogled-chromium-71.0.3578.98-2/build/src/out/Default/
Command: C:/Python27/python.exe C:/Users/TCB13/Desktop/ungoogled-chromium-71.0.3578.98-2/build/src/build/vs_toolchain.py get_toolchain_dir
Returned 1 and printed out:

Please follow the instructions at https://chromium.9oo91esource.qjz9zk/chromium/src/+/master/docs/windows_build_instructions.md

See //third_party/angle/gni/angle.gni:80:3: whence it was imported.
  import("//build/config/win/visual_studio_version.gni")
  ^----------------------------------------------------
See //third_party/angle/src/tests/BUILD.gn:5:1: whence it was imported.
import("../../gni/angle.gni")
^---------------------------
See //BUILD.gn:202:7: which caused the file to be included.
      "//third_party/angle/src/tests:angle_white_box_tests",
      ^----------------------------------------------------

C:\Users\TCB13\Desktop\ungoogled-chromium-71.0.3578.98-2\build\src>exit
Traceback (most recent call last):
  File "ungoogled_packaging\build.py", line 187, in <module>
    main()
  File "ungoogled_packaging\build.py", line 179, in main
    _run_build_process('out\\Default\\gn.exe', 'gen', 'out\\Default', '--fail-on-unused-args')
  File "ungoogled_packaging\build.py", line 65, in _run_build_process
    **kwargs)
  File "C:\Program Files\Python36\lib\subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '('cmd.exe', '/k')' returned non-zero exit status 1.

Update: I manage get around the errors above by reinstalling Python 3 and Python 2 and depot_tools from https://storage.googleapis.com/chrome-infra/depot_tools.zip (as explained here https://chromium.googlesource.com/chromium/src/+/64.0.3282.168/docs/windows_build_instructions.md#install).
After this I was able to proceed with the build until I hit...
In file included from ../..\third_party/skia/include/core/../private/SkOnce.h:13:
../..\third_party/skia/include/core/SkTypes.h(13,10):  fatal error: 'SkUserConfig.h' file not found
#include "SkUserConfig.h"
         ^~~~~~~~~~~~~~~~

Looks like I'm not properly disabling the safe browser, how am I supposed to do that?

4. Disable safe-browsing patches by:

Search in the folder C:\ungoogled-chromium-71.0.3578.98-2\patches for safe and delete all files related to safe search

you dont have to delete any files. Only edit the patch.list and gn_flags.map files

Run:

mkdir build\src
py get_package.py windows build\src\ungoogled_packaging
cd build\src

Search once again for safe inside build\src and also remove any files there
Finally build by running py ungoogled_packaging\build.py

Step 9. is not neccessary, because step 8. copies your changes in step 7. into build\src\ungoogled_packaging.
Here the steps for the build process of ungoogled-chromium 71.0.3578.98-2:

checkout tagged version 71.0.3578.98-2 of ungoogled-chromium

delete build/src folder completely (after prior build process)

edit like described for deleting all "safe-browsing" references:
config_bundles\windows\patch_order.list
config_bundles\common\patch_order.list
config_bundles\common\gn_flags.map

Run
mkdir build\src
py get_package.py windows build\src\ungoogled_packaging
cd build\src
py ungoogled_packaging\build.py
py ungoogled_packaging\package.py

thats all

@TN0X should I worry about this error?
[546/20133] ACTION //chrome/test/chromedriver:embed_version_in_cpp(//build/toolchain/win:win_clang_x64)
ERROR:root:Git error: rc=0, output='

@TN0X should I worry about this error?
[546/20133] ACTION //chrome/test/chromedriver:embed_version_in_cpp(//build/toolchain/win:win_clang_x64)
ERROR:root:Git error: rc=0, output='
``

No, I got the same error at this point.

@TN0X I finally managed to get a working build!! Only possible with your precious help!
I did a few things differently this way:

Used PowerShell instead of cmd.exe to compile it
I was downloading the tag as a zip from github, this time I did:

git clone https://github.com/Eloston/ungoogled-chromium.git
git checkout tags/71.0.3578.98-2

Not sure if there was some issue with that zip download or the path in my computer was way too big, because it used to be C:\ungoogled-chromium\71.0.3578.98-2 and this time, it was just C:\ungoogled-chromium.
Thank you very much for all your help!

safe-browsing patches needs to be updated for windows, i will see what i can do ;)

i guess this would be solved here https://github.com/Eloston/ungoogled-chromium/issues/619 later on
@Eloston

Closing per @intika's request.


Support starttls for client connections

Problem
If I want to implement an upstream protocol that supports a form of
start TLS I can't because unlike the downstream transport, their isn't
an upstream start-tls transport socket.
Solution
Add support for starttls on the upstream side. This is exactly the same
code as used for the server implementation, just reused.
Risk Level: Low
Testing:
Docs Changes:
Release Notes:
Platform Specific Features: None
Signed-off-by: Bryce Anderson<EMAIL_ADDRESS>

I doubt that the .proto changes are what we want, but I'd appreciate some pointers as to what the best way forward would be.

@lizan, I'd love your input wrt the api change.

This should be in a pretty good state, but I'm not sure why the CI jobs are failing. The logs look unrelated to my changes, but please let me know if that's not the case. :)

@lizan, @cpakulski, what do I need to do to get this moving forward?

@mattklein123, do you have any pointers on how to get at the failure logs from Azure? This passes for my locally and on a Ubuntu 20.04 instance in AWS. I've been having trouble with docker on osx due to this bug or I'd try the provided containers.

@mattklein123, do you have any pointers on how to get at the failure logs from Azure? This passes for my locally and on a Ubuntu 20.04 instance in AWS. I've been having trouble with docker on osx due to this bug or I'd try the provided containers.

AFAIK you should be able to click on details and drill into the logs? I don't think you need special permissions? If that's not the case can you reach out on Slack?

@cpakulski, @lizan, can I get another look at this?

@bryce-anderson Can you merge main to see if it fixes failing CI tests?

/retest

@cpakulski, thanks for the review. It looks like most of the CI jobs are taking a lot longer with this patch than other recent runs. I'm not familiar with the Azure pipeline structure in this case, but could it be due to the base of this patch and maybe a squash/rebase would help?

I do not know why macos CI always fails here. I did a PR yesterday and it passed all checks.

/retest

Can you merge main to your branch?

Done.

The MacOS build passed this time but it looks like an unrelated test failed on windows.

@lizan @cpakulski, looks like CI's finally all green. Anything left for me to do?


When will release 0.32 will be ready?

Do you have any ETA?

Based on that you are asking...you have an idea when would be a good time to release ? ;-)

today? :-)

Ok not really today...but i can see if can make it over the weekend...Ok too ?

10x a lot!

So done a release in Central. Should be available within a few hours (synch takes some time)...

Hello! Has this release been tested? Thank you!

So release is out...so we can close this.


Running mad_icp with a folder of pcds

Hello,
Are you planning to release a version of mad_icp that includes the <pcd_folder/> option as input, like for kiss_icp standalone?
Thank you.

Hi @Efesendil ! At the moment we do not have plans to release a PCD reader. However, if you manage to implement such compatibility, we welcome you to open a pull request. Alternatively, I suggest converting your PCD files into one of the supported formats.


Makefile: add ability to use custom go bin

Now there's no need to switch go bin via ln -s.
$ go install golang.org/dl/gotip@latest
$ gotip download

$ make init GO=gotip

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

I'm afraid that's not enough. For example, we also use go there: https://github.com/FerretDB/FerretDB/blob/c52ea450072bd3453f27146c49a6869a8a7c25fb/tools/tools.go#L30-L36
I would rather do nothing until 1.18 release in Februrary, when all this complexity will go away.

Ah, I see. Ok, let's just close it.


Extensively Test HashStore for 1.0.0

To prepare for production release 1.0.0, we need to ensure that our Public API methods have been tested extensively, especially with regards to the object and sysmeta locks.
To Do:

Test with real datasets
Explore multiple instances of HashStore attempting to store the same duplicate objects and sysmeta concurrently
Consider other race conditions and add new pytests where applicable

Update:

Stored all objects from var/metacat/data in knbvm (test.arcticdata.io) to a HashStore: /home/mok/testing/knbvm_hashstore via Python Client app.

Overview:

Total Objects Possible to Convert: 585371
Total Objects Stored: 582898
Objects unaccounted for (no pids found): 2473

Retrieved objects via Java Client app:

579770 checksums match
3128 do not match, it appears that the checksums stored are incorrect

Stored all metadata from var/metacat/documents in knbvm (test.arcticdata.io) to a HashStore: /home/mok/testing/knbvm_hashstore via Python Client app.

Overview:

Total Metadata Possible to Convert: 34915
Total Metadata Stored: 34915

Storage/retrieval based on values obtained by accessing the identifier and systemmetadata tables in metacat postgres db

Update:

Using values obtained by querying metacat postgres db:

Stored list of objects (~580k) via Java and retrieved with Python
Stored list of objects (~580k) via Python and retrieved with Java
Deleted list of objects via Python
Stored, retrieved and deleted list of metadata via Python

Next Up:

Test for sensible responses when attempting to store the same pid with asynchronous parallel writes
Test deleting objects via Java
Test re-storing, retrieving and deleting metadata via Java

Summary/Update:

Tested deleting objects via Java
Tested storing, retrieving and deleting metadata via Java & Python
Confirming that we can store, retrieve and delete objects using both Java and Python interchangeably.

The testing process uncovered a few latent bugs (that were fixed) and provided peace of mind that the code base is robust. There were no exceptions encountered that I didn't expect to see.
Storing data objects via the metacat db with both Java/Python produced the same number of files stored (and not stored due to the same reason, which was that the checksum found in the db does not match what has been calculated). The mismatch is not due to a calculation error, but likely an input error from test data in knbvm.
Data objects were also retrieved and validated (calculating the checksum from the stream and comparing against the db). Data objects were also all deleted as expected.
Storing metadata via the metacat db with both Java/Python produced the same number of metadata documents stored. Metadata objects were retrieved, validated and deleted without any concerns.

Up Next:

Create race conditions to test the locking (and not storing duplicates unintentionally). I will report back here with my logic for review with the backend team.

Testing Strategy for Review #1:
Python:

Multiprocessing Library Test

Using multiprocessing, I can create separate processes that do not have any shared resources/variables.
This type of test means that the pid lock would be ineffective and all processes created would attempt to store the same object.
Race Condition: Multiple processes all attempting to store different objects but with the same pid:

Multiprocessing.pool() receives a list with 20 integers, either 1 or 2. A maximum of 10 processes execute at any one time.
This list gets unpacked as individual arguments and are passed into a method which will either call store_object on a large file (5GB) or smaller file (500MB) based on the integer with the same pid.
Since the pid lock is bypassed due to separate memory space from using processes, one process will be the first to create a (large/small) temporary file and attempt to move it to its final location.
Results:

All 10 processes immediately begin to create a temporary file after moving past the pid lock
One process completes before the other for the smaller tmp file (500MB), and moves the file to its permanent location
The processes creating the smaller tmp file (500mb) that finish creating their temporary file after detects that the object already exists at the permanent location, and deletes its respective temporary file.
The processes that get spawned afterwards after the pool opens up can see that a file already exists, and exits immediately.
The processes that finish creating their temporary file late (because they are copying a large file (5GB)), see that a file already exists, and deletes its respective temporary file.

Conlusion:

The behaviour seems reasonable, and the results are reproducible. When the list provided to store triggers the same size, the same behaviour is seen.
Objects should always be stored with a checksum and checksum algorithm to be completely sure that the correct file is stored.

Storing 1 type object. 1 = 5GB, 2 = 500MB
Storing 1 type object. 1 = 5GB, 2 = 500MB
Storing 2 type object. 1 = 5GB, 2 = 500MB
Storing 2 type object. 1 = 5GB, 2 = 500MB
Storing 1 type object. 1 = 5GB, 2 = 500MB
Storing 2 type object. 1 = 5GB, 2 = 500MB
Storing 1 type object. 1 = 5GB, 2 = 500MB
Storing 2 type object. 1 = 5GB, 2 = 500MB
Storing 1 type object. 1 = 5GB, 2 = 500MB
Storing 2 type object. 1 = 5GB, 2 = 500MB
Storing 1 type object. 1 = 5GB, 2 = 500MB
ERROR:root:FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Storing 2 type object. 1 = 5GB, 2 = 500MB
WARNING:root:FileHashStore - _move_and_get_checksums: Object exists at: /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b, deleting temporary file.
Storing 1 type object. 1 = 5GB, 2 = 500MB
ERROR:root:FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Storing 2 type object. 1 = 5GB, 2 = 500MB
WARNING:root:FileHashStore - _move_and_get_checksums: Object exists at: /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b, deleting temporary file.
Storing 1 type object. 1 = 5GB, 2 = 500MB
ERROR:root:FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Storing 2 type object. 1 = 5GB, 2 = 500MB
WARNING:root:FileHashStore - _move_and_get_checksums: Object exists at: /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b, deleting temporary file.
Storing 1 type object. 1 = 5GB, 2 = 500MB
ERROR:root:FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Storing 2 type object. 1 = 5GB, 2 = 500MB
WARNING:root:FileHashStore - _move_and_get_checksums: Object exists at: /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b, deleting temporary file.
Storing 1 type object. 1 = 5GB, 2 = 500MB
ERROR:root:FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Storing 2 type object. 1 = 5GB, 2 = 500MB
ERROR:root:FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
ERROR:root:FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
ERROR:root:FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
ERROR:root:FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
ERROR:root:FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
WARNING:root:FileHashStore - _move_and_get_checksums: Object exists at: /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b, deleting temporary file.
WARNING:root:FileHashStore - _move_and_get_checksums: Object exists at: /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b, deleting temporary file.
WARNING:root:FileHashStore - _move_and_get_checksums: Object exists at: /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b, deleting temporary file.
WARNING:root:FileHashStore - _move_and_get_checksums: Object exists at: /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b, deleting temporary file.
WARNING:root:FileHashStore - _move_and_get_checksums: Object exists at: /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b, deleting temporary file.

Testing Strategy for Review 2:
Python:

Threading Library Test

With threads, resources are shared so creating multiple threads will interact with the pid lock as expected.
Race Condition: Multiple threads all attempting to store the same object with the same pid

Using a loop, I create 20 threads that all call store_object on a large file (5GB)
Once the first thread is executed, the pid is locked and all threads must wait

Only one temporary file is generated, and once completed, is moved to its permanent address

The pid is released, and every other thread sees that there is already a file for the given pid and raises a FileExistsError exception.
Results:

The pid lock behaves as I expect it to, and I can confirm that they are queued up as there is only one tmp file generated in the /tmp folder while it serializes the data.
Increasing the number of threads produces the same results. Testing with a 500MB file and the same pid also behaves as I expect it to.

Thread 0 Started
Exception in thread Thread-1:
...
FileExistsError: FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Thread 1 Started
Exception in thread Thread-2:
Traceback (most recent call last):
...
FileExistsError: FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Thread 2 Started
Exception in thread Thread-3:
Traceback (most recent call last):
...
FileExistsError: FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Thread 3 Started
Exception in thread Thread-4:
Traceback (most recent call last):
...
FileExistsError: FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Thread 4 Started
Exception in thread Thread-5:
Traceback (most recent call last):
...
FileExistsError: FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Thread 5 Started
Exception in thread Thread-6:
Traceback (most recent call last):
...
FileExistsError: FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Thread 6 Started
Exception in thread Thread-7:
Traceback (most recent call last):
 ...
FileExistsError: FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Thread 7 Started
Exception in thread Thread-8:
Traceback (most recent call last):
...
FileExistsError: FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Thread 8 Started
Exception in thread Thread-9:
Traceback (most recent call last):
...
FileExistsError: FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Thread 9 Started
Exception in thread Thread-10:
Traceback (most recent call last):
...
FileExistsError: FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Thread 10 Started
Exception in thread Thread-11:
...
FileExistsError: FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Thread 11 Started
Exception in thread Thread-12:
Traceback (most recent call last):
...
FileExistsError: FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Thread 12 Started
Exception in thread Thread-13:
Traceback (most recent call last):
...
FileExistsError: FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Thread 13 Started
Exception in thread Thread-14:
Traceback (most recent call last):
...
FileExistsError: FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Thread 14 Started
Exception in thread Thread-15:
Traceback (most recent call last):
...
FileExistsError: FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Thread 15 Started
Exception in thread Thread-16:
Traceback (most recent call last):
...
FileExistsError: FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Thread 16 Started
Exception in thread Thread-17:
Traceback (most recent call last):
...
FileExistsError: FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Thread 17 Started
Exception in thread Thread-18:
Traceback (most recent call last):
...
FileExistsError: FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Thread 18 Started
Exception in thread Thread-19:
Traceback (most recent call last):
...
FileExistsError: FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b
Thread 19 Started
Exception in thread Thread-20:
Traceback (most recent call last):
...
FileExistsError: FileHashStore - _move_and_get_checksums: File already exists for pid: doutest1 at /Users/doumok/Code/hashstore/douhs/objects/29/25/40/b0f6f28d760c2660f3ac580911504eb6e06d3ef104d323e148b117595b

Testing Strategy for Review 3:
Python:

Multiprocessing with threading test

Race Condition: Simulate a scenario where multiple processes (separate memory, no shared resources) attempts to store the same pid but different objects with multiple threads (shared resources)

5 processes that each calls store object 10 times via threading, storing both large (5GB) and small (500MB) data file
The first process to store the object successfully is the 500MB file, and causes all remaining threads to raise the appropriate FileExistsError exception
The other 4 processes attempting to store the small object delete their temporary file and raises FileExistsError exception on every thread
The remaining 5 processes attempting to store the large object once completing the creation of the tmp file see that there is already a file at the permanent address. They delete their respective tmp files, and the remaining threads raise FileExistsError exceptions as expected.
Results:

The expected exception is raised each time, and the pid lock does not throw any unexpected errors.
The behaviour is reproducible with an increased amount of processes and threads.

Conclusion:

HashStore (python) appears to be working as expected, and throws FileExistsError exception at every point.
The pid lock for threading behaves as expected, and blocks other threads while store_object is executing.
If there are no shared resources, the first process that moves its file to the permanent address blocks all others - so it's important to store files with their checksums to be 100% safe that the correct file is stored.

Testing Update:

Strategy 1 Follow-up: What happens when we store large files with duplicate pids and different sizes?

Objects stored are validated and not being overwritten
Exceptions raised are for the correct reason (object already exists) and the temporary files are deleted.
Other processes are able to move forward to store their respective objects
Cancelling store_object causes tmp files generated in objects/tmp that have not finished to remain in an incomplete state (and would need to be deleted manually)

Strategy 2 Follow-up: What happens to other threads while a pid is locked? Are they able to move forward with storing?

Pids that are not being stored are able to move past the pid lock
Objects are validated and stored, and the expected exception is raised for a duplicate object
The original test was not actually very accurate (threads created through a for loop), so the test was amended to manually start() and join() threads that call store_object.

Manually increasing the number of Threads also resulted in sensible responses

Cancelling midway through the process left temporary files in an incomplete state

Strategy 3 Follow-up: Store lots of different pids with multiprocessing and threading

Up to 10 processes were created to call the threading test method which manually started/joined threads (which stored 4 different files: 500MB, 1GB, 3GB, 3.5GB, using the same pids for each respective file)
Once files were completed, validated and moved to their final destination, the remaining threads exited as expected and tmp files created were all deleted as expected.

Strategy 4 Follow-up: Open multiple terminals to simulate the same results from the combined multiprocessing/threading test in Python but with Java

The pid synchronization to store the same objects appears to be working as expected
store_object is called with a checksum and a checksum algorithm, and the object that I expect to be stored is stored.
Exceptions are raised, multiple temporary files are all created and deleted when the JVM sees that an object already exists at the target location
Objects with different pids are able to move forward while the rest are in progress.
Staggering the commands does not cause any unexpected issues

Strategy 5: Test deleting objects, what happens when we cancel the command halfway through?

Python:

A multiprocessing pool is created, which is fed a list of different and repeated pids of different sizes and calls delete_object on each item in the list
When the file or parent folders no longer exists, the expected exception is raised.
Cancelling right after executing the command (to interrupt the deletion process) is proving to be difficult, the deletion happens almost instantly. I think it's safe to say that delete_object is working as expected.

Java:

Similarly, a list of repeated pids of different sizes are called upon by .parallelStream() to delete and cancelling right after is also proving to be difficult.
Executing the command on multiple terminals is also producing sensible responses (file does not exist). I think it is also safe to say that deleteObject in Java is also working as expected.

That all sounds awesome, @doulikecookiedough !

Cancelling midway through the process left temporary files in an incomplete state

Is there anything we can do about this?

@mbjones What do you think about using the last modified date to determine whether a tmp file is in progress or not? So as a last step during the instantiation/initialization of a HashStore, we will get the list of blobs in the /objects/tmp directory, and if the last modified date of any blob is past 30 days from the current date, it will be deleted. A bit of overhead, but we can automate the clean up of /objects/tmp and prevent it from building up and taking up disk space.

Maybe that would work, but it seems a bit late as a means of detection of a problem. Would be better to have a more proactive way of assessing whether the connection that was transferring a data object was lost, and therefore the temp file will never be completed? Maybe you could discuss architecturally with @taojing2002 and @artntek how we can reliably determine if a data transfer process was severed with the client, and then a proactive approach to cleaning temp when that happens.

Sounds good and will do!

I have added code to ensure that temporary files are cleaned up (removed from /objects/tmp) and executed the testing strategies described above. HashStore-Java appears to be working smoothly, but Python is not behaving consistently with a forceful termination.
Java:

Whether I let the JVM finish on its own or forcefully terminate the process in the console (Ctrl + C) for the above testing strategies, redundant temporary files were deleted.

Python:

Allowing the process to complete in all strategies will remove the temporary files as expected.
However, a KeyboardInterrupt (forceful termination) can at times leave temp files behind.

It inconsistently leaves behind 2-10 files, based on the amount of processes run, and is not easily reproduce-able. Most of the time, all temporary files are removed/cleaned up.
This looks to be an expected/observed issue with Python (unpredictable outcomes with forced termination), as atexit and finally blocks are not always executed in fatal situations.

Specifically catching and handling KeyboardInterrupt exceptions did not change the results (and feels like it left more temporary files).

If the cleanup process must be absolute, it looks like a "monitor" or "watchdog" would be necessary to keep on top of processes. Managing and facilitating processes feels a bit out of scope of this library - so I am opting to skip exploration here unless we want to make it a requirement.

Summary:

The cleanup process is much more reliable in Java than in Python. Overall, the Public API calls in both Java/Python are behaving in an expected manner.

Good summary.
For the Java process, if you do a kill -9 on the command line for the JVM, do you still see clean removal of temp files? I'd be surprised if you do, as a KILL signal is much more immediate than a SIGHUP.  In both that case and the python case, it seems like the only alternative would be to properly handle cleanup as well on startup after a terminating event like this.
A couple of other things that you might want to carefully check:

What happens if the storage system disk hits 100% full in the middle of a store request?
what happens if the storage mount is removed in the middle of an operation (can simulate by forcefully unmounting a temporary filesystem.
what happens if, in a kubernetes deployment, the process is moved from one node to another on a new pod?

we can talk about how to test this one with @artntek -- it should be fairly easy to setup a temporary image to test this

Thank you for the feedback @mbjones! You're right, when I do a kill -9, the temp files are not cleaned up. I'm currently in the process of answering the three questions raised and will update this issue once I've finished testing.

Following up on points 1) and 2) raised above with a testing on a new image/volume that I mounted to my machine with a max capacity of 5GB:
HashStore-Java:

When calling store_object to this volume that does not have enough space for additional objects, the exit/termination code & JVM works as expected and deletes the temporary files.

Every thread thereafter also encounters the same IOException (No space left on device)

When forcefully unmounting the volume (sudo diskutil unmount force /Volumes/5GB) while store_object is in progress, the temporary files do not get removed and are left behind.

HashStore-Python:

When calling store_object to this volume that does not have enough space for additional objects, the exit/termination code executes and deletes the temporary files

OSError (No space left on device) is encountered for both threads and processes
All temporary files are cleaned up if Python Interpreter is not interrupted

When forcefully unmounting the volume:

Threads:

PermissionError: [Errno 13] Permission denied: '/Volumes/5GB' errors are encountered and temp files are left behind.

Processes:

OSErrors 'Input/output error' are thrown followed by PermissionErrors. Temp files are left behind as well.

I will update this issue regarding Point #3 (what happens in kubernetes deployment) when I've made progress.

Update:

Testing with the latest storeObject refactor is ongoing in knbvm in both Java and Python libraries.
HashStore-java

Successfully stored the first 100000 records from knbvm's metacat pgdb where the object actually exists in var/metacat/data.

However, when an object already exists when calling storeObject when a pid is available, I am currently rejecting the request rather than creating the pid reference file and updating the cid reference file. Need confirmation of the behaviour in this scenario.

Currently, all objects that exist in var/metacat/data are being stored

HashStore (python) will be used to test the delete process (objects and reference files) once the Java process is completed to store all objects.

Afterwards, Python will be used to store all objects and Java will be used to delete objects.

TODO: Execute the previous tests to confirm behaviour during race conditions

Update (after refactors to delete_object and clarifications to processes)

HashStore (python)

Successfully stored/deleted all data objects in knbvm that are <1gb
There are several data objects with multiple pids, updating and removing pids from the cid refs file is working as intended
Both thread and multiprocessing locks appear to be working as expected, and is executing efficiently with the usage of Conditions to help with synchronizing the process

HashStore-java

Successfully stored/deleted all data objects in knbvm that are <1gb, reference files are updating as intended
When called to store 50 pids for 1 data object, then to delete the same 50 pids after - while another process adds another 50 pids to the same data object - the cid reference file is updated correctly with the right amount of values/pids
Parallelization with ExecutorService works well when running a Runnable class with a for loop as well as using the existing built in .parallelStream.forEach() on a list.

TODO:

Test storing/deleting metadata in both libraries after new updates
Store with java, delete with python library & vice versa

Update:

Storing/deleting objects & metadata with the python & the java library is behaving as expected
Cross-language testing looks good with storing/deleting objects & metadata
Testing continues as HashStore is integrated into Metacat

HashStore and HashStore-java has been and continues to be tested - both libraries are on their way to their respective releases. Closing this issue and new issues will be created as bugs are discovered


Dependency Required

com.strumenta.langmodules.sas-parser
ast
${project.version}

How can I get this dependency and run the sample?

Hello @luckyxu , you contacted us also by email, and we continued the discussion there, however I will reply also here in case someone ready this issue in the future.
This particular project demonstrates the capabilities of our SAS Parser. However the SAS Parser itself is a commercial product, so it is available only for those who acquire a license from us.


feat: Support for wasmd >=0.29 by generating files via buf

Since wasmd >= 0.29 started using buf, it broke proto-build. This PR fixes it by instead generating the files via buf. Needless to say this changes requires users to have buf installed. See here for more context: https://github.com/cosmos/cosmos-rust/issues/318#issuecomment-1443670490

@apollo-sturdy can you rebase? that should take care of the MSRV issues

@apollo-sturdy can you rebase? that should take care of the MSRV issues

Done!

Looks like cosmrs needs to be updated due to the schema changes.
@apollo-sturdy not sure if you want to take that on or not. If not, I can

Looks like cosmrs needs to be updated due to the schema changes.
@apollo-sturdy not sure if you want to take that on or not. If not, I can

Would be great if you could do it. I'm not too familiar with it or the main Cosmos SDK types as I mostly work with CosmWasm.

Merged in #358


Added a 'grep' command.

Added a 'grep' command to the basic linux command list.

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] My article does not contain shortened URLs or affiliate links.

If your pull request closes a GitHub issue, replace the XXXXX below with the issue number.
Closes #XXXXX

One of the most important and basic command that everyone should learn how to use is certainly 'grep' command. I have added it to the list.

Thank you for your PR @SimasJan, but this is quite confusing can you add an explanation of what your example commands do? And perhaps some line breaks? This is what your changes look like - https://github.com/freeCodeCamp/freeCodeCamp/blob/fe05b1fffa792eb482aa8facf692a6cc5e841a6a/guide/english/linux/basic-linux-commands/index.md
Thanks, and happy coding!

Looks good to me, and congratulations on your first Pull Request!  🎉🎉🎉
We look forward to future contributions!


Chrome cannot Download Files when Volumes are mounted

So I need to refer to a file in the created selenoid containers so I used the following in browser.json:
"volumes":["/home:/home/selenium:[:rw]"]
The problem when Chrome attempts to download a file, it gives insufficient permission. Is there something wrong with volume permission?

No this is related to get the download file name, When chrome attempts to download a File, It failed because of insufficient permission shared /home Volume with the host. When I tried to change the mount to /tmp/new it works successfully


Software citation

Regarding the last issue mentioned in https://github.com/tgkolda/siam-latex/blob/master/requirements.md
@misc{clawpack,
  title={Clawpack software},
  author={{Clawpack Development Team}},
  url={http://www.clawpack.org},
  note={Version x.y},
  year={2013}}

with two sets of braces around the "author" results in "Clawpack Development Team" in the bibliography (alphabetized at C) rather than "C. D. Team".
More generally -- how much guidance do we want to give on how to cite software?  Some packages explain how to cite in their documentation, many don't.

It would be nice to have an example of citing software, just to show some best practices. For instance, you probably want the version to be in the title and not as a note, because otherwise the version is at the very end of the citation, after the year.

This has been added in the last commit, though I forgot to explicitly cite this issue.


cleanup useless param of InitDebug

Please provide a description of this PR:

@hzxuzhonghu: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-telemetry-mc_istio
58337e8e64288057f39957656c7540306694c6e9
link
true
/test integ-telemetry-mc

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


DTQ - B-bit binary array

From what I understood, both from the paper and the code, the concept here is to train a neural network through triplets to produce a B-bit array binary output that will allow an effective yet fast search image retrieval search. More so, I assume that when you state "compact binary codes" you are mentioning to b and, implicitly, to the "codes" of each image has in the code of this project.
However, when you try 8 bits, 16 bits, 32 bits on your benchmark, which of the args are you changing on the training script? the subspace? or the product of the subspace and the subcenter? I say this because in your code a default value is
parser.add_argument('--subspace', default=4, type=int)
parser.add_argument('--subcenter', default=256, type=int)
In this case, is it 1024 bits?

256 centers can be indexed with 8bit, so for 4 spaces we need 8x4=32bit, where each 8bit indicates a center from the appropriate space.

Thanks @hbellafkir
I am closing this issue due to a long time of inactivity, feel free to reopen it if there is any question. Thanks!


Own certs does not seems to work when placed in mnt/docker/nginx/certs/live/mail.domain.tld

Hi there,
I have a wildcard cert purchased via positive SSL. As per the doc, we must place the certs in the mnt/docker/nginx/certs/live/mail.domain.tld directory; however when I place, the PEM files (buddle+key) in this particular directory, the mail server ignores it and use a self-signed cert instead. When configuring NGINX, I'm able to point it to the directory and it works fine.
Is anyone experiencing this issue?
Let me know.
Thanks

Can you provide some logs and the content of your docker-compose.yml file ?
Logs :
docker logs mailserver

TLS handshake :
openssl s_client -connect mail.domain.tld:993 -tlsextdebug
openssl s_client -connect mail.domain.tld:587 -starttls smtp -tlsextdebug

Content of the live directory :
ls /mnt/docker/nginx/certs/live/mail.domain.tld

FQDN value inside the container :
docker-compose exec mailserver bash
# hostname --fqdn

Hello there,
See mailserver log file:
mailserver_logs.txt
See ls of live directory

hostname --fqnd : mail.mydomain.com

Can you provide your docker-composer.yml file ?
And check the content of the live directory inside the container :
docker exec -ti mailserver bash
> ls /etc/letsencrypt/live/mail.domain.tld

docker-compose.yml.txt

Your custom certificat seems recognized correctly :
# mailserver_logs.txt

2017-05-07 17:09 :
[INFO] Let's encrypt live directory found
[INFO] Using /etc/letsencrypt/live/mail.mydomain.com folder
[INFO] Found DKIM key pair for domain mydomain.com - skip creation
[INFO] No extra postfix settings loaded because optional custom configuration file (/var/mail/postfix/custom.conf) is not provided.
[INFO] Database hostname not found in /etc/hosts, try to find container IP with docker embedded DNS server
[INFO] Container IP found, adding new record in /etc/hosts
[INFO] Fetchmail forwarding is disabled.
[INFO] ManageSieve protocol is enabled.

Try openssl client command to confirm the certificate used :
openssl s_client -connect mail.domain.tld:993 -tlsextdebug
openssl s_client -connect mail.domain.tld:587 -starttls smtp -tlsextdebug

You must wait until the container is fully initialized before executing these commands. Wait for "clamd[xxx]: Self checking every 3600 seconds." message.

when I do openssl s_client -connect mail.mydomain.com:993 -tlsextdebug and
openssl s_client -connect mail.mydomain.com:587 -starttls smtp -tlsextdebug
I get  connect: Connection refused connect:errno=111

netstat -tulpn | grep -E -w '993|587'

Here is the output:

Ok. Here is the scoop on why the private SSL issues:
I deleted everything and removed everything including volumes. What I did instead, placed all my certs in a directory on the host, then use my docker-compose to orchestrate another container (FROM tianon/true ADD certs ) to push the certs  into the /etc/letsencrypt directory at the end of the orchestration (depends_on: mailserver, mariadb...). It worked! now I can use my purchased SSL. Otherwise, when your first create the mailserver, it does not have access to the certs, I had to manually place the certs in the directory, and then restart the mailserver, and sometimes it was able to found the letsencrypt directory and other times not.
Let me know if this makes sense. Thank You for all your help!

So the problem is on your side ? Can i close this issue ?

The problem is not on my side. I just figure out a work around.  If you want to close this issue go ahead, but there is an issue with the current implementation of purchased SSL. That's all!


PHILOV - FORUM

[ ] Check if everything works fine.

[x] posting/commenting with photos
[ ] See if the links are opening (websits)
[x] public profile view....

[ ] Share link (dynamic link)

Bug
When opening a post and scrolling to the bottom of the comment section, especially the one with many comments, the screen looks like it is shaking up and down and scrolling indefinitely causing to get data over and over
Confirmed Devices

iPhone 14 Pro
iPhone 14 Pro Max


Update create-and-edit-legacy-transactional-templates.md

Description of the change:
Reason for the change:
Link to original source:

Closes #

Hello @Whatthefoxsays,Thanks again for the PR! We appreciate your contribution and look forward to                                              continued collaboration. Thanks!                                             Team SendGrid DX


Add Proper Winget Version

Problem:
Current document has an underspecified requirement for Winget. It says: "xxx or later?"
That is super confusing for any user.
Fix:
Add a specific working version

@mdanish-kh do you know who should approve this PR to get it properly merged? It has been out for 2 weeks already. Thanks.

I believe that would be @mattwojo looking at past WinGet PRs

Apologies for my delay on this and thanks for the ping. Merged!


Update storage-network-security.md

Normally the scenario here is that you want to add the outbound addresses of the other service (Web Apps being the most common) to the whitelist on the Storage account. I don't think the inbound address of the other service is very relevant, since traffic won't originate from Storage to that service.
Let me know if I'm somehow misunderstanding. We have to go over this issue on an almost weekly basis in support and it would be great to have clear documentation. There may be other wordings/examples to convey the issue more clearly here, just let me know if we want to try something different.

@dcbrown16 : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@santoshc1 - FYI
@dcbrown16 - that sounds right to me.
#sign-off


Bump to 0.2.16

@alexcrichton: no appropriate reviewer found, use r? to override

@bors: r+

:pushpin: Commit 351df2d has been approved by alexcrichton

:hourglass: Testing commit 351df2d with merge ecf4845...

:broken_heart: Test failed - travis

@bors: retry
On Thu, Sep 8, 2016 at 5:10 PM, bors<EMAIL_ADDRESS>wrote:

💔 Test failed - travis
https://travis-ci.org/rust-lang/libc/builds/158589551
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rust-lang/libc/pull/386#issuecomment-245781185, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAD95BjoJAEZrFvFxEaZlYvwbZ9O9KWxks5qoKP0gaJpZM4J4f-3
.

:hourglass: Testing commit 351df2d with merge b987f0c...

:sunny: Test successful - status-appveyor, travis
Approved by: alexcrichton
Pushing b987f0ccce1d26ceb0de9a982939c97a406531b2 to master...


[INTERNAL] Introduce build task replaceBuildtime

Introduce the build task 'replaceBuildtime' used in
the LibraryBuilder. It replaces the placeholder
${buildtime} with a current timestamp in the
format "YYYYMMDD-HHmm".
JIRA: CPOUI5FOUNDATION-327

Coverage increased (+0.009%) to 94.705% when pulling e3675fba3ab5a165321fb90fa2c0faeb43d4cfa1 on replaceBuildtime into 155dd943f4dbe207e557685a5487937bed051c36 on master.

Looking good! But I think this should be a [FEATURE] commit. Also, please mention in the commit message that the placeholder is only replaced in Global.js by default.
Also, an integration test would be good to make sure that the "Global.js"-pattern actually works in a real library. I think we can add another test in test/lib/builder/builder.js.

The integration test should also ensure that Global-dbg.js contains the replacement as well!


Remove error when empty for multi[Remove|Set]

Fixes #6 by just invoking the callback without the error.

:tada: This PR is included in version 1.1.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Is this project abandoned?

Long time no change!

No, it's not. I'm just extremely busy, working at a startup. I guess I'll have a time window within 2 months to continue.


unexpected T_FUNCTION

Hi,
After including the autoloader php file, I've got an error inside at line 2: syntax error, unexpected T_FUNCTION, expecting ')'
Thanks

Sorry, found using an old php version.


#20818 Close previous console created by the same include command on repeated execution

https://github.com/dbeaver/dbeaver/assets/28875055/69e7f256-569b-44a2-9fdd-cc3e702514d6

verified


How set service url for spawn a Microsoft Teams bot

Hi,
I’m trying to create a bot for microsoft team and I can't receive any message on it.
In the documentation it's written: "When spawning a bot for Microsoft Teams, you must pass in a serviceUrl field as part of the options parameter. The serviceUrl can be extracted from the incoming message payload at message.serviceUrl." but how do I get a message if the bot hasn’t spawned yet?
I tried to spawn the bot with a ngrok url to get the message.serviceUrl but I don't receive message anyway.
The bot work when I use botkit.botframeworkbot process.
var controller = botkit.teamsbot({
 clientId: process.env.MICROSOFT_APP_ID,`
 clientSecret: process.env.MICROSOFT_APP_PASSWORD,`
  storage: mongoStorage,
  debug: false,
});

var bot = controller.spawn({serviceUrl: process.env.NGROK_URL});

controller.setupWebserver(process.env.PORT || 3000, function(err, webserver) {
    controller.createWebhookEndpoints(webserver, function() {
        console.log("BOTKIT: Webhooks set up!");
    });
});

controller.on('message_received', function(bot, message) {
  console.log("message_received"); // Not printed
  bot = controller.spawn({serviceUrl: message.serviceUrl});
});
var controller = botkit.teamsbot({
 clientId: process.env.MICROSOFT_APP_ID,`
 clientSecret: process.env.MICROSOFT_APP_PASSWORD,`
  storage: mongoStorage,
  debug: false,
});

var bot = controller.spawn({serviceUrl: process.env.NGROK_URL});

controller.setupWebserver(process.env.PORT || 3000, function(err, webserver) {
    controller.createWebhookEndpoints(webserver, function() {
        console.log("BOTKIT: Webhooks set up!");
    });
});

controller.on('message_received', function(bot, message) {
  console.log("message_received"); // Not printed
  bot = controller.spawn({serviceUrl: message.serviceUrl});
});

`var controller = botkit.teamsbot({
clientId: process.env.MICROSOFT_APP_ID,
clientSecret: process.env.MICROSOFT_APP_PASSWORD,
storage: mongoStorage,
debug: false,
});
var bot = controller.spawn({serviceUrl: process.env.NGROK_URL});
controller.setupWebserver(process.env.PORT || 3000, function(err, webserver) {
controller.createWebhookEndpoints(webserver, function() {
console.log("BOTKIT: Webhooks set up!");
});
});
controller.on('message_received', function(bot, message) {
console.log("message_received"); // Not printed
bot = controller.spawn({serviceUrl: message.serviceUrl});
});`

Are you using the starter kit?  https://glitch.com/edit/#!/botkit-teams
You have the route for the serviceUrl there over in https://github.com/howdyai/botkit-starter-teams/blob/master/components/routes/teams.js#L7

To elaborate on @peterswimm's answer -- you get the serviceUrl from the incoming messages. You have to store it for later use.

Thanks for responses, i didn't used starter kit and with it,  it's works. :)

Every doc I see seems to be saying "get the serviceURL from the conversation", which is chicken and egg if there is no existing conversation(s). If a bot wants to initiate a new conversation with a new user, where would the serviceURL come from?

I agree the documentation could definitely be improved.
Bot's can be set up with a Bot endpoint address.
Whenever the app is installed by a user this endpoint will be called and sent a conversationUpdate event.  This event contains the serviceUrl.
You can also go one step further and use the Microsoft Graph Api to automatically install Teams applications for users.
One question I have that isn't clear is whether the serviceUrl is User specific or Teams specific (e.g. could I potentially have a different serviceUrl for each user, or is it OK to store it on a per-teams basis).


handle C++ name mangling during symbol lookup

Workaround for SWDEV-225389 w/ HCC.

Could you please create test case for the scenario that you have fixed here.
Thanks

I don't quite think this is the right place to fix this, since this is an implementation detail. I suspect you might have some user-facing API that is not passing the right symbol name (probably because the user is not passing the right thing), but that should be addressed there and not in the implementation, which assumes stricter preconditions.

Closing re: email discussion


Rebuttal Stage: Expire invitations that are not being used

If a venue select in the RebuttalStage "Multiple author rebuttals per paper" and then change it to "One author rebuttal per posted review". The process function creates different invitation ids.
We should expire the previous invitations and create the new ones.
How to reproduce this issue?:

Set the rebuttal stage for a venue selecting "Multiple author rebuttals per paper"
This creates invitation like venue.cc/Submission1/-/Rebuttal
Change the rebuttal stage for the same venue selecting "One author rebuttal per posted review"
This created the invitation like venue.cc/Submission1/Review1/-/Rebuttal

So both invitations are active, we should expire the previous. @celestemartinez I think there is something done in the edit process function that expire invitations but it doesn't contemplate this caae, should we add the fix there?

We can make this change. Neuirps did use 2 rebuttal stages at some point:

Neurips 2023 used both an Author_Rebuttal invitation  (one per paper) and a Rebuttal invitation .

However, we don't allow changing the name of the Rebuttal Stage from the request form, so perhaps we only should allow one being run from the UI. For Neurips, it looks like the Author_Rebuttal was run from python.

I would use two different stages if they want to allow both options.


Dont localize item.quantity in JS Code

Fixes #2505
Fixes a Javascript SyntaxError on different locales with "," as decimal seperator

@matmair the new "welcome message" integration is failing here. Can you take a look at it?


Won't/can't unit test the generated state_id_to_string() or a ROOT_exit() functions

Hi, this is a great project! =)
I like unit testing the generated source code to verify the behavior but I don't want to unit test these functions. Since I don't unit test them, my code coverage metrics show them as not covered.

...state_id_to_string() -- I don't personally use this function at all so it would be nice if this was excluded from the code coverage metrics
ROOT_exit() -- This function is unreachable as my state machine doesn't typically exit (they just run forever).

The following comments could be added before/after these functions to exclude them from code coverage metrics:
// LCOV_EXCLUDE_START
// GCOV_EXCLUDE_START
// GCOVR_EXCLUDE_START
... function goes here ...
// LCOV_EXCLUDE_STOP
// GCOV_EXCLUDE_STOP
// GCOVR_EXCLUDE_STOP

This should exclude the function from being picked by lcov, gcov, and gcovr.

Great suggestion!

Completed as part of #114. See https://github.com/StateSmith/StateSmith-examples/tree/main/user-post-process
Video tutorial will be uploaded soon-ish to https://www.youtube.com/@statesmith

Video links:

Removing state id to string function
Surrounding a function with code coverage annotations


docker build error "System error: read parent: connection reset by peer"

Steps to reproduce:

On Ubuntu 14.04, have Docker 1.9.1 installed using AUFS.
git clone https://github.com/boot2docker/boot2docker; cd boot2docker
docker build -t username/boot2docker .

It will take a long time because boot2docker takes a while to build, but eventually the build fails with:
Step 36 : RUN mkdir -p /prl_tools &&     curl -fL http://download.parallels.com/desktop/v${PRL_MAJOR}/${PRL_VERSION}/ParallelsTools-${PRL_VERSION}-${PRL_BUILD}-boot2docker.tar.gz         | tar -xzC /prl_tools --strip-components 1 &&    cd /prl_tools &&    cp -Rv tools/* $ROOTFS &&        KERNEL_DIR=/linux-kernel/ KVER=$KERNEL_VERSION SRC=/linux-kernel/ PRL_FREEZE_SKIP=1     make -C kmods/ -f Makefile.kmods installme &&        find kmods/ -name \*.ko -exec cp {} $ROOTFS/lib/modules/$KERNEL_VERSION-boot2docker/ \;
 ---> Running in d5fa827db3d0
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  111k  100  111k    0     0  1427k      0 --:--:-- --:--:-- --:--:-- 1435k
'tools/usr/local/etc/init.d/prltoolsd' -> '/rootfs/usr/local/etc/init.d/prltoolsd'
'tools/usr/bin/prltoolsd' -> '/rootfs/usr/bin/prltoolsd'
'tools/usr/bin/prl_nettool' -> '/rootfs/usr/bin/prl_nettool'
'tools/usr/bin/prlvmcheck' -> '/rootfs/usr/bin/prlvmcheck'
make: Entering directory '/prl_tools/kmods'
cd prl_eth/pvmnet && make
make[1]: Entering directory '/prl_tools/kmods/prl_eth/pvmnet'
make -C /linux-kernel/ M=/prl_tools/kmods/prl_eth/pvmnet
make[2]: Entering directory '/linux-kernel'
Cannot start container d5fa827db3d0c74fe15b7f652c9a490b032a1fc1fdbcfd14e1249c541cbeec60: [8] System error: read parent: connection reset by peer

In the logs:
WARN[0280] signal: killed
WARN[0280] failed to cleanup ipc mounts:
failed to umount /var/lib/docker/containers/d5fa827db3d0c74fe15b7f652c9a490b032a1fc1fdbcfd14e1249c541cbeec60/shm: invalid argument
failed to umount /var/lib/docker/containers/d5fa827db3d0c74fe15b7f652c9a490b032a1fc1fdbcfd14e1249c541cbeec60/mqueue: invalid argument

Hi!
Please read this important information about creating issues.
If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.
If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.
This is an automated, informational response.
Thank you.
For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues

BUG REPORT INFORMATION
Use the commands below to provide key information from your environment:
docker version:
docker info:
uname -a:
Provide additional environment details (AWS, VirtualBox, physical, etc.):
List the steps to reproduce the issue:
1.
2.
3.
Describe the results you received:
Describe the results you expected:
Provide additional info you think is important:
----------END REPORT ---------
#ENEEDMOREINFO

I did a git bisect to see where this issue first occurs with building boot2docker, and https://github.com/boot2docker/boot2docker/commit/f651ad2017f8b5f2ec7e0c83078dffc37bcab2e2 seems to be the offending commit.  My guess is that bumping the VirtualBox guest additions does not play nicely with the Parallels tools step, but I'm pretty baffled why the Docker server simply resets the connection and causes the build to fail.

Closing as this is a dup of #14203


x/website:

What version of Go are you using (go version)?

$ go version
1.16

Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
Ubuntu 20.04 amd64
go env Output
$ go env

What did you do?
run example on site under func os.WriteFile

https://play.golang.org/p/mCCjFLaE9Ua
What did you expect to see?
nothing
What did you see instead?
2009/11/10 23:00:00 open testdata/hello: no such file or directory

Thanks for reporting. See #9679. Closing as a dup..


error: esp_event.h: No such file or directory

Hi there
I'm using idf v5.0.1
esp_websocket_client (1.0.0)
I am getting a complier error-
In file included from D:/Projects/discord-test/discord-test/components/esp-discord/include/discord/private/_models.h:5,
from D:/Projects/discord-test/discord-test/components/esp-discord/include/discord/private/_json.h:5,
from D:/Projects/discord-test/discord-test/components/esp-discord/src/discord/private/_json.c:1:
D:/Projects/discord-test/discord-test/components/esp-discord/include/discord.h:5:10: fatal error: esp_event.h: No such file or directory
5 | #include "esp_event.h"
But the project has this file in-
D:/Projects/discord-test/discord-test/components/components/esp_event/include/esp_event.h

hello, i have the same error, did you manage to solve it? :)

I just got a successful build by switching to ESP-IDF v4.4.4

#7


Scaffold bulk person find endpoint

This adds a skeleton bulk person find endpoint for review.
I've also fixed up the naming of the existing find endpoint and the associated operation.

 Bulk person search API for CTR


Fix repeatable compiler options handling from the command line

Fixes #2653
Repeatable compiler options (-P:*:* and -Xplugin:*, specifically) were being filtered out on some code paths.
This was happening as we do treat them differently, as they can't be shadowed. We have special handling for compiler options with keys which can be passed multiple times, as those cannot be overridden from the CLI when they're also being declared in a using directive.
@sequencer This may already be enough to cover your usecase from #2621. It still doesn't seem to work correctly with Bloop, but I can no longer replicate the problem with --server=false (although it may be more complicated in your mill build, so let's verify it after this gets merged).

Thank you @Gedochao, is possible to release a new minor version of scala-cli for this bug fix?


Question about generated update input types from SDLInputs()

Hi!
My team and I are currently upgrading from prisma1 to prisma2. We are making use of pal g to help generate graphql type definitions to replace the ones generated previously by prisma1.
We are having issues with the generated UpdateInput graphql input types
A example model in our schema.prisma file below:
model Project {
  id                 String            @id @default(cuid())
  projectName        String
  projectDescription String
  shortDescription   String

paljs generated below
type Mutation {
  # example update mutation
  updateOneProject(where: ProjectWhereUniqueInput!, data: ProjectUpdateInput!): Project!
}

input ProjectUpdateInput {
  id: StringFieldUpdateOperationsInput
  projectName: StringFieldUpdateOperationsInput
  projectDescription: StringFieldUpdateOperationsInput
  shortDescription: StringFieldUpdateOperationsInput
}

input StringFieldUpdateOperationsInput {
  set: String
}

# other definitions...

prisma1 generated below:
input ProjectUpdateInput {
  projectName: String
  projectDescription: String
  shortDescription: String
}

using the graphql types by paljs, we are now required to always use a set even for scalar or enum types.
But the prisma2 typescript definitions do allow for writing either just the string to update or with the help of a nested set property
prisma2 typescript definition from .prisma/client/index.d.ts below:
export type ProjectUpdateInput = {
  id?: string | StringFieldUpdateOperationsInput
  projectName?: string | StringFieldUpdateOperationsInput
  projectDescription?: string | StringFieldUpdateOperationsInput
  shortDescription?: string | StringFieldUpdateOperationsInput
}

export type StringFieldUpdateOperationsInput = {
  set?: string
}

so this requires us to change lots of update mutations that weren't using set on the frontend.
Just wondering if there is a way to add a config to not use xxxxFieldUpdateOperationsInput by default?
Maybe you could also shed some light as to why xxxxFieldUpdateOperationsInput was chosen when generating sdlInputs. That'll be awesome.
Our current workaround is making a change to the source sdlInputs.ts file to detect if the type includes FieldUpdateOperationsInput
export const getInputType = (field: DMMF.SchemaArg) => {
  let index: number = 0;

  // new code below
  if (field.inputTypes.length > 1 && field.inputTypes[1].type.includes('FieldUpdateOperationsInput') ) {
    index = 0;
    return field.inputTypes[index]
  }
  // new code above

  if (field.inputTypes.length > 1 && field.inputTypes[1].kind === 'object') {
    index = 1;
  }
  return field.inputTypes[index];
};

Thank you!
our pal.js config below if you're wondering about it
module.exports = {
  backend: {
    generator: 'sdl',
    output: 'gql/src/typedefs',
};

Why we use Object not string  type?
Because the Number fields have many options like:
from prisma release notes https://github.com/prisma/prisma/releases/tag/2.6.0

We are introducing five atomic operations for Int and Float fields :
increment: x: Adds x to the current value
decrement: x: Subtracts x from the current value
multiply: x: Multiplies the current value by x
divide: x: Divides the current value by x
set: x: Sets the value to x (equivalent to data: { age: 18 })

And Also for Json fields must with set
In the end, I can add an option for you to pass into SDLInputs() function

Thanks for info!
and yes if there could be some kind of flag in the paljs config that we could use together with generator:'sdl' so that sdlInputs() will not choose the types that require setand it'll be more "prisma1-friendly"
I guess it could look something like the following?
// pal.js config
module.exports = {
  backend: {
    generator: 'sdl',
    output: 'gql/src/typedefs',
    doNotuseFieldUpdateOperationsInput: true // defaults to false.  This is just a suggestion. I'm not sure what would be a good name or a good place to indicate this flag
};

Will probably have to indicate in the documentation that this will not generate graphql type defs that support prisma2.6 and above.

Can you tell me if you using my generator with Javascript or Typescript?

Can you tell me if you using my generator with Javascript or Typescript?

we are using typescript 👍🏻


missing implicit

Positionable is missing an implicit conversion from Value -> Position1D
Needs something like:
implicit def V2P[T <: Value](t: T): Positionable[Position1D] = {
  new Positionable[Position1D] { def apply(): Position1D = Position1D(t) }
}

fixed by #43


fix: revert to bullseye

Bookworm is still in testing phase so better to move to bullseye instead
Issue

Link:
Summary:

How

Screenshots

References

Checklist

[ ] Tests added
[ ] Cleaned up commit history (rebase!)
[ ] Documentation added
[ ] Thought about variable and method names

Waiting for these before merging:
https://github.com/radxa/apt/issues/51
https://github.com/NebraLtd/hm-config/pull/220


Fix linting error

Summary
Change Type
Mark the type of change your PR introduces:

[ ] Bug fix (resolves an issue without affecting existing features)
[ ] Feature (adds new functionality without breaking changes)
[ ] Breaking change (may impact existing functionalities or require documentation updates)
[ ] Documentation (updates or additions to documentation)
[x] Refactoring or test improvements (no bug fixes or new functionality)

Testing
Outline how the changes were tested, including steps to reproduce and any relevant configurations.
Attach screenshots if helpful.
Review Checklist:

[x] Reviewed my own code for quality and clarity.
[ ] Added comments to complex or tricky code sections.
[ ] Updated any affected documentation.
[ ] Included tests that validate the fix or feature.
[ ] Checked that related changes are merged.

coverage: 53.333%. remained the same
when pulling b32feeb1db172543a58f2bcb5a07123847c63a10 on eleftherias:fix-lint
into 6cf650307eb984d458b876c744f256fb09af1538 on stacklok:main.


How to install external libraries

Hi,
how can I install external libraries so that the packed tar.gz file already has a set of libraries?
At the moment there is an option to unpack the files, install the libraries and repack the files into tar.gz files, but could it be simplified?

Do you mean additional python libraries? As in .../bin/pip install ...?
If so, that could be a nice additional feature, not possible right now, but could be added via a configuration entry

Yes because I would like to add additional libraries like Pillow, PyQt5, PyYaml etc

My current solution is additional script which will build for me tar.gz file in docker and then update it:
`
#!/bin/sh
set -e
PYTHON3_VERSION="3.9.11"
TMP_PATH="/tmp/portable-python"
portable-python -c portable-python.yml build ${PYTHON3_VERSION} -m libffi,zlib,xz,bzip2,readline,openssl,sqlite,bdb,gdbm,uuid
if [ ! -x /src/dist/cpython-${PYTHON3_VERSION}-linux-x86_64.tar.gz ]; then
set -x
mkdir -p "${TMP_PATH}"
tar -C ${TMP_PATH}/ -xf /src/dist/cpython-${PYTHON3_VERSION}-linux-x86_64.tar.gz
${TMP_PATH}/${PYTHON3_VERSION}/bin/pip install Pillow==10.0.0 ansible-core==2.12.3
tar -czvf /src/dist/cpython-${PYTHON3_VERSION}-linux-x86_64_UPDATED.tar.gz -C ${TMP_PATH}/ ${PYTHON3_VERSION}
fi
`

Added additional option "cpython-additional-packages" to portable-python.yml file (list of packages).


Add a curriculum vitae

Expand the user profile with a resume view, that is loaded from standard format. It should be possible to link structured information about my skills for matching purposes.

https://github.com/jsonresume/resume-schema
https://github.com/josephcslater/vitae
https://github.com/poldrack/autoCV

See #147

Initial support for this, allowing only admins to upload JSON resumes, has been added.


Improve API documentation

The README does not provide enough information about the API endpoints. Things like:

Type of parameter
Is this parameter optional or required?

Sure, no problem at all. It's indeed a personal project but any contribution is always welcome.


nodeos crashes - Exception Details: 3010008 block_id_type_exception: Invalid block ID

I am able to spin up a fresh instance of a server and consistently crash it when calling a simple hello world contract.
I've put debug files here:
https://vsolution.box.com/s/5gzvmcj237j64xaet1tn6ic7yh82apa5

Server output
There is a windows fiddler trace showing the exact http requests.

If you don't have a windows box to view the fiddler trace. Here's some screenshots showing the raw HTTP requests.

This can be reproduced by going to this URL: http://pennstation.eosdocs.io/ and clicking the 1st button on the page repeatedly.
http://pennstation.eosdocs.io:7001/v1/chain/get_info
{"server_version":"75635168","chain_id":"cf057bbfb72640471fd910bcb67639c22df9f92470936cddc1ade0e2f2e7dc4f","head_block_num":73,"last_irreversible_block_num":59,"last_irreversible_block_id":"0000003bd342916b0ec1f06bf766195c3585460e9647fff9a2afbdffb85a0c1b","head_block_id":"00000049591e08a5b9b377ad692a2570938e592aa1a0cbda667b3aaa8a55c81e","head_block_time":"2018-07-19T03:43:00.500","head_block_producer":"bp1","virtual_block_cpu_limit":214903,"virtual_block_net_limit":1126859,"block_cpu_limit":199900,"block_net_limit":1048576}

This is a small debug environment running as a Docker image in case that makes a difference.

@warrick-eosny please give me step by step tutorial to install and download blockchain

unable to reproduce this again.


Android tutorial outdated

completely and utterly useless to someone just starting out

look at android tutorial
Steps to Reproduce
look at android tutorial

You must provide a minimal and easy-to-run reproduction of your issue - how to create a minimal reproduction?. We're a small team of maintainers and do not have time to try reproduce bugs ourselves.

Please try to reproduce the bugs on the provided example app. Either provide a link to the repo that reproduces the bug or provide your version of the index.js file that reproduces the issue and that we can easily use.

Explain the steps we need to take to reproduce the issue.

Include a gif if you think it may help.

If you're asking for help, please make it easy for us to help you!

Expected Behavior

Actual Behavior

Environment
Please provide

react-native version
@react-native-community/google-signin version
your Podfile, output of running pod install and cocoapods version (if applicable)
version of Android, gradle, android plugin for gradle, play-services-auth and version of the Google Play Services app on your phone (if applicable)
version of iOS (if applicable)

Hello and thanks for your feedback. Please note that for any feedback to be actionable, it needs to be specific - otherwise it's utterly useless, if you will.
If you don't want to bother explaining the details, please do consider forking the repo and contributing a PR with a fix. If you don't want to contribute and don't want to spend time giving proper feedback, just don't give any, it's better that way.


General Recursion

What would it look like to add on general recursion? What would typing letrec look like (I'm assuming it would be letrec)? Are there specific designs you already had in mind?
Adding general recursion was mentioned as a possible future update here https://github.com/AndrasKovacs/elaboration-zoo/issues/7#issuecomment-546970285

Here is a gist from Andras: https://gist.github.com/AndrasKovacs/6b1270d9edb702641f42fd6440f0f750
I think this typing rule should work:
G, rec : (a : A) -> B[a], a : A |- t : B[a]
----------------------------
G |- fix rec a. t : (a : A) -> B[a]

If you want to have datatypes with induction then a pretty minimal setup is (I'm assuming consistency is not important):

Add unit and boolean types
Add sigma types
Add a fixpoint type, such as:

Fix : (U -> U) -> U
con : {F : U -> U} -> F (Fix F) -> Fix F
elimFix : {F : U -> U} (P : Fix F -> U) (alg : (rec : (z : Fix F) -> P z) (y : F (Fix F)) -> P (con {F} y)) (x : Fix F) -> P x

with computation rule:
elimFix {F} P alg (Con {F} x) ~> alg (\z. elimFix {F} P alg z) x

If you want indexed datatypes you have to add a native identity type and "upgrade" the fixpoint to:
Fix : {I : U} -> ((I -> U) -> I -> U) -> I -> U

You can even get indexed inductive-recursive types, with the following fixpoint:
Fix : {I : U} {R : I -> U}
  (F : (S : I -> U) -> ({i : I} -> S i -> R i) -> I -> U) -> ({S : I -> U} -> (T : {i : I} -> S i -> R i) -> {-i : I} -> F S T i -> R i) ->
  I -> U

If you want efficiency and/or consistency, things become more complicated...

Can you explain what the B[a] notation means?

It's a type where the a variable may appear. I probably should've just written only B.

Another way is to add fix with typing rule:
A : Type
B : A -> Type
f : (x : A) -> Either A (B x)
x : A
--------------------------------------------------------
fix {A} {B} f x : B x

With computation rule:
fix f x ~> either (\a. fix f a) (\b. b) (f x)

For some Either type, for example if you have sigma and booleans:
Either : Type -> Type -> Type = \A B. (tag : Bool) ** if tag then A else B
Left : {A B} -> A -> Either A B = \x. (True, x)
Right : {A B} -> B -> Either A B = \x. (False, x)
either : {A B R} -> (A -> R) -> (B -> R) -> Either A B -> R = \f g x. elimBool (\tag. (if tag then A else B) -> R) f g (fst x) (snd x)


req.user is undefined

Hello. I've set up Twitter, Github, Facebook and Google authentication.
So far I get a nice user in req with Twitter and Github, but when logging in with Facebook and Google I get req.user undefined.
I've added client id and secret for all passports and scope for Google. I see redirect_uri is set, but still with Facebook and Google I'm redirected back to the login page. Is there something I'm missing?
My app: https://github.com/cenobitedk/lego-exchange/tree/dev-sails/app

Maybe @kasperisager can help me?

Have you tried req.session.req.user?

@kdelmonte thanks, but req.sessionis undefined.

have you checked the passport policy is getting executed? for example, the route for homepage is bypassing all controller and hence the policies applied. Make sure the passport policy is being executed.
  '/': {
    view: 'homepage'
  },

I think that's fairly normal because default route is not under any controller. Basically you just display HTML page. Put your default page under a controller and works like a charm.


Add more options in config.ts to make it easier to switch to different modes

Currently some configs are hardcoded, and one have to modify multiple places to switch to another mode (like 3-player mahjong or only want to review the games in Jade Room), which is inconvenient and not elegant.
For example, one have to do the following:

Change multiple occurances of CONFIGURATIONS.DEFAULT to the configuration of 3p (CONFIGURATIONS.ikeda) in src/amae-koromo/utils/conf.ts. (can also replace the DEFAULT with ikeda)
Change hardcoded reviewCompatibleModes ([16, 12, 9]) to eg. [26,25,24,23,22,21] for 3p in src/utils.ts
Change mjaiReviewer, mortal, mortalCfg in src/config.ts to the 3p-compatible reviewer and mortal binary path

Maybe more options can be added to config.ts as well as refactoring some code in other places to make it easier to change these internal configurations

This script is not intended for use in 3p. However, I will be happy to see a pull request for this feature.


the pull_request_review_comment event from fork repo cannot access secrets

bad run:
https://github.com/Climatik-Project/Climatik-Project/actions/runs/9164252790
The pull_request_target worked fine:
https://github.com/Climatik-Project/Climatik-Project/actions/runs/9142107503

from https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request_target:~:text=Warning%3A For,with this event.

I tried to redirect the event to manually dispatch and then passing the payload.
However, when we use
  curl -L -X POST -H "Accept: application/vnd.github+json" \
    -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" \
    https://api.github.com/repos/${{ github.repository }}/actions/workflows/$workflow_filename/dispatches \
    -d '{"ref": "main", "inputs": {"payload": "${{ toJson(github.event) }}", "event_name": "${{ github.event_name }}"}}'

the secrets.GITHUB_TOKEN doesn't have the required actions:write permission. It is read-only token. We will get this error:
{
  "message": "Resource not accessible by integration",
  "documentation_url": "https://docs.github.com/rest/actions/workflows#create-a-workflow-dispatch-event"
}

This https://stackoverflow.com/questions/77882485/github-actions-resource-not-accessible-by-integration
provides a solution: rely on a GitHub App to provide the token for the step.
But it again needs to store the APP private key in the secrets, which cannot be accessed in pull_request_review_comment event.
Replacing secrets.GITHUB_TOKEN with an repo admin's personal access token have the same issue.
So, it seems that the only way is to use a backend service and store the API Keys on the server.


change logging mechanism

The current logging mechanism has huge performance impacts.
When something is done like "logger.debug('functionName -'+parmValue)"
this goes thru the expensive nashorn ADD function (to concatenate the string) before it is even
determined if the logger is logging at that level.
For the examples/mllib/largedataset.js example, it take 100s when logging is on in the Serialize.js,
and only 50s when the logging is commented out.
We probably want something like "logger.debug('functionName',parmvalue)" and then within the logger
implementation, first check if it is at the correct logging level, and if it is, do a string.join on the log parameters.

This may help but if the problem is string concatenation we have that throughout our code  fro example
var Logger = require(EclairJS_Globals.NAMESPACE + '/Logger');

Is String.conca()t in Nashorn any faster than using the + operator?


Default cidr blocks to empty list

what

Default cidr blocks to empty list

why

Otherwise on each plan, this is shown for each sg rule

  # module.msk_cluster.module.broker_security_group.aws_security_group_rule.keyed["_m[0]#tls#sg#0"] has been changed
  ~ resource "aws_security_group_rule" "keyed" {
      + cidr_blocks              = []
        id                       = "sgrule-snip"
      + ipv6_cidr_blocks         = []
        # (9 unchanged attributes hidden)
    }

references
N/A

/test all

Closing because this plan drift is a provider issue and is not possible to solve because you cannot supply both cidr_block and ipv6_cidr_block at the same time, even if either or both of them are empty lists.


Add completion for jobs and variable files.

cc @chendrix

@jmcarp This looks great and worked when we tried it locally.
Before we merge these changes, would you please write a test for the variable files completion?

Thanks for reviewing @andrewedstrom ! It turns out that go-flags fixed the issue that required a custom slice type for slice completions the other day, so I reverted the commit for variable files completion and filed https://github.com/concourse/concourse/pull/1383.

@jmcarp, thanks for the PR!
@andrewedstrom, sorry for I accepted the PRs in the wrong order. Didn't mean to take the satisfaction of pressing that button.


build: Adjust ci for updated Vulkan-Loader

Adapt to Vulkan-Loader getting rid of Vulkan-Headers submodule.
Get and build Vulkan-Headers and fix Vulkan-Loader build to use it.

Travis jobs are not being scheduled.


Removing "Loading" text after loading?

Hello,
I couldn't find anything in the docs about how to get rid of the "Loading..." text fully after the loading is "finished"? I tried it in a Jupyter Notebook and this is the result:

Any suggestions?

Thanks for the report. It seems like in the notebook, the sequence used to erase a line and flush is somehow interpreted differently? Relevant code is here:
https://github.com/ines/wasabi/blob/2beb8fe98cdfe9cafcae0a7c7b18c9738119e337/wasabi/printer.py#L201-L202
I'd need to investigate this. The whole cursor movement stuff, erasing lines etc. is definitiely one of the trickier parts – at least, I've always found it kinda difficult to get right across platforms and configurations and there might still be cases where it fails.

I know!! I am so happy to have come across your package though! (Found it when I looked over the spaCy code!) I have had so much trouble with flushing... ugh.

I have looked into this issue myself now and cannot find a solution!

@ines do you think the newest release will help with this issue?

@kallewesterling  any solution ?

@ardyfeb I have nothing to do with the dev of this package. Ask @ines! :)

@ines Revisited this issue and it now looks like it works better, in a way. At last, the .good() message displays at the end of the loop:

However, the context manager does not generate any output in the Notebook. This is what it looks like when inside the with msg.loading() manager:


Add ElasticsearchSinkOptions.BufferFileRollingInterval option

What issue does this PR address?
Relates to https://github.com/serilog-contrib/serilog-sinks-elasticsearch/issues/410
Add ElasticsearchSinkOptions.BufferFileRollingInterval option
Does this PR introduce a breaking change?
No.
Please check if the PR fulfills these requirements

[x] The commit follows our guidelines
[ ] Unit Tests for the changes have been added (for bug fixes / features)

Other information:
Add ElasticsearchSinkOptions.BufferFileRollingInterval option details:

Using this option we can customize buffer file rolling interval. The default is RollingInterval.Day (so no changes here). In some cases higher granularity may be needed.
Changed regular expression for FileSet to get buffer files to support different rolling interval file name formats - from Infinite to Minute. All of them are different amount of digits representing date - 0(Infinite), 4(Year),..., 12 (Minute). So replaced expression part for day format (?<date>\\d{8}) with the expression for all interval date (?<date>\\d{0,12}).
Unit tests were not added because it is hard to do with the current architecture: classes need to be tested are internal, objects are instantiated inside constructors (so cannot be injected). Only complex integration tests are viable option, which would require a lot of effort.

Found a problem with ElasticsearchPayloadReader, so this needs more changes. Understood that I can at least add tests for them, so they support different intervals.

Closed in favor of https://github.com/serilog-contrib/serilog-sinks-elasticsearch/pull/416


2.0"

@rultor release tag is 2.0

@rultor release tag is 2.0

@portlek OK, I will release it now. Please check the progress here

@rultor release tag is 2.0

@portlek Done! FYI, the full log is here (took me 4min)


Metadata (EML) link gives 404

Metadata (EML) link gives 404
Clicking on Source data > Metadata (EML) does not find any results and delivers the 404 Dodo

fbitem-dataset025769ee-df28-40de-bead-69fe1259bbad
Reported by: @ahahn-gbif
System: Firefox 51.0.0 / Windows 10 0.0.0
Referer: https://demo.gbif.org/dataset/025769ee-df28-40de-bead-69fe1259bbad/origin
Window size: width 1395 - height 604
API log
Site log

yes that isn't good. thanks for reporting - no more it is so.


[feat] Implement prompt

Describe the problem
Hi! Great framework!
I was trying to create a quick app, and I wanted to use browser's prompt function
There's a @tauri-apps/api/dialog package which provides you with confirm, alert analogues, and even file selection. But, unfortunately, there is no prompt
Describe the solution you'd like
In browsers, it looks like this:

Would be cool to see the same thing:
import { prompt } from '@tauri-apps/api/dialog'

const name = await prompt("Type your name")

Alternatives considered
I was also trying to use window.prompt. Console says this function exists, but it does nothing
Additional context
No response

I believe the plugin:notification label is wrong (also looking for this feature, the browser-default prompt is very ugly with the JavaScript - tauri://localhost/)

whoops, thanks for catching that :)


APEXMALHAR-2200 Enabled checkstyle for demos. Fixed checkstyle errors.

Great to see this cleanup happening! Please split the change into 2 commits. One for the build change and the second for the formatting changes. The second commit should be attributed to the CI user so there is no confusion when questions arise WRT the logic itself. You can do that by running:
git commit --amend --author "MalharJenkins<EMAIL_ADDRESS>


Should be able to find all installers for all versions of unity in one call

Right now Find-UnitySetupInstaller requires you to specify a version. It'd be great if I could specify -Latest or neither to find all versions.

[ ] Should patch and beta versions be included by default if no args are passed?
[ ] Should we add flags to control the versions you'll search for, use param sets to avoid Version being specified.
[ ] Should we add a Find-UnitySetupVersion which finds available versions which could be piped to 'Find-UnitySetupInstaller'?

Folding into 4.0 for now as it sits alongside #87