TFA can't load _image_ops.so - says file doesn't exist, but its there?

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10 64-bit

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

TensorFlow installed from (source or binary): TF 2.1.0 installed via conda

TensorFlow version: 2.1.0

Python version: 3.7.0

Installed using virtualenv? pip? conda?: TF installed via conda, TFA via pip.
TFA installed via pip install tensorflow_addons==0.9.1

Bazel version (if compiling from source): n/a

GCC/Compiler version (if compiling from source):

CUDA/cuDNN version:

packages in environment at C:\ProgramData\Anaconda3\envs\tf:

Name                    Version                   Build  Channel
cudnn                     7.6.5                cuda10.1_0

GPU model and memory:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 442.19       Driver Version: 442.19       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108... WDDM  | 00000000:01:00.0  On |                  N/A |
| 23%   38C    P8    17W / 250W |   1049MiB / 11264MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108... WDDM  | 00000000:03:00.0 Off |                  N/A |
| 23%   29C    P8     9W / 250W |    137MiB / 11264MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

Describe the problem
TFA can't load _image_ops.so
File "C:\ProgramData\Anaconda3\envs\tf\lib\site-packages\tensorflow_addons\image\distance_transform.py", line 69, in euclidean_dist_transform
output = _image_so.ops.addons_euclidean_distance_transform(images)
File "C:\ProgramData\Anaconda3\envs\tf\lib\site-packages\tensorflow_addons\utils\resource_loader.py", line 56, in ops
self._ops = tf.load_op_library(get_path_to_datafile(self.relative_path))
File "C:\ProgramData\Anaconda3\envs\tf\lib\site-packages\tensorflow_core\python\framework\load_library.py", line 57, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: C:\ProgramData\Anaconda3\envs\tf\lib\site-packages\tensorflow_addons\custom_ops\image_image_ops.so not found
The file exists in the specified directory
Provide the exact sequence of commands / steps that you executed before running into the problem
Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
As you can see, the library file is clearly there...
(tf) C:\ProgramData\Anaconda3\envs\tf\Lib\site-packages\tensorflow_addons\custom_ops\image>dir
Volume in drive C has no label.
Volume Serial Number is 62E0-530C
Directory of C:\ProgramData\Anaconda3\envs\tf\Lib\site-packages\tensorflow_addons\custom_ops\image
08/15/2020  02:19 PM              .
08/15/2020  02:19 PM              ..
08/15/2020  02:19 PM           172,544 _distort_image_ops.so
08/15/2020  02:19 PM           977,408 _image_ops.so
08/15/2020  02:19 PM           169,984 _resampler_ops.so
3 File(s)      1,319,936 bytes
2 Dir(s)  255,060,709,376 bytes free
Here is the code.
"""
    Debug Euclidean distance
"""
from silence_tensorflow import silence_tensorflow
silence_tensorflow()

import os
os.system('cls' if os.name == 'nt' else 'clear')

import warnings
warnings.filterwarnings('ignore', category=FutureWarning)

import tensorflow
import numpy as np
import tensorflow_addons as tfa
import matplotlib.pyplot as plt

with tensorflow.device('/GPU:1'):

    M = 500
    img = np.zeros((M, M))
    top = 50
    bottom = 270
    left = 300
    right = 450
    img[np.ix_(np.arange(top, bottom + 1), np.arange(left, right + 1))] = 1
    img = tensorflow.convert_to_tensor(img)
    img = tensorflow.expand_dims(img, -1)
    img = img.numpy()

    # TFA - Euclidean distance transform
    img = tensorflow.cast(img, dtype=tensorflow.uint8)
    d_img = tfa.image.euclidean_dist_transform(img)  # See https://github.com/tensorflow/addons

    # Plot it
    plt.figure()
    plt.title('True TFA Distance Image')
    _ = plt.imshow(d_img)

    # Tell user how to end code.
    print("\nPress ctrl-c or ctrl-break or close windows to exit.\n")
    plt.show()

Thanks for the help. I can't understand why the system is telling me this library doesn't exist.

@jeball
I am facing a different error when i run your code, please find gist here.
Can you please share a colab gist with the error faced for us to analyse.

Just remove the first lines. Here is the code.
import tensorflow
import numpy as np
import tensorflow_addons as tfa
import matplotlib.pyplot as plt
with tensorflow.device('/CPU:0'):
M = 500
img = np.zeros((M, M))
top = 50
bottom = 270
left = 300
right = 450
img[np.ix_(np.arange(top, bottom + 1), np.arange(left, right + 1))] = 1
img = tensorflow.convert_to_tensor(img)
img = tensorflow.expand_dims(img, -1)
img = img.numpy()

# TFA - Euclidean distance transform
img = tensorflow.cast(img, dtype=tensorflow.uint8)
d_img = tfa.image.euclidean_dist_transform(img)  # See https://github.com/tensorflow/addons

# Plot it
plt.figure()
plt.title('True TFA Distance Image')
_ = plt.imshow(d_img)

# Tell user how to end code.
print("\nPress ctrl-c or ctrl-break or close windows to exit.\n")
plt.show()

I didn't mean to close this. Hit wrong button.

@jeball
I ran the code and face an error please refer to gist here.

Thanks
I think the problem is tensorflow 2.1 GPU was installed with anaconda and tensorflow addons with pip and they seem to be incompatible. The error you get is different from the error I am receiving. I think to resolve I need to install tensorflow with pip.
Can you verify – after more research this seems to be the main problem.
John Ball, Ph.D.
Associate Professor and Robert Guyton Endowed Chair for Teaching Excellence | Electrical and Computer Engineering
<EMAIL_ADDRESS>Associate Editor | IEEE Signal Processing Letters
Associate Editor | Journal of Applied Remote Sensing
Editorial Board Member | MDPI Electronics
Faculty Advisory Council | Transactions on Stem Education
Lab Director | Simrall Radar Laboratory
Lab Co-Director | Center for Advanced Vehicular Systems (CAVS) Sensor Lab
Research Fellow | Geosystems Research Institute (GRI)
Simrall 233, 406 Hardy Rd., Mississippi State, MS 39762
p:662.325.4169  e:jeball<EMAIL_ADDRESS>From: Saduf2019<EMAIL_ADDRESS>Sent: Tuesday, August 18, 2020 6:26 AM
To: tensorflow/tensorflow<EMAIL_ADDRESS>Cc: Ball, John<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [tensorflow/tensorflow] TFA can't load _image_ops.so - says file doesn't exist, but its there? (#42398)
@jeballhttps://secure-web.cisco.com/1DDekOIZVGxDAV-tZQNItaihharROf6u3uBl31EIejO9FW9mY4TYvvwjn6UevtqJ7s0W6o2Vg_5sUOUc3JjeKt228diMMZdiJHTiUkoorG1jTv27lUhQkHdZEg9baKwSRdFirgy_uUZPpUHmU_2aycPJdLLEJFVSUHr-06jkSnYYU0WVHp1xncP8SySfPZpPQfbSxzU5UyBWBXHdfckU4RtAwEcCqgtonuEVcob9LZ-iQoyHpfutFJjrd9ivML_v0dnsiNgi8W9711niMxYwRyukWdpJ59-GdvRf8OU1cCBadxQuHM4WoJBvue4o5Vdibiqv9gIbuNFKVFEzVv_4N8w/https%3A%2F%2Fgithub.com%2Fjeball
I ran the code and face an error please refer to gist herehttps://secure-web.cisco.com/19PUsvm2D2Q95oxVa_ROLqTNqECJ-TSXBB_cEzFWSAh25BkEvbpP2CMdH4Vmc5TYUUDHv_SqOF4T2UzB2bLEfsDDJnI9E_C5_eTyASCFG8r_zz0hAJasIWHujCTiPWIQzV6wj8LLzYvndtZoeQ4JLkNZ2D8UpgStQa7K72zx2bCQm5PmikLcemdRptzt4ym8CUmiZDGngSmOkPo1z3v_vL6g2yKkNUZnWVTY5bada875BJhJ7d5ZDAH7_tzfw2wzKeC49MwSUgccd8aPLwsVPnhrWarp9Qe7AUQXnH3cADuowBMsQrNzCGXwvHA50CLImMZLEMdMhLqSUI7Q82xb6hw/https%3A%2F%2Fcolab.research.google.com%2Fgist%2FSaduf2019%2Faac88f6c369b8d569071c5929e44cf48%2Funtitled367.ipynb.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://secure-web.cisco.com/1E4m66aWdZdg2xm2BgY8h5wr28oREcY7rJlmwNHvTlFw2nO46AYAd7dhmYwUD32_dk-AoKUpccuBJ1zghlRS4cGiHdBXbLdF-I4QZgYjgdYN4b8gRSgXqwHbhIo2uqKiGpn4d35tfLoyREcYP1ZYnHC5mltSbdW2LBp3MmZqljxgLfR13WuBF7iAhGERgStk9hzn0XLuY_eKuzJv7jtqZcn2yrpPQsSi7URhoENzfXrGaoNT83OEqvRtlRr7Aog_CFBHt2_vOIEpEx763MyNmDpCycoH9OsqfTRGCWAemACzd1Dp7-l8F_6ytBEs6VOuJ9ip-kIGHefpf_zf_CIcR7Q/https%3A%2F%2Fgithub.com%2Ftensorflow%2Ftensorflow%2Fissues%2F42398%23issuecomment-675421983, or unsubscribehttps://secure-web.cisco.com/1ncuZ-OAFmyTrwq_pw-gxL9qLwhX80cY8Q3XGFgCmCltlykctyf5ijUQ9YpqAPBT6L2xAFo5W_YIMz_o8ZfgxQ4CROC6muUbQT7MWh9cFxhEL2pe3HpySX1TUHaoL-fq7qn5YMvZWFIoXG7ZS9yS3vRtHssNVaGsJItbB2tDQJ1GFcKU1iXdjBgoKHH_-aA9irTPj0gaAG8HwTwiNPw8SEvsSP7aFtyUvsG64dyODtICg6uwkAgJrRqOsQap2__ztqTWam2J5CGKhHG-cQtV8Xg558lXjzlKkhmEHR6T-A8Zb56DzR7U02FvoKDzRfnzS2M9kF5EY-ximc6lA6I5daQ/https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAETBXODUD3GA4XKXA7IUMO3SBJQKXANCNFSM4QANRIQQ.

My error was with TF 2.1.0, not 2.3.0
John Ball, Ph.D.
Associate Professor and Robert Guyton Endowed Chair for Teaching Excellence | Electrical and Computer Engineering
<EMAIL_ADDRESS>Associate Editor | IEEE Signal Processing Letters
Associate Editor | Journal of Applied Remote Sensing
Editorial Board Member | MDPI Electronics
Faculty Advisory Council | Transactions on Stem Education
Lab Director | Simrall Radar Laboratory
Lab Co-Director | Center for Advanced Vehicular Systems (CAVS) Sensor Lab
Research Fellow | Geosystems Research Institute (GRI)
Simrall 233, 406 Hardy Rd., Mississippi State, MS 39762
p:662.325.4169  e:jeball<EMAIL_ADDRESS>From: Saduf2019<EMAIL_ADDRESS>Sent: Tuesday, August 18, 2020 6:26 AM
To: tensorflow/tensorflow<EMAIL_ADDRESS>Cc: Ball, John<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [tensorflow/tensorflow] TFA can't load _image_ops.so - says file doesn't exist, but its there? (#42398)
@jeballhttps://secure-web.cisco.com/1DDekOIZVGxDAV-tZQNItaihharROf6u3uBl31EIejO9FW9mY4TYvvwjn6UevtqJ7s0W6o2Vg_5sUOUc3JjeKt228diMMZdiJHTiUkoorG1jTv27lUhQkHdZEg9baKwSRdFirgy_uUZPpUHmU_2aycPJdLLEJFVSUHr-06jkSnYYU0WVHp1xncP8SySfPZpPQfbSxzU5UyBWBXHdfckU4RtAwEcCqgtonuEVcob9LZ-iQoyHpfutFJjrd9ivML_v0dnsiNgi8W9711niMxYwRyukWdpJ59-GdvRf8OU1cCBadxQuHM4WoJBvue4o5Vdibiqv9gIbuNFKVFEzVv_4N8w/https%3A%2F%2Fgithub.com%2Fjeball
I ran the code and face an error please refer to gist herehttps://secure-web.cisco.com/19PUsvm2D2Q95oxVa_ROLqTNqECJ-TSXBB_cEzFWSAh25BkEvbpP2CMdH4Vmc5TYUUDHv_SqOF4T2UzB2bLEfsDDJnI9E_C5_eTyASCFG8r_zz0hAJasIWHujCTiPWIQzV6wj8LLzYvndtZoeQ4JLkNZ2D8UpgStQa7K72zx2bCQm5PmikLcemdRptzt4ym8CUmiZDGngSmOkPo1z3v_vL6g2yKkNUZnWVTY5bada875BJhJ7d5ZDAH7_tzfw2wzKeC49MwSUgccd8aPLwsVPnhrWarp9Qe7AUQXnH3cADuowBMsQrNzCGXwvHA50CLImMZLEMdMhLqSUI7Q82xb6hw/https%3A%2F%2Fcolab.research.google.com%2Fgist%2FSaduf2019%2Faac88f6c369b8d569071c5929e44cf48%2Funtitled367.ipynb.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://secure-web.cisco.com/1E4m66aWdZdg2xm2BgY8h5wr28oREcY7rJlmwNHvTlFw2nO46AYAd7dhmYwUD32_dk-AoKUpccuBJ1zghlRS4cGiHdBXbLdF-I4QZgYjgdYN4b8gRSgXqwHbhIo2uqKiGpn4d35tfLoyREcYP1ZYnHC5mltSbdW2LBp3MmZqljxgLfR13WuBF7iAhGERgStk9hzn0XLuY_eKuzJv7jtqZcn2yrpPQsSi7URhoENzfXrGaoNT83OEqvRtlRr7Aog_CFBHt2_vOIEpEx763MyNmDpCycoH9OsqfTRGCWAemACzd1Dp7-l8F_6ytBEs6VOuJ9ip-kIGHefpf_zf_CIcR7Q/https%3A%2F%2Fgithub.com%2Ftensorflow%2Ftensorflow%2Fissues%2F42398%23issuecomment-675421983, or unsubscribehttps://secure-web.cisco.com/1ncuZ-OAFmyTrwq_pw-gxL9qLwhX80cY8Q3XGFgCmCltlykctyf5ijUQ9YpqAPBT6L2xAFo5W_YIMz_o8ZfgxQ4CROC6muUbQT7MWh9cFxhEL2pe3HpySX1TUHaoL-fq7qn5YMvZWFIoXG7ZS9yS3vRtHssNVaGsJItbB2tDQJ1GFcKU1iXdjBgoKHH_-aA9irTPj0gaAG8HwTwiNPw8SEvsSP7aFtyUvsG64dyODtICg6uwkAgJrRqOsQap2__ztqTWam2J5CGKhHG-cQtV8Xg558lXjzlKkhmEHR6T-A8Zb56DzR7U02FvoKDzRfnzS2M9kF5EY-ximc6lA6I5daQ/https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAETBXODUD3GA4XKXA7IUMO3SBJQKXANCNFSM4QANRIQQ.

@jeball
Please try latest version of tf and let us know if the issue exist.

Last I knew TF 2.1.0 was the latest version supported on Windows 10. I will try TF 2.3.
I still have a question – if I install via Conda will it work or does it require a pip install?
Thanks
John Ball, Ph.D.
Associate Professor and Robert Guyton Endowed Chair for Teaching Excellence | Electrical and Computer Engineering
<EMAIL_ADDRESS>Associate Editor | IEEE Signal Processing Letters
Associate Editor | Journal of Applied Remote Sensing
Editorial Board Member | MDPI Electronics
Faculty Advisory Council | Transactions on Stem Education
Lab Director | Simrall Radar Laboratory
Lab Co-Director | Center for Advanced Vehicular Systems (CAVS) Sensor Lab
Research Fellow | Geosystems Research Institute (GRI)
Simrall 233, 406 Hardy Rd., Mississippi State, MS 39762
p:662.325.4169  e:jeball<EMAIL_ADDRESS>From: Saduf2019<EMAIL_ADDRESS>Sent: Wednesday, August 19, 2020 9:44 AM
To: tensorflow/tensorflow<EMAIL_ADDRESS>Cc: Ball, John<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [tensorflow/tensorflow] TFA can't load _image_ops.so - says file doesn't exist, but its there? (#42398)
@jeballhttps://secure-web.cisco.com/1PyB5G81NqPq9F_ni-OsoXOJZ9A533-gi3irHU1H4yjkP-Zb9wnXzTgocslEFvKrUo564k83gPWaHCL4yQbnOcLfCs0c9uhXNZB6sF0R399Pm7k_yUr9Fb39jdTrbIagVAPaNLrGyYyhIvBCtTjW8khebiYU4r46onZ9NpG0nbWL9y0iTqcp6ulME0ybfQK0qJyoBcejoZGeK70M6FGzfDAilmu4ZkVw2F5xuMBXD3lYnkn0t_-NAGl9QIgwm69Ve2zD3KvKDvQ4rPpGn-4mbshADVFit-8fNq9oBi4OYBIly43JKFMNsfOyZ8fM9fO5_coA7gnQC-s8tiRJiLPO5KA/https%3A%2F%2Fgithub.com%2Fjeball
Please try latest version of tf and let us know if the issue exist.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://secure-web.cisco.com/10SU9AVYOjJxqShETogFAW9E4dLhGH-k0pToRybLaNOYBYxKGxlUmFscepC1PXA5s2ClFPfLUCFwOfXCsSWrCF2ZQvXyZ6sal1xGWJNlp6R05zqWpviF9G04P5tr1JlG1teX3alr5YWMxbpoe1Wx-u5-q5t8ChjQe-Vv3moe5iOqosQ_m--_i2iX_Ri9TY9aaNhJfV1jxm8PpA8Mdik4Yaqvmu8rkwbBzGAhyRbWcH2N3vf7S5vO4UBTPkVuXVCpb05FO9POXWMBWow0VGWvRAiKplW5tGLK1V65CdV3WcNcy91dZhoJdkteS9K8XgP_T0U4kEKMiyeoGQpXJAYhtIw/https%3A%2F%2Fgithub.com%2Ftensorflow%2Ftensorflow%2Fissues%2F42398%23issuecomment-676470335, or unsubscribehttps://secure-web.cisco.com/1c3wyqohvsQkRG0Fk1eAhA5VBZZIvuTEV5eepG608iNTjFvzB-Nv1biKLa8Elwh34YtDvWHjCHax5MaBRPoD0rPDxdElMZfG_KzwcjsfCLKSd0aR55kbJ4TFk3izW5OEiJIZ3p7tTj2FMgmq7r_C6HpYwyimWtCTumdD4G8zCTAGruI6KUQ5wpzw6EKSewm44z8GnDPxf8DVrfAlGpnCHIOLaMpQs-Fg3Pfq_HqP6K3woYfFoAMo1BotatseZyhEGY7C3GCvyVmHETXc7DwuSseZwl3G206QFZFnwGmyN8emCVd0DohKkrOUj_kcdZbsWdP4_u3v6HjKlIWC6hCEoGQ/https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAETBXOBCTBSRRKMZ2CDJ3KTSBPQLFANCNFSM4QANRIQQ.

@jeball

I still have a question – if I install via Conda will it work or does it require a pip install?
You can go ahead with either of them, and let us know.


Is there a problem with me creating a PR to add support for ESP_BLE_AD_TYPE_NAME_SHORT = 0x08

Guys,
just before I do it i just wanted to make sure there was no issue in me creating a PR to add support for the ESP_BLE_AD_TYPE_NAME_SHORT type in BLEAdvertisedDevice. The devices that I am using use this field and it would make my down stream processing a lot easier.  If not I will hack because i really don't want to fork
Thanks

Feel free to make PR

Cool will do - now I have to learn how to do the fork and create the pull request :-)


Allow zfsnap.sh to be renamed

Removed hardcoded file name, which would allow for the script to be renamed - i.e. to 'zfsnap'

Thanks for the patch. I modified the patch slightly to avoid the use of $() which is non-portable.
In the end, I'd love to get rid of readlink, since it's not POSIX and isn't available on Solaris w/o GNU. But I wasn't able to solve it quickly. :-/
---Alex


Feat/region activity

Added module for social risks. Used ruff default linter.

lgtm
just a small comment, it would be nice if last changes from dev branch would be incorporated. By this I mean the new module structure

btw it is also would be nice to add to gitignore any other files that are not the initial sampled data. I dont think we actually need all that csv files in the repo

I will close this branch and add changes together with other changes merged from other branches

I am not convinced that current module structure (here is used the old one) is still appropriate
could you match it to the latest modules structure?

also please clean your temporary example file in the examples folder

the regional activity example contains errors and undeclared vars :(


ThreatStream: Bug Fix - PAPP-26387: bug fix for whois domain action

Pull Request Checklist
Please check if your PR fulfills the following requirements:

[x] Testing of all the changes has been performed (for bug fixes / features)
[x] The readme.html has been reviewed and added / updated if needed (for bug fixes / features)
[x] Use the following format for the PR description: <App Name>: <PR Type> - <PR Description>
[x] Provide release notes as part of the PR submission which describe high level points about the changes for the upcoming GA release.
[ ] Verify all checks are passing.
[x] Do NOT use the next branch of the forked repo. Create separate feature branch for raising the PR.
[x] Do NOT submit updates to dependencies unless it fixes an issue.

Pull Request Type
Please check the type of change your PR introduces:

[ ] New App
[x] Bugfix
[ ] Feature
[x] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation
[ ] Other (please describe):

Security Considerations (REQUIRED)

[ ] If you are exposing any endpoints using a REST handler,
please document them in the readme.html.
[ ] If this is a new connector or you are adding new actions

[ ] Please document in the readme.html all methods (eg, OAuth) used to authenticate
with the service that the connector is integrating with.
[ ] If any actions are unable to run on SOAR Cloud, please document this in the readme.html.

[ ] Are you introducing any new cryptography modules? If yes, please elaborate their purpose:
[ ] Are you are accessing the file system? If yes, please verify that you are only accessing paths returned through
the Vault API.
[ ] Are you are marking code to be ignored by Semgrep with nosemgrep?
If yes, please provide justification in an additional comment next to the ignored code.

Custom pipeline link:
All tests: https://cd.splunkdev.com/phantom-apps/app-tests/-/jobs/39004920
Only cloud playbooks: https://cd.splunkdev.com/phantom-apps/app-tests/-/jobs/38506434
As there are many playbooks for the app and some of them are flaky, we have run two pipelines and the playbook failing in 1st pipeline is passing in the second one.


allow edge to gc chunks

Try uploading > 5GB with edge. It does not matter if chunks are used or not. Edge will kill the tab process once it hits 5GB. This can be monitored quite nicely when keeping the EDGE dev tools memory tab open. When the process is killed the tools get closed. Sadly, memory is not even freed after an upload completes.
This PR allows edge to cleanup chunks. Currently, by only omitting the actual upload data from the options used to initialize the progress. Maybe it makes more sense to always omit it ... my js is not good enough to wrap my head around that.
Comments welcome.

Thanks for your contribution @butonic
As far as I understand you, you're encountering a memory leak with Edge and file upload progress reporting.
The part that I don't understand is how your code snippet is supposed to fix things with this snippet:
var clone = Object.assign({}, o);
delete clone.data;

You're assigning all option properties to a new object and then remove the data property from that new object.
Since you're not using this new clone object it should be simply dismissed for garbage collection by the browser at the end of the function block.
And since you're not mutating any other objects (o will be unchanged), I fail to see how this can have an impact?
Could you confirm that this change fixes the memory leak on edge?
And if yes, do you have an explanation for this behavior?
Sorry but as of right now, I fail to see how this fix could have any actual impact, despite maybe triggering undocumented behavior in edge.

updated pr incoming ... I forgot to actually use clone as a parameter to _initProgressListener.
That method seems to indirectly reference the object by itself, or rather by an anonymous handler listening on progress events.

AFAICT _initProgressListener() is the culprit:
https://github.com/blueimp/jQuery-File-Upload/blob/a451d60bb8034e2736082e24cf7aa7f9dbed5b8a/js/jquery.fileupload.js#L417-L435
In 419 xhr might be taken from the options.
If xhh.upload exists line 423 will bind an anonymous function to 'progress' events of the xhh.upload object.
Finally, line 431 replaces the options.xhr with another anonymous function, creating another indirection.
Im not too much into garbage collection, but I have a feeling that the edge implementation does not yet handle that good enough.

hm, indeed, making the options.xhr replacement conditional also allows the objects to be garbage collected.
--- apps/files/js/jquery.fileupload.js	(date<PHONE_NUMBER>000)
+++ apps/files/js/jquery.fileupload.js	(date<PHONE_NUMBER>000)
@@ -428,9 +428,11 @@
                     e.total = oe.total;
                     that._onProgress(e, options);
                 });
-                options.xhr = function () {
-                    return xhr;
-                };
+                if (options.xhr && options.xhr() !== xhr) {
+					options.xhr = function () {
+						return xhr;
+					};
+				}
             }
         },

hm the condition is wrong, let me see if I can polish this.

debugging shows:

options.xhr is unset in 419
commenting lines 431-433 allows gc to happen, upload still completes successfully

@blueimp Why is the xhr added to the options when initializing the progress listener?
It was added with the rewrite 7 years ago. It seems back then handleGlobalProgess had the xhr as a parameter, but it was never used:
https://github.com/blueimp/jQuery-File-Upload/blob/8ae0b060acb3e08ad745908c28b29030581bddc4/jquery.fileupload.js#L211-L234
Also if settings.onProgess wes a function it would be called with the xhr in handleLoadEvent
https://github.com/blueimp/jQuery-File-Upload/blob/8ae0b060acb3e08ad745908c28b29030581bddc4/jquery.fileupload.js#L246
and handleProgressEvent
https://github.com/blueimp/jQuery-File-Upload/blob/8ae0b060acb3e08ad745908c28b29030581bddc4/jquery.fileupload.js#L284
Access to a .xhr property is nowhere to be found in https://github.com/blueimp/jQuery-File-Upload/tree/b0b003c74cfbccf5801588602d147a6256d10e17 , the jQuery UI Widget rewrite.
Today, the only place that might rely on the objects having the xhr property is
https://github.com/blueimp/jQuery-File-Upload/blob/a451d60bb8034e2736082e24cf7aa7f9dbed5b8a/js/cors/jquery.postmessage-transport.js#L66
After fiddling around a bit, it seems that just cloning the options already allows edge to gc the chunks.
Currently, this PR should allow code relying on options.xhr() to continue to work. While getting here I was under the impression that  options.xhr should just not be set. But to be safe and backwards compatible it seems this is the right workaround.

It seems to be a variant of https://blog.meteor.com/an-interesting-kind-of-javascript-memory-leak-8b47d2e7f156

the options are also referenced by the jquery cache ... so maybe https://makandracards.com/makandra/31325-how-to-create-memory-leaks-in-jquery is related .... too late to dig further. will have another look tomorrow.

it seems I can clean this up on our side:
				fileupload.on('fileuploadchunkdone', function(e, data) {
					$(data.xhr().upload).unbind('progress').remove();
				});

will test tomorrow ... really getting too late

Ok, so actually unbinding the progress listener is enough to let gc happen:
				fileupload.on('fileuploadchunkdone', function(e, data) {
					$(data.xhr().upload).unbind('progress');
				});

The cleanest way I could come up with was using always() to remove the progress listener. Updated this PR accordingly, Maybe this saves others some time, trying to figure out why uploads fail randomly.

Thanks a lot for your investigation and fix, I've included a slightly modified version of it in the latest release.
And sorry for the late response. :|


Native transformation of Arel to ES AST

So I've nearly fallen in love with the approach you're taking with Chewy.
One piece I'm still hung up about is first class transformation between Arel and ES AST.
I'd like to do something similar to ...
foo = Foo.arel_table.(some long chain of arel)
es_foo = foo.as_es_ast chewy_type: ESFoo
es_foo.filter

Is this (1) already a feature that I just haven't discovered in Chewy, (2) feasible but not implemented or on the roadmap, (3) infeasible?
Kindest regards.
Great work.

Actually, this seems to be a good idea, it would be better to implement it like:
UsersIndex.filter(...).merge(User.some_scope_or_arel)
UsersIndex::User.filter(...).merge(User.some_scope_or_arel)
# User is an AR model here

But since ES and AR functionality is quite different, the implementation would be really limited, I suppose. So, this is partially feasible and it was not on the roadmap, feel free to try.


Istio reconciler support istio 1.13.x

Description
Implement necessary changes to install / upgrade istio in 1.13 version. For this story we do not need multiple version support.

[ ] istioctl 1.13.x is used for installation / upgrade
[ ] istio webhook patching updated if needed
[ ] gardener cluster can be hibernated
[ ] istio running in 1.13.x version
[ ] sidecars running in 1.13.x version
[ ] reconciler bumped in kyma cli

Reasons
Keeping Istio up to date
Attachments
https://github.com/kyma-project/test-infra/issues/5109

Tested installation with images available after bump from here:

https://github.com/kyma-project/test-infra/pull/5202

Local Istio chart config that got adjusted for testing purposes looked this way:
# the rest of the config
# ...

global:
  containerRegistry:
    path: eu.gcr.io/kyma-project
  images:
    # these definitions aren't used by the chart, but they are required by external tool for list generation
    istio_proxyv2:
      name: "proxyv2"
      version: "1.13.2-distroless"
      directory: "external/istio"
      containerRegistryPath: "eu.gcr.io/kyma-project"
    istio_pilot:
      name: "pilot"
      version: "1.13.2-distroless"
      directory: "external/istio"
      containerRegistryPath: "eu.gcr.io/kyma-project"
  tracing:
    enabled: true

So far so good. I did not encounter any problems with installation or upgrade:
{"level":"INFO","time":"2022-04-07T13:16:10.853+0200","message":"Istio in version 1.13.2-distroless successfully installed"}
{"level":"INFO","time":"2022-04-07T13:16:16.275+0200","message":"Component 'istio' has status 'running'"}
{"level":"INFO","time":"2022-04-07T13:16:27.154+0200","message":"Found 1 pods with different istio proxy image ({istio/proxyv2 1.13.2-distroless-distroless})"}
{"level":"INFO","time":"2022-04-07T13:16:41.972+0200","message":"Proxy reset for 1 pods successfully done"}
{"level":"INFO","time":"2022-04-07T13:16:41.972+0200","message":"Component 'istio' has status 'success'"}

cli bump waits for #1031

CLI bumped: kyma-project/cli#1244


Add filter by price in collection page

It's already have, why need to add? @thienvip107

update pls check @thienvip107


.net project output {} when serializing the model in publish mode (v9-v13.0.3)

User userInfo = new User { id="11011", name="zhangsan", age=12,sex= false,address= "sicuan" };
Console.WriteLine(userInfo.id);
string json=JsonConvert.SerializeObject(userInfo);
Console.WriteLine(json);

Since i don't know what settings you used for the publish mode, i would ask you to narrow down the problem space.
First, in case you use AOT compiling, disable it and see whether the json serialization will start behaving.
If not, also disable code trimming for publishing and see whether this will help.
If that doesn't seem to make the code behave either, you'll need to tell us more details about your problem: What is the build/platform target you are publishing for? What are the publish settings you are using? What build environment/tools are you using?

The problem found is the use of 'Obfuscar' plugin caused by the bug

The problem found is the use of 'Obfuscar' plugin caused by the bug

I just had a similar problem, I was using JObject.ToString() and was also getting "{}" as the output. Disabling "Trim unused code" in publish options fixed the problem.
Publish options were Release | Any CPU, net7.0, Self-Contained, linux-x64, produce single file


Fix R

Motivation for this change
R was broken on aarch64 because

It now seems to expect java to exist
A recently added regression test fails on aarch64.

ZHF: #80379
Things done

Removed the conditional exclusion of jdk as a dependency on aarch64 (and aarch32), as aarch64 now has openjdk.
Patched the sources to remove a test which fails on aarch64 (Reported upstream. Believe test to be incorrect).

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

If preferred, we could only apply the patch on platforms where the test fails.

I don't know R, but I think it's a bug that on aarch64 there are no warnings. Asking for 2^64 %% 3 is nonsensical as at this point it's not possible to represent integers with double floating point numbers (you can only do up to 2^53 - 1).
I think it could make sense to limit disabling this test to aarch64.

The calculation is done with long double on aarch64

Although that's probably not relevant, since the value has already lost precision before the mod is calculated, so there should be a warning.
I do still think that it makes sense to suppress the test until it's fixed upstream. It's arguably no worse than the version before the test was added.


Future drop is in the past drops

Context
I was looking for turbulent drop and found it as a past drop

However per DB it will be dropped 17th of August

Task
make sure that future drop is considered as future drop, not in the past section

https://koda.art/base/drops/turbulent

seems to be fixed

seems to be fixed


ESP32-S3 PSRAM Not Maintained

Hi,
while experimenting with the MMU (tried mmaping flash) I noticed that the PSRAM on the ESP32-S3 does not predictably maintain the set value.
I would assume it is only maintained while it is in cache.
One can replicate the results by modifying the PSRAM example as shown below.
The example also seems to be broken in its default state (Exception occured 'LoadProhibited' when creating the string)
I tested this on two ESP32-S3s with 8MB of PSRAM.
Is this reproducible or did I misinterpret how it can be used?
Thanks for any hints.
    ...
    println!("Going to access PSRAM");

    unsafe {
        let start = psram::psram_vaddr_start() as *mut u8;
        for i in 0..psram::PSRAM_BYTES {
            *start.add(i) = 0xAB;
            if(*start.add(i) != 0xAB) {
                println!("failed set");
            }
        }
        for i in 0..psram::PSRAM_BYTES {
            if(*start.add(i) != 0xAB) {
                println!("failed check");
            }
        }
    }

    loop {}

What exact ESP32-S3 boards are that?
Background: there are different PSRAM chips and we don't support all of them. (Probably more a thing for older ESP32)
Might your PSRAM be octal-SPI-RAM?
If the example already fails unmodified then something is quite wrong

Ok, sorry for causing a fuss here, turns out I just misread the datasheet.
I was most testing on a https://www.lilygo.cc/products/t5-4-7-inch-e-paper-v2-3, which has a ESP32-S3-WROOM-1-N16R8m, which actually uses octal-SPI RAM. With the correct RAM type selected this issue does not not exist.

Thanks for confirmation! Great to hear it's working for you
I guess we can close this, then


Propel inserts twice

When using sqlite, rows get inserted twice.
Here's my setup,
composer.json
"propel/propel": "~2.0@dev"

schema.xml
<table name="usertype" phpName="Usertype">
  <column name="id" phpName="Id" type="integer" required="true" primaryKey="true" autoIncrement="true"/>
<column name="name" phpName="Name" type="VARCHAR" required="true"/>
</table>

test.php
$usertype = new Usertype;
$usertype->setName('X_' . time());
$usertype->save();

sqlite select after running test.php
sqlite> select * from usertype;
1|X_1426856872
2|X_1426856872

Update : When running in sandbox, it correctly inserts one row.
See it here : http://sandbox.propelorm.org/9d8003c
Thanks!

Hard to say what the error here really is, since we can not reproduce it easily. Does it happen only with SQLite?

Figured it out and Propel has nothing to do with it. Sorry for wasting your time :(
I recently switched from using mod_rewrite to 'fallbackresource' under Apache.
When using 'fallbackresource', all calls to non-existing files are routed to a specific file usually index.php.
A non-existing favicon triggered a second call to index.php which resulted in a duplicated row.


How to change the size of the block?

Hello sir
My project uses blockly. The block size looks too big and how to change the size of the drawing to make it look more beautiful

Blocks in the workspace can change size by modifying the zoom levels in VirtualWorkspaceView.java. We don't currently have an API to change block sizes globally, so renaming this into a feature request.


Alias column selection in group_by_columns to enable timestamp wrapping

Describe the feature
group_by_columns does not allow you to group by higher aggregates. For example, if I have a timestamp column, I might want to group by date_trunc('week', timestamp) to monitor that values are expected each week.
Right now, if my yml is set to:
- dbt_utils.not_null_proportion:
      at_least: 0.95
      group_by_columns: [date_trunc('week', timestamp)]

The code compiles to:
select
      count(*) as failures,
      count(*) != 0 as should_warn,
      count(*) != 0 as should_error
    from (

with validation as (
  select
    date_trunc('week' ,timestamp),
    sum(case when visit_reasons is null then 0 else 1 end) / cast(count(*) as numeric) as not_null_proportion
  from <my_table>
  group by date_trunc('week',timestamp)
),
validation_errors as (
  select
    date_trunc('week' ,timestamp),
    not_null_proportion
  from validation
  where not_null_proportion < 0.95 or not_null_proportion > 1
)
select
  *
from validation_errors
    ) dbt_internal_test

which, as you can see, in the validation_errors cte does not work. That said, if we just aliased the column in the validation cte and referenced it in validation_errors, we'd be all set. My suggested change in macros/generic_tests/not_null_proportion.sql:
with validation as (
  select
    {% for c in select_gb_cols %}
    {{select_gb_cols}} as group_col_{{loop.counter}}{% if not loop.last %},{%endif%}
    {% endear %}
    sum(case when {{ column_name }} is null then 0 else 1 end) / cast(count(*) as {{ dbt.type_numeric() }}) as not_null_proportion
  from {{ model }}
  {{groupby_gb_cols}}
),
validation_errors as (
  select
   {% for c in select_gb_cols %}
    group_col_{{loop.counter}}{% if not loop.last %},{%endif%}
    {% endear %}
    not_null_proportion
  from validation
  where not_null_proportion < {{ at_least }} or not_null_proportion > {{ at_most }}

Describe alternatives you've considered
I could make my own custom test to do this
Additional context
Don't think database will impact anything here
Who will this benefit?
anybody trying to group by time
Are you interested in contributing this feature?
I hope the above is enough to get started
Let us know if you want to contribute the feature, and whether would need a hand getting started

My team and I would need this feature to assess data quality on a day-by-day basis for data that we receive every hour thanks to a DATE_TRUNC('day', received_date) !


One-line command to avoid missing paths

Following the description from https://github.com/tiredofit/docker-osticket/issues/25 this is the update to fix the last error in the build docker process.

Thanks!


07/02/2018 22:15 : Create Pull Request for branch master -> prTesting1_2018070222

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

split/updateTestRef1.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Added minder provider update CLI command.

Summary
Update command allows changing configuration for provider on a per-field basis. It operates on a read-modify-write fashion, retrieving current configuration from the backend for modification.
Modification is done by reflection by walking over the struct's JSON tags. Implementation is recursive, but the maximum depth is determined by the deepest field in the configuration struct.
Argument parsing is trivial and assumes that arguments are either of the form field1.field2.field3 (for --unset-attribute) or field1.field2.field3=value (for --set-attribute). The right parser for value is determined once the correct struct field is found.
It is not currently possible to modify "github-app" or "github" fields.
Fixes #3509
Change Type
Mark the type of change your PR introduces:

[ ] Bug fix (resolves an issue without affecting existing features)
[X] Feature (adds new functionality without breaking changes)
[ ] Breaking change (may impact existing functionalities or require documentation updates)
[ ] Documentation (updates or additions to documentation)
[ ] Refactoring or test improvements (no bug fixes or new functionality)

Testing

unit tested
locally tested
tested against SAAS (staging)

Review Checklist:

[X] Reviewed my own code for quality and clarity.
[X] Added comments to complex or tricky code sections.
[ ] Updated any affected documentation.
[X] Included tests that validate the fix or feature.
[X] Checked that related changes are merged.

coverage: 53.115% (-0.01%) from 53.125%
when pulling a3229ac5d4a210e358739afa794631b73705058b on update_prov_cli
into cfeaed8b8ae83889018cd2ee3f9cac43e31df871 on main.

coverage: 53.12% (-0.005%) from 53.125%
when pulling a3229ac5d4a210e358739afa794631b73705058b on update_prov_cli
into cfeaed8b8ae83889018cd2ee3f9cac43e31df871 on main.

coverage: 52.78% (-0.3%) from 53.125%
when pulling 2df5588047410d491875d21562364c2bdc9fadd3 on update_prov_cli
into cfeaed8b8ae83889018cd2ee3f9cac43e31df871 on main.

coverage: 52.775%. remained the same
when pulling 103448c0c593990cb951d35e988f8d8d21cf44d4 on update_prov_cli
into 7a3bce6e3403dcc7d9d710f2643c960e6675c418 on main.


Diagnostics docs re #204

New docs to describe how to add diagnostics to LFRic atmosphere.

Fixed the requested issues. Ready for review again.


(WIP) New mechanism for colored stdout on terminal

Description
Using the termcolor package, this PR would make the raised exceptions appear red on the terminal with no changes to the background color.
Added the termcolor package in the requirements.txt file and compatible with both Python 2 and 3.
Fixes #2506

no more maintained


Associate Issue with Project

Description:
This issue is created to associate the current task with Project ID 12. The goal is to ensure that all relevant discussions, updates, and progress related to this task are tracked within the project.
Please provide any specific details or requirements that need to be addressed in this project association.

Testing


Error in calculating overlap

Hello, I have a problem when I use your cal_overlap.py file to get a point cloud with a degree of overlap higher than 30%. I only get a file with a size of 5b. It feels like I have not read the ply point cloud or fragments. Thanks.

Total 7 scenes, 0 point cloud fragments.
Load pts file from ~/data/3DMatch/raw_data/cal_over/
Reload overlap info from ~/data/3DMatch/raw_data/cal_over/3DMatch_train_0.300_overlap.pkl
Reload keypts info from ~/data/3DMatch/raw_data/cal_over/3DMatch_train_0.300_keypts.pkl
Begin processing scene sun3d-home_at-home_at_scan1_2013_jan_1
Finish sun3d-home_at-home_at_scan1_2013_jan_1, Done in 0.0s
Begin processing scene sun3d-home_md-home_md_scan9_2012_sep_30
Finish sun3d-home_md-home_md_scan9_2012_sep_30, Done in 0.0s
Begin processing scene sun3d-hotel_uc-scan3
Finish sun3d-hotel_uc-scan3, Done in 0.0s
Begin processing scene sun3d-hotel_umd-maryland_hotel1
Finish sun3d-hotel_umd-maryland_hotel1, Done in 0.0s
Begin processing scene sun3d-hotel_umd-maryland_hotel3
Finish sun3d-hotel_umd-maryland_hotel3, Done in 0.0s
Begin processing scene sun3d-mit_76_studyroom-76-1studyroom2
Finish sun3d-mit_76_studyroom-76-1studyroom2, Done in 0.0s
Begin processing scene sun3d-mit_lab_hj-lab_hj_tea_nov_2_2012_scan1_erika
Finish sun3d-mit_lab_hj-lab_hj_tea_nov_2_2012_scan1_erika, Done in 0.0s

I carefully studied the file that should be missing the beginning of seq, what is this?

The pose.npy file should be missing, how can I get it? thanks

Hello, I have solved this problem by downloading the pose file in txt format from the official website and converting it into a matrix. But I don't quite understand why this rotation matrix is being multiplied. Thanks.

I carefully studied the file that should be missing the beginning of seq, what is this? It looks like a sequence file.

Hi, I met the same problem. How did you solve it? As so far, I suppose the seq_ is just the naming rule of training dataset folder.


mpfr: fix download link

Closes gh-2448.

good job on finding the old releases on their website!

Thanks!


python310Packages.angr: 9.2.32 -> 9.2.33

Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 210103 run on x86_64-linux 1

16 packages built:

python310Packages.ailment
python310Packages.angr
python310Packages.angrcli
python310Packages.angrop
python310Packages.archinfo
python310Packages.claripy
python310Packages.cle
python310Packages.pyvex
python39Packages.ailment
python39Packages.angr
python39Packages.angrcli
python39Packages.angrop
python39Packages.archinfo
python39Packages.claripy
python39Packages.cle
python39Packages.pyvex


cleanup gregor

@keybase/react-hackers @zanderz @akalin this cleans up gregor

[x] modernizes saga
[x] moves all this git/kbfs/team/chat stuff out of gregor and into their own sagas
[x] removes some deprecated folder stuff (cc: @jzila)

phew, thanks Nojima! I tried it, LGTM :thumbsup:

@songgao for the favorites changes

@zanderz can you try with this lateest commit of the kbfs service check stuff . i had to move it


Easily create custom form operations (using HasForm trait and general form view)

WHY
Note: This PR points to https://github.com/Laravel-Backpack/CRUD/pull/5118 - it needs that to be merged first.
BEFORE - What was wrong? What was happening before this PR?
It was clunky and difficult to create custom operations that included Backpack forms. We have a tutorial for it but it's a multi-step process, and after following it you still end up with non-working code that you need to fix yourself.
AFTER - What is happening after this PR?
It is MUUUUCH easier to create custom operations that include Backpack forms. Especially the most common ones:
Step 1. Create a new operation trait using php artisan backpack:crud-operation Email.

Note: This can be made even easier, by creating a new command with a stub specifically for this kind of operations: php artisan backpack:crud-form-operation Email (TODO - could/should).

Step 2. In the CrudController where you want that operation, use this new operation trait and add the fields you want in its setup method:
class UserCrudController extends CrudController
{
    // ...
    use \App\Http\Controllers\Admin\Operations\EmailOperation;
    // ...

    protected function setupEmailOperation()
    {
        CRUD::field('to')->type('email')->size(4);
        CRUD::field('cc')->type('text')->size(4);
        CRUD::field('bcc')->type('text')->size(4);
        CRUD::field('subject')->type('text');
        CRUD::field('message')->type('textarea');
    }

Step 3. Inside EmailOperation.php use the brand-new HasForm trait and the methods it provides:
<?php

namespace App\Http\Controllers\Admin\Operations;

use Illuminate\Support\Str;
use Backpack\CRUD\app\Library\CrudPanel\CrudPanelFacade as CRUD;
use Backpack\CRUD\app\Http\Controllers\Operations\Concerns\HasForm;

trait EmailOperation
{
    use HasForm;

    /**
     * Define which routes are needed for this operation.
     *
     * @param string $segment    Name of the current entity (singular). Used as first URL segment.
     * @param string $routeName  Prefix of the route name.
     * @param string $controller Name of the current CrudController.
     */
    protected function setupEmailRoutes($segment, $routeName, $controller)
    {
        $operationName = Str::of(__FUNCTION__)->after('setup')->before('Routes');
        $routesHaveIdSegment = true;

        $this->formRoutes($operationName, $routesHaveIdSegment, $segment, $routeName, $controller);
        // this will set up two routes:
        // - GET that points to getEmailForm()
        // - POST that points to postEmailForm()
    }

    /**
     * Add the default settings, buttons, etc that this operation needs.
     */
    protected function setupEmailDefaults()
    {
        // Access
        CRUD::allowAccess('Email');

        // Config
        $this->crud->operation($this->crud->getCurrentOperation(), function () {
            $this->crud->loadDefaultOperationSettingsFromConfig();
            $this->crud->setupDefaultSaveActions();
        });

        // Button
        $this->crud->operation('list', function () {
            $this->crud->button('email')->stack('line')->view('crud::buttons.quick');

            // this will add a quick line button, but you can also
            // configure how it looks and works by passing parameters:
            // $this->crud->button('email')->stack('line')->view('crud::buttons.quick')->meta([
                // 'access' => 'Email',
                // 'label' => 'Email',
                // 'icon' => 'la la-envelope',
                // 'classes' => '',
                // 'wrapper' => [
                    // 'href' => url($this->crud->route . '/email'),
                    // 'class' => 'btn btn-sm btn-link',
                    // 'target' => '_blank',
                // ],
            // ]);

        });
    }

    /**
     * Method to handle the GET request and display the View with a Backpack form
     *
     * @param  int  $id
     * @return \Illuminate\Contracts\View\View
     */
    public function getEmailForm(int $id = null)
    {
        $this->crud->hasAccessOrFail('Email');

        return $this->getForm($id);
    }

    /**
     * Method to handle the POST request and perform the operation
     *
     * @param  int  $id
     * @return array|\Illuminate\Http\RedirectResponse
     */
    public function postEmailForm($id = null)
    {
        $this->crud->hasAccessOrFail('Email');

        // You logic goes here...
        dd('got to ' . __METHOD__, request()->all());

        return true;
    }
}

HOW
How did you achieve that, in technical terms?

added a new trait HasForm, that custom operations can use to quickly do stuff they needed to do anyway (create the GET and POST routes, show the form);
added a quick.blade.php button that is PERFECT for both form use cases (general-purpose form with top button and item form with line button);
added a new inc/form.blade.php view, which is a general-purpose version of create.blade.php and update.blade.php (those had quite a few things hardcoded);

Is it a breaking change?
Completely non-breaking. But it can be turned into a breaking change, by:

🟡 changing the Create and Update operations to use this new form.blade.php (nice!);
⚫ changing the Create and Update operations to use he HasForm trait (probably overkill);
🔵 moving form_content.blade.php to inc/form_content.blade.php so that the crud directory only contains direct operation views; all partials would then live in inc;

How can we test the before & after?
Before - the steps in https://backpackforlaravel.com/articles/tutorials/how-to-create-a-custom-operation-that-uses-backpack-fields
After - the steps outlined above

TODOs (keeps it non-breaking):

🔴 what do we do about save actions? maybe custom forms should just have "save and back" by default;
🔴 are any config values set up by default? should they?
🔴 if the developer creates a config file, what name should it have? does the operation really pick it up?
🔴 I noticed the operation names have the first letter in capital by default... why? is that standard?
🔴 test multiple likely scenarios to see that it fits them well;
🟡 can we make setupEmailRoutes() shorter or easier to write?
🟡 can we make setupEmailDefaults() shorter or easier to write?
🔵 create a new command to generate a form operation - php artisan backpack:crud-form-operation Email;

TODOs (turns it into breaking):

🟡 change the Create and Update operations to use this new form.blade.php;
⚫ change the Create and Update operations to use the HasForm trait (probably overkill);
🔵 move form_content.blade.php to inc/form_content.blade.php so that the crud directory only contains direct operation views; all partials would then live in inc;

@pxpm I want to finish this up tomorrow morning. Do you see any reason this approach wouldn't work well for some realistic 80% use cases? What do you think overall? Any concerns?

Hey Cristian.
This is very similar to the approach I started but I was going a little bit further, and I think here we will need to go a little bit further too to fix some issues or provide a little better DX.
Let's talk about a real scenario:
BanOperation - An operation that allow you to ban entries, in this case users.

It should provide a "quick ban" inline button that will ask only for the reason and ban the user for 1 week.
a general form to "ban users" where admin can select one or more users and provide the reason and duration of the ban. (top button)
an inline button to view the ban history of a specific user (no form)

So I am assuming here that the BanOperation will have multiple forms, and not only one form, in that regard I think the routing/form definition could be improved, just not 100% sure  how:
I was looking to define something like this for routes:
trait BanOperation
{
    use HasForm;

    protected function setupBanRoutes($segment, $routeName, $controller)
    {
        // this make sense for forms, so can be provided by HasForms
        // the first the form to ban a single user:
        $this->setupFormRoute('ban-system', 'id'); // admin/user/ban-system/{id}/ban [GET/POST]
        // and then a general "ban users" form
        $this->setupFormRoute('ban-system');// admin/user/ban-system/ban [GET/POST]

        // for "consistency" we can optionally have a similar approach for regular urls that work in 80%+
        $this->setupRoute('history', 'id'); // admin/user/history/{id}/ban [GET]

        // as examples
        $this->setupRoute('ban-system/ban-history', 'id', 'POST'); // admin/user/ban-system/{id}/ban [POST]
        $this->setupRoute('ban-system'); // admin/user/ban-system/ban [GET]

        // notice the usage of named arguments to omit some of them
        $this->setupRoute('ban-system', method: 'POST');

        // any complex scenario can still use Route facade directly
        Route::post();
    }
}

We maybe use a little bit of PHP magic here to ease developer life a bit and automatically get the function parameters with debug_back_trace() ($segment, $routeName, $controller)  in the setup routes methods without explicitly passing them?
Imagine that this is all good and working, we got the routes setup. AFAIK, this is a non-bc.
Next thing in line is the configurations, mainly the default ones.

save actions: I agree with you, we should only probably have 1 save action in the form.
as a developer I expect to create a config/backpack/operations/ban.php and that it would be picked as my config file.

protected function setupBanDefaults()
    {
        // what users can do in this operation
        CRUD::allowAccess(['view-user-bans', 'ban-users', 'ban-user']);

        // I don't expect to be creating the "save form button" etc, so the HasForms trait can provide some initialization function for some simple defaults:
        $this->setupForms();

        //translates to :
        $this->crud->operation('ban', function () {
            //load my config/backpack/operations/ban.php
            $this->crud->loadDefaultOperationSettingsFromConfig();

            // the save action should probably be automatically added by `setupParameterFormRoute/setupFormRoute` functions,
            // as the save action href is dependant on the form routes.
        });

        // add the operation stuff, buttons, columns etc
        $this->crud->operation('list', function () {
            $this->crud->button('ban-user')->stack('line')->view('crud::buttons.quick')->form('ban-user');
            $this->crud->button('ban-users')->stack('top')->view('crud::buttons.quick')->form('ban-users');
            $this->crud->button('view-user-ban')->stack('line')->view('crud::buttons.quick')->url(route('ban.get.history.ban')); //naming routes is a bit confusing here
        });
    }

// we can check if the CrudController is using `HasForms`, in case it is we can register a macro `form()` on the CrudButton ? This will link the button with the "show form url".

My ideas are a bit confusing, I am sure they make more sense in my head than when writing, but let's focus on the essential, a "real usecase", because of what I think is missing here is working with a "real operation".
What you think in general from the above ? Do your implementation/thinking allow us to do something similar ?
Cheers

Pedro... I think we have something here... I've fixed the above... and did your Ban operation example. And man... it took me just a few minutes to create all 3 of them 🤯 This is super SUPER powerful, what we've done here. You can take a look at the working code for BanOperation, GeneralBanOperation and BanHistoryOperation - https://gist.github.com/tabacitu/aa7b3ecbe2c43c9bb4fa81433f6ba29f - notice I've split up your functionality into 3 distinct operations. Because... that's what they are! Sure you can keep them in one operation, but you have to override a lot of things. Follow the protocol (one form = one operation) and you will do things amazingly fast.

After the changes above, here's how an EmailOperation looks:
<?php

namespace App\Http\Controllers\Admin\Operations;

use Illuminate\Support\Str;
use Backpack\CRUD\app\Library\CrudPanel\CrudPanelFacade as CRUD;
use Backpack\CRUD\app\Http\Controllers\Operations\Concerns\HasForm;

trait EmailOperation
{
    use HasForm;

    /**
     * Define which routes are needed for this operation.
     *
     * @param string $segment    Name of the current entity (singular). Used as first URL segment.
     * @param string $routeName  Prefix of the route name.
     * @param string $controller Name of the current CrudController.
     */
    protected function setupEmailRoutes($segment, $routeName, $controller)
    {
        $this->formRoutes(
            operationName: 'email',
            routesHaveIdSegment: true,
            segment: $segment,
            routeName: $routeName,
            controller: $controller
        );
    }

    /**
     * Add the default settings, buttons, etc that this operation needs.
     */
    protected function setupEmailDefaults()
    {
        $this->formDefaults(
            operationName: 'email',
            buttonStack: 'line',
            buttonMeta: [
                'icon' => 'la la-envelope',
                'label' => 'Send Email',
            ]
        );
    }

    /**
     * Method to handle the GET request and display the View with a Backpack form.
     *
     * @param  int  $id
     * @return \Illuminate\Contracts\View\View
     */
    public function getEmailForm(int $id = null)
    {
        $this->crud->hasAccessOrFail('email');

        return $this->formView($id);
    }

    /**
     * Method to handle the POST request and perform the action.
     *
     * @param  int  $id
     * @return array|\Illuminate\Http\RedirectResponse
     */
    public function postEmailForm($id = null)
    {
        $this->crud->hasAccessOrFail('email');

        return $this->formAction($id, function($inputs, $entry) {
            // You logic goes here...
            // dd('got to ' . __METHOD__, $inputs, $entry);

            // show a success message
            \Alert::success('Email was sent!')->flash();
        });
    }
}

this also needs in the CrudController to have:
    use \App\Http\Controllers\Admin\Operations\EmailOperation;

    protected function setupEmailOperation()
    {
        CRUD::field('to')->type('email')->size(4)->validationRules('required|email');
        CRUD::field('cc')->type('text')->size(4);
        CRUD::field('bcc')->type('text')->size(4);
        CRUD::field('subject')->type('text')->validationRules('required');
        CRUD::field('message')->type('textarea')->validationRules('required');
    }

but that's no different from what was before.

In total, I've checked:

✅ validation works as expected;
✅ one save action by default (save_and_back); gets performed automatically by form action;
✅ operation names are now camelCase as expected;
✅ if you create a config file in config/operations it will be picked up; if not, the default backpack.operations.form will get used (as you did in your branch https://github.com/Laravel-Backpack/CRUD/compare/v6...wip-form-operations )
✅ HasForm now helps with all fours steps of creating a form operation:

✅ formRoutes() helps set up the GET and POST routes;
✅ formDefaults() helps set up the config, save action and button;
✅ formView() returns the form view (used in GET method);
✅ formAction() performs the form validation, model events, then runs the callback the dev gave it to perform whatever custom action is needed of that form, then redirects to the right page using save_action;

I'm pretty happy with the state of this. I think it's the 20% that covers 80% of use cases. Any more than this... and we'd be falling down the rabbit hole of developing too much, making it too configurable or catering to use cases are too rare.
Take another look please... try it out if you want... and let me know if anything else triggers your spidey sense.
Thanks!

Update: I've just created a command to generate a form operation too. Now... it's even faster 🤯 It's crazy-fast, actually! Pedro if you're actually testing, please use this too, you'll love it https://github.com/Laravel-Backpack/Generators/pull/191

I am so excited about this... that I've created a short video to show it off to you. THIS. IS. FREAKING. AWESOME!!!

@tabacitu supa-dupa work 🙏
I am just "transfering" here what I said to you on discord, and where I think our ideas diverge here:

notice I've split up your functionality into 3 distinct operations. Because... that's what they are! Sure you can keep them in one operation, but you have to override a lot of things. Follow the protocol (one form = one operation) and you will do things amazingly fast

I know at the moment "this is what is happening", and it's valid to have 1 - 1 form/operation. But I also think that rationale is the culprit of some issues we have.
For example, InlineCreate should be part of the Create, with just a setting: allowInlineCreate => true/false in the operation. They are "technically" the same operation but with two different interfaces. Having them separated led to issues like defining the InlineCreate before Create breaks havoc.
Let's get back to the "real example":
The BanOperation is now split into three operations like you said. You are using the BanOperation in users, tags, books, maybe something else.
If you now want to add the shadow ban functionality enabled by default you need to change the operation and all cruds that use it to add the ShadowBanOperation.
This gets worst if you have your operation as a composer package to use in multiple projects, you will need to change all cruds x multiple projects after a composer update.
Imagine the nightmare two weeks later to add the UnbanOperation that requires a reason too.
Bare with me, it works the way we have it at the moment, I just don't think is "fewer work" as it would be if our operations were ready to deal with those scenarios.
What will happen most of the times is that you will just create the rest of the forms manually, so you will endup with a formRoutes() for "the form" and then the additional Route::get/post for other stuff that you think relates to that operation.
protected function setupBanRoutes($segment, $routeName, $controller)
    {
        $this->formRoutes(
            operationName: 'ban',
            routesHaveIdSegment: true,
            segment: $segment,
            routeName: $routeName,
            controller: $controller
        );
        Route::get($segment.'/{id}/unban', [
            'as'        => $routeName.'.unbanForm',
            'uses'      => $controller.'@unbanForm',
            'operation' => 'ban',
        ]);
        Route::put($segment.'/{id}', [
            'as'        => $routeName.'.unban',
            'uses'      => $controller.'@unban',
            'operation' => 'ban',
        ]);
    }

Or you create the UnbanOperation totally fair, it just has caveats.
Cheers

For example, InlineCreate should be part of the Create, with just a setting: allowInlineCreate => true/false in the operation. They are "technically" the same operation but with two different interfaces. Having them separated led to issues like defining the InlineCreate before Create breaks havoc.

InlineCreate is PRO, Create is free. No way they can be part of the same operation.

The BanOperation is now split into three operations like you said. You are using the BanOperation in users, tags, books, maybe something else. If you now want to add the shadow ban functionality enabled by default you need to change the operation and all cruds that use it to add the ShadowBanOperation.

You can bundle all 3 operations into a mega-operation 😀 BanOperations can contain all 3 operations. When you want to add ShadowBan, you add create that operation trait, then add it there. Easy-peasy-japanesy. The only thing you DO NOT have is one "access" for all operations inside it. You are forced into more granularity when it comes to access, which is unnecessary in most cases. But it's not a huge tradeoff if you ask me.

What will happen most of the times is that you will just create the rest of the forms manually, so you will endup with a formRoutes() for "the form" and then the additional Route::get/post for other stuff that you think relates to that operation.

Sure, you can do that too. And that's fine. You can also copy-paste code from HasForm if you want. And that's fine too. This isn't meant to solve all your problems. This feature is only meant to solve one problem - creating an operation with a form.

I understand what you're proposing. It would be cool to be able to have an operation with multiple forms. But it would be just that... cool. It's a COULD-HAVE. When you can easily create one trait that contains as many operations as you want - that gives you 90% of the benefit with 0% cost.
So let's analyze what this COULD-HAVE implies:

change to formRoutes() signature (to add $actionName or something like that);
change to route (it has one more parameter now);
change to configs (all configs we have now need to be in an array);

All of those are either a DX downgrade (if we force them on developers who only need one form) or clutter in our code. Plus, this change in scope (⚠) brings all sorts of dilemmas we will have to face, to account for the added complexity. Just think of the mess we'll be having with the configs. My point here is: This added complexity is not worth it. If a COULD-HAVE means added complexity, it's most likely not worth it.

This is exactly the kind of thing I'm trying to force us NOT to do any more. We DO NOT increase the scope of a task, for a COULD-HAVE. No mas! Not trying to be a hypocrite here, I do this sometimes too - you know I do. But it's up to me to tell you when you're doing it. And up to you to tell me when I'm doing it. We're accountability buddies in this. Let's stop adding unnecessary complexity!
And another thing I'm trying to make us not do any more: we DO NOT let issues baloon any more. I've estimated this task to be an M. To take me one week from start to finish. If we change the scope of this task, from "create an operation with a form" to "create an operation with one form or multiple forms", this will no longer be an M. It won't exactly be an L either, but it will be 2 Ms, most likely. Which is simply not doable - I have other priorities next week, that are FAR more important than this.
So please tell me. What do you think is smarter for us to do?
(A) Merge this feature like it was first defined?
(B) Change the scope and increase its duration?
You see where I'm getting at here Pedro? Let's please stop doing this. Let's please stop saying "oh but what if this also..." and change the scope of a task. That is bad. That is preventing us from shipping features. It's making us slow. Let's stop being slow and be fast instead 😅
You think it's a good idea to make it easier for developers to create multi-form operations? Open an issue for it please. We can just as well create a HasMultipleForms trait that helps them do that. Bring up the idea (that's the perfect moment to do so) but... as a separate task, mindful of the fact that we cannot and should not do it, because

it changes the scope
it baloons the task
it has a different priority than the original task (COULD-DO)

Now let's focus on the task at hand, with its defined scope. We set up to do one thing - make it easier for devs to create an operation with a form. To create their own operation like Create and Update. We did it, right? If you have no other objections, or found any bugs, I want to merge this 🎉
Can I merge this?

We should probably do some validation between stack / routesHaveIdSegment ?
I just ran the command to generate a new operation with a form, and added a TOP button.
Since it generates with routesHaveIdSegment => true, the top button don't work (method not allowed exception) as there is no id if not used in the rows (line button).
To fix it, I need to either change routesHaveIdSegment => false or add the button to the line.
Since there seems to be a "connection" between the property and the button stack, shouldn't we validate it ? The error is not quite clear of the issue and may be confusing. https://flareapp.io/share/NPGEW395
Apart from that it seems to be working, I've also reviewed Generators PR, there is a bug at the moment, the generated stub does not allow to create forms with routesHaveIdSegment = false https://github.com/Laravel-Backpack/Generators/pull/191#discussion_r1232568857
Good job Cristian 🙏


Please add a function to somehow pull subs option override and according to their timing

https://www.youtube.com/watch?v=Py21QCndbxc
From this video, we can see in some part the subs, there's a white bg while others don't, this is video override option and it changes according to the timing
The available subs format currently doesn't seems to provide the video override settings, as if the subs and the video override is two different thing
So would it be possible to get video override?
Also is it possible to get playlist video ids without downloading every single information of the video in the playlist?(I just want the ids thats all)

The available subs format currently doesn't seems to provide the video override settings, as if the subs and the video override is two different thing
So would it be possible to get video override?

Good question. Maybe it's available as a separate .vtt that is currently ignored? No-one ever seems to have looked at this. Further information/study/advice needed.

Also is it possible to get playlist video ids without downloading every single information of the video in the playlist?(I just want the ids thats all)

--get-id (RTFM)

timedtext.txt
Should look somewhat like this in json format

@dirkf Idk if you see this but can I know whether the plain timed text is possible or not?

Utilities exist to strip HTML from a file, so you could try that.
The encoding in your timedtext.txt looks a bit weird, an issue with CP1252 vs UTF8, I guess.

No the problem came when trying to download it
When looking around, I found out when you try to play a vid with subtitles, it sent a file named timedtext?v=[video_id]&[whatever_variables_they_put_here]&expire=[expire_id]&[more_stuff]
I used chrome, and look through the network inspection tab, but I do not if the timed text can be obtained by ytdl
I mean it's kinda tedious to go to every vid just to download subs

@dirkf I know this is an old stuff but can you check on if it's possible? I just do not know how to automate since it require expire_id and the timedtext is sent with the video making getting it from url with just video id is hard

Also last time you said strip HTML from a file?
Here it's more like strip a file from a page

Nvm I found how to do it

By all means let everyone else know what your solution was.


fix rendering of joins

Esqueleto exhibits strange behaviour with respect to joins. All joins seem to be left-associated even though the join Constructors are right associative. Explicitly adding parenthesis to left-associate the constructors adds seemingly superfluous parenthesis to the generated SQL. On the other hand there is no way to generate right-associated joins in the SQL.
This pullrequest takes a stab at fixing this.
While the tests run though fine,  I'm a bit unsure about this. I don't understand SQL well enough to be certain that this is correct in all cases.
This is related to #8
The downside of this patch is that the default associativity of the join operators now leads to different SQL code, e.g.
foo = from $ \((p1 :: SqlExpr (Entity DB.Product))
              `InnerJoin` (p2 :: SqlExpr (Entity DB.Product))
              `InnerJoin` (p3 :: SqlExpr (Entity DB.Product))) -> do
    on (p3 ^. DB.ProductName ==. p1 ^. DB.ProductName)
    on (p2 ^. DB.ProductName ==. p1 ^. DB.ProductName)
    return ( p1 ^. DB.ProductName
           , p2 ^. DB.ProductName
           , p3 ^. DB.ProductName
           )

used to compile into
SELECT "product"."name", "product2"."name", "product3"."name"
FROM "product" INNER JOIN "product" AS "product2" ON "product2"."name" = "product"."name" INNER JOIN "product" AS "product3" ON "product3"."name" = "product"."name"

But after this patch it becomes
SELECT "product"."name", "product2"."name", "product3"."name"
FROM "product" INNER JOIN ("product" AS "product2" INNER JOIN "product" AS "product3" ON "product3"."name" = "product"."name") ON "product2"."name" = "product"."name"

However, this is arguably the correct behaviour, as long as InnerJoin etc. are right-associative. Changing the associativity to infixl 2 restores the old behaviour and seems more intuitive to begin with, but can change the type of existing code.

Thanks for looking into this, @Philonous!
I'm also wary of changing this behaviour, especially since it's possible to change the meaning of a query while shuffling parenthesis (source).
I have a few questions I'd like to be answered before we land this:

Disregarding backwards compatibility, is it better to associate constructors to the left or to the right?

Can you write test cases showing current esqueleto failing and your new version working?  BTW, would the example I linked above be enough?

Are we going to make any valid queries in current esqueleto invalid?  If yes, we need at least a major version bump.

Are we going to change the meaning of any valid queries in current esqueleto?  If yes, we'll need to be extra careful, make a blog post and some mailing list announcements.

Note to self: regardless of the answers to the questions above, it's probably going to be a good idea to ask people on the mailing lists to test their code against new esqueleto just to be sure we didn't miss anything.

@Philonous, after staring at your commits for some time, I think you got it right.  But I don't feel too confident on my own judgement.  Actually, I'm the one who introduced this bug in the first place ;-).  So, I'd still appreciate if you could take a stab into the questions I've stated above.

I feel that left-associating would be the more natural choice. It avoids parens in the SQL and generally seems to be the intuitive thing to do (Also, the old behaviour was left-associative, even though the constructors right-associated)

An example for the old behaviour failing would be visible for (p LeftOuterJoin (q InnerJoin r)). The expected behaviour is to have a row for every p even when the inner join is empty. In the old version, this join would be rendered as ((p LeftOuterJoinq) InnerJoin r), where the inner join might drop rows from p. I will try to write this up as a formal test.

3 and 4) I think, but am not entirely certain, that the semantics of existing queries should be preserved as long as the constructors are changed to be left-associative. However, the type of expressions and patterns like (p InnerJoin q InnerJoin r) would change, leading to possible breakage, so a major version bump would be necesary.

+1

@Philonous, instead of editing older comments, in the future please write a new comment.  I just saw that you wrote the test case due to @jonkri's comment.  I don't get e-mail notifications for new commits nor comment edits.
I'll merge and ask for feedback later, but thanks in advance!

Oops sorry

Great, thanks! Any idea when this could be merged and released onto Hackage?

@Philonous, thank you for your PR, it's merged!  I've bumped the version to 2.2.
@jonkri, I'm going to send a few e-mails asking people to test this new version.  If everything goes well, I'll release it to Hackage in the next couple of weeks.  Please let me know if it works for your codebase and if you had to make any adjustements for it to work.

Tying the knot: https://groups.google.com/forum/#!topic/yesodweb/duwc5jcxgDY .

Thanks! The sooner this can be released, the better it would be for us; we have some features lined up that would need this patch. :) For us, only a few type signatures changed, and so it was just a matter of deleting them and having GHC infer them. Take care!

All fine for my biggest Esqueleto using project.

Thank you for your reports, @jonkri and @erikd, I'm getting more confident with these changes.

Well, then I should probably mention that me and @Philonous are working on the same code base, so... :)

I've released esqueleto-2.2.  Thanks, everyone!

Great! Thanks! :)


Add debug level logging for failed processing of events in parsing.py

logging.debug level logging for exceptions thrown out of processEvent, processCredential, etc.

logging.debug level logging for exceptions thrown out of processEvent, processCredential, etc.

Awesome! Adding this one to the vLEI top 10 discussion https://github.com/WebOfTrust/keri/discussions/77#discussioncomment-9149252


add keras to ml-notebook

We need keras on gcp for a project with @anirban89 (see https://github.com/pangeo-data/pangeo-cloud-federation/pull/706).
Will this cause any problems?

I don't expect that this will cause issues as long as we already have tensorflow.

Ah interesting. This image is explicitly installing tensorflow-gpu>=2. I have no idea how conda is going to handle that, since keras (via conda-forge) requires tensorflow >=2.2 https://github.com/conda-forge/keras-feedstock/blob/331732988551fe81a5e363d1f4d076a94b8905da/recipe/meta.yaml#L25. So this might indeed cause issues. We'll see how things build.

Doh!
from tensorflow import keras

This shows how old my deep learning experience is. Keras is now part of tensorflow.


docs: Include our ADRs in GitHub

Feature or Bugfix

Documentation

Detail

Moving our Architectural Decision Records (ADR) into GitHub
The ADRs will also appear in our documentation

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 19b7d4e771381bcd6f7b175c1248f2f946712989
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
Commit ID: 19b7d4e771381bcd6f7b175c1248f2f946712989
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
Commit ID: 63bd4a41728cca2e11eccea220264b203d858a66
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 63bd4a41728cca2e11eccea220264b203d858a66
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository


Navigation hide/sort options

Two configuration options are added:

enableNavigation: When set to false, the entire navigation section
is hidden for the user.
defaultNavigationSortKey: Allows specifying the initial sort key,
this is especially useful when navigation is disabled.

Merged with master with minor modification ("Newest" as default sort key)


Adjusting the size of the calendar

I've got a question. I've been evaluating the calendar component for a project, and keep getting an assert "Calendar metrics and size resulted in a negative-or-zero amount of remaining height" when placing it in a container view (about 375 x 375 points). I have to size it up to about 385 height to get it to appear. I couldn't find anything in the docs about sizing the calendar, the day items or adjusting the layout beyond the CalendarViewContent .with methods. Is there something I'm missing?

Hi @say-when - I know that this is a frustrating behavior. At a minimum, the error that's printed should be improved to explain why the issue is occurring. I also think we could improve the docs for this - I'll do that soon.
If you must fit your calendar in a 375x375 container, you'll need to increase the number of months visible using the horizontal layout config. By displaying more months horizontally, each month will have smaller day items to fit the width, which will also decrease the height, allowing you to fit your calendar in the 375pt tall container.
Try doing this:
CalendarView(
  visibleDateRange: startDate...endDate,
  monthsLayout: .horizontal(options: HorizontalMonthsLayoutOptions(maximumFullyVisibleMonths: 2)))

Try changing the maximumFullyVisibleMonths until the content fits correctly. Hope this helps!

Starting in version 1.11.0 (just released), CalendarView now has an intrinsicContentSize.height equal to the exact amount of space needed to fit all of the rows of a month, given the current width / other layout metrics. That means that you can add a horizontally-scrolling CalendarView to your view hierarchy, constrain its width, and the height will be set based on the intrinsicContentSize.

Version 1.11.0


add data fields to container logs

I'm currently harvesting the docker logs on the host. The JSON format makes it great for importing via logstash and I seem to be able to rotate/purge them as well.
One thing that would make it even better if additional fields were available.
I was thinking along the lines of adding some environment variables that could be added as a field to the logs. For example
DOCKER_LOG_VERSION=0.7
DOCKER_LOG_NAME=indexer
could then be added to the logs in JSON format adding '{ "docker_log_version" : "0.7", "docker_log_name" : "indexer" }'. In essence any variable that starts with a certain prefix would show up as a field in the JSON logs.
That would be a great value add for pushing to logstash.
I'm happy to give this a shot if this makes sense.

I think this is generally handled by the logging drivers, and the addition of the --log-opt for customizing drivers.
wdyt?

yeah, i think the logging driver should do the trick. Thanks!


Points spending mechanism

Hi! I'm new here and I do not know chinese language, I use google translate to read tushare.pro. Please, help me understand, are we spending points every time we making request?
For example, I made a donation and have 5000 points. It says, that I need 5000 points. Will I spend this amount of points when making request?

Using different API need different points level. you can try to make sense, not one time request using xxx points. note: every year will empty the points.

@flycomte-mzx Thank you!


Skip the backtick after inlined code so it doesn't remain in generated docs

Currently,
`inlined code`

generates something like:

inline code
`

That is, the trailing backtick is still in the output. This breaks harbored-mod, and probably harbored as well.
This pull request should fix this (but please check; I don't know if this change may cause other issue).

Could you add an unittest ?

I have a unit test that I'll check in after I merge this.


Document advertising additional startd attributes (SOFTWARE-5324)

This is one of those "if you don't know why you'd want to use it, you probably shouldn't" features but there isn't really a place for reference documentation and I don't want to have it be undocumented, so here it is.

Since @osg-cat has been talking about how he wants to rework this doc, I'd like to leave it up to him to determine whether or not this should go here. I think I'd prefer reference docs to go into the README

I’m fine adding this for now, so that it doesn’t get lost.


Clear out errors from manifest.json

Seems we've got some errors

too much glitches in this desert cake

on it


multiple users and 1-to-1 mapping ?

Is it possible for a single tty.js server to support multiple users with each user mapped to a unique linux user ? Currently it seems all the users access the same linux user's shell.
I have userZ running the tty.js server and two users userA, userB specified in config.json. But both userA and userB end up having the access to userZ's shell [whoami returns userZ for both userA and userB]. How can I map userA to linux userA's shell and userB to linux userB's shell ?
Thank you

Not right now, but pty.js has the ability to set the uid and gid of the forked process. So, it is technically possible to map each browser user to a unix user. I think this is a good idea and shouldn't be too difficult to implement. Will accept PRs.
That being said, you could also set the default shell to a unix login. That way users would just login using the terminal. I haven't tested it myself, but I've heard of people doing it.

Thank you. I was also thinking of the unix login shell approach.

chjj I would like to know how to setup tty.js or the inside implementation of pty for the default unix login. I'm currently running tty.js project because of the express server.

FWIW, I had trouble getting tty.js to use the default unix login, so set my shell parameter to be a tiny script that I call 'ssh-localhost' which looks like this:
#!/usr/bin/perl
print "Login: ";
my $username = <STDIN>;
chomp $username;
$username =~ s/[^a-z]//g;
exec ("/usr/bin/ssh -e none $username\@localhost");

This allows me to use tty.js to login as any user on the local host where tty.js is running.


feat: Add color to rat logo

Why doesn't the logo have color ?

Another question: Why does this logo isn't in ratatui::widgets::RatatuiLogo?

I feel indifferent about adding a color to the rat. If we are going to add a color there's a number of considerations

It should work in both a light and dark theme on the website.
It should have good contrast to the background in both cases

I didn't like going too grey on the rat. Take a look at the colors on the actual rat logo on the website for some ideas. It took a while to find the right grey there, but we stuck with white on the demo.

I don't think adding colors to that demo is necessary.

Another question: Why does this logo isn't in ratatui::widgets::RatatuiLogo?

This one is a demo/example rather than a widget.
We should probably rename that already existing widget to RatatuiTextLogo or RatatuiText and add the rat logo as RatatuiLogo.

I think that the logo is the Ratatui word, while the Rat holding the terminal is the mascot. That's kinda how I think of it at least (and could be wrong here).

Yeah, that also makes sense if we think that way. Then how about adding this mascot as RatatuiMascot widget (or even just Rat :D)
Though I'm not sure if that's super useful. It could be nice just as a gag.

Thanks for the great review
Here is the updated code with the updated result

Hello
I've updated the code with corrections :)

LGTM - I pushed a last change to make the terminal border color 237 because it's (close enough to) the dark mode color for macOS borders.

Thanks for sticking with this to get it over the line. This is one of those changes that definitely was always going to have some back and forth due to the subjective nature, and I appreciate you for the work you spent getting it over the line.


2.0: Delaring Any Model with @extras_features("relationships") Empties Those from #826

Environment

Nautobot version (Docker tag too if applicable): next or v2.0.0-alpha.3
Python version: 3.X
Database platform, version: Any
Middleware(s): N/A

Same applies for Custom Fields.
Steps to Reproduce

Clone the repo
Add @extras_features("relationships") on any model inheriting from PrimaryModel
Run Tests/Edit or Create a Relationship

Expected Behavior
Have access to all models that inherit from PrimaryModel to associate a Relationship to.

Observed Behavior
Only have the decorated model(s).

Closed with #3594


comment out webkit-animation-name: fadeOutWeb & changed fadeoutkey animation to 0px at 100%

I saw the code you demoed on your machine so functionality was good. The clean up of old unneeded code chunks is also appreciated.


Fix gcs manifest

Fixes #208
Appending is not possible in an object storage like GCS. If having the full update history of the manifest is a requirement, a loss-less version is also available here but it seems unnecessary and overly complex as it relies on writing to different objects each time and merging them in the end, and Firestore manifest is also overwriting.

I update the change to remove GCS Manifest, we can continue the discussion in #60.

Closing this PR as it has been addressed by the changes made in PR #295 which has been merged.


HomeKit not getting motion detection

Hi, I've installed this hack successfully for both Xiaofang and Dafang. However, despite troubleshooting and opening tickets to Dafang Hacks (not getting response), I'm still not getting any motion detection notification on the Home app.
I have enabled Motion -MQTT Message, enabled notification on the Home app. When I run mosquitto_sub -v -t '#' command I believe the motion was recognized by the camera.

I'm really stuck here, as these cameras were recently blocked by Xiaomi due to geographical restrictions :(

Please share your config.json, looks like it probably isn’t set up correctly.

Actually, looks like something is up with the MQTT configuration on your camera too. Seems like you don’t have any topic set at all, and motion is being published to just /motion.

Here is my config.json...
{
"name": "Camera FFmpeg",
"mqtt": "<IP_ADDRESS>",
"portmqtt": 1883,
"topic": "homebridge/motion",
"cameras": [
{
"name": "Entrance",
"manufacturer": "Xiaomi",
"model": "Xiaofang 1S",
"serialNumber": "2016DP5557",
"firmwareRevision": "<IP_ADDRESS>",
"motion": true,
"motionTimeout": 0,
"videoConfig": {
"source": "-re -i rtsp://<IP_ADDRESS>:8554/unicast",
"stillImageSource": "-i<EMAIL_ADDRESS>"maxStreams": 2,
"vcodec": "copy",
"packetSize": 564,
"audio": true
}
},
{
"name": "Shelf",
"manufacturer": "Xiaomi",
"model": "Dafang",
"serialNumber": "2017DP3894",
"firmwareRevision": "5.5.1_0426",
"motion": true,
"videoConfig": {
"source": "-re -i rtsp://<IP_ADDRESS>:8554/unicast",
"audio": true
}
}
],
"platform": "Camera-ffmpeg"
},
{
"cameras": [
{
"name": "Entrance",
"dafang_topic": "Foyer/Entrance",
"cooldown": 30,
"accessories": {
"rtsp_h264_server": true
}
},
{
"name": "Shelf",
"dafang_topic": "Den/Shelf",
"cooldown": 30,
"accessories": {
"rtsp_h264_server": true
}
}
],
"name": "Camera FFmpeg",
"server": "<IP_ADDRESS>",
"port": 1883,
"homebridge_topic": "homebridge/motion",
"platform": "dafangMqtt"
}

Forgive me if its too tiny to see. I suck at this :(

That all looks about right to me, the problem must be on the configuration on the cameras themselves then.
I’m really not sure why you’re seeing the control stuff coming through on the proper topic, but not motion though.

Do you think the MQTT is the one to blame? I followed the official Dafang Hacks guide (https://github.com/EliasKotlyar/Xiaomi-Dafang-Hacks/blob/master/hacks/install_cfw.md) to install the CFM for both cameras, but neither has motion detection enabled. Any more things I should look out to?
Or was it the microSD card that caused this? As I am using 16GB card, deleted the partition and set with only 512MB active.

Issue resolved. Thanks!


Status tooltip

This pull request contains a component called CellTooltipText that displays the contents of the cell's factors json object as a two-row table.

Using auto width seems to actually solve the overflow problem so I have made that update. The status pane does seem redundant to me now. Implementing for the heatmap should not be hard, I can implement it on this branch once the PR #245 is merged.

I updated, and haven't seen any problems. Will merge.


[trivial] Fix macOS build

Apple LLVM is stricter about headers.
Test plan: local build, CI

fixed in https://github.com/flashlight/flashlight/pull/1096/files