bfloat16 running 4x slower than fp32

🐛 Bug
I'm training a hybrid Resnet18+Conformer model using A100 GPUs. I've used both fp16 and fp32 precision to train the model and things work as expected: fp16 uses less memory and runs faster than when using fp32. However, when I try to switch to bf16 through Trainer(precision='bf16'), training speed is dramatically lower than with fp32, from ~1.3s/it to ~5.5s/it. The only difference between the runs is the precision argument, where I used '16' for fp16, '32' for fp32, and 'bf16' for bfloat16. The slowness of bf16 is experienced when using only 1 GPU, 8 GPUs (single node), or 32 GPUs (multiple nodes).
The docs state

BFloat16 is also experimental and may not provide significant speedups or memory improvements, offering better numerical stability.

But shouldn't it work faster than fp32 on A100 GPUs? Any help would be hugely appreciated.
Expected behavior
For bf16 to work faster than fp32 and probably slower than fp16.
Environment

CUDA:

GPU:

A100-SXM4-40GB

available:         True
version:           11.3

Packages:

numpy:             1.21.2
pyTorch_debug:     False
pyTorch_version:   1.10.2
pytorch-lightning: 1.5.9
tqdm:              4.62.3

System:

OS:                Linux
architecture:

64bit
ELF

processor:         x86_64
python:            3.8.12

Additional context
Although speed is dramatically lower with bf16 than with fp32, the GPU memory allocation is lower (~55% vs ~95%), as expected.
cc @carmocca @justusschock @awaelchli @akihironitta @rohitgr7 @borda

Do you have a pointer to your code for us to investigate?

Hi @ananthsub, I have the same problem and my trial code is available here: (https://github.com/PyTorchLightning/pytorch-lightning/issues/12600). In my case, fp32 is about x1.3 faster than bf16 and fp16 is ~x1.9 faster than fp32.
I tested with A100 and A5000, with PyTorch 1.11 and Lightning 1.6.

@ahaliassos It could be caused by the specific operations you have in your LightningModule. AFAIK torch's einsum is not the most performant.
Do you see experience speed-downs with simpler modules?

Hi @carmocca, I removed the einsum operation parts from (https://github.com/PyTorchLightning/pytorch-lightning/issues/12600) and checked the training speeds for each precision.
The training speeds are as follows

fp16: 13.10it/s
fp32: 8.78it/s
bf16: 4.55it/s

Briefly, my code calculates the cross-entropy loss during training and updates the ResNet18 model with ImageNet dataset.

We will try to repro using this script https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pl_examples/domain_templates/imagenet.py and report back

Hi, @joshua840 - Thank you so much for reporting this issue! Acknowledging that we were able to reproduce this issue on our servers, and this is something I'll debug further. We will get back to you as soon as possible, whenever we have some pointers on this. Thank you again!

Hi, so I was finally able to find some time to benchmark for possible precisions (apologies for the delay).

Precision
PyTorch (iters/s)
PyTorch Lightning (iters/s)

float16
3.92
4.27

float32
3.77
4.22

bfloat16
4.08
2.5

Note:

Tested on: Image Net dataset
Scripts used:

PL: https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pl_examples/domain_templates/imagenet.py
PyTorch: https://github.com/pytorch/examples/blob/main/imagenet/main.py (used AMP for float16 and bf16 precisions)

Tested on ~2500 iterations. Numbers are averaged out.

Analysis:

From the numbers, it looks like PyTorch Lightning is around 1.6 times slower than PyTorch for bfloat16 precision, while for other precisions - there doesn't seem to be a huge difference. PL performs a little faster, but I assume it will be near to equal if tested for >1 epochs.
Also, I agree that bf16 is running slower than fp32 in PyTorch Lightning, which is not expected.

We'll try investigating why performance is slower for bf16 in PyTorch Lightning and will get back to this issue once we have some numbers. :) Thank you for your patience and for reporting this issue to us!
cc: @carmocca @SeanNaren @Borda for visibility! :)

Working with @krshrimali to try to investigate!
I'm excluding Lightning and using the pytorch imagenet example to gather the below results on an A100. I ran for more than 2500 iterations.

Precision
seconds per iteration

FP32
0.103

FP32 (No TF32)
0.160

FP16
0.079

BFloat16
0.415

I also tried the latest NGC container which improved results marginally but showed a massive slowdown for BFloat16. I also tried resnet101 and still showed a slowdown.
I also tried adding torch.cuda.synchronize() before measurements were taken, and a negligible difference was made.
I think it will be worth checking other model types with a minimal script to see if we see the same slow down without PyTorch Lightning before investigating.
Note: no TF32 meant adding this:
# The flag below controls whether to allow TF32 on matmul. This flag defaults to True.
torch.backends.cuda.matmul.allow_tf32 = False

# The flag below controls whether to allow TF32 on cuDNN. This flag defaults to True.
torch.backends.cudnn.allow_tf32 = False

ping @SeanNaren @ananthsub, any news on that ?

ping @SeanNaren @ananthsub, any news on that ?

ok, synced offline and seems that it's not a priority :(

Here are my results running a pytorch-lightning minGPT script (here). This was a transformer model, with the below cmd:
python min_gpt.py --accelerator gpu --devices 1 --precision X

Precision
Iterations/s
nvidia-smi Memory (MiB)

16
2.26
30038

bf16
2.24
30038

bf16 + model.bfloat16()
2.40
24550

I also used torch.set_float32_matmul_precision('medium') which gave roughly the same results.
We can see that the performance is roughly the same when using AMP, but when converting the model we get a small speedup + a significant memory drop as expected.
It seems for transformer architectures BFloat16 works as expected.
I found this forum post and this comment describing slowdowns when using Conv2d + Bfloat16.
The model type in OP's post is based on convs, and the above reference issue could explain the reason why we see a 4x slowdown.

That's great, thanks so much @SeanNaren ! The memory benefits are huge, for a similar speed it's still very useful. Cc @matthieutphr @quentindrx

I still have some problems regarding to the training speed of bfloat16 when enabling autocast.
Could anyone see the problem here?
https://discuss.pytorch.org/t/performance-training-speed-of-autocast-bfloat16/158183


Video models, two outputs?

Hi,
I cannot figure out why the video models published has two different outputs, why is that?
Thanks

Check the section: Techniques For Improving Robustness, Accuracy and Speed in readme and refer the related research paper for  more information.

Ok thanks!


Extracting EXIF location from images as location field values

Just an idea. Mobile users can turn on geotagging for their cameras. They often already have it on, so when they fill out a response with an image and a location question they are having to capture location twice. Also for my use cases (natural science field sightings) the Enketo UI is often insufficiently immediate for point of capture, so users will take a bunch of pictures in the field and then submit them later.
Something like https://github.com/exif-js/exif-js could be used to parse the EXIF headers, shouldn't be too memory intensive given that it only needs to look at the first few bytes.
The main difficulty would seem to be coming up with a sensible interface design for this feature which would be ergonomic for survey designers and reporters. Maybe each image question could point to a location question for which it would provide a default value. The location field could perhaps indicate that it had found a value via the attached image, or simply hide itself.
Feel free to close as out of scope or direct to a different repo.

Cool idea! From a technical perspective this would be an awesome (fun) feature to develop. Could perhaps be implemented as a new XPath function location(${image}) or exif(${image}) though this will be a challenge because XPath only has access to the file name stored in the XML instance (not the file itself). We'd have to agree with the ODK Collect devs too - needs to be supported in ODK Validate.
Not sure about the demand for this. Let's leave this open.


java.lang.UnsatisfiedLinkError: dtxjpi when running maps in DataStage for windows

While we are working with WTX for InfoSphere Datastage that executes on top of the Java Integration stage, we could see some issues
Error 1:
hipaa_837p_5010_flat,0: java.lang.UnsatisfiedLinkError: dtxjpi (The specified procedure could not be found. )
at java.lang.ClassLoader.loadLibraryWithPath (ClassLoader.java: 1542)
at java.lang.ClassLoader.loadLibraryWithClassLoader (ClassLoader.java: 1506)
at java.lang.System.loadLibrary (System.java: 562)
After setting Jvm argument :
-Djava.library.path="C:\IBM\WebSphere Transformation Extender 8.4.1\dtxjpi.dll"
facing with below error,
Error 2 :
hipaa_837p_5010flat: java.lang.NoClassDefFoundError: com.ascential.e2.connector.CCConnectorLibrary (initialization failure)
at java.lang.J9VMInternals.initializationAlreadyFailed (J9VMInternals.java: 121)
hipaa_837p_5010flat: Java runtime exception occurred: java.lang.NoClassDefFoundError: com.ascential.e2.connector.CCConnectorLibrary (initialization failure) (java.lang.J9VMInternals::initializationAlreadyFailed, file J9VMInternals.java, line 121)

Can you provide, the version number of WTX ? and the Java Version of Data Stage ?  Version of dtxjpi.dll ?
To get the version of dtxjpi, run dtxver on the library

We are using wtx 8.4.1 and java 7 32-bit and C:\IBM\WebSphereTransformationExtender8.4.1\dtxjpi.dll
--> <IP_ADDRESS>(37) <-- Thu Nov 21 00:17:42 2013 - 267264 bytes.
We also tried with itx 9.0 64-bit , java 7 64-bit and dtxjpi.dll 9.0 but still facing same error

we are using wtx 8.4.1 , java 7 32-bit and C:\IBM\WebSphereTransformationExtender8.4.1\dtxjpi.dll
--> <IP_ADDRESS>(37) <-- Thu Nov 21 00:17:42 2013 - 267264 bytes
we also tried with wtx 9.0, java 7 64-bit and dtxjpi 9.0 but still same error (java.lang.UnsaticifiedLinkError dtxjpi)

Did you try to re-build the WTXStage.jar with ITXv9.0 libraries ? I would have the latest dtxpi.jar from ITX v9.0 and ccjava-api.jar (Datastage jar)  in the classpath, rebuild a new WTXStage.jar and try in v9.0. FYI ..ITXv9.0 is a 64Bit Platform.

We have don't have dstxpi.jar file in ITXv9.0 ,So is it necessary to add in a classpath at java integration stage . or only the new bulided  WTXStage.jar file is enough

a) Its renamed as dtxpi.jar. b) Yes. it is required to have dtxpi.jar in the class path.

After rebuilding the WTXStage.jar file ,  as am facing the issue as:
MapAllTypes: java.lang.UnsupportedClassVersionError: JVMCFRE003 bad major version; class=com/ibm/is/cc/javastage/wtx/WTXStage, offset=6
at java.lang.ClassLoader.defineClassImpl (ClassLoader.java)
at java.lang.ClassLoader.defineClass (ClassLoader.java: 364)
at java.security.SecureClassLoader.defineClass (SecureClassLoader.java: 154)
at java.net.URLClassLoader.defineClass (URLClassLoader.java: 777)
at java.net.URLClassLoader.access$400 (URLClassLoader.java: 96)
at java.net.URLClassLoader$ClassFinder.run (URLClassLoader.java: 1225)
at java.security.AccessController.doPrivileged (AccessController.java: 366)
at java.net.URLClassLoader.findClass (URLClassLoader.java: 660)
at com.ibm.is.cc.javastage.common.CC_JavaChildFirstClassLoader.loadClass (CC_JavaChildFirstClassLoader.java: 67)
at java.lang.ClassLoader.loadClass (ClassLoader.java: 827)
at com.ibm.is.cc.javastage.connector.CC_JavaConnection.connect (CC_JavaConnection.java: 171)

we are getting this kind of error

8.4.1,3 :
WTXStage.zip
9.0.0
WTXStage.zip
Try these jars. (I Built with  <IP_ADDRESS> and one with 9.0 dtxpi.jar)

we are still facing the same issue as
master,1: java.lang.UnsatisfiedLinkError: dtxjpi (Not found in java.library.path)
at java.lang.ClassLoader.loadLibraryWithPath (ClassLoader.java: 1542)
at java.lang.ClassLoader.loadLibraryWithClassLoader (ClassLoader.java: 1506)
at java.lang.System.loadLibrary (System.java: 562)
will u share which version of datastage , java & wtx you have worked with..
Waiting for ur reply.


"No response received" error

Thank you for this amazing repo! I have a Midea AC and I'm getting "No response received" error when I try to run your code.
I'm using my custom TCP-Serial bridge using the standard Aliexpress components which worked normally via Home Assistant app. However, since I would like to create my custom app for the control of my a/c and get rid of Home Assistant I decided to try your repo. So I used the exact same hardware, I just flashed ESP-Link to my ESP-01, added my wifi credentials and set my static ip <IP_ADDRESS>.
However I keep getting above error, you can also find log with details to the bottom of my message.
When I connect it to the serial of my AC, the IP works normally, so I think that the only part missing is the connection between the ESP-01 and the AC. Since I use the exact same cable that I use with HA, connecting it to the exact same way as with HA app, the only I can think of is that maybe I did something wrong with the pin assignment within the settings of ESP-Link, where I chose the ESP-01 preset.
Is there any suggestion you can give me to help me about the pin assignment, as I really don't know what to change there? Or maybe do you have any other idea what I may did wrong?
Thank you in advance!
My code:
const appliances = require('node-mideahvac');

// Specify your specific information
const ac = appliances.createAppliance({
  communicationMethod: 'serialbridge',
  host: '<IP_ADDRESS>',
  port: '23'
});

// Any updated status properties will be printed
ac.on('status-update', data => {
  console.log(`Received updates: ${JSON.stringify(data)}`);
});

ac.initialize()
  .then(response => {
    console.log('Initialized, start polling');

    // The status will be polled each 30s, when properties are changed these will be emitted as a
    // status-update
    setInterval(() => {
      ac.getStatus()
        .catch(error => {
          console.log(`Error: ${error.message}`);
        });
    }, 30000);
  })
  .catch(error => {
    console.log(`Error: Failed to initialize (${error.message})`);
  });

const logger = require('winston')

logger.remove(logger.transports.Console)
logger.add(new logger.transports.Console({
  format: logger.format.combine(
    logger.format.timestamp(),
    logger.format.colorize(),
    logger.format.printf(event => {
      return `${event.timestamp}: ${event.message}`
    })
  ),
  level: 'silly'
}))

My log:
2023-11-11T01:46:36.571Z: SerialBridge._processQueue: Entering
2023-11-11T01:46:36.580Z: SerialBridge._processQueue: Sending 'sendNetworkStatusNotification' command
2023-11-11T01:46:38.589Z: SerialBridge._processQueue: No response received in time for 'sendNetworkStatusNotification' command
2023-11-11T01:46:38.590Z: SerialBridge.initialize: Failed to send network status notification for <IP_ADDRESS>:23 - No response received
2023-11-11T01:46:38.591Z: AC.getCapabilities: Entering
2023-11-11T01:46:38.592Z: SerialBridge._processQueue: Entering
2023-11-11T01:46:38.592Z: SerialBridge._processQueue: Sending 'getCapabilities' command
2023-11-11T01:46:39.090Z: SerialBridge._processQueue: Entering
2023-11-11T01:46:39.091Z: SerialBridge._processQueue: Command in progress
2023-11-11T01:46:40.606Z: SerialBridge._processQueue: No response received in time for 'getCapabilities' command
2023-11-11T01:46:40.607Z: SerialBridge._write: Retry getCapabilities, 0 retries left
2023-11-11T01:46:41.109Z: SerialBridge._processQueue: Entering
2023-11-11T01:46:41.110Z: SerialBridge._processQueue: Sending 'getCapabilities' command
2023-11-11T01:46:43.113Z: SerialBridge._processQueue: No response received in time for 'getCapabilities' command
2023-11-11T01:46:43.114Z: SerialBridge.initialize: Failed to get capabilities of <IP_ADDRESS>:23 - No response received
2023-11-11T01:46:43.116Z: AC.getStatus: Entering
2023-11-11T01:46:43.116Z: SerialBridge._processQueue: Entering
2023-11-11T01:46:43.117Z: SerialBridge._processQueue: Sending 'getStatus' command
2023-11-11T01:46:43.629Z: SerialBridge._processQueue: Entering
2023-11-11T01:46:43.630Z: SerialBridge._processQueue: Command in progress
2023-11-11T01:46:45.132Z: SerialBridge._processQueue: No response received in time for 'getStatus' command
2023-11-11T01:46:45.133Z: SerialBridge._write: Retry getStatus, 0 retries left
2023-11-11T01:46:45.647Z: SerialBridge._processQueue: Entering
2023-11-11T01:46:45.648Z: SerialBridge._processQueue: Sending 'getStatus' command
2023-11-11T01:46:47.661Z: SerialBridge._processQueue: No response received in time for 'getStatus' command
2023-11-11T01:46:47.662Z: SerialBridge.initialize: Failed to get current status of <IP_ADDRESS>:23 - No response received
Error: Failed to initialize (undefined)

Probably the pin assignment configuration in esp-link is wrong. Mine, but I use an ESP12, looks as follows:
Reset: gpio12
ISP/Flash: gpio14
Conn LED: gpio0
Serial LED: gpio2/TX1
UART pins:  normal
RX pull-up: V

Probably the pin assignment configuration in esp-link is wrong. Mine, but I use an ESP12, looks as follows: Reset: gpio12 ISP/Flash: gpio14 Conn LED: gpio0 Serial LED: gpio2/TX1 UART pins: normal RX pull-up: V
And did you set the uConsole to 9600 8N1?

had same problem, setting 9600 helped. Thanks!


Bug report: The problem in the dark mode Mentorship cards

Describe the bug
In the dark mode the dates just below the heading are not properly visible needs to b change its color to white when the dark mode is on.

To Reproduce
Steps to reproduce the behavior:

just changing the color for the text in the date as the other text in the card.

Expected behavior
A clear and concise description of what you expected to happen.
Actual behavior
A clear and concise description of how the code performed w.r.t expectations.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional details

Will my score be added for raising this issue @anushbhatia

Will my score be added for raising this issue @anushbhatia

Yes. We update the scores every 5 days

Hey @smaranjitghose  Can I try this issue?


MAINT: sparse: change coo_matrix.indices to coo_matrix.coords

To avoid confusion between formats, the recently implemented attribute name indices for COO format is changed in this PR to coords.
This is a public attribute, but did not exist in 1.12, so merging this before 1.13 will not have any backward compatibility issues.
The CSR/CSC formats already use the attribute name indices. So while the word indices is fine for COO, it may cause confusion between compressed and COO especially when working with both, converting etc. The name coords is arguably better anyway because it matches the "COO" of the COO format name. Both words seem to mean the same thing in English.

Good catch -- I didn't think enough about docs and comments. But I suspect that its better that you make those calls anyway. I've made those changes.


change little2

👷 Deploy Preview for amazing-trifle-94ae4f processing.

Name
Link

🔨 Latest commit
6278cc859e3e652deafdf83885fc4b23dfa9947a

🔍 Latest deploy log
https://app.netlify.com/sites/amazing-trifle-94ae4f/deploys/64c6a14590215300081506b1


Extend Vulmix's Icon component with Iconify

The original idea is to use the name prop to use custom icons on the assets/icons folder and the icon prop for the Iconify icons.


Goal codes in credential exchange protocols

I think we've made a mistake by not including optional goal codes in the credential exchange protocols. I know this is a late add, so we may need to skip it for AIP 2.0 purposes.
Signed-off-by: Sam Curren<EMAIL_ADDRESS>

Approved on community B call, 24 Mar 2021


Feature request : Face Mask detection system using Computer Vision

Is your feature request related to a problem? Please describe.
A Face mask detection model which will identify whether someone has a mask or not
Describe the solution you'd like
Face Mask Detection System built with OpenCV, Keras/TensorFlow using Deep Learning and Computer Vision concepts in order to detect face masks in static images as well as in real-time video streams.
Additional context

@UppuluriKalyani can you assign me this issue


Unsupported URL: Yandex.com

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I remove or skip any mandatory* field

Checklist

[X] I'm reporting a broken site
[X] I've verified that I'm running yt-dlp version 2022.09.01 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[X] I've read about sharing account credentials and I'm willing to share it if required

Region
USA
Provide a description that is worded well enough to be understood
See the output, below.
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
$ yt-dlp -vU 'https://yandex.com/video/preview/?text=dossier%2051%20film%201978&path=yandex_search&parent-reqid=1664361087754492-8727541069609384458-sas2-0340-sas-l7-balancer-8080-BAL-8045&noreask=1&from_type=vast&filmId=5794987234584444632'
[debug] Command-line config: ['-vU', 'https://yandex.com/video/preview/?text=dossier%2051%20film%201978&path=yandex_search&parent-reqid=1664361087754492-8727541069609384458-sas2-0340-sas-l7-balancer-8080-BAL-8045&noreask=1&from_type=vast&filmId=5794987234584444632']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.09.01 [5d7c7d6] (pip)
[debug] Python 3.10.6 (CPython 64bit) - Linux-5.15.0-48-generic-x86_64-with-glibc2.35 (glibc 2.35)
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.09.24, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Loaded 1670 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.09.01, Current version: 2022.09.01
yt-dlp is up to date (2022.09.01)
[debug] [generic] Extracting URL: https://yandex.com/video/preview/?text=dossier%2051%20film%201978&path=yandex_search&parent-reqid=1664361087754492-8727541069609384458-sas2-0340-sas-l7-balancer-8080-BAL-8045&noreask=1&from_type=vast&filmId=5794987234584444632
[generic] ?text=dossier 51 film 1978&path=yandex_search&parent-reqid=1664361087754492-8727541069609384458-sas2-0340-sas-l7-balancer-8080-BAL-8045&noreask=1&from_type=vast&filmId=5794987234584444632: Downloading webpage
WARNING: [generic] Falling back on generic information extractor
[generic] ?text=dossier 51 film 1978&path=yandex_search&parent-reqid=1664361087754492-8727541069609384458-sas2-0340-sas-l7-balancer-8080-BAL-8045&noreask=1&from_type=vast&filmId=5794987234584444632: Extracting information
[debug] Looking for Brightcove embeds
[debug] Looking for embeds
ERROR: Unsupported URL: https://yandex.com/video/preview/?text=dossier%2051%20film%201978&path=yandex_search&parent-reqid=1664361087754492-8727541069609384458-sas2-0340-sas-l7-balancer-8080-BAL-8045&noreask=1&from_type=vast&filmId=5794987234584444632
Traceback (most recent call last):
  File "/home/dev/.local/pipx/venvs/yt-dlp/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 1459, in wrapper
    return func(self, *args, **kwargs)
  File "/home/dev/.local/pipx/venvs/yt-dlp/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 1535, in __extract_info
    ie_result = ie.extract(url)
  File "/home/dev/.local/pipx/venvs/yt-dlp/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 670, in extract
    ie_result = self._real_extract(url)
  File "/home/dev/.local/pipx/venvs/yt-dlp/lib/python3.10/site-packages/yt_dlp/extractor/generic.py", line 3078, in _real_extract
    raise UnsupportedError(url)
yt_dlp.utils.UnsupportedError: Unsupported URL: https://yandex.com/video/preview/?text=dossier%2051%20film%201978&path=yandex_search&parent-reqid=1664361087754492-8727541069609384458-sas2-0340-sas-l7-balancer-8080-BAL-8045&noreask=1&from_type=vast&filmId=5794987234584444632

This can be fixed by allowing .com in the regex for YandexVideoPreviewIE, but it might also make sense to add all yandex domains into each regex.

Assuming there are no known false positives, something like yandex\.\w+ is acceptable too

Great, thanks.  If you go to that URL in the OP, the film is in French (that's the original).  If you change it to ".ru" from ".com", then you get a Russian dubbed version.  So I couldn't find any easy way to do this.
I'll assume I have to git clone the repo and then modify that one or two lines that I see where YandexVideoPreviewIE is located?

I didnt notice before bc of an automated redirect: The link you are trying to download is a search(?), not a movie link. The real link you want to download would be<PHONE_NUMBER>584444632, which already works with the .ru prefix. If you use .ru for the search, it points you to<PHONE_NUMBER>269031239 instead (notice how the ids differ).
As a temporary workaround use "Share" + "Copy link" and change that to .ru. That should already work as is.


Improve Bounties and Child Bounties Deposit Logic

It was suggested that the deposit logic for Bounties and Child Bounties was not flexible enough.
Specifically:

If a Bounty paid no fee, it would have no deposit requirements.
If a Child Bounty had a huge value, it could end up for a huge deposit for the end user.
If a Child Bounty curator was also the parent curator, and there was no fee to be earned, they were being asked to put an additional deposit down.

To solve this, we refactored the logic around calculating the curator deposit into its own function, and put a little bit of simple logic in there to handle different situations.
For Bounties:

If there is a fee, then the curator deposit is a fraction of that fee.
If there is no fee, then the curator deposit is a fraction of the overall bounty value.

For Child Bounties:

The logic above is the same, except in the case there is no fee, and the parent and child curator is the same, then we need not to take an additional deposit from the curator.

These fractions are all configurable in the runtime.
New tests ensure each of these different scenarios are handled correctly.

I think this should be voluntary curator friendly that when curator is doing voluntary work, they shouldn't need to lock deposit from their own pocket?
Otherwise we simply require a minimal curator fee so that is ok to ask curator to lock deposit which will be compensated later.

Can you make a specific suggestion?

I just don't find this useful. If we are not going to set NoFee to a non zero value, this code will be redundant and I don't think we want to set it to a non zero value.
What value are you going to pick anyway? The Polkadot Pioneers Price is 993286 DOT, so if you make it 0.1%, the deposit will be 993 DOT, which is still a big number for curator.
But for 10 DOT (the current minimal bounty value for Polkadot), 0.1% will be 0.01 DOT, which is less than ED.

What I want to see, is a maximum value for curator deposit. Similar to the treasury proposal deposit. That ensures the deposit won't be too large for some large curator payout. e.g. for the ORML security bounty, the curator fee will be 10% of the payout, which can be large if the payout is big.

I agree with @xlc that there should be a cap on the curator deposit of a couple hundred DOT. For cases when the fee is zero, the percentage I proposed for the deposit was 1%, which for a 20000 DOT bounty would be 200 DOT, which I think is reasonable to be the cap as well.
IMO the fee should be zero only when the curator is part of an organisation and they're doing it as part of their job. In those cases the organisation can front the deposit. Otherwise community members should get rewarded for their efforts.
But we do need a deposit. Otherwise a malicious curator could set a zero fee in order not to place a deposit and then take advantage of the fact the have nothing to lose.

Related: #11026

The min/max bounds implementation looks good to me.

Thanks for the feedback @xlc. Things should be up to date now, and pretty straightforward.
The only other thing I did here was remove the redundant configurations from child bounties, since I don't think they should really ever be different from the parent bounties itself.

bot merge

bot merge


Analyze what type of information can we get out of OEM

As a CE and/or SCE, I want to be able to get as much information from customer database environments with as little effort as possible even if that means only high level information. So, I can get started with planning and discussing with customer migration options sooner.
Expected Results:

OEM Script that generates spool files such as current SQL Scripts
OR
If this is not possible/feasible/worth it

Initial Target Completion:
21 / Q4

Is it about accepting the terms of cookies or not?

@cofin


esp32c6: Waiting for download after flashing

I have a esp32c6 here that I flash using the command espflash flash --monitor --flash-size 8mb --partition-table ./partitions.csv <elf>
Sometimes I have the situation that after flashing the monitor shows:
Commands:
    CTRL+R    Reset chip
    CTRL+C    Exit

ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0x15 (USB_UART_HPSYS),boot:0x4 (DOWNLOAD(USB/UART0/SDIO_FEI_FEO))
Saved PC:0x4080041e
0x4080041e - get_buffer_malloc
    at /tmp/backtrace-std/.embuild/espressif/esp-idf/master/components/spi_flash/spi_flash_os_func_app.c:193
waiting for download

Why is it still waiting for download? I have not pulled the strapping pin GPIO9 to low.
I have not yet found out when this happens exactly, but it seems to somehow depend on the firmware I flash...

Minimal example // repro:

start with an erased esp32c6 (I'm using a esp32-c6-devkit-c1 v1.2 here)
Run cargo generate esp-rs/esp-idf-template to generate a hello world with std, based on idf v5.1.
Connect the target via JTAG USB. The pins are not connected to any external components, just a naked dev kit. (hook up the uart0 pins to the logic analyzer if you like)
Flash this to the target with espflash flash --monitor <elf>  (flashproblems.elf.zip )
So far everything works and the monitor prints the hello world. I can also press CTRL+R, still works
Exit the monitor (from step 4), then run espflash monitor --no-stub

espflash shows:
espflash monitor --no-stub
.... (version check with crates.io)
[2024-01-22T12:54:31Z INFO ] Serial port: '/dev/ttyACM0'
[2024-01-22T12:54:31Z INFO ] Connecting...
[2024-01-22T12:54:31Z DEBUG] Attempting flash enable with: SpiAttachParams { clk: 0, q: 0, d: 0, hd: 0, cs: 0 }
[2024-01-22T12:54:31Z DEBUG] Flash detect OK!
Commands:
    CTRL+R    Reset chip
    CTRL+C    Exit

Error:   × Broken pipe

Logic analyzer started before step 6:

The first bunch of messages shows:

ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0x15 (USB_UART_HPSYS),boot:0xc (SPI_FAST_FLASH_BOOT)
Saved PC:0x40802990
SPIWP:0xee
mode:DIO, clock div:2
load:0x4086c410,len:0xd48
load:0x4086e610,len:0x2d68
load:0x40875720,len:0x1800
SHA-256 comparison failed:
Calculated: 377c26481ad6ab0cce508450f353a55f4bd7eb9159aaf85fde3111071f164514
Expected: 0af544a033ab3492852b8232c904c578d5f07c9d4a423a64473f060db374ab32
Attempting to boot anyway...
entry 0x4086c410
\x1B[0;32mI (41) boot: ESP-IDF v5.1-beta1-378-gea5e0ff298-dirt 2nd stage bootloader\x1B[0m
\x1B[0;32mI (42) boot: compile time Jun  7 2023 08:02:08\x1B[0m
\x1B[0;32mI (43) boot: chip revision: v0.0\x1B[0m
\x1B[0;32mI (46) boot.esp32c6: SPI Speed      : 40MHz\x1B[0m
\x1B[0;32mI (51) boot.esp32c6: SPI Mode       : DIO\x1B[0m
\x1B[0;32mI (56) boot.esp32c6: SPI Flash Size : 8MB\x1B[0m
\x1B[0;32mI (60) boot: Enabling RNG early entropy source...\x1B[0m
\x1B[0;32mI (66) boot: Partition Table:\x1B[0m
\x1B[0;32mI (69) boot: ## Label            Usage          Type ST Offset   Length\x1B[0m
\x1B[0;32mI (77) boot:  0 nvs              WiFi data        01 02 00009000 00006000\x1B[0m
\x1B[0;32mI (84) boot:  1 phy_init         RF data          01 01 0000f000 00001000\x1B[0m
\x1B[0;32mI (92) boot:  2 factory          factory app      00 00 00010000 007f0000\x1B[0m
\x1B[0;32mI (99) boot: End of partition table\x1B[0m
\x1B[0;32mI (103) esp_image: segment 0: paddr=00010020 vaddr=42000020 size=413f4h (267252) map\x1B[0m
\x1B[0;32mI (166) esp_image: segment 1: paddr=0005141c vaddr=40800000 size=06bfch ( 27644) load\x1B[0m
\x1B[0;32mI (174) esp_image: segment 2: paddr=00058020 vaddr=42048020 size=1da78h (121464) map\x1B[0m
\x1B[0;32mI (\xFFESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0x15 (USB_UART_HPSYS),boot:0x4 (DOWNLOAD(USB/UART0/SDIO_FEI_FEO))
Saved PC:0x4001a36e
waiting for download

then there is a 8 second delay, during which espflash says "broken pipe".
then the board boots up normally with:
ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0x10 (LP_WDT_SYS),boot:0xc (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:2
load:0x4086c410,len:0xd48
load:0x4086e610,len:0x2d68
load:0x40875720,len:0x1800
SHA-256 comparison failed:
Calculated: 377c26481ad6ab0cce508450f353a55f4bd7eb9159aaf85fde3111071f164514
Expected: 0af544a033ab3492852b8232c904c578d5f07c9d4a423a64473f060db374ab32
Attempting to boot anyway...
entry 0x4086c410
\x1B[0;32mI (42) boot: ESP-IDF v5.1-beta1-378-gea5e0ff298-dirt 2nd stage bootloader\x1B[0m
\x1B[0;32mI (43) boot: compile time Jun  7 2023 08:02:08\x1B[0m
\x1B[0;32mI (44) boot: chip revision: v0.0\x1B[0m
\x1B[0;32mI (47) boot.esp32c6: SPI Speed      : 40MHz\x1B[0m
\x1B[0;32mI (52) boot.esp32c6: SPI Mode       : DIO\x1B[0m
\x1B[0;32mI (57) boot.esp32c6: SPI Flash Size : 8MB\x1B[0m
\x1B[0;32mI (62) boot: Enabling RNG early entropy source...\x1B[0m
\x1B[0;32mI (67) boot: Partition Table:\x1B[0m
\x1B[0;32mI (71) boot: ## Label            Usage          Type ST Offset   Length\x1B[0m
\x1B[0;32mI (78) boot:  0 nvs              WiFi data        01 02 00009000 00006000\x1B[0m
\x1B[0;32mI (85) boot:  1 phy_init         RF data          01 01 0000f000 00001000\x1B[0m
\x1B[0;32mI (93) boot:  2 factory          factory app      00 00 00010000 007f0000\x1B[0m
\x1B[0;32mI (100) boot: End of partition table\x1B[0m
\x1B[0;32mI (105) esp_image: segment 0: paddr=00010020 vaddr=42000020 size=413f4h (267252) map\x1B[0m
\x1B[0;32mI (167) esp_image: segment 1: paddr=0005141c vaddr=40800000 size=06bfch ( 27644) load\x1B[0m
\x1B[0;32mI (175) esp_image: segment 2: paddr=00058020 vaddr=42048020 size=1da78h (121464) map\x1B[0m
\x1B[0;32mI (200) esp_image: segment 3: paddr=00075aa0 vaddr=40806bfc size=0193ch (  6460) load\x1B[0m
\x1B[0;32mI (207) esp_image: segment 4: paddr=000773e4 vaddr=40808540 size=00fcch (  4044) load\x1B[0m
\x1B[0;32mI (211) boot: Loaded app from partition at offset 0x10000\x1B[0m
\x1B[0;32mI (212) boot: Disabling RNG early entropy source...\x1B[0m
\x1B[0;32mI (229) cpu_start: Unicore app\x1B[0m
\x1B[0;32mI (229) cpu_start: Pro cpu up.\x1B[0m
\x1B[0;33mW (238) clk: esp_perip_clk_init() has not been implemented yet\x1B[0m
\x1B[0;32mI (244) cpu_start: Pro cpu start user code\x1B[0m
\x1B[0;32mI (244) cpu_start: cpu freq: 160000000 Hz\x1B[0m
\x1B[0;32mI (245) cpu_start: Application information:\x1B[0m
\x1B[0;32mI (247) cpu_start: Project name:     libespidf\x1B[0m
\x1B[0;32mI (252) cpu_start: App version:      1\x1B[0m
\x1B[0;32mI (257) cpu_start: Compile time:     Jan 22 2024 13:37:02\x1B[0m
\x1B[0;32mI (263) cpu_start: ELF file SHA256:<PHONE_NUMBER>000000...\x1B[0m
\x1B[0;32mI (269) cpu_start: ESP-IDF:          v5.1\x1B[0m
\x1B[0;32mI (274) cpu_start: Min chip rev:     v0.0\x1B[0m
\x1B[0;32mI (278) cpu_start: Max chip rev:     v0.99 \x1B[0m
\x1B[0;32mI (283) cpu_start: Chip rev:         v0.0\x1B[0m
\x1B[0;32mI (288) heap_init: Initializing. RAM available for dynamic allocation:\x1B[0m
\x1B[0;32mI (295) heap_init: At 4080A430 len 000721E0 (456 KiB): D/IRAM\x1B[0m
\x1B[0;32mI (301) heap_init: At 4087C610 len 00002F54 (11 KiB): STACK/DIRAM\x1B[0m
\x1B[0;32mI (308) heap_init: At 50000010 len 00003FF0 (15 KiB): RTCRAM\x1B[0m
\x1B[0;32mI (315) spi_flash: detected chip: generic\x1B[0m
\x1B[0;32mI (319) spi_flash: flash io: dio\x1B[0m
\x1B[0;33mW (323) timer_group: legacy driver is deprecated, please migrate to `driver/gptimer.h`\x1B[0m
\x1B[0;32mI (335) sleep: Configure to isolate all GPIO pins in sleep state\x1B[0m
\x1B[0;32mI (339) sleep: Enable automatic switching of GPIO sleep configuration\x1B[0m
\x1B[0;32mI (346) coexist: coex firmware version: ebddf30\x1B[0m
\x1B[0;32mI (351) coexist: coexist rom version 5b8dcfa\x1B[0m
\x1B[0;32mI (356) app_start: Starting scheduler on CPU0\x1B[0m
\x1B[0;32mI (361) main_task: Started on CPU0\x1B[0m
\x1B[0;32mI (361) main_task: Calling app_main()\x1B[0m
\x1B[0;32mI (361) flashproblems: Hello, world!\x1B[0m
\x1B[0;32mI (371) main_task: Returned from app_main()\x1B[0m

If I'm doing  espflash monitor (using stub), I don't get the broken pipe, but the target also hangs at "waiting for download" (it does not reset after 8 seconds though)
I'm on ubuntu 22.04 and I'm using espflash 2.1.0 (although it is the same with espflash from git (master))

I've also tried to use the esp idf provided bootloader, e.g. flash with espflash flash --bootloader target/riscv32imac-esp-espidf/debug/build/esp-idf-sys-7af63acd3e33c9a6/out/build/bootloader/bootloader.bin --monitor target/riscv32imac-esp-espidf/debug/flashproblems
for v5.1.1: same problem.
for v5.2-beta2: same problem.

Just tried reproducing the error and I was not able to reproduce it:

Thanks for trying to reproduce this @SergioGasquez .
I can also not reproduce this reliably.  There are times where this issue occurs constantly and then there are times where it does not occur for hours/days.
I'll keep this issue open for a while and try to add more info, as soon as the issue occurs again.

Just as an experiment, could you try the following?
cargo install espflash --git https://github.com/SergioGasquez/espflash --branch feat/reset-args
espflash monitor --before no-reset-no-sync --chip esp32c6

I'm just dropping in here to say I have seen what seems to be the same issue with my new esp32c6. With a fresh minimal esp-template project:
$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.39s
     Running `espflash flash --monitor target/riscv32imac-unknown-none-elf/debug/exp3`
[2024-01-31T18:43:17Z INFO ] Detected 6 serial ports
[2024-01-31T18:43:17Z INFO ] Ports which match a known common dev board are highlighted
[2024-01-31T18:43:17Z INFO ] Please select a port
✔ Remember this serial port for future use? · no
[2024-01-31T18:43:19Z INFO ] Serial port: '/dev/cu.usbmodem141401'
[2024-01-31T18:43:19Z INFO ] Connecting...
[2024-01-31T18:43:20Z INFO ] Using flash stub
Chip type:         esp32c6 (revision v0.0)
Crystal frequency: 40MHz
Flash size:        8MB
Features:          WiFi 6, BT 5
MAC address:       40:4c:ca:4e:6a:28
App/part. size:    202,480/8,323,072 bytes, 2.43%
[00:00:00] [========================================]      13/13      0x0
[00:00:00] [========================================]       1/1       0x8000
[00:00:01] [========================================]      79/79      0x10000                                                                                [2024-01-31T18:43:21Z INFO ] Flashing has completed!
Commands:
    CTRL+R    Reset chip
    CTRL+C    Exit

ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0x15 (USB_UART_HPSYS),boot:0x4 (DOWNLOAD(USB/UART0/SDIO_FEI_FEO))
Saved PC:0x408007fe
0x408007fe - core::option::Option<T>::unwrap
    at /Users/<user>/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/src/rust/library/core/src/option.rs:929
waiting for download

I don't have time right this moment, but I wanted to chime in and say there's at least one other occurrence.  It looks like I should investigate updating the firmware and I'll try the suggested experiment, but I probably won't get to those until tomorrow.

I have seen this too, to reproduce my case:

Manually put the chip into download mode, hold boot whilst resetting the chip
flash it with espflash
Observe the rom download mode message

Part of me was wondering if this might be a tooling configuration issue (until today I had not gotten any Rust to run on an esp32c6), and wondered if there was some weirdness in my macos setup. But it seems to not be the case.
My experience above was with a Waveshare ESP32-C6-DEV-KIT-N8, which is a custom package around a ESP32-C6-WROOM-1. It still exhibits this issue.
However, 20 minutes ago I received a delivery of a proper Espressif ESP32-C6-DevKitC-1, which also uses the ESP32-C6-WROOM-1. I popped it out of the box, plugged it in, ran cargo run, and it worked perfectly:
$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 7.03s
     Running `espflash flash --monitor target/riscv32imac-unknown-none-elf/debug/exp3`
[2024-02-01T21:08:07Z INFO ] Detected 4 serial ports
[2024-02-01T21:08:07Z INFO ] Ports which match a known common dev board are highlighted
[2024-02-01T21:08:07Z INFO ] Please select a port
[2024-02-01T21:08:15Z INFO ] Serial port: '/dev/cu.usbserial-1410'
[2024-02-01T21:08:15Z INFO ] Connecting...
[2024-02-01T21:08:15Z INFO ] Using flash stub
Chip type:         esp32c6 (revision v0.0)
Crystal frequency: 40MHz
Flash size:        8MB
Features:          WiFi 6, BT 5
MAC address:       40:4c:ca:51:39:7c
App/part. size:    202,480/8,323,072 bytes, 2.43%
[00:00:01] [========================================]      13/13      0x0
[00:00:00] [========================================]       1/1       0x8000
[00:00:07] [========================================]      79/79      0x10000                                                                                [2024-02-01T21:08:25Z INFO ] Flashing has completed!
Commands:
    CTRL+R    Reset chip
    CTRL+C    Exit

ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0x1 (POWERON),boot:0xc (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:2
load:0x4086c410,len:0xd48
0x4086c410 - _ESP_HAL_DEVICE_PERIPHERALS
    at ??:??
load:0x4086e610,len:0x2d68
0x4086e610 - _ESP_HAL_DEVICE_PERIPHERALS
    at ??:??
load:0x40875720,len:0x1800
0x40875720 - _ESP_HAL_DEVICE_PERIPHERALS
    at ??:??
SHA-256 comparison failed:
Calculated: 377c26481ad6ab0cce508450f353a55f4bd7eb9159aaf85fde3111071f164514
Expected: 0af544a033ab3492852b8232c904c578d5f07c9d4a423a64473f060db374ab32
Attempting to boot anyway...
entry 0x4086c410
0x4086c410 - _ESP_HAL_DEVICE_PERIPHERALS
    at ??:??
I (42) boot: ESP-IDF v5.1-beta1-378-gea5e0ff298-dirt 2nd stage bootloader
I (42) boot: compile time Jun  7 2023 08:02:08
I (43) boot: chip revision: v0.0
I (47) boot.esp32c6: SPI Speed      : 40MHz
I (52) boot.esp32c6: SPI Mode       : DIO
I (57) boot.esp32c6: SPI Flash Size : 8MB
I (61) boot: Enabling RNG early entropy source...
I (67) boot: Partition Table:
I (70) boot: ## Label            Usage          Type ST Offset   Length
I (78) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (85) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (92) boot:  2 factory          factory app      00 00 00010000 007f0000
I (100) boot: End of partition table
I (104) esp_image: segment 0: paddr=00010020 vaddr=42000020 size=28ab4h (166580) map
I (147) esp_image: segment 1: paddr=00038adc vaddr=40800000 size=0001ch (    28) load
I (147) esp_image: segment 2: paddr=00038b00 vaddr=42028b00 size=078f4h ( 30964) map
I (159) esp_image: segment 3: paddr=000403fc vaddr=4080001c size=012c4h (  4804) load
I (162) boot: Loaded app from partition at offset 0x10000
I (167) boot: Disabling RNG early entropy source...
INFO - Logger is setup
Hello world!
Loop...
Loop...
Loop...
Loop...
<you get the idea>

My new (working) board has the same ESP-ROM and Build date as the one that doesn't work.
For completeness, I'll mention that the espflash board-info reports the exact same for both boards aside from the MAC (below happens to be for the Espressif kit):
$ espflash board-info
[2024-02-01T21:10:43Z INFO ] Detected 4 serial ports
[2024-02-01T21:10:43Z INFO ] Ports which match a known common dev board are highlighted
[2024-02-01T21:10:43Z INFO ] Please select a port
[2024-02-01T21:10:45Z INFO ] Serial port: '/dev/cu.usbserial-1410'
[2024-02-01T21:10:45Z INFO ] Connecting...
[2024-02-01T21:10:45Z INFO ] Using flash stub
Chip type:         esp32c6 (revision v0.0)
Crystal frequency: 40MHz
Flash size:        8MB
Features:          WiFi 6, BT 5
MAC address:       ...

I dont think this is hardware related.
I had the issue occur with various boards (also different types) and then again it does not appear for days on the exact same hardware. Also, if I remember correctly, idf.py monitor always works, even if espflash monitor fails.
This looks like some timing issue to me...
Without having looked at the source, I imagine there is a memory location or register which tells the first stage bootloader to stay in bootloader mode after reset and do not continue with the normal boot process (so that you can send more commands to it). Can it be that theres old data in this location/register? Just strawballing here...

does this timeout behavior persist after a cold boot? I also saw it multiple times with current HEAD, even after cold boot, i could only got the timeout cleared by manually forcing it into download mode, but not via cold boot, indicating that it didn't reset the esp correctly after a first attach.

Yes, the issue persists over unplug/replug until I manually put the device into download mode.

Tried reproducing the issue with @MabezDev steps and using<EMAIL_ADDRESS>

Manually put the chip into download mode, hold boot whilst resetting the chip

esp32c6 on  main [?] is 📦 v0.1.0 via 🦀 v1.78.0-nightly
❯ espflash monitor --before no-reset-no-sync --chip esp32c6
[2024-02-20T14:56:44Z WARN ] Pre-connection option 'NoResetNoSync' was selected. Connection may fail if the chip is not in bootloader or flasher stub mode.
[2024-02-20T14:56:44Z INFO ] Serial port: '/dev/ttyUSB0'
[2024-02-20T14:56:44Z INFO ] Connecting...
Commands:
    CTRL+R    Reset chip
    CTRL+C    Exit

Loop...
Loop...
ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0x1 (POWERON),boot:0x4 (DOWNLOAD(USB/UART0/SDIO_FEI_FEO))
waiting for download

flash it with espflash
Observe the rom download mode message

esp32c6 on  main [?] is 📦 v0.1.0 via 🦀 v1.78.0-nightly took 6s
❯ cargo r
    Finished dev [unoptimized + debuginfo] target(s) in 0.02s
     Running `espflash flash --monitor target/riscv32imac-unknown-none-elf/debug/esp32c6`
[2024-02-20T14:57:00Z INFO ] Serial port: '/dev/ttyUSB0'
[2024-02-20T14:57:00Z INFO ] Connecting...
[2024-02-20T14:57:01Z INFO ] Using flash stub
[2024-02-20T14:57:02Z WARN ] Setting baud rate higher than 115,200 can cause issues
Chip type:         esp32c6 (revision v0.0)
Crystal frequency: 40 MHz
Flash size:        8MB
Features:          WiFi 6, BT 5
MAC address:       60:55:f9:f7:1b:6c
App/part. size:    172,384/4,128,768 bytes, 4.18%
[2024-02-20T14:57:02Z INFO ] Flashing has completed!
Commands:
    CTRL+R    Reset chip
    CTRL+C    Exit

ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0x1 (POWERON),boot:0xc (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:2
load:0x4086c410,len:0xd48
load:0x4086e610,len:0x2d68
load:0x40875720,len:0x1800
entry 0x4086c410
I (23) boot: ESP-IDF v5.1-beta1-378-gea5e0ff298-dirt 2nd stage bootloader
I (24) boot: compile time Jun  7 2023 08:02:08
I (25) boot: chip revision: v0.0
I (29) boot.esp32c6: SPI Speed      : 40MHz
I (33) boot.esp32c6: SPI Mode       : DIO
I (38) boot.esp32c6: SPI Flash Size : 4MB
I (43) boot: Enabling RNG early entropy source...
I (49) boot: Partition Table:
I (52) boot: ## Label            Usage          Type ST Offset   Length
I (59) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (67) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (74) boot:  2 factory          factory app      00 00 00010000 003f0000
I (82) boot: End of partition table
I (86) esp_image: segment 0: paddr=00010020 vaddr=42000020 size=22dd0h (142800) map
I (123) esp_image: segment 1: paddr=00032df8 vaddr=40808000 size=00020h (    32) load
I (124) esp_image: segment 2: paddr=00032e20 vaddr=42022e20 size=06044h ( 24644) map
I (134) esp_image: segment 3: paddr=00038e6c vaddr=40808020 size=012c4h (  4804) load
I (139) boot: Loaded app from partition at offset 0x10000
I (144) boot: Disabling RNG early entropy source...
Hello world!
Loop...
Loop...

@SergioGasquez the initial issue was about the usb/jtag connection not the uart one. There might be differences here how this is influencing it

Yes sorry I should have been more clear, I've only seen it happen with USB jtag.

Oh! Completely forgot that “small” detail.

Manually put the chip into download mode, hold boot whilst resetting the chip

Loop...
       ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0x1 (POWERON),boot:0x67 (DOWNLOAD(USB/UART0/SDIO_REI_REO))
waiting for download

flash it with espflash
Observe the rom download mode message

esp32c6 on  main [?] is 📦 v0.1.0 via 🦀 v1.78.0-nightly took 9s
❯ cargo r
    Finished dev [unoptimized + debuginfo] target(s) in 0.02s
     Running `espflash flash --monitor target/riscv32imac-unknown-none-elf/debug/esp32c6`
[2024-02-20T15:32:41Z INFO ] Serial port: '/dev/ttyACM0'
[2024-02-20T15:32:41Z INFO ] Connecting...
[2024-02-20T15:32:41Z INFO ] Using flash stub
[2024-02-20T15:32:41Z WARN ] Setting baud rate higher than 115,200 can cause issues
Chip type:         esp32c6 (revision v0.0)
Crystal frequency: 40 MHz
Flash size:        8MB
Features:          WiFi 6, BT 5
MAC address:       60:55:f9:f7:1b:6c
App/part. size:    172,384/4,128,768 bytes, 4.18%
[2024-02-20T15:32:41Z INFO ] Flashing has completed!
Commands:
    CTRL+R    Reset chip
    CTRL+C    Exit

ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0x15 (USB_UART_HPSYS),boot:0x67 (DOWNLOAD(USB/UART0/SDIO_REI_REO))
Saved PC:0x4080042e
waiting for download

Serial port shows:
<logs from step1>
ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0x15 (USB_UART_HPSYS),boot:0x67 (DOWNLOAD(USB/UART0/SDIO_REI_REO))
Saved PC:0x40017600
waiting for download
ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0x15 (USB_UART_HPSYS),boot:0x67 (DOWNLOAD(USB/UART0/SDIO_REI_REO))
Saved PC:0x4080042e
waiting for download

I've also managed to reproduce this behavior with esptool.

Manually put the chip into download mode, hold boot whilst resetting the chip:

...
I (283) main_task: Started on CPU0
I (283) main_task: Calling app_main()
Hello world!
This is esp32c6 chip with 1 CPU core(s), WiFi/BLE, 802.15.4 (Zigbee/Thread), silicon revision v0.0, 2MB external flash
Minimum free heap size: 473304 bytes
Restarting in 10 seconds...
Restarting in 9 seconds...
ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0x1 (POWERON),boot:0x4 (DOWNLOAD(USB/UART0/SDIO_FEI_FEO))
waiting for download

Flash it with espflash
Observe the rom download mode message

…ples/get-started/hello_world (11eaf41) via △ v3.24.0 via 🐍 v3.11.7 took 7s
❯ idf.py -p /dev/ttyACM0 flash monitor
Executing action: flash
Running ninja in directory /home/sergio/Documents/Espressif/espressif/esp-idf_v5.2/examples/get-started/hello_world/build
Executing "ninja flash"...
[1/5] cd /home/sergio/Documents/Espre...ted/hello_world/build/hello_world.bin
hello_world.bin binary size 0x22c30 bytes. Smallest app partition is 0x100000 bytes. 0xdd3d0 bytes (86%) free.
[1/1] cd /home/sergio/Documents/Espre...world/build/bootloader/bootloader.bin
Bootloader binary size 0x5440 bytes. 0x2bc0 bytes (34%) free.
[2/3] cd /home/sergio/Documents/Espre...ents/esptool_py/run_serial_tool.cmake
esptool.py --chip esp32c6 -p /dev/ttyACM0 -b 460800 --before=default_reset --after=hard_reset write_flash --flash_mode dio --flash_freq 80m --flash_size 2MB 0x0 bootloader/bootloader.bin 0x10000 hello_world.bin 0x8000 partition_table/partition-table.bin
esptool.py v4.7.0
Serial port /dev/ttyACM0
Connecting...
Chip is ESP32-C6 (QFN40) (revision v0.0)
Features: WiFi 6, BT 5, IEEE802.15.4
Crystal is 40MHz
MAC: 60:55:f9:00:00:f7:1b:6c
BASE MAC: 60:55:f9:f7:1b:6c
MAC_EXT: 00:00
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Flash will be erased from 0x00000000 to 0x00005fff...
Flash will be erased from 0x00010000 to 0x00032fff...
Flash will be erased from 0x00008000 to 0x00008fff...
Compressed 21568 bytes to 12983...
Writing at 0x00000000... (100 %)
Wrote 21568 bytes (12983 compressed) at 0x00000000 in 0.2 seconds (effective 710.9 kbit/s)...
Hash of data verified.
Compressed 142384 bytes to 74629...
Writing at 0x0002eb10... (100 %)
Wrote 142384 bytes (74629 compressed) at 0x00010000 in 0.7 seconds (effective 1707.7 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 103...
Writing at 0x00008000... (100 %)
Wrote 3072 bytes (103 compressed) at 0x00008000 in 0.0 seconds (effective 743.6 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...
Executing action: monitor
Running idf_monitor in directory /home/sergio/Documents/Espressif/espressif/esp-idf_v5.2/examples/get-started/hello_world
Executing "/home/sergio/.espressif/python_env/idf5.2_py3.11_env/bin/python /home/sergio/Documents/Espressif/espressif/esp-idf_v5.2/tools/idf_monitor.py -p /dev/ttyACM0 -b 115200 --toolchain-prefix riscv32-esp-elf- --target esp32c6 --revision 0 --decode-panic backtrace /home/sergio/Documents/Espressif/espressif/esp-idf_v5.2/examples/get-started/hello_world/build/hello_world.elf -m '/home/sergio/.espressif/python_env/idf5.2_py3.11_env/bin/python' '/home/sergio/Documents/Espressif/espressif/esp-idf_v5.2/tools/idf.py' '-p' '/dev/ttyACM0'"...
--- esp-idf-monitor 1.4.0 on /dev/ttyACM0 115200 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0x15 (USB_UART_HPSYS),boot:0x4 (DOWNLOAD(USB/UART0/SDIO_FEI_FEO))
Saved PC:0x40017604
0x40017604: ets_delay_us in ROM

waiting for download

If I flash with espflashand then run idf.py monitor, I still get the same output:
…ples/get-started/hello_world (11eaf41) via △ v3.24.0 via 🐍 v3.11.7 took 3s
❯ idf.py -p /dev/ttyACM0 monitor
Executing action: monitor
Running idf_monitor in directory /home/sergio/Documents/Espressif/espressif/esp-idf_v5.2/examples/get-started/hello_world
Executing "/home/sergio/.espressif/python_env/idf5.2_py3.11_env/bin/python /home/sergio/Documents/Espressif/espressif/esp-idf_v5.2/tools/idf_monitor.py -p /dev/ttyACM0 -b 115200 --toolchain-prefix riscv32-esp-elf- --target esp32c6 --revision 0 --decode-panic backtrace /home/sergio/Documents/Espressif/espressif/esp-idf_v5.2/examples/get-started/hello_world/build/hello_world.elf -m '/home/sergio/.espressif/python_env/idf5.2_py3.11_env/bin/python' '/home/sergio/Documents/Espressif/espressif/esp-idf_v5.2/tools/idf.py' '-p' '/dev/ttyACM0'"...
--- esp-idf-monitor 1.4.0 on /dev/ttyACM0 115200 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0x15 (USB_UART_HPSYS),boot:0x4 (DOWNLOAD(USB/UART0/SDIO_FEI_FEO))
Saved PC:0x40017600
0x40017600: ets_delay_us in ROM

waiting for download

Which is strange because @t-moe mentioned:

but using idf.py monitor (from idf master branch) instead of espflash monitor works.

I again today got the problem,  running with RUST_LOG=debug espflash monitor didnt bring anything new to the table. But as soon as i run it only with espflash monitor --no-stub i at least could connect again and then could ctrl + r to reset and start normally. So the timeout happens when trying to write the stub on espflash monitor and than it will put the board in a non working state with espflash.

I'll try to capture the raw usb packets, the next time I observe this.
( https://wiki.wireshark.org/CaptureSetup/USB )

I again today got the problem, running with RUST_LOG=debug espflash monitor didnt bring anything new to the table. But as soon as i run it only with espflash monitor --no-stub i at least could connect again and then could ctrl + r to reset and start normally. So the timeout happens when trying to write the stub on espflash monitor and than it will put the board in a non working state with espflash.

Looks like the behavior described in https://github.com/esp-rs/esp-flasher-stub/issues/45

I'm having this problem as well with a new Waveshare ESP32-C6-DEV-KIT-N8 board. I have not been able to see "Hello, World!" at all yet.
I'm using espflash v3.0.0.
I'm selecting the "USB JTAG_serial debug unit" when flashing, as selecting the "USB Single Serial" doesn't seem to work.
Flashing completes successfully, but the output is then stuck on "waiting for download" instead of printing "Hello, World!".
I'm using a clean template, with the only added thing being CRATE_CC_NO_DEFAULTS = "1" (see https://github.com/esp-rs/esp-idf-template/issues/174).
$ cargo run --release
    Finished `release` profile [optimized] target(s) in 0.21s
     Running `espflash flash --monitor target/riscv32imac-esp-espidf/release/esp32-airplay-demo`
[2024-05-09T17:08:35Z INFO ] Detected 4 serial ports
[2024-05-09T17:08:35Z INFO ] Ports which match a known common dev board are highlighted
[2024-05-09T17:08:35Z INFO ] Please select a port
[2024-05-09T17:09:00Z INFO ] Serial port: '/dev/tty.usbmodem1401'
[2024-05-09T17:09:00Z INFO ] Connecting...
[2024-05-09T17:09:00Z INFO ] Using flash stub
Chip type:         esp32c6 (revision v0.0)
Crystal frequency: 40 MHz
Flash size:        8MB
Features:          WiFi 6, BT 5
MAC address:       40:4c:ca:43:a1:48
App/part. size:    332,752/4,128,768 bytes, 8.06%
[2024-05-09T17:09:00Z INFO ] Segment at address '0x0' has not changed, skipping write
[2024-05-09T17:09:00Z INFO ] Segment at address '0x8000' has not changed, skipping write
[2024-05-09T17:09:00Z INFO ] Segment at address '0x10000' has not changed, skipping write
[2024-05-09T17:09:01Z INFO ] Flashing has completed!
Commands:
    CTRL+R    Reset chip
    CTRL+C    Exit

ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0x15 (USB_UART_HPSYS),boot:0x4 (DOWNLOAD(USB/UART0/SDIO_FEI_FEO))
Saved PC:0x408005b4
0x408005b4 - spi_flash_cache_enabled
    at ??:??
waiting for download

Pressing CTRL+R to reset the chip seems to work, but it then stops at "waiting for download" again.
Will continue to investigate...

I have the same "waiting for download" issue while flashing on Xiao ESP32s3. After a day, I can work well will the following command. Let try and see if it works for you: (they key is --before & --after)
espflash flash --list-all-ports --before usb-reset --after hard-reset --monitor --chip esp32s3 --flash-size 8mb target/xtensa-esp32s3-none-elf/release/the_no-std_elf_file
Hope that useful,
Trinh Tuan

Same issue here with ESP32H2 WROOM-03 (espflash v3.0.0) using minimal connections for USB flash (IO9 pulled down, D+, D-, 3.3V, GND).
Running `espflash flash --monitor target/riscv32imac-esp-espidf/debug/floor-light`
[2024-05-25T16:02:41Z INFO ] Serial port: '/dev/ttyACM0'
[2024-05-25T16:02:41Z INFO ] Connecting...
[2024-05-25T16:02:42Z INFO ] Using flash stub
Chip type:         esp32h2 (revision v0.1)
Crystal frequency: 32 MHz
Flash size:        2MB
Features:          BLE
MAC address:       74:4d:bd:61:6f:e8
App/part. size:    559,504/4,128,768 bytes, 13.55%
[2024-05-25T16:02:42Z INFO ] Segment at address '0x0' has not changed, skipping write
[2024-05-25T16:02:42Z INFO ] Segment at address '0x8000' has not changed, skipping write
[00:00:05] [========================================]     268/268     0x10000                                           [2024-05-25T16:02:49Z INFO ] Flashing has completed!
Commands:
    CTRL+R    Reset chip
    CTRL+C    Exit

ESP-ROM:esp32h2-20221101
Build:Nov  1 2022
rst:0x15 (USB_UART_HPSYS),boot:0x0 (USB_BOOT)
Saved PC:0x408005b6
0x408005b6 - i2c_ll_read_rxfifo
    at /home/bitmagier/Workspace/opensource-me/floor-light/code/.embuild/espressif/esp-idf/v5.1.2/components/hal/esp32h2/include/hal/i2c_ll.h:587
wait usb download

@bitmagier did you try the the different -b and -a options like mention previously? you can run espflash flash help to get a list of the possible params

@bitmagier did you try the the different -b and -a options like mention previously? you can run espflash flash help to get a list of the possible params

Just tried - there is no positive effect.
(Doing the same flash on the development board works just fine)
BTW: I'm wondering if my PIN configuration is really sufficient.
Specifically GPIO8 - for "joint boot mode" according to spec it should be pulled UP (default is floating), but there is no GPIO8 exposed on this module.

Yeah – I believe my issue is caused by a missing hardware reset button wired to EN to properly enable the Join Download Boot Mode.
Thanks for your attention.


[Bug] JWE header alg attribute with value http://www.w3.org/2001/04/xmlenc#rsa-oaep instead of RSA-OAEP

Which version of Microsoft.IdentityModel are you using?
Microsoft.IdentityModel 6.30.1
Where is the issue?

[ X] M.IM.JsonWebTokens
[ ] M.IM.KeyVaultExtensions
[ ] M.IM.Logging
[ ] M.IM.ManagedKeyVaultSecurityKey
[ ] M.IM.Protocols
[ ] M.IM.Protocols.OpenIdConnect
[ ] M.IM.Protocols.SignedHttpRequest
[ ] M.IM.Protocols.WsFederation
[ ] M.IM.TestExtensions
[ X] M.IM.Tokens
[ ] M.IM.Tokens.Saml
[ ] M.IM.Validators
[ ] M.IM.Xml
[X ] S.IM.Tokens.Jwt
Other (please describe)

Is this a new or an existing app?
The app is in production and next release requires interoperability with other apps written in different languages (ie: GoLang)
Repro
*SigningCredentials and EncryptingCredentials was created using a X509Certificate2 with public and private keys
            SecurityTokenDescriptor tokenDescriptor = new SecurityTokenDescriptor
            {
                Issuer = _openIdConfigurationService.OpenIdConfiguration.Issuer,
                IssuedAt = issuedAt,
                Subject = request.Claims,
                NotBefore = issuedAt.AddSeconds(-1),
                Expires = expires,
                SigningCredentials = credentials.SigningCredentials,
                EncryptingCredentials = credentials.EncryptingCredentials,
                TokenType = "JWE"
            };

            JwtSecurityTokenHandler tokenHandler = new JwtSecurityTokenHandler();
            JwtSecurityToken token = tokenHandler.CreateJwtSecurityToken(tokenDescriptor);

Expected behavior
Algorithm (alg) should be one of the registered at IANA "JSON Web Signature and Encryption Algorithms" (https://www.iana.org/assignments/jose/jose.xhtml) as described on topic 4.1.1 of RFC7516 (https://datatracker.ietf.org/doc/html/rfc7516#page-12)
{
  "alg": "RSA-OAEP",
  "enc": "A128CBC-HS256",
  "kid": "0758B2A8DF6B980E569421870D7CFE29E65D26EB",
  "typ": "JWE",
  "cty": "JWT"
}

Actual behavior
{
  "alg": "http://www.w3.org/2001/04/xmlenc#rsa-oaep",
  "enc": "A128CBC-HS256",
  "kid": "0758B2A8DF6B980E569421870D7CFE29E65D26EB",
  "typ": "JWE",
  "cty": "JWT"
}

Possible solution
Change the KeyWrapper providers to declare header alg complaint to IANA document

@RogerioWagner sorry for the hassle, we will fix this.

@brentschmaltz, thanks for the reply! I have a test environment where a Go client need to validate and decrypt a JWE issued by a dotnet core service, count on me to help test the new version identity model assembly

Just saw this @RogerioWagner , thanks for offering to test the new version. It's already out, we had to hit .NET 8's RC 2 with a non-preview version. If you have any feedback on 7.0.0, we would love to hear it. I assume this issue is still present in 7.0.0?

@brentschmaltz, thanks for the reply! I have a test environment where a Go client need to validate and decrypt a JWE issued by a dotnet core service, count on me to help test the new version identity model assembly

Just saw this @RogerioWagner , thanks for offering to test the new version. It's already out, we had to hit .NET 8's RC 2 with a non-preview version. If you have any feedback on 7.0.0, we would love to hear it. I assume this issue is still present in 7.0.0?

Hi @jennyf19, sorry for the delay I was OOF and did not saw your post.
I think it's not fixed yet; I bumped the System.IdentityModel.Tokens.Jwt to version 7.0.1 and the JWE issued had the below header:
{
  "alg": "http://www.w3.org/2001/04/xmlenc#rsa-oaep",
  "enc": "A128CBC-HS256",
  "kid": "1359A14E054F33548B84BF8521997162EE6CC443",
  "typ": "JWE",
  "cty": "JWT"
}

Let me know if I can contribute with any further information.

@RogerioWagner thanks, we will look into this.


Added support for any type of channel.

Added support for any type of channel.

Can you remind me what's the current limitation?
Also we'll likely need some tests if there's behavioral change.

Can you remind me what's the current limitation?

We can't use the typed channel with linq package.
Before:
origCh := make(chan int)
srcCh := make(chan interface{})

// Forward
go func() {
    for {
        srcCh <- origCh
    }
}()

From(srcCh)...

Now:
origCh := make(chan int)
From(origCh)...

Also we'll likely need some tests if there's behavioral change.

I'm not sure about additional tests because logic was not changed. The old test is good for these changes.

I think at least we can change tests to be chan int instead of chan interface{} to verify that it indeed works the way you wanted.

Agreed. I changed from chan interface{} to chan int.

@casskir I am generally ok with this idea, though since you use reflection to receive from the channel I expect the performance of this method to be worse. So, it would probably be better to introduce a new method, say, FromTypedChannel().
I did a benchmark
BenchmarkFromChannel-12                  	      10	 189264414 ns/op
BenchmarkFromTypedChannel-12             	       3	 758983592 ns/op

And I did a pr #90 with the new method.
@casskir @ahmetb what do you think?

I think it more flexible, especially in case if performance is important. @ahmetb please close pr #89 if you agree.

Yeah #90 is preferable, since we have a generic worse-performing From for syntactic sugar and FromXxx for less-guesswork and therefore better performance.
Thanks a lot for this patch though. Make sure #90 addresses your problem.


Mekton Zeta Equipment Name Width

The width for the Equipment name table row is not set to 100% of the available space.
I did some experimentation with Chrome's Inspect tool, and the value is set to 120px.
Here is all of the information that I found, along with a screenshot of the issue.

After setting the width to 100%, the following was achieved:

I had a quick look through the files, but I'm not a web developer, so I'm unsure where the correct place to make a change is. Sorry!

(Bonus while anybody is looking at the above: There appears to be no section for putting your character's current funds into your sheet. Not sure if intentional or not!)

@CoalPoweredPuppet I think this was one of yours. Are you interested in doing some updates?

I will see what I can do.

Fantastic!
Also works a charm on mobile, which was an unexpected joy - first time I’ve logged into a Roll20 game on one.

Hi @CoalPoweredPuppet
After playing a session, my group and I discovered some other issues with the sheet which are hopefully simple to fix. I thought I'd put them back in here instead of raising another issue.
(I'll also update the main issue post to reflect everything once I'm done!)

The "Endurance" skill has no entry in the Total column
Each Intelligence skill beneath "Know Language" is rolled with the word "Know" before it in the chat log - not an issue as such, but just a cosmetic thing I thought could be pointed out
When rolling "Jury Rig", the chat log displays "First Aid" in the message
It does still use the correct values for the "Jury Rig" skill
The skill "Photography & Film" is mistyped as "Photgraphy"
When rolling "Endurance" and "Aircraft/Shuttle Pilot", you get errors in the chat log as below. This persists even with a non-zero skill level.

No attribute was found for @{Character|Level-Endurance}
No attribute was found for @{Character|Attribute-Aircraft_Shuttle_Pilot}

...joy.  I will work on it.
In the mean time, I can set up a game under my account, make you the GM, and let you run a game if timing is an issue

... sorry for being a pain 😬
There's no need to rush, though! We're alternating games between sessions, so it might be a while before we get back to Mekton....


Fixing console errors in tabs and conversations

💬 Description
Cleaning up some prop-types to reduce the console noise in tabs + conversations.
✅ Checklist

[ ] Tests written
[ ] Browser tested
[ ] Added to documentation
[ ] Added to storybook

 Tidy up console errors for Tabs & Conversations


How about show [Type] when the label of field is repeated?

for example,
message AdRequest {
  // List of important key words from the current page describing the context.
  repeated string context_keys = 1;
}

The Type of context_keys is string.
I think [string] is more intuitive.

I'd also like this! May make a PR if I have a moment


Create mirror repository from API

Feature Description
I recently saw that Gitea did not provide a way to create a mirror from API, the strange thing is that you can actually update the mirror interval from the endpoint /repos/{owner}/{repo}.
Do you guys open PR for this implementation? Is that possible?
Screenshots
No response

This is indeed possible via the migration API by supplying mirror: true and/or an optional interval.


[RFC] nativefier-catalog and interactivity

Nativefier catalog is a catalog of configs for nativefier apps. It currently stores app configs as a json file with the following structure:
{
 "app-token": {
   "name": "App Name",
   "targetUrl": "https://appurl.example/",
   "counter": true,
   "bounce": true
 }
}

The options are passed directly to the nativefier Programmatic API.
The UI is built using enquirer autocomplete prompts and shows a searchable list of available app configs.

I also could make a graphical way of viewing it to go along with my nativefier-gui project.
The json file is curently stored in the same repo as the code but should probably be moved to a different repo to seperate app adition prs from code pr like how Homebrew/brew (package manager code) is separate from Homebrew/homebrew-core (package repo). Having a repo of just configs could allow multiple different clients to all access it and share the same configs. (the default nativefier cli (see question 3), nativefier-catalog cli, nativefier-gui, something else?... )
Questions

Should this integrate with the existing nativefier-icons repo somehow?

Should there be a folder for userscripts to go along with apps.json or should they be linked as gist urls?

Should there be some kind of url (regex?) matching so the configs are used by default on the standard nativefier cli? (eg. use counter by default on gmail because the gmail entry in apps.json has counter enabled.

Any other ideas? Comments? Questions? Different directions?
@ronjouch @TheCleric

@mattruzzi awesome! Working on it now.

@mattruzzi @TheCleric here are my thoughts. Before you read, I want to stress out that the size of this brain dump shouldn't make it overwhelming/intimidating. I just happen to have seen this idea bounced a lot, so natually I have a few ideas. I have zero expectations about this, and me suggesting somthing doesn't mean I'd expect it to be here yesterday on a first release. Alright, here it goes:
Client UX
Your interactive prompt with a list & autocomplete looks rad, and would help a lot making Nativefier easier to use for people not familiar with the command-line.
I don't think your new interactive UI obviates the need for current Nativefier; some users will want control over specific flags, add custom --injects, etc. Use cases I can picture:

Users could want to extend a catalog config: Will build Google Docs. Are there extra flags you would like to add?
On users launching a regular Nativefier build with their flags, we could auto-detect that the domain matches a catalog domain, and offer to use its config: You requested to build an app on mail.google.com. Nativefier-catalog has a config for this site, named "Gmail", with an icon, options [counter, bounce], and injectable CSS+JS. Do you want to use it? . This "do you want to use it" could then be selective: a user could opt in to just the icon, or icon + one JS but no custom CSS, etc.

This I think lets us answer the question "Where should this client live? Should it keep being a wrapper (like your current project is, @mattruzzi), or should the client & logic live inside Nativefier proper?". My thinking about this is twofold:

Super extra happy to see you experiment in your wrapper for now.
But later on, given the use cases listed above (extending, noticing a manual build that could be helped by a config) and the need for extra interactivity in core Nativefier that your project surfaces, I think we will want this in Nativefier proper. A wrapper won't do.

Also, if/when the interactive wrapper matures, we'll want to publicize it to users. To do that and for it to be the default experience, it should belong to main Nativefier. Leaving the wrapper outside would be like "hey there's this better easier thing over here, go use it", to which a frank new user would reply "if it's better, why isn't it the default?"

Configs repository
I agree that the json file should live in its own repo, to let it be its own thing, with maybe its own maintainers helping digest quickly community contributions if the volume increases, and yes it opens the way to other clients.
CSS & JS injects ("userscripts") must live alongside in the repo, yes. No gist URLs, especially since we'd be giving these userscripts more visibility and accessibility, they should live in the repo, for security.
This repo should absolutely replace the severely-uncared-for nativefier-icons repo and its gitcloud client, yes. These things are two of the weirdest things in Nativefier, leftovers of Jia (Nativefier creator) doing dubious things as novice dev, generating HTML using Jekyll, that is later on parsed in gitcloud (using html-parsing npm dep cheerio) to grab icons, rube-goldberging the act of fetching data, that would be a simple JSON.parse if it were JSON to begin with. It's nonsensical and Jia himself jokes about it 😄. Yes, let's merrily burn this and replace it all with a simple JSON. While doing this, we should merge the mountain of icons accumulating in open PRs and that I'm ignoring 😬.
The new repo of configs could be entirely JS-less / only a JSON (plus CSS+JS injects). Contributors would submit PRs modifying the JSON (it's reasonable, JSON is readable) and potential css/js injects. We'd have GitHub Actions-based CI that does sanity checks, for maximally automatic checks and minimal PR-reviewing overhead:

Validate the JSON schema is respected
Validate the JSON keys are sorted
Validate that css/js injects referenced in the JSON exist in the tree
Require js & css inject to have a MIT license header
Check & lint: js injects (with eslint) and css injects (with stylus)
Crush PNG icons (with pngquant)

Schema
In your RFC, you indexed the JSON by an "app key/token". My initial thinking was to index by domain, but I may be wrong. Domain/subdomain stuff might be tricky, change with time, so maybe it's not a good idea and you're right with an arbitrary key. Maybe this key should be the app name shown to the user, though? It's totally fine to have any character and spaces in a JSON key.
Then, I was imagining something like what you're suggesting:

url
options (with nested options like the counter and bounce you already mention, but any other options should be usable here)

Options should probably not be raw key: value couples, we'll definitely want an object per option, letting us add extra metadata about each option. I imagine we could want each option to have:

A required default / recommended flag, conditioning whether this is used by default. This would let users contribute flags/injects that we know are not desirable for everybody, but are still nice to offer, ready for cherry-picking in interactive mode.
An optional description field, helping explain to users what some options do (e.g. if WhatsApp has 6 JS injects offered, we want to let users know what each of them does)

Misc
As experimentation happens in your interactive client, please keep an eye on npm dependencies, and align them with Nativefier's. If picking a dep, review its activity, popularity, and sub-dependencies weight; I want to keep Nativefier dependencies weight minimal. I'm not putting a hard no on extra deps, and for example the interactive prompt deserves a dep and enquirer looks like a sane choice, but for example I'd ask you replace meow with commander we already use.

Todo List:

[ ] Move apps.json to separate repo
[ ] Implement new schema
[ ] Migrate icons from nativefier-icons
...

@ronjouch feel free to edit.

For adding some new features (eg. auto detecting and using settings based on a url or customizing the settings used) I'm not quite sure where to start. I could add the ability to customize args or pass a url to this wrapper but if it eventually gets integrated into nativefier it would be awkward to have a wrapper for nativefier as a part of nativefier. I could try to add code to auto detect urls and use settings from the catalog into my fork the main nativefier repo (I think I would put the code into https://github.com/nativefier/nativefier/blob/master/src/cli.ts) but I'm not sure exactly what to do.

I'm working on a adding a way of specifying css/js injects in nactivefier-catalog but I am having problems related to #458 when passing multiple files to inject where it seems to overwrite it each time I pass it a --inject so only the last one is actually included.

Follow-up of https://github.com/nativefier/nativefier/issues/1147 : udemy.com is a case where the catalog should suggest using our flag --widevine

@ronjouch did you mean to close this?

@ronjouch did you mean to close this?

Whoopsie 😄. No! Just meant to add the link to #1147. Thanks.

I was thinking for every site that is in nativefier-catalog we also could put it in a markdown file with explanations for all of the flags used so that the json schema for nativefier-catalog can remain clean and not have the explanations for the flags in it.
See https://github.com/nativefier/nativefier/issues/1166#issuecomment-830527310

Eventually when this gets combined with the main nativefier codebase maybe we could make it so that if you ran nativefier without passing a site or any flags it could enter an interactive mode asking you to choose from an existing site or build an app for your own with an interactive wizard asking you questions eg.
$ nativefier
(we could have some way of asking if you want to create your own custom config for the app or if you want to use a premade one here, or we could default to the custom app maker if the user types an app that is not in the catalog})
(custom app maker:)
? What is the url to that website you want to nativefy … https://mail.google.com/
? What name should the app be saved as … Gmail
? Do you want a counter (insert explanation of counter?) (y/n) … y
? Do you want to use disk cache (insert other explanation) (y/n) … n

I was thinking for every site that is in nativefier-catalog we also could put it in a markdown file with explanations for all of the flags used so that the json schema for nativefier-catalog can remain clean and not have the explanations for the flags in it.

@mattruzzi I disagree: I don't see anything dirty about having a comment key for each catalog entry. I'd prefer it to an explosion of small markdown files. Everything remains closely at the same place, obvious to understand & maintain without having to go fish for a separate file.
To which someone may argue "But in a md file we'll have space, access to formatting, and multiline". I'm not sure we really need these, and being "cramped" in a JSON file will push us to remain concise, and that's good.
Worse is better.

In the current version the json is directly passed to nativefier. I am working on adding injects to it and have added the following to the schema:
"example": {
    "name": "Example",
    "targetUrl": "https://example.com/",
    "injects": {
      "does something ": {
        "path": "./injects/example/do-thing1.js",
        "desc": "does something "
      },
      "does something else": {
        "path": "./injects/example/do-thing2.js",
        "desc": "does something else"
      },
      "important fix": {
        "path": "./injects/example/fix-thing3.js",
        "desc": "fix something else"
      }
    }
  }

I am using the following code prompt for the userscripts and remove the unused ones in memory at runtime:
if (apps[name].hasOwnProperty("injects")) {
		var injectsList = Object.keys(apps[name].injects);
		console.log(injectsList)
		const prompt = new MultiSelect({
			name: 'injects',
			message: 'Available userscripts/stylesheets',
			hint: '(Use <space> to select, <return> to submit)',
			choices: injectsList
		});

		prompt.run()
			.then(answer => runNativefier( getOptions(name, answer)))
			.catch(console.error);
	}}

function getOptions(name = {}, injects = {}) {
		var toInject = [];
		injects.forEach(element => {
			toInject.push(apps[name].injects[element].path)
		});
		var  options = apps[name];
		delete options.injects;
		options.inject = toInject;
		console.dir(options);
		return options;
	}

function runNativefier(config = {}) {
	nativefier(config, function (error, appPath) {
		if (error) {
			console.error(error);
			return;
		}
		console.log("App has been nativefied to", appPath);
	});
}

The current way I am adding extra features to the schema is by adding code that will translate them to standard nativefier configs at runtime (see above where I translate "injects" (my schema addon for injects) to "inject" (a nativefier flag).
I don't know how scalable this is for adding more options/descriptions for options to the schema. Can you elaborate more on the schema you were proposing in the first reply on this issue or provide an example?

What do you think of the following schema?
"example": {
	"Set App name": {"name": "Example"},
    "Set App Url": {"targetUrl": "https://example.com/"},
	"Fix something with inject": {"inject": "injects/example/fix-thing3.js"},
	"Do something else with inject": {"inject": "injects/example/do-thing2.js"},
	"Use a Widevine-enabled version of Electron to allow video playback":{"widevine":true},
	"Explanation": {"flag":"value"}
  }

What do you think of the following schema? Do you have any different ideas?

@mattruzzi I like the new schema you're proposing:

I like that it's "solution-oriented": you offer a "bag of solutions" ("Fix something with inject", "Use Widevine ...", etc), each customizing one or several options (in your examples, all the "solutions" override a single flag, but we could imagine an option/solution setting e.g. inject, and widevine, and userAgent, if they're all needed cohesively to enable something) and cherry-pickable. I think that's great.

To follow-up on our "explaining why" discussion above, each cherry-pickable solution/option could have an optional "comment" or "catalogComment" key, that is used by the interactive UI to explain the solution in detail and e.g. link to discussion in an issue, if users ask for details. This key would be of course stripped when passing flags to Nativefier.

For schema validation, what about nesting the solutions inside a "solutions" (or "options") root key. If we don't do such nesting, we'll have to allow arbitrary keys at the root, and I don't like that as I'd like to be able to validate required keys like "name" and "url" (see next comment).
I find strange to see required "App name" and "App Url" spelled like the rest of the "bag of cherry-pickable options". These two (and maybe more) aren't like the others, in that they're mandatory. Feels like the keys should be standard to "name" and "url", to offer a consistent experience and to do a good validation job. But I like that they use the same "it's just a define/override" approach.
One question: how/where will search work? I had in mind to search in [ catalog keys (`"example"`), `name (`"Example"`), and `url` (`"https://example.com/"`). Makes sense?

We could provide some kind of interactive prompt that would allow users to toggle/edit each option.

Yeah, that fits nicely with the workflow you're suggesting.

"example": {
	"name": "Example",
     "targetUrl": "https://example.com/",
     "Additional Options": {
	     "Allow video playback": {
		 	"flags": {"inject": "injects/example/fix-video.js", "widevine":true},
			"default": true,
			"comment":"Longer explanation"},
			"linktomoreinfo?" : "https://explanation.example/something"
}
  }

Should the link be part of the comment or a different field? Maybe we could standardize it and have a GitHub issue for each one?

@ronjouch I just committed some Proof of concept code for the new schema. I'm starting to see a lot of duplication in it as a lot of apps use the same fix (all Google apps need user agent fix). Should we add a section for common fixes that can be referenced from multiple apps?

I'm starting to see a lot of duplication in it as a lot of apps use the same fix (all Google apps need user agent fix). Should we add a section for common fixes that can be referenced from multiple apps?

@mattruzzi yeah indeed, a section for common fixes (that are reference-able-by-name by apps) seems legit. So:

At the root of the json there could be a "__commonOptions" key containing an object of the common options, named e.g. "useragent-firefox" in your case.
Then, next to additionalOptions we'd have something like importedOptions (or referencedOptions ? or libraryOptions ? , and in your case it would be [{ "libraryOptionName": "useragent-firefox", "default": true }] . Something like this?

@ronjouch @TheCleric
I noticed there is another very similar project to both nativefier-catalog and nativefier-gui.
https://github.com/vinifmor/bauh#native-web-applications
https://github.com/vinifmor/bauh-files/blob/master/web/suggestions.yml
https://github.com/vinifmor/bauh-files/tree/master/web
I don't know if this changes anything.

Is there any reason switch to yaml instead of json?
Could we collaborate with them and share the catalog of apps and fixes?

Could we collaborate with them and share the catalog of apps and fixes?

@mattruzzi two thoughts:

Yeah, seems legit to want to collaborate
At the same time, I see a few objections to trying hard to re-use their stuff.

It's not as if their suggestions.yml had hundreds of apps and important stuff. It's mostly naming/localization and minor things.
They seem to re-implement icon handling logic (for good reason as I/nobody maintains our nativefier-icons repo) and use icons from the web
Many of the default options they set are debatable

Sooo, if I were in your shoes I'd err on the side of doing my thing, keeping in mind the features they support, and then later inviting them to contribute.

Is there any reason switch to yaml instead of json?

I was suggesting json initially for absolute maximal platform (node/js) platform friendliness at the detriment of maintainer friendliness (see HACKING.md / point 4. about deps). That being said, for such a database that is bound to be frequently modified by humans, it's probably reasonable to use YAML, as it's much more readable. Also, I see that yaml and js-yaml both seem well-maintained and are very light on dependencies (0 for one, 1 for the other). So, no objection to YAML.

If we use yaml, should we require keys with spaces to be quoted? It isn't required by yaml, but I don't know if there is any reason we would want to do it. For things like Change user agent to prevent Google from blocking access, instead of having them in this file as keys, we might want to move them to a different file and give them simpler names to reference by like google-ua-fix or something, and then add a key like referencedOptions: [ google-ua-fix ]
I used an online json to yaml converter and got this:

gmail:
  name: Gmail
  targetUrl: https://mail.google.com/
  additionalOptions:
    Show counter on app icon in dock:
      flags:
        counter: true
        bounce: true
      default: true
      comment: (macOS only) Use a counter that persists even with window focus for
        the application badge for sites that use an "(X)" format counter in the page
        title (i.e. Gmail).
    Change user agent to prevent Google from blocking access:
      flags:
        user-agent: firefox
      default: true
      comment: 'Note: lying about the User Agent is required, else Google will notice
        your "Chrome" isn''t a real Chrome, and will refuse access.'
google-meet:
  name: Google Meet
  targetUrl: https://meet.google.com/
  additionalOptions:
    Change user agent to prevent Google from blocking access:
      flags:
        user-agent: firefox
      default: true
      comment: 'Note: lying about the User Agent is required, else Google will notice
        your "Chrome" isn''t a real Chrome, and will refuse access.'
google-docs:
  name: Google Docs
  targetUrl: https://docs.google.com/
  additionalOptions:
    Change user agent to prevent Google from blocking access:
      flags:
        user-agent: firefox
      default: true
      comment: 'Note: lying about the User Agent is required, else Google will notice
        your "Chrome" isn''t a real Chrome, and will refuse access.'
google-sheets:
  name: Google Sheets
  targetUrl: https://sheets.google.com/
  additionalOptions:
    Change user agent to prevent Google from blocking access:
      flags:
        user-agent: firefox
      default: true
      comment: 'Note: lying about the User Agent is required, else Google will notice
        your "Chrome" isn''t a real Chrome, and will refuse access.'
google-slides:
  name: Google Slides
  targetUrl: https://slides.google.com/
  additionalOptions:
    Change user agent to prevent Google from blocking access:
      flags:
        user-agent: firefox
      default: true
      comment: 'Note: lying about the User Agent is required, else Google will notice
        your "Chrome" isn''t a real Chrome, and will refuse access.'
  example2:
    name: Example2
    targetUrl: https://example.net/
    additionalOptions:
      Allow video playback:
        flags:
          inject: injects/example/fix-video.js
          widevine: true
        default: true
        comment: Longer explanation

@mattruzzi

If we use yaml, should we require keys with spaces to be quoted? It isn't required by yaml, but I don't know if there is any reason we would want to do it.

Quoting not necessary indeed, and given the main goal of YAML is to be human-friendly, I'd err on the side of omitting them.

For things like Change user agent to prevent Google from blocking access, instead of having them in this file as keys, we might want to move them to a different file and give them simpler names to reference by like google-ua-fix or something, and then add a key like referencedOptions: [ google-ua-fix ]

👍, I like the idea of a "named options referrable to" section, containing common fixes that we can refer to by name.

Should I use the .yml or .yaml extension?

Dunno; I don't see any objection to one or the other.

@ronjouch I noticed another similar project made by @Nickersoft. I don't know if that changes anything about our plans.
https://github.com/Nickersoft/hop

@ronjouch Wow it's been a year!
I just pushed code that switches the apps file from json to yaml.
We could use yaml anchors to fix the duplication. I switched the Google user agent fix to use them.
https://www.educative.io/blog/advanced-yaml-syntax-cheatsheet#anchors


facebook connect popup doesn't close

I'm not familiar with facebook connect and I may be missing something. But I'm in a hurry on this project and don't have time to retrace the whole facebook connect process (which I'd need to learn first.) I'd be grateful for assistance and a quick resolution would definitely be beer money donation worthy.
Anyways, here's the symptoms

facebook connect window looks correct when the application has no permissions for a user on facebook
facebook connect window doesnt close automatically when clicking "allow", instead it goes blank with the title XD Proxy
after manually closing facebook connect window the first time it redirects to /social/facebook/login and says there was an error (not specific about what kind of error)
trying the whole thing again behaves the same until you close the FB window manually, then it goes to social/login and asks for username/email
after submitting then im redirected to my LOGIN_REDIRECT_URL correctly
after logging back out then clicking facebook button the blank XD Proxy window appears again
when you close it then it redirects properly

I'm gonna jump in and start looking but I'd like to get past this quickly since I'm trying to use it on a quick project i need ready by monday so I'd love to know what I did wrong/didn't do. I've gone over the README instructions and triple checked my settings. I was really hoping for a drop in solution that wouldn't require me to learn how to deal with facebook just yet.
I guess this issue would be with the documentation then, since I ran into this issue someone else probably has too and it's probably because of a setup procedure that was assumed in the documentation.

There is no more popup since 0.5


Include .editorconfig support in Code by default

I want to pick up the discussion from https://github.com/microsoft/vscode/issues/56742 again. I think .editorconfig support should be included by default, not requiring an extension. VS2019 did that and it was a huge productivity boost. I know installing VS extensions is more difficult than VSCode extensions but it would still help a lot of people, knowing that support is included by default and being able to rely on it.

Default support for .editorconfig files and applying it's rules in VS Code.

:-1: it's already a recommended extension which shows a notification when you have .editorconfig in the folder. Bundling = bloating. Plus settings sync is around the corner so you wouldn't need to worry about it ever again after installing the first time if you sign in.

My personal use case is having to tell teammates that they have to install it instead of it just working out of the box. I agree on bloating concerns but yeah...

I guess they ignore the recommendation 🤷♂

Pretty much yes, I see it as a requirement not a recommendation.

Even without the bloat point, the extension is an effort by the community and I don't think we could include it in the product we ship without taking ownership of it as well. I get your argument and I was on that side as well a while ago, but don't think you'll be able to convince anyone in the team to bundle it 🙂


Windows path resolving issues

Hello there,
I'm currently trying to figure out how to use Fast Data Transfer and I did encounter an error which prevents me from using library. I will provide an exact command I'm executing with no luck.
java -jar fdt-0.26.1-SNAPSHOT.jar -pull -r -c <IP_ADDRESS> -d C:/Users/dev/Desktop/ /Users/danny/NetBeansProjects/fdt/target/fdt.jar — from Mac to Windows and its working perfectly fine. File is being transferred.
But in reverse order,
java -jar fdt-0.26.1-SNAPSHOT.jar -pull -r -c 192.168.105 -d /Users/danny/desktop/ C:/Users/dev/Desktop/fdt.jar  — from Windows to Mac FDT is always prompting an error.
Caused by: java.lang.IllegalArgumentException: Illegal syntax! You can use either Client/Server (-c/-d) syntax, either SCP syntax.
I'm doing something wrong? Is there syntax to format windows path to be accepted as last parameter?
I will appreciate any informations regarding my issue.

Okey I made quick dive into code and spotted an issue. Its line 486 in Utils.java. There is check if path contains ':' and then if (File.separatorChar == '\') {// "Windowns" baby!. I think checking File.separatorChar is not necessary there since its preventing from parsing windows formatted path in unix like system.

Hey, how did you solve the problem eventually? I have the same problem....

Hey, how did you solve the problem eventually? I have the same problem....

If I remeber correctly I just removed mentioned if statment and builded project myself.

So you mean you edited the fdt directly?