Framerate < 1 fps, display of out of date frames, in example code "Using NetGear_Async with Variable Parameters"

Description
The client for the example Using NetGear_Async with Variable Parameters does:

display the first frame from the server on connection rather than the most recent
display frames updating at a frame rate < 1 per second

Acknowledgment

[x] I have searched the issues for my issue and found nothing related or helpful.
[x] I have read the Documentation.
[x] I have read the Issue Guidelines.

Environment
Server:
Python 3.8
Raspberry Pi 4 4Gb
Current Raspbian
Current test branch of vidgear installed today (however issue occurred with regular pip3 branch too)
Running on a local network, (other video-streaming approaches do work at expected frame rates, seemingly implying it's not the network)
Client:
Mac OS Catalina
Python 3.8
Current pip3 install vidgear[asyncio]
Expected Behavior
Whether the server or the client starts first, the client will display the latest frame from the server with reasonable frame rate (eg, > 5 frames per second).
Actual Behavior
When the server starts first, then when the client connects, it starts showing frames one by one from when the server started at an extremely low rate (frame rate < 1 per second).
When the client starts first, and then the server starts, the client does start with the servers first (current) frame, but the frame rate on the client is so slow as to quickly get behind (< 1 frame per second)
Possible Fix
Maybe there is config required to make the client/server ignore past frames, and start from the most recent frame when they connect?
Also, this warning prints out when the server starts:
CamGear :: WARNING :: Threaded Queue Mode is disabled for the current video source!
If I wave my arms like a madman while starting the server, and then start the client, I can see my arms waving in super super slow motion. That indicates to me that the server is in fact taking frames at the right frame rate (or else the arm wave would be jerkier), and the issue may be more with the client side, or in the transport of frames.
Steps to reproduce

On the server, make sure to get the test branch of vidgear[asyncio] as described here: https://abhitronix.github.io/vidgear/installation/source_install/#installation

On the server, setup the example code "Using NetGear_Async with Variable Parameters" and run it:

# import libraries
from vidgear.gears.asyncio import NetGear_Async
import asyncio

#initialize Server with suitable source
server=NetGear_Async(source=0, address='<IP_ADDRESS>', port='5454', protocol='tcp',  pattern=3, logging=True).launch()

if __name__ == '__main__':
    #set event loop
    asyncio.set_event_loop(server.loop)
    try:
        #run your main function task until it is complete
        server.loop.run_until_complete(server.task)
    except KeyboardInterrupt:
        #wait for keyboard interrupt
        pass
    finally:
        # finally close the server
        server.close()

On the client, setup the code from the same example and run it:

# import libraries
from vidgear.gears.asyncio import NetGear_Async
import cv2, asyncio

#define and launch Client with `receive_mode=True`. #change following IP address '192.168.x.xxx' with yours
client=NetGear_Async(address='<IP_ADDRESS>', port='5454', protocol='tcp',  pattern=3, receive_mode=True, logging=True).launch()

#Create a async function where you want to show/manipulate your received frames
async def main():
    # loop over Client's Asynchronous Frame Generator
    async for frame in client.recv_generator():

        # do something with received frames here

        # Show output window
        cv2.imshow("Output Frame", frame)
        key=cv2.waitKey(1) & 0xFF

        #await before continuing
        await asyncio.sleep(0.00001)

if __name__ == '__main__':
    #Set event loop to client's
    asyncio.set_event_loop(client.loop)
    try:
        #run your main function task until it is complete
        client.loop.run_until_complete(main())
    except KeyboardInterrupt:
        #wait for keyboard interrupt
        pass

    # close all output window
    cv2.destroyAllWindows()
    # safely close client
    client.close()

Attaching a video showing what I mean:
trimmed.mov.zip
I started the server, and a few seconds after it ran, brought my arms up and wiggled them.
When I then started the client, I saw only myself sitting (no arms up). I closed the client, re-opened it about 10 minutes later, and at that point the client showed my arms up, and you can see the relative speed issue from the wiggling of the arms, and someone else's hand coming into frame at the end.
Based on this, I am pretty confident all the frames are taken by the server, and then getting cached somewhere, and read out very slowly from the client.
The behavior I'm trying to achieve is: when the client connects, all frames that are older than the connection would be ignored, and it would start on the most current frame. Further, it would keep displaying frames at the rate they're being sent. I can solve the out of date starting frames by starting the server only after the client is started, but the low frame rate still results in it immediately going out of date.

extremely low rate (frame rate < 1 per second).

@whogben Thanks for this detailed info. I have few questions like, Is something is wrong with your Network/router/wifi setting, ans are you using any proxy settings for your network? Also, are you using connected camera for any other purpose/task while running these examples?

**Maybe there is config required to make the client/server ignore past frames, and start from the most recent frame when they connect?
Also, this warning prints out when the server starts:
CamGear :: WARNING :: Threaded Queue Mode is disabled for the current video source!**

No, this a normal warning in Threaded Queue Mode which is disabled for live source, and has nothing to do with NetGear_Async API.
Possible Solutions
Please follow following steps:
Step 1: Update latest ZMQ manually on both machines:

On macOS machine, do this:

# uninstall zmq
pip3 uninstall pyzmq -y

# brew install zmq
brew install zmq

# reinstall pyzmq
pip3 install -U pyzmq

Then, On Raspberry Pi

# uninstall zmq
pip3 uninstall pyzmq -y

# apt install zmq
sudo apt update && sudo apt install libzmq3-dev

# reinstall pyzmq
pip3 install -U pyzmq

Step-2: On Raspberry Pi, Run following command
iw wlan0 set power_save off   # disables powersaver setting for wifi

Step-3 Reinstall VidGear's testing branch:
Just now, I've pushed some commits to testing branch for increasing performance, so you can re-install it again by following these instructions.
Step-4 Run following Modified example:
Now run following modified example, instead of previous one:
Client's End
Open a terminal on Client System (where you want to display the input frames received from the Server) and execute the following python code:

"Note down the IP-address of this system(required at Server's end) by executing the command: hostname -I and also replace it in the following code."

"Client will throw TimeoutError if it fails to connect to the Server in given timeout value!"

"You can terminate client anytime by pressing [Ctrl+C]/[⌘+C] on your keyboard!"

# import libraries
from vidgear.gears.asyncio import NetGear_Async
import cv2, asyncio

#define and launch Client with `receive_mode=True`. #change following IP address '192.168.x.xxx' with yours
client=NetGear_Async(address='192.168.x.xxx', port='5454', protocol='tcp',  pattern=2, receive_mode=True, logging=True).launch()

#Create a async function where you want to show/manipulate your received frames
async def main():
    # loop over Client's Asynchronous Frame Generator
    async for frame in client.recv_generator():

        # do something with received frames here

        # Show output window
        cv2.imshow("Output Frame", frame)
        key=cv2.waitKey(1) & 0xFF

        #await before continuing
        await asyncio.sleep(0.00001)

if __name__ == '__main__':
    #Set event loop to client's
    asyncio.set_event_loop(client.loop)
    try:
        #run your main function task until it is complete
        client.loop.run_until_complete(main())
    except KeyboardInterrupt:
        #wait for keyboard interrupt
        pass

    # close all output window
    cv2.destroyAllWindows()
    # safely close client
    client.close()

Server's End
Now, Open the terminal on another Server System (with a webcam connected to it at index 0), and execute the following python code:

"Replace the IP address in the following code with Client's IP address you noted earlier."

"You can terminate stream on both side anytime by pressing [Ctrl+C]/[⌘+C] on your keyboard!"

# import library
from vidgear.gears.asyncio import NetGear_Async
from vidgear.gears.asyncio.helper import reducer
import cv2, asyncio

#initialize Server
server=NetGear_Async(source=None, address='192.168.x.xxx', port='5454', protocol='tcp',  pattern=2, logging=True)

#Create a async frame generator as custom source
async def my_frame_generator():

        #Open any video stream such as live webcam video stream on first index(i.e. 0) device
        stream=cv2.VideoCapture(0)

        # loop over stream until its terminated
        while True:

            # read frames
            (grabbed, frame)=stream.read()

            # check if frame empty
            if not grabbed:
                #if True break the infinite loop
                break

            # reducer frames size if you want more performance, otherwise comment this line
            frame = reducer(frame, percentage = 30) #reduce frame by 30%

            # do something with the frame to be sent here

            # yield frame
            yield frame
            # sleep for sometime
            await asyncio.sleep(0.00001)

if __name__ == '__main__':
    #set event loop
    asyncio.set_event_loop(server.loop)
    #Add your custom source generator to Server configuration
    server.config["generator"]=my_frame_generator()
    #Launch the Server
    server.launch()
    try:
        #run your main function task until it is complete
        server.loop.run_until_complete(server.task)
    except KeyboardInterrupt:
        #wait for keyboard interrupt
        pass
    finally:
        # finally close the server
        server.close()

and see if this boost performance and then report your results here. Goodluck!

Wow thanks for all that!
Question Responses

I don't control the network, but I don't think it's running any proxies. It is a verizon fios modem assigning IPs, connected via ethernet to a netgear router that the machines are actually connected to. The RPi is connected to the 2.4ghz version while the Mac is on the 5ghz version. It's possible traffic is routed out to the Verizon modem and then back into the netgear before reaching me - but other video solutions I've been testing have not experienced as much trouble, and VNC streams the screen well.
I don't believe any other process is using the camera.

Suggested Tests Results:

Rerunning original setup:

when server starts first, same behavior as original issue
when client starts first:, the client gives the following output:

NetGear_Async :: DEBUG :: Launching NetGear asynchronous generator!
NetGear_Async :: DEBUG :: Successfully Binded to address: tcp://<IP_ADDRESS>:5454 with pattern: 3.
NetGear_Async :: DEBUG :: Receive Mode is activated successfully!
Traceback (most recent call last):
  File "vidgeartest.py", line 31, in <module>
    client.loop.run_until_complete(main())
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "vidgeartest.py", line 12, in main
    async for frame in client.recv_generator():
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/vidgear/gears/asyncio/netgear_async.py", line 363, in recv_generator
    recvd_msg = await asyncio.wait_for(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/tasks.py", line 490, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

After reinstalling ZMQ on Mac and Pi (old version on both was 19.0.1,  new version installed was also 19.01)

when server starts first:  same behavior as original issue
when client starts first:  same error as test 1

unexpected behavior:  I stopped the server on the raspberry pi, and ran the client on the mac again (in preparation to start the server again), even though the server was closed, out of date frames were displayed.  I then started the server again, and after a short time, a new up to date frame was displayed - though the speed stayed <1fps and got out of date again.  It's as though the frames were buffered somewhere, but restarting the server cut through that and sent a new one over the top.

reinstalling vidgear testing branch (on Mac and RPi):

running server first: same original delay behavior
running client first: same error as test 1 & 2

Running modified example:

Running the server code gives this output:

NetGear_Async :: WARNING :: Given source is of NoneType!
NetGear_Async :: DEBUG :: Creating NetGear asynchronous server handler!
NetGear_Async :: DEBUG :: Successfully connected to address: tcp://<IP_ADDRESS>:5454 with pattern: 2.
NetGear_Async :: DEBUG :: Send Mode is successfully activated and ready to send data!
NetGear_Async :: DEBUG :: Terminating various Send Mode Processes.
Traceback (most recent call last):
  File "/home/pi/Desktop/netgear_example.py", line 46, in <module>
    server.loop.run_until_complete(server.task)
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/home/pi/.local/lib/python3.7/site-packages/vidgear/gears/asyncio/netgear_async.py", line 299, in __server_handler
    if not (frame.flags["C_CONTIGUOUS"]):
AttributeError: 'coroutine' object has no attribute 'flags'

Additional Tests:

Try running both client and server on the pi together, using original example code, hopefully to rule out the network / router setup as an issue:

when client starts first:  same timeout error as previous tests
when server starts first: the behavior is slightly less slow, but it still took 18 seconds to see my hand appear after I moved it in front of the camera

Try running both modified examples on the pi together:

client first, timeout error
server first, same 'coroutine' object has no attribute 'flags' error

Try running both client and server on the mac together, using original example code:

client first, timeout error
server first, took 24 seconds to see my hand in the video after I put it in front of the camera

TLDR:

probably ruled out the network being the issue: 18 and 24 second delays when running locally to <IP_ADDRESS>,
discovered I'm using python 3.7 on the pi not 3.8 as I thought, not sure if that's considered to be a possible problem.
General question: Is it working / streaming smoothly for you when you run the examples? If so, what computer/config?

Anyway, thank you so much for all the great help with this - I am really really excited to use vidgear to get low latency streaming from the pi - this is an amazing framework and your response has been amazing too!

Rerunning original setup:
when server starts first, same behavior as original issue
when client starts first:, the client gives the following output:
NetGear_Async :: DEBUG :: Launching NetGear asynchronous generator!
NetGear_Async :: DEBUG :: Successfully Binded to address: tcp://<IP_ADDRESS>:5454 with pattern: 3.
NetGear_Async :: DEBUG :: Receive Mode is activated successfully!
Traceback (most recent call last):
File "vidgeartest.py", line 31, in
client.loop.run_until_complete(main())
File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
File "vidgeartest.py", line 12, in main
async for frame in client.recv_generator():
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/vidgear/gears/asyncio/netgear_async.py", line 363, in recv_generator
recvd_msg = await asyncio.wait_for(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/tasks.py", line 490, in wait_for
raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

@whogben this TimeoutError can be avoided  by setting a high value  on timeout parameter at receiver's end. But problem is somewhere else, maybe in your network, since I'm not able to reproduce this much latency <1fps on my Raspberry pi 3B and Windows computer connected over 2.4ghz Wi-fi. Are you using USB Camera instead of Pi Camera Module?

After reinstalling ZMQ on Mac and Pi (old version on both was 19.0.1, new version installed was also 19.01)
when server starts first: same behavior as original issue
when client starts first: same error as test 1

Re-installing ZMQ binaries from apt or brew does helps, according my tests. But again it's something else in your case that is throttling the performance.

unexpected behavior: I stopped the server on the raspberry pi, and ran the client on the mac again (in preparation to start the server again), even though the server was closed, out of date frames were displayed. I then started the server again, and after a short time, a new up to date frame was displayed - though the speed stayed <1fps and got out of date again. It's as though the frames were buffered somewhere, but restarting the server cut through that and sent a new one over the top.

Reason: You can see if it will block or drop by looking through the documentation for the socket type for ZMQ::HWM option action which will either be Block or Drop. The default ZMQ::HWM value of zero means “no limit”, meaning that it will not enter an exceptional state until the system runs out of memory, which is happening your case.
Solution: I've pushed the update to set ZMQ::HWM value automatically in testing branch, that;s why I asked you to reinstall it from scratch.

Running modified example:
Running the server code gives this output:
NetGear_Async :: WARNING :: Given source is of NoneType!
NetGear_Async :: DEBUG :: Creating NetGear asynchronous server handler!
NetGear_Async :: DEBUG :: Successfully connected to address: tcp://<IP_ADDRESS>:5454 with pattern: 2.
NetGear_Async :: DEBUG :: Send Mode is successfully activated and ready to send data!
NetGear_Async :: DEBUG :: Terminating various Send Mode Processes.
Traceback (most recent call last):
File "/home/pi/Desktop/netgear_example.py", line 46, in
server.loop.run_until_complete(server.task)
File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
File "/home/pi/.local/lib/python3.7/site-packages/vidgear/gears/asyncio/netgear_async.py", line 299, in __server_handler
if not (frame.flags["C_CONTIGUOUS"]):
AttributeError: 'coroutine' object has no attribute 'flags'

Apologies, I've missed a await in above example. Anyways, download following file:
netgear_async test.zip
It contains proper working code for both Server and Client end. And, You need to do following task with the files contained in it:

⚠ Important : Install testing branch again.
The files in local folder must be run in different terminals on same machine (i.e server on terminal-1 and client on terminal-2), make sure you have your camera connected to your this machine (at index 0).
The files in multiple folder must be run on different machine(Pi and MacOS), like above. Please make sure you change the address parameter value to yours.
Post result here.

Try running both client and server on the pi together, using original example code, hopefully to rule out the network / router setup as an issue:
when client starts first: same timeout error as previous tests

Meaning, your are letting your Client alone for long time without Server, thus, triggering this Timeout error.

when server starts first: the behavior is slightly less slow, but it still took 18 seconds to see my hand appear after I moved it in front of the camera

This is unbelievable, there is something wrong with your machine. Are you using this bare-minimum code to test the performance on same machine?
See my results on much less powerful Raspberry Pi 3B than your Raspberry Pi 4:
2020-06-15-12-48-03.zip
and I'm recording VNC output which is usually slower than in real.

Try running both client and server on the mac together, using original example code:
client first, timeout error
server first, took 24 seconds to see my hand in the video after I put it in front of the camera

That's weird because You can see exact real-time output on my Windows 10 Machine:
2020-06-15-12-34-05.zip

TLDR:
probably ruled out the network being the issue: 18 and 24 second delays when running locally to <IP_ADDRESS>,
discovered I'm using python 3.7 on the pi not 3.8 as I thought, not sure if that's considered to be a possible problem.

Nope. It maybe these problems. Can you test NetGear API too, to cross-check everything?

General question: Is it working / streaming smoothly for you when you run the examples? If so, what computer/config?

I'm working with AMD laptop and Raspberry Pi 3B with 2.4ghz standard Wi-Fi as local network.

Closed and marked invalid for now, as it can't be reproduced. Kindly share your results when you're ready. Good luck!

Under testing branch, using the provided files (thank you!):

local test works great on mac and raspberry pi
multiple test has the same issue

I will be by a different wifi network and able to try something new on Monday, will report back when I know more.

Under testing branch, using the provided files (thank you!):
local test works great on mac and raspberry pi

@whogben Thanks for confirming.

multiple test has the same issue
I will be by a different wifi network and able to try something new on Monday, will report back when I know more.

Good approach, please do notify the results here, as It will be helpful for the community too.
@whogben Also, try NetGear API examples too and see if it is any faster on your machine than NetGear_Async API.

Testing non async approach seems to be much much faster, 1.5 - 2 seconds latency, 2-3 frames per second, and it doesn't get further behind over time - the latency stays about the same from connecting to several minutes later. Not sure if it's relevant, but the server constantly prints out:
NetGear :: DEBUG :: b'Data received on device: 24404 !'
When I added a printout on the server side to the loop, showing when it captures a frame, the frame rate it's sending is about 1FPS:
# import required libraries
from vidgear.gears import VideoGear
from vidgear.gears import NetGear

# define various tweak flags
options = {'flag' : 0, 'copy' : False, 'track' : False}

# Open live video stream on webcam at first index(i.e. 0) device
stream = VideoGear(source=0).start()

# Define Netgear server at given IP address and define parameters (!!! change following IP address '192.168.x.xxx' with client's IP address !!!)
server = NetGear(address = '<IP_ADDRESS>', port = '5454', protocol = 'tcp',  pattern = 1, logging = True, **options)

# loop over until KeyBoard Interrupted
while True:

  try:

    # read frames from stream
    frame = stream.read()

    # check for frame if Nonetype
    if frame is None:
        break

    # {do something with the frame here}
    print('frame')

    # send frame to server
    server.send(frame)

  except KeyboardInterrupt:
    break

# safely close video stream
stream.stop()

# safely close server
server.close()

My theory of the issue is now that it's the network. The server should be able to take frames much faster than 1 fps - it must be getting stuck waiting during the send() for acknowledgement from the client. As compared to asyncio version, where it likely queued up loads of frames (server showed sending rapidly) and then the network somehow buffered / delayed them down to the low rate observed. I will post again when I have the opportunity to test on new network hardware.

Not sure if it's relevant, but the server constantly prints out:
NetGear :: DEBUG :: b'Data received on device: 24404 !'

@whogben just turn the logging off (logging = False).

My theory of the issue is now that it's the network. The server should be able to take frames much faster than 1 fps - it must be getting stuck waiting during the send() for acknowledgement from the client. As compared to asyncio version, where it likely queued up loads of frames (server showed sending rapidly) and then the network somehow buffered / delayed them down to the low rate observed. I will post again when I have the opportunity to test on new network hardware.

Yes, indeed its network, otherwise  others would have reported this behavior too. According to my tests, NetGear_Async is sometimes faster, and other times its slower than NetGear. I don't actually know what causes this but its certain that NetGear_Async is far memory efficient than  NetGear due to its asyncio structure. I'll advise you to stick with NetGear as it worked much better for you, but do try different pattern values, as each of them suits different settings.
On top of this see FAQ question.

I can confirm that changing to a new router resolved the issue, thank you so much for the help in troubleshooting this and I'm sorry it turned out to be something so simple!
For others who might be having a similar problem in the future and reading this issue - previously, the WiFi was acting in bridge mode to a second router was doing the DHCP work. My guess is something about this network topology or the second router (a verizon modem) was causing variable latency spikes, and would get backed up causing the delays described. Moving to another router setup, where a different router both provides DHCP and the WiFi network resolved the issue for me. Thanks again abhiTronix!

@whogben Kudos 🍻


A few language and content improvements in main.go

Signed-off-by: Julius Volz<EMAIL_ADDRESS>

Codecov Report

Merging #60 (6671989) into master (b43c75a) will not change coverage.
The diff coverage is 0.00%.

@@           Coverage Diff           @@
##           master      #60   +/-   ##
=======================================
  Coverage   48.24%   48.24%
=======================================
  Files          54       54
  Lines        2390     2390
=======================================
  Hits         1153     1153
  Misses       1112     1112
  Partials      125      125

Impacted Files
Coverage Δ

cmd/perses/main.go
0.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b43c75a...6671989. Read the comment docs.


Integrate WDS CheckboxGroup component with appsmith

CheckboxGroup in Storybrook
CheckboxGroup code

In order to get acquainted with what properties should be in property pane, please read this doc.

Hey team! Please add your planning poker estimate with Zenhub @dhruvikn @jsartisan @KelvinOm


Implement console log parser

This PR contains implementation for console log parser.

Codecov Report

Merging #31 (54a47c7) into main (6556131) will increase coverage by 1.19%.
The diff coverage is 91.02%.

@@            Coverage Diff             @@
##             main      #31      +/-   ##
==========================================
+ Coverage   69.92%   71.12%   +1.19%
==========================================
  Files          37       37
  Lines        1320     1392      +72
==========================================
+ Hits          923      990      +67
- Misses        397      402       +5

Impacted Files
Coverage Δ

src/twister2/log_parser/factory.py
73.91% <50.00%> (-3.36%)
:arrow_down:

src/twister2/log_parser/console_log_parser.py
92.59% <92.00%> (+10.77%)
:arrow_up:

src/twister2/exceptions.py
100.00% <100.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Drop the automation for #118661

Describe the bug
GitHub action for monitoring direct pushes to master generates random false positives with no true positives in recent memory
Steps To Reproduce
Steps to reproduce the behavior:

Merge a PR using the GitHub button
Get slightly unlucky.
Get noise notification from the action.

Expected behavior
Nothing after the merge.
Screenshots
#118661 is full of references to the noise notifications.
Notify maintainers
@domenkozar I guess?

I just opened an RFC to disallow direct pushes, which would resolve this: https://github.com/NixOS/rfcs/pull/156

The bot is now removed, see https://github.com/NixOS/nixpkgs/issues/249117 and https://github.com/NixOS/nixpkgs/pull/249151 :smile:


Revision of material - team Grasshoppers

Dear @open-sci/grasshoppers,
Please find here attached my comments of all your material. You have to address all of them and, once finalised, to close this issue with a comment containing your reply to each of the points I have highlighted. There is no specific deadline to complete this task, thus please take your time.
Please, be aware that some modifications in some document may affect also modifications in other documents. As a final note, please remember to keep your notebooks up-to-date.
After closing this issue, please remember to update your material.md file by specifying the references to the new version of all your documents.
As usual, for further doubts, do not hesitate to contact me in the Signal group or just comment this issue here.

General comments from the presentation

You showed the result of your research by using a subset of the input citations. Would it be possible to have a complete coverage of the results by considering all the citations, instead of just a part of it?

It would be great to compare the results you obtained if you use only the procedure (i.e. regular expressions) described in Xu et al. vs all those that you obtained by using your updated methodology, i.e. that one which includes more regex as you have presented.

You said that it is pretty impossible that the process stop to compute the various values.  Do you have a way to restart it from the last data produced, avoiding to start it again from the beginning?

An invited expert identified a bug in the code for cleaning the DOIs. Is it possible that this bug affected the results presented? If so, it would be necessary to recompute the whole data, of course.

Did you check, in some way, if the cleaning of a DOI produced a valid DOI which is indeed the article that is indeed cited by the citing entity? In other words, is it possible that your cleaning process produce a citation between two entities which is not the one that was originally defined in the citing paper?

DMP
The tense used in the DMP is the future, since it is developed before any data have been produced. Thus, the answers of the various points should reflect this.
Title: Classes of errors in DOI names: output dataset
Section 3.1.3: you have to specify the vocabulary used, avoiding the sentence "Couldn't find it? Insert it manually"
Section 3.1.8: since you specified (in section 3.1.7) that you will use naming convention, you have to specify which naming convention is adopted.
Section 3.1.9: here you should explain how to associate version numbers to your dataset, and not which version is currently in use.
Section 4.4: the answer provided is "[Other]" but not additional comment has been added there to clarify what that "[Other]" means.
Classes of errors in DOI names: code
Section 2.1: in this context the term "data" should be interpreted as "software". Thus, the question refers to if you are going to reuse existing software for accomplishing your goal. Under this meaning, please be aware that also the Section 2.2 and Section 2.3 could be populated with some information. Please notice that this interpretation of "data" as "software" may affect also other points of the DMP.
Section 3.1.4: the DataCite Ontology mentioned is not the official DataCite specification, bur rather a community effort to provide an OWL representation of (some portions of) the DataCite schema. Zenodo is not using SPAR DataCite Ontology in any way. Please correct.
Section 3.1.9: here you should explain how to associate version numbers to your dataset, and not which version is currently in use.
Section 3.1.15: I do not believe you use/share Python Compiled Files (.pyc) but rather simple Python files (.py), right?
Section 3.4.5: in case you have developed unit tests to check the correctness of the code, then you have a documented procedure for quality. However, I do not know if you did tests or not.
Protocol
"The data needed for this work is provided by COCI as a CSV": it is not provided by COCI (which is a collection), it is provided by myself.
"The records are provided on public licence by": change in "license in" and put the bibliographic reference in a shape it is clear (e.g. by using italic or other appropriate format).
"40'228": write "40,228" (the comma in English notation is visual separator).
"there are two main classes of errors: author errors, made by authors when creating the list of cited articles for their publication, and database mapping errors, related to a data-entry error": it would be good to acknowledge here the fact that we have, in general, "human errors", i.e. those introduced by authors, editors and publisher representatives when managing the references - as explained during the workshop event.
"we manually isolated from a subset of 100 DOIs recurrent strings": how did you selected such 100 DOIs?
"eventual unwanted characters": remove "eventual"
Software
The README.md in the GitHub repository does not contain an appropriate introduction to the software and, in particular, how to and in which order to call the various scripts to run the process correctly. In addition, you should also specify with which configuration (i.e. computer, processor, RAM, HD, etc.) you have run the scripts to get your final output, since this is crucial to foster reproducibility. Finally, if the software is somehow related with other documents (the protocol, the article, website, etc.), please mention them here in the README.md.
Article
"OpenCitations Index Of Crossref Open DOI-To-DOI References": it should be "OpenCitations Index Of Crossref Open DOI-To-DOI Citations"
"As the data provided by publishers to Crossref is not double- checked, the references integrated in COCI can still contain errors, which would mean a drawback for the goal of open scholarly citations": the references in Crossref may contain errors, but in COCI these are excluded.
"COCI validates the DOIs using the DOI API and provides a CSV list with all the DOI names that prove as invalid": it is not entirely true. COCI does not provide the DOI names that are invalid, simply they are not added in COCI. I (Silvio) provided such names. Then, I agree with you it would be good to share systematically this invalid DOI names within the OpenCitations website, something that can be done in the future.
"Consequently, the taxonomy...": you should mention in which section all the things you are summarising here will be presented. E.g.: "In Section 'Methods and Material' we present ...".
"1.223.296 DOI-to-DOI citations": please use the English notation for numbers, since "." is used to separate the decimal part of a number from the floating part. Thus, that number (and others, if any) should be written as "1,223,296".
"are highlighted in red in the table below" (and similar): when referring to tables, listings, figures, etc., you should avoid below/abobe/etc. but rather explicitly point to the particular table. E.g.: "are highlighted in red in Table 5".
"Table 5 Regular expressions...": explicitly add in the caption the meaning of the two colours adopted.
"As mentioned in the Materials and Methods chapter": it is a section, not a chapter.

General comments from the presentation

You showed the result of your research by using a subset of the input citations. Would it be possible to have a complete coverage of the results by considering all the citations, instead of just a part of it?

Yes, the results have been recalculated considering the entire dataset of input citations you provided. The new version of the output dataset was published with the following DOI: http://doi.org/10.5281/zenodo.4892551.

It would be great to compare the results you obtained if you use only the procedure (i.e. regular expressions) described in Xu et al. vs all those that you obtained by using your updated methodology, i.e. that one which includes more regex as you have presented.

It was interesting indeed. The procedure of Xu et al. was reproduced and their results compared with ours. The data were reported and interpreted in the new version of the article and the visualization has been updated to take into account this comparison. The visualization can be seen at the following address: https://open-sci.github.io/2020-2021-grasshoppers-code/.

The comparison to the Xu et al. reference study showed that the methods used in this research could be employed to clean more DOIs overall. However, which could be seen as surprising, applying the methods from Xu et al. allowed 17,395 more suffix errors to be cleaned. On the other hand, the reference procedure did not clean any prefix errors in the given dataset. This result can be explained by taking a closer look at the regular expressions and the definition of prefix- and suffix-type errors used by Xu et al. In the study presented here, the regular expression for prefix errors was broader, as any additional strings before the DOI were considered prefix errors, even if they were not at the beginning of the string. Meanwhile, the reference procedure considered prefix errors only those at the beginning of the string, hence the absence of matches in the given dataset. The current study, for instance, in the DOI "10.1016/J.JLUMIN.2004.10.018.HTTP://DX.DOI.ORG/10.1016/J.JLUMIN.2004.10.018", considered the presence of the DOI proxy server as a prefix-error, because it appeared before the DOI. The higher number of suffix errors cleaned with the methods by Xu et al. comes as a consequence. Since the present algorithm subsequently checks a DOI name for prefix, suffix and, finally, other-type errors, the errors not caught as prefix errors are cleaned as suffix errors in the next step. Presumably, the mistake related to the presence of the DOI proxy server was fixed as a suffix error in the Xu et al. procedure, which led to a higher number of matches.

You said that it is pretty impossible that the process stop to compute the various values.  Do you have a way to restart it from the last data produced, avoiding to start it again from the beginning?

Indeed, it is true, if the PC were to shut down, the entire work (except for the cache of requests to the DOI Proxy server) would have been lost. To deal with this problem, a new support method was implemented, read_cache. If a cache file was created, this support method reads the data processed up to that moment and restarts the process from the last CSV line read and not from the beginning. This function has been integrated into both check_dois_validity and procedure methods, that create or update a cache file every N DOI. The user can customize the number of DOIs after which the cache updates, as well as the location of the cache file.
The code was inspired by the strategy adopted by the Leftovers 2.0 team to solve the same problem.

An invited expert identified a bug in the code for cleaning the DOIs. Is it possible that this bug affected the results presented? If so, it would be necessary to recompute the whole data, of course.

Yes, the results related to the application of regular expressions had to be recalculated and they were. However, it was not necessary to recalculate the results from the verification of the previous validity of the DOIs and they have been reused to speed up the process.

Did you check, in some way, if the cleaning of a DOI produced a valid DOI which is indeed the article that is indeed cited by the citing entity? In other words, is it possible that your cleaning process produce a citation between two entities which is not the one that was originally defined in the citing paper?

To check if the cleaned DOI names were part of the citing entity references, 100 DOIs were randomly selected from the results and manually checked. Three cases were differentiated:

the reference to the cleaned DOI was verified;
no reference to the cleaned DOI could be found;
the reference list and metadata of the given articles were not accessible, making it impossible to produce a statement about the correctness of the DOI.

98% of the cleaned cited DOIs proved as correct after the manual check. However, a closer investigation of the two DOI names which resulted incorrect showed that the reason is not to be imputed to the algorithm adopted. The DOI "10.1007/978-3-319-90698-0_26" should appear among the references of the article corresponding to the DOI "10.17660/actahortic.2020.1288.20", which unfortunately is behind a paywall and therefore inaccessible. This article was not the only one published on a toll-access journal among the 100 checked DOIs, but the University of Bologna, which the authors of this research are affiliated to, is not subscribed to the magazine in question, namely “ISHS Acta Horticulturae”. On the other hand, the DOI "10.1007/s10479-011-0841-3" should be cited by "10.1101/539833", but, although reported by Crossref, it does not appear in the original article, not only in its correct version but also in the invalid one, that is "10.1007/s10479-011-0841-3." with a point after the DOI. This mismatch can be explained as a processing error made by the platform that published the article, that is bioRxiv, since the same invalid DOI appears 118 times in the dataset analyzed and not even one in the articles that mention it.

Data Management Plan

The tense used in the DMP is the future, since it is developed before any data have been produced. Thus, the answers of the various points should reflect this.

The answers to the various DMP points were converted to the future tense.

Title: Classes of errors in DOI names: output dataset

Section 3.1.3: you have to specify the vocabulary used, avoiding the sentence "Couldn't find it? Insert it manually"

The vocabulary used for metadata has been specified. Since the dataset is published on Zenodo, the metadata about it is provided by Zenodo
itself using the Datacite Metadata Schema (https://schema.datacite.org/).

Section 3.1.8: since you specified (in section 3.1.7) that you will use naming convention, you have to specify which naming convention is adopted.

The naming convention adopted was specified, in this way: "The field titles in the CSV file will be indicated in snake case, with each space replaced by an underscore. The first letter of each field will be capitalized, as well as the acronym "DOI" will always appear in capital letters. In addition, the DOI names reported in the cells of the file will always be in lowercase. Finally, whether or not a value belongs to a class will be indicated with 1 or 0, having the Boolean value of true or false respectively".

Section 3.1.9: here you should explain how to associate version numbers to your dataset, and not which version is currently in use.

This section has also been checked and edited. We used Semantic Versioning 2.0.0 (https://semver.org/) for both datasets.

Section 4.4: the answer provided is "[Other]" but not additional comment has been added there to clarify what that "[Other]" means.

We added a comment to clarify what "Other" means, that is "The data is available on GitHub Repository and Zenodo under an ISC license".

Classes of errors in DOI names: code

Section 2.1: in this context the term "data" should be interpreted as "software". Thus, the question refers to if you are going to reuse existing software for accomplishing your goal. Under this meaning, please be aware that also the Section 2.2 and Section 2.3 could be populated with some information. Please notice that this interpretation of "data" as "software" may affect also other points of the DMP.

All libraries used within our code have been fully referenced, specifying which use has been made of them. References were made to the specific versions used, published on the Software Heritage Archive.

Section 3.1.9: here you should explain how to associate version numbers to your dataset, and not which version is currently in use.

The answer to this question has been modified. Semantic Versioning 2.0.0 (https://semver.org/) was used for both dataset.

Section 3.1.15: I do not believe you use/share Python Compiled Files (.pyc) but rather simple Python files (.py), right?

Right. The data format was corrected from Python Compiled Files (.pyc) to Python files (.py).

Section 3.4.5: in case you have developed unit tests to check the correctness of the code, then you have a documented procedure for quality. However, I do not know if you did tests or not.

The answer is no, the test-driven development procedure has not been adopted. However, we think it would have been better to adopt it. Although TDD requires tests to be developed before code, we believe that it still has value to develop them afterwards, to have a solid foundation in case of extensions or code changes. For this reason, we developed unit tests, using the Python built-in "unittest" framework.

Protocol

  "The data needed for this work is provided by COCI as a CSV": it is not provided by COCI (which is a collection), it is provided by myself.

In our protocol, we removed all the mentions of COCI as a provider of the dataset and we substituted them with the reference to the Zenodo repository.

  "The records are provided on public licence by": change in "license in" and put the bibliographic reference in a shape it is clear (e.g. by using italic or other appropriate format).

The text was corrected as suggested. Any in-text reference now follows the APA citation style, and we added the reference list as the last section of the protocol.

 "40'228": write "40,228" (the comma in English notation is visual separator).

As suggested, we used a comma instead of an apostrophe as visual separator for groups of thousands.

  "there are two main classes of errors: author errors, made by authors when creating the list of cited articles for their publication, and database mapping errors, related to a data-entry error": it would be good to acknowledge here the fact that we have, in general, "human errors", i.e. those introduced by authors, editors and publisher representatives when managing the references - as explained during the workshop event.

In the newer version of the protocol we described the class of human-made errors, in addition to the two author errors and database-mapping errors classes.

  "we manually isolated from a subset of 100 DOIs recurrent strings": how did you selected such 100 DOIs?

Since we didn't store the subset of 100 DOIs from which we extracted recurrent patterns of errors and we were not able to reproduce it, we removed any mention to this subset of DOIs and all we stated was that we did a manual identification of a subset of DOI names having recurrent prefixes and suffixes errors.

  "eventual unwanted characters": remove "eventual"

We removed the occurrence of the word "eventual" in section 3.3 of the newer version of the protocol.

Software

The README.md file was added to the Github repository. In particular, the chapter "Getting started" describes how to reproduce the experiment, which methods to apply and in what order. The chapter "Hardware configuration" reports relevant specifications of the PC on which the research was conducted. The chapter "Results" refers to the website on which the outcomes' visualizations were published. Finally, the chapter "References" refers to the resources mentioned or linked, such as the DMP, the protocol and the article.

Please note: Due to technical problems attributable to Zenodo, it was not possible to publish the latest release of the code and then provide the DOI of the latest version. The technical problem is widespread and documented in the following issue: https://github.com/zenodo/zenodo/issues/2181. We do not know when it will be resolved, therefore, the DOI that refers to all versions of the software and that, by default, redirects to the latest version was used as a reference. Anyway, we will monitor the situation and make sure that the latest release is uploaded correctly to Zenodo as soon as the problem is resolved.

Article
All the mentioned issues have been addressed, correcting former inconsistencies and mistakes. In addition, the article has been updated with new content, including descriptions of the added methods applied to improve the research.


DCOS-10991: Add host volume

As noted here: https://github.com/dcos/dcos-ui/pull/1409#issuecomment-263444396
This PR adds the following features:

Host Volume
Remove External Volumes Drop down

This also adds Host Volumes to the review table:

LGTM 💯 ✨


 We have been using this UI library for about a year now.
However, this commit seems to have broken our ButtonDropdown.
When a user picks an option, the menu dropdown closes but the onItemClick does not get triggered anymore.

See: How event retargeting and how event propagation works inside Shadow DOM

Related links, issue #, if available: n/a
How has this been tested?

Review checklist
The following items are to be evaluated by the author(s) and the reviewer(s).
Correctness

Changes include appropriate documentation updates.
Changes are backward-compatible if not indicated, see CONTRIBUTING.md.
Changes do not include unsupported browser features, see CONTRIBUTING.md.
Changes were manually tested for accessibility, see accessibility guidelines.

Security

If the code handles URLs: all URLs are validated through the checkSafeUrl function.

Testing

Changes are covered with new/existing unit tests?
Changes are covered with new/existing integration tests?

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

is there a way to add unit or integ tests for this case?

Thanks for the quick review!
Absolutely - just added tests to cover uses cases inside Shadow DOM.


dummy change to kick pipelines integration

Description
Changes proposed in this pull request:

dummy change to kick pipelines integration
...
...

Related issue(s)

/test pre-control-plane-components-provisioner


Associating Issues With Pull Requests V2

Our old approach involved scraping the DOM from a public issue view. This does not work on private repositories (we could not extract an auth token from the app private key authenticated octokit client.)
After some research by @wannacfuture and I, we came to the conclusion that the only way to do this is to read the events of an issue and pull request e.g. https://api.github.com/repos/ubiquibot/production/issues/77/timeline https://api.github.com/repos/ubiquibot/production/issues/78/timeline
And cross reference the connected or cross-referenced events e.g.
Issue
  {
    "id":<PHONE_NUMBER>6,
    "node_id": "COE_lADOJkXFU85-DOthzwAAAAK4YB4E",
    "url": "https://api.github.com/repos/ubiquibot/production/issues/events/11683241476",
    "actor": {
      "login": "pavlovcik",
      "id": 4975670,
      "node_id": "MDQ6VXNlcjQ5NzU2NzA=",
      "avatar_url": "https://avatars.githubusercontent.com/u/4975670?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/pavlovcik",
      "html_url": "https://github.com/pavlovcik",
      "followers_url": "https://api.github.com/users/pavlovcik/followers",
      "following_url": "https://api.github.com/users/pavlovcik/following{/other_user}",
      "gists_url": "https://api.github.com/users/pavlovcik/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/pavlovcik/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/pavlovcik/subscriptions",
      "organizations_url": "https://api.github.com/users/pavlovcik/orgs",
      "repos_url": "https://api.github.com/users/pavlovcik/repos",
      "events_url": "https://api.github.com/users/pavlovcik/events{/privacy}",
      "received_events_url": "https://api.github.com/users/pavlovcik/received_events",
      "type": "User",
      "site_admin": false
    },
    "event": "connected",
    "commit_id": null,
    "commit_url": null,
    "created_at": "2024-02-02T12:00:52Z",
    "performed_via_github_app": null
  }

Pull Request
  {
    "id":<PHONE_NUMBER>9,
    "node_id": "COE_lADOJkXFU85-DO11zwAAAAK4YB39",
    "url": "https://api.github.com/repos/ubiquibot/production/issues/events/11683241469",
    "actor": {
      "login": "pavlovcik",
      "id": 4975670,
      "node_id": "MDQ6VXNlcjQ5NzU2NzA=",
      "avatar_url": "https://avatars.githubusercontent.com/u/4975670?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/pavlovcik",
      "html_url": "https://github.com/pavlovcik",
      "followers_url": "https://api.github.com/users/pavlovcik/followers",
      "following_url": "https://api.github.com/users/pavlovcik/following{/other_user}",
      "gists_url": "https://api.github.com/users/pavlovcik/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/pavlovcik/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/pavlovcik/subscriptions",
      "organizations_url": "https://api.github.com/users/pavlovcik/orgs",
      "repos_url": "https://api.github.com/users/pavlovcik/repos",
      "events_url": "https://api.github.com/users/pavlovcik/events{/privacy}",
      "received_events_url": "https://api.github.com/users/pavlovcik/received_events",
      "type": "User",
      "site_admin": false
    },
    "event": "connected",
    "commit_id": null,
    "commit_url": null,
    "created_at": "2024-02-02T12:00:52Z",
    "performed_via_github_app": null
  }

We can see that the only thing matching here is the created_at property.
Be sure to invalidate pairing disconnected and un-cross-referenced events (not sure the exact names.)
Glossary
connected refers to when a collaborator selects the issue from the pull request UI (sidebar)
cross-referenced is when a contributor who opens the pull request, opens with a "closing" keyword e.g. Resolves #1234

@wannacfuture this is yours to take

Just did a test. Once not linked, and once linked.
import { Octokit } from "@octokit/rest";
import { Endpoints } from "@octokit/types";
import { emitterEventNames } from<EMAIL_ADDRESS>export type GitHubEventClassName = (typeof emitterEventNames)[number];

export type GitHubIssue = Endpoints["GET /repos/{owner}/{repo}/issues"]["response"]["data"][0];
export type GitHubComment = Endpoints["GET /repos/{owner}/{repo}/issues/{issue_number}/comments"]["response"]["data"][0];
export type GitHubLabel = Endpoints["GET /repos/{owner}/{repo}/labels"]["response"]["data"][0];
export type GitHubPullRequest = Endpoints["GET /repos/{owner}/{repo}/pulls"]["response"]["data"][0];
export type GitHubEvent = Endpoints["GET /repos/{owner}/{repo}/issues/{issue_number}/timeline"]["response"]["data"][0];

// Initialize Octokit with authentication
const octokit = new Octokit();
// { auth: `YOUR_GITHUB_APP_PRIVATE_KEY` }

async function fetchIssueOrPullRequestEvents(owner: string, repo: string, issueOrPrNumber: number): Promise<GitHubEvent[]> {
  try {
    const response = await octokit.rest.issues.listEvents({
      owner,
      repo,
      issue_number: issueOrPrNumber,
    });
    return response.data as GitHubEvent[];
  } catch (error) {
    console.error("Error fetching events:", error);
    return [];
  }
}

async function processEventsForConnections(owner: string, repo: string, issueNumber: number, pullRequestNumber: number) {
  const issueEvents = await fetchIssueOrPullRequestEvents(owner, repo, issueNumber);
  const prEvents = await fetchIssueOrPullRequestEvents(owner, repo, pullRequestNumber);

  // Track connections and disconnections
  const connections = new Map<number, GitHubEvent>(); // Use issue/pr number as key for easy access

  issueEvents.forEach((event) => {
    if (event.event === "connected" || event.event === "cross-referenced") {
      connections.set(issueNumber, event);
    } else if (event.event === "disconnected") {
      connections.delete(issueNumber); // Invalidate connection if disconnected
    }
  });

  prEvents.forEach((event) => {
    if (event.event === "connected" || event.event === "cross-referenced") {
      connections.set(pullRequestNumber, event);
    } else if (event.event === "disconnected") {
      connections.delete(pullRequestNumber); // Invalidate connection if disconnected
    }
  });

  if (connections.size > 0) {
    console.log("Valid connections found:", Array.from(connections.values()));
  } else {
    console.log("No valid connections or all connections invalidated.");
  }
}

async function main() {
  const result = await processEventsForConnections("ubiquibot", "production", 77, 78);
  console.trace({ result });
}

try {
  void main();
} catch (error) {
  console.error(error);
}

m1:static nv$ yarn tsx static/link-pull.ts
yarn run v1.22.21
$ /Users/nv/repos/pavlovcik/link-pull-request-prototype/node_modules/.bin/tsx static/link-pull.ts
No valid connections or all connections invalidated.
Trace: { result: undefined }
    at main (/Users/nv/repos/pavlovcik/link-pull-request-prototype/static/link-pull.ts:62:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
✨  Done in 1.28s.
m1:static nv$ yarn tsx static/link-pull.ts
yarn run v1.22.21
$ /Users/nv/repos/pavlovcik/link-pull-request-prototype/node_modules/.bin/tsx static/link-pull.ts
Valid connections found: [
  {
    id:<PHONE_NUMBER>4,
    node_id: 'COE_lADOJkXFU85-DOthzwAAAAK4eOs6',
    url: 'https://api.github.com/repos/ubiquibot/production/issues/events/11684866874',
    actor: {
      login: 'pavlovcik',
      id: 4975670,
      node_id: 'MDQ6VXNlcjQ5NzU2NzA=',
      avatar_url: 'https://avatars.githubusercontent.com/u/4975670?v=4',
      gravatar_id: '',
      url: 'https://api.github.com/users/pavlovcik',
      html_url: 'https://github.com/pavlovcik',
      followers_url: 'https://api.github.com/users/pavlovcik/followers',
      following_url: 'https://api.github.com/users/pavlovcik/following{/other_user}',
      gists_url: 'https://api.github.com/users/pavlovcik/gists{/gist_id}',
      starred_url: 'https://api.github.com/users/pavlovcik/starred{/owner}{/repo}',
      subscriptions_url: 'https://api.github.com/users/pavlovcik/subscriptions',
      organizations_url: 'https://api.github.com/users/pavlovcik/orgs',
      repos_url: 'https://api.github.com/users/pavlovcik/repos',
      events_url: 'https://api.github.com/users/pavlovcik/events{/privacy}',
      received_events_url: 'https://api.github.com/users/pavlovcik/received_events',
      type: 'User',
      site_admin: false
    },
    event: 'connected',
    commit_id: null,
    commit_url: null,
    created_at: '2024-02-02T14:24:51Z',
    performed_via_github_app: null
  },
  {
    id:<PHONE_NUMBER>6,
    node_id: 'COE_lADOJkXFU85-DO11zwAAAAK4eOsy',
    url: 'https://api.github.com/repos/ubiquibot/production/issues/events/11684866866',
    actor: {
      login: 'pavlovcik',
      id: 4975670,
      node_id: 'MDQ6VXNlcjQ5NzU2NzA=',
      avatar_url: 'https://avatars.githubusercontent.com/u/4975670?v=4',
      gravatar_id: '',
      url: 'https://api.github.com/users/pavlovcik',
      html_url: 'https://github.com/pavlovcik',
      followers_url: 'https://api.github.com/users/pavlovcik/followers',
      following_url: 'https://api.github.com/users/pavlovcik/following{/other_user}',
      gists_url: 'https://api.github.com/users/pavlovcik/gists{/gist_id}',
      starred_url: 'https://api.github.com/users/pavlovcik/starred{/owner}{/repo}',
      subscriptions_url: 'https://api.github.com/users/pavlovcik/subscriptions',
      organizations_url: 'https://api.github.com/users/pavlovcik/orgs',
      repos_url: 'https://api.github.com/users/pavlovcik/repos',
      events_url: 'https://api.github.com/users/pavlovcik/events{/privacy}',
      received_events_url: 'https://api.github.com/users/pavlovcik/received_events',
      type: 'User',
      site_admin: false
    },
    event: 'connected',
    commit_id: null,
    commit_url: null,
    created_at: '2024-02-02T14:24:51Z',
    performed_via_github_app: null
  }
]
Trace: { result: undefined }
    at main (/Users/nv/repos/pavlovcik/link-pull-request-prototype/static/link-pull.ts:62:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
✨  Done in 1.45s.
m1:static nv$

@pavlovcik I already stumbled upon this before, and the problem with cross-referenced is that referencing another issue in a comment also creates that same event. connected does not always happen either, depending on some scenarios. I've seen somewhere that the GraphQl API actually does make it possible to retrieve, but didn't test myself. Second problem is that the timeline is paginated and could eventually get tedious to parse. I can have a look if @wannacfuture has something else on his plate.

I guess I technically finished implementing this already at https://github.com/ubiquibot/conversation-rewards
Which is a brand new codebase with tests for this.


Development release process document is out of date

Development release process document
In section 3. Update version number, the following need to be changed:

This file needs to be removed earth_enterprise/src/fusion_version.txt as its not used any more in versioning
This file earth_enterprise/src/version.txt needs the version number to be upgraded
No changes are needed in earth_enterprise/src/third_party/apache2/apache-ge-index.shtml

Also in the process of creating a release (final) we need to document that this script needs to be executed to create an archive that include the files stored in Git LFS.
Some where in the document the following needs to be documented:

Updated version number in BUILD.md and Upgrade.md
earth_enterprise/BUILD.md
earth_enterprise/Upgrade.md

Update copyright year if the year changed. Example, if the year when the release is made is 2019, the copyright notice needs to be extended to include that year. Change '2017-2018' to '2018-2019'.

Done


Sync or Async?

Just to be clear, if I use this:
// Load a script and don't worry about a callback
load("http://foo.com/foo.js");

is this going to run synchronously?

It will be asynchronous.  The load function will finish in your main event loop  sometime in the future the script will load.


Map layer manager UI - refresh

If the top Map Layer Toggle is off - should the entire Map Layer UI below that component be disabled?

If the top Map Layer Toggle is off - should the entire Map Layer UI below that component be disabled?

If the top Map Layer Toggle is off - should the entire Map Layer UI below that component be disabled?

Good point, we will add this to the backlog.  Tx.

If the top Map Layer Toggle is off - should the entire Map Layer UI below that component be disabled?

Good point, we will add this to the backlog.  Tx.

@Mergifyio backport release/2.11.0

@Mergifyio backport release/2.11.0


Fixes 4897: env_created field reset after updating template

Summary
Fixes query that sets the rhsm_environment_created field. Without this change, any subsequent template update task would set all the other templates' rhsm_environment_created fields to false.
Testing steps

Create a template. rhsm_environment_created should be false initially in the response
Fetch the template. rhsm_environment_created should be true
Create another template and then list them after a couple seconds to let the update-template-content task complete
Both templates should have their rhsm_environment_created fields set to true

https://issues.redhat.com/browse/HMS-4897

test failing with expired gpg key:
 Error Trace:    /Users/bhouse/Repos/content-sources-backend/pkg/dao/repository_configs_test.go:2277
                Error:          Should be true
                Error Trace:    /Users/bhouse/Repos/content-sources-backend/pkg/dao/repository_configs_test.go:2278
                Error:          Not equal:
                                actual  : "Error validating signature: openpgp: key expired. Is this the correct GPG Key?"
                                +Error validating signature: openpgp: key expired. Is this the correct GPG Key?

i'm guessing this is the gpg key in pkg/test/gpg/gpgkey.pub?

yeah, i think so?
$ gpg --show-keys --with-fingerprint pkg/test/gpg/gpgkey.pub
pub   dsa1024 2007-03-08 [SC]
      4CCA 1EAF 950C EE4A B839  76DC A040 830F 7FAC 5991
uid                      Google, Inc. Linux Package Signing Key<EMAIL_ADDRESS>sub   elg2048 2007-03-08 [E]

pub   rsa4096 2016-04-12 [SC]
      EB4C 1BFD 4F04 2F6D DDCC  EC91 7721 F63B D38B 4796
uid                      Google Inc. (Linux Packages Signing Authority<EMAIL_ADDRESS>sub   rsa4096 2016-04-12 [S] [expired: 2019-04-12]
sub   rsa4096 2017-01-24 [S] [expired: 2020-01-24]
sub   rsa4096 2019-07-22 [S] [expired: 2022-07-21]
sub   rsa4096 2021-10-26 [S] [expired: 2024-10-25]
sub   rsa4096 2023-02-15 [S] [expires: 2026-02-14]

doesn't look like we're using it in a test to actually validate a repo, try replacing it with this one? https://packages.microsoft.com/keys/microsoft.asc

hmmm with that one i see this error in the same test: Error validating signature: openpgp: signature made by unknown entity. Is this the correct GPG Key?

oh then maybe we are testing that, it looks like maybe its a google repo?  try updating it to https://www.google.com/linuxrepositories/

yep looking at that one now, looks like that's the one we had that was expired. or well i get the same failure (openpgp: key expired), but the output here does look a bit different:
bhouse@mac:content-sources-backend $ gpg --show-keys --with-fingerprint pkg/test/gpg/gpgkey.pub
pub   dsa1024 2007-03-08 [SC]
      4CCA 1EAF 950C EE4A B839  76DC A040 830F 7FAC 5991
uid                      Google, Inc. Linux Package Signing Key<EMAIL_ADDRESS>sub   elg2048 2007-03-08 [E]

pub   rsa4096 2016-04-12 [SC]
      EB4C 1BFD 4F04 2F6D DDCC  EC91 7721 F63B D38B 4796
uid                      Google Inc. (Linux Packages Signing Authority<EMAIL_ADDRESS>sub   rsa4096 2016-04-12 [S] [expired: 2019-04-12]
sub   rsa4096 2017-01-24 [S] [expired: 2020-01-24]
sub   rsa4096 2019-07-22 [S] [expired: 2022-07-21]
sub   rsa4096 2021-10-26 [S] [expired: 2024-10-25]
sub   rsa4096 2023-02-15 [S] [expires: 2026-02-14]
sub   rsa4096 2024-01-30 [S] [expires: 2027-01-29]

Hi @xbhouse
I deployed content-sources-backend=pr-866-7232dbb
I created a repo
I created a template and then fetched it. In both cases rhsm_environment_created was False:
In [2]: template_dict = dict(arch='x86_64', version='8', repository_uuids=['ef361385-3470-45ae-b99e-799ae4a14db1'], name='test template', use_latest=True)

In [3]:  app.content_sources.rest_client.templates_api.create_template(template_dict)
2024-10-28 10:25:50.829 [    INFO] [iqe.base.rest_client] REST: POST https://ee-2woml2fz.apps.crc-eph.r9lp.p1.openshiftapps.com/api/content-sources/v1/templates/ with query params [] and x-rh-insights-request-id=7475312b-0959-45f4-bd97-cd956aec3af8
Out[3]:
{'arch': 'x86_64',
 'created_at': '2024-10-28T10:25:50.7602466Z',
 'created_by': 'ephemeral-user',
 'date': '0001-01-01T00:00:00Z',
 'description': '',
 'last_update_snapshot_error': '',
 'last_updated_by': 'ephemeral-user',
 'name': 'test template',
 'org_id': '3340851',
 'repository_uuids': ['ef361385-3470-45ae-b99e-799ae4a14db1'],
 'rhsm_environment_created': False,
 'rhsm_environment_id': 'db774ff31d084963bd476287b7f9a57e',
 'updated_at': '2024-10-28T10:25:50.7602466Z',
 'use_latest': True,
 'uuid': 'db774ff3-1d08-4963-bd47-6287b7f9a57e',
 'version': '8'}

In [4]: app.content_sources.rest_client.templates_api.get_template('db774ff3-1d08-4963-bd47-6287b7f9a57e')
2024-10-28 11:03:28.049 [    INFO] [iqe.base.rest_client] REST: GET https://ee-2woml2fz.apps.crc-eph.r9lp.p1.openshiftapps.com/api/content-sources/v1/templates/db774ff3-1d08-4963-bd47-6287b7f9a57e with query params [] and x-rh-insights-request-id=eeb8696c-e18a-4d3d-a62f-85f9bf5aed4a
Out[4]:
{'arch': 'x86_64',
 'created_at': '2024-10-28T10:25:50.760246Z',
 'created_by': 'ephemeral-user',
 'date': '0001-01-01T00:00:00Z',
 'description': '',
 'last_update_snapshot_error': '',
 'last_updated_by': 'ephemeral-user',
 'name': 'test template',
 'org_id': '3340851',
 'repository_uuids': ['ef361385-3470-45ae-b99e-799ae4a14db1'],
 'rhsm_environment_created': False,
 'rhsm_environment_id': 'db774ff31d084963bd476287b7f9a57e',
 'updated_at': '2024-10-28T10:25:50.760246Z',
 'use_latest': True,
 'uuid': 'db774ff3-1d08-4963-bd47-6287b7f9a57e',
 'version': '8'}

In [5]:

Thank you

@swadeley ah yep this isn't testable in ephemeral. this field only ever gets set to true if candlepin is enabled

@swadeley ah yep this isn't testable in ephemeral. this field only ever gets set to true if candlepin is enabled. i think what we've been doing in these cases is just testing in stage

OK, will do.

thank you @swadeley


Improve documentation for Mac OS users

I'd like to try jazzer with my Java library under Mac OS and have already created a target class: https://github.com/marco-schmidt/ufxcoder/blob/master/src/main/java/ufxcoder/formats/JazzerTarget.java
I've also followed your README.md and installed bazel.
I don't understand what do next. It seems like there are no released binaries for Mac OS and bazel is required to compile them. However, the documented bazel calls are for Linux as far as I can see.
Could you include a small guide on what to do as a Mac user? Or point me to a section in the README I may have overlooked?
If someone has time to run jazzer on my library, I've added the build and jazzer call which I think should work to the target class: https://github.com/marco-schmidt/ufxcoder/blob/bf14f126e963620275f59106e068448d3bafd5a9/src/main/java/ufxcoder/formats/JazzerTarget.java#L30
Thanks for any feedback and for providing jazzer,
Marco

I agree that the installation instructions for macOS were less than ideal. I don't own a macOS device myself and thus can't test binaries, but have improved the instructions in #128. If you encounter any issues while going through the steps, please update this issue and I will try to be of help.
I tried running the command at https://github.com/marco-schmidt/ufxcoder/blob/bf14f126e963620275f59106e068448d3bafd5a9/src/main/java/ufxcoder/formats/JazzerTarget.java#L30 and it worked well after a small fix: The jar files live in a lib subdirectory of build/install/ufxcoder on my machine.
The target produces quite a few log line of the form
09:37:27.884 [main] DEBUG ufxcoder.formats.jpeg.JpegProcessor - 2 ffc4 40

Fuzzing should be faster if there is a way to disable these.

@fmeum Thanks for you quick response. I could follow the new guide easily, but running jazzer fails in the compilation stage on my machine:
$ ../../tools/bazelisk-darwin run //:jazzer -- --cp=... [lengthy arguments snipped]
INFO: Analyzed target //:jazzer (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
INFO: From Compiling absl/strings/internal/utf8.cc:
warning: unknown warning option '-Wno-implicit-int-float-conversion'; did you mean '-Wno-implicit-float-conversion'? [-Wunknown-warning-option]
1 warning generated.
...
ERROR: /Users/marco/dev/code/jazzer/driver/BUILD.bazel:4:11: Compiling driver/sanitizer_hooks_with_pc.cpp failed: (Exit 1): cc_wrapper.sh failed: error executing command external/local_config_cc/cc_wrapper.sh -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -fcolor-diagnostics -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG ... (remaining 22 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
driver/sanitizer_hooks_with_pc.cpp:51:16: error: 'asm goto' constructs are not supported yet
  asm volatile goto(
               ^
1 error generated.
Target //:jazzer failed to build

As for the dependencies mentioned under Using Bazel on my machine:
$ clang --version
Apple clang version 11.0.0 (clang-1<IP_ADDRESS>)
Target: x86_64-apple-darwin20.4.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
$ java -version
openjdk version "11.0.10" 2021-01-19
OpenJDK Runtime Environment (build 11.0.10+9)
OpenJDK 64-Bit Server VM (build 11.0.10+9, mixed mode)

This problem was fixed in https://github.com/CodeIntelligenceTesting/jazzer/issues/116 by updating XCode. Could you give that a try?

Hi @marco-schmidt ! We have made a lot of changes since you opened this ticket nearly 3-years ago. I'm following up with users of Jazzer to see how/what they achieved and if there's any way we can help.
Ping me if you can discuss?
david[dot]merian [at] code-intelligence[dot]com


NavViewItem AutomationPeer should support Invoke pattern

NavViewItem AutomationPeer was intended to support Invoke Pattern, but didn't return correctly for GetPatternCore.
Not supporting Invoke causes problems when interacting with NavView via UIA, particularly when using Scan Mode in Narrator.
Closes #5998

/azp run

Looks like NavViewItem AP not supporting Invoke pattern was an intentional change:
NavigationViewItem tab control type supports invoke pattern with PaneDisplayMode="Top" #2702

/azp run

Since we don't want to add Invoke back to NavViewItem AP (because of #2702), we instead double up UIA SelectionItem.Select() to also do an invoke. This makes SelectionItem.Select() equivalent to clicking on the item which is what desired in most cases.

@ojhad can you take a look at the latest iteration?

/azp run


Fixed: 5.6 double audio, ads Fixed

To use, click this link below and click update
https://github.com/TheRealJoelmatic/RemoveAdblockThing/raw/180c55ef13ac8f87866caa7df4afd1594cb531cc/Youtube-Ad-blocker-Reminder-Remover.user.js
thanks to @statchamber

https://github.com/CatPieLeaf/RemoveAdblockThing/raw/test/Youtube-Ad-blocker-Reminder-Remover.user.js
this one has all changes from the recent PRs.

Seems to have only worked briefly, dreaded ad blockers not allowed on YouTube message is back with 5.6


Add the ability to specify a different console for a new process on an existing container

There is a limitation regarding the ability to provide a different console than container's one in case we are trying to execute a new process on this existing container. Indeed, because the console is tied to the ContainerConfig structure, we have no way to specify a different console for another process.
This PR moves Interactive and Console fields from the structure ContainerConfig to Cmd, solving our current limitation.
This PR fixes issue #204.

Coverage decreased (-0.5%) to 67.42% when pulling 109eaf380672e52dac7543edd00610caa1f67293 on sboeuf/fix_issue_204 into<PHONE_NUMBER>caba45f0630d7ac0442aefb4e93023 on sboeuf/fix_issue_202.

Do not merge until #205 is merged.

Coverage decreased (-0.5%) to 67.42% when pulling b988315d73cb6d306d322b8c3bc73724b28d82ff on sboeuf/fix_issue_204 into<PHONE_NUMBER>caba45f0630d7ac0442aefb4e93023 on sboeuf/fix_issue_202.

lgtm

Coverage decreased (-0.2%) to 67.405% when pulling 5a3ffc391e6ebb185b908237bc34f5c160bfd63f on sboeuf/fix_issue_204 into 3778d5167ffdb76de98d3fd6389ba4a932b4ae02 on sboeuf/fix_issue_202.


[Android]About onresume event

When Android app is resumed, I want to do some check. So right now I cannot subscribe an event from js code.

+1 my android app broken because of onresume

@mkonicek What do you think about adding
mCurrentReactContext
  .getJSModule(DeviceEventManagerModule.RCTDeviceEventEmitter.class)
  .emit("event", null);

onResume and onPause in ReactInstanceManagerImpl.java?
Just tested it real quick and it seems to work.

+1

+1

@christopherdro - I tried implementing that piece of code (with 'onResume' and 'onPause' instead of 'event') in ReactInstanceManagerImpl.java to onResume and onPause, and listening to them via DeviceEventManagerModule  but I can't get them to fire..
When I listen to toggleElementInspector (it's in the same file) and open the inspector, it does fire.

Closing as Android has AppState now - https://github.com/facebook/react-native/commit/1b7d7b0a78484bda82618246dab34f5d554464ba


fix: Reference ScaledObject's/ScaledJob's name in the scalers log

Signed-off-by: Zbynek Roubalik<EMAIL_ADDRESS>
Fixing outstanding issues introduced by this change.
Checklist

[x] Commits are signed with Developer Certificate of Origin (DCO - learn more)

Relates to #3419

/run-e2e


Adding Filter on Element

Offering a solution to the feature request  #405  opened (by me ...)

Thanks.
Could you break the menu into sub-menus? Like the Model Tree - Business, Application, etc... It's a bit large!
@jbsarrodie Could you look too?

@phillipus Yes I will. BTW Happy Birthday!
@vinceaxe how the hell are you pushing a PR while you're at work ;-)

BTW Happy Birthday!

Thanks. So I got a PR for my birthday ;-)

Thanks. So I got a PR for my birthday ;-)

I can offer you a free copy of a very good ArchiMate modelling tool ;-)
I'll pay you a bear the next time we meet !

I needed some coffee break, and looked at it during evening anyway
I thought about splitting... But there is quite some complexity due to the
fElementAction that is full fill from another list ... It would have
required to fully split all actions and elementMenu by their layer ... A
lot of duplicated code (I already think about duplicated code between
element and relationship....)
And so Happy Birthday !!!
Le jeu. 8 nov. 2018 à 18:17, Jean-Baptiste Sarrodie <
<EMAIL_ADDRESS>a écrit :

Thanks. So I got a PR for my birthday ;-)
I can offer you a free copy of a very good ArchiMate modelling tool ;-)
I'll pay you a bear the next time we meet !
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/archimatetool/archi/pull/407#issuecomment-437082942,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABM66PksLB9NQslkMhFLa4AnjxFZNK6iks5utGcfgaJpZM4YVBmW
.

Could you break the menu into sub-menus? Like the Model Tree - Business, Application, etc... It's a bit large!

Done !

Cool. Will look at this soon. Question - why are the string like this:
"Element Filter" /* Messages.ZestView_9*/
?
Best to leave them as is and externalise last. :-)

I'll be taking a look at this next week some time. Leave it with me. :-)

Thanks for the latest PR. The element list is easier to mange now that we have sub-menus.
However there is a side-effect.
Because the menu items use AS_RADIO_BUTTON style then selecting one will not deselect the current one from another sub-menu. This is because menu items are grouped together in more than one group so exclusive selection does not apply. So either we go back to a huge single sub-menu list or we figure out a hack to deselect the current menu item when selecting a new one.

I'm working on it...

Thanks. This pull request has been integrated manually.

Thanks. This pull request has been integrated manually.

(That is an automatic message that is generated when I do a PR merge)
Still need to discuss this...
I refactored your code to simplify it, avoiding duplication and a fix for the menu selection issue above.
Could you test the functionality to ensure that the filter works as expected? And also whether it works in combination with the relationship filter?

Tested and approved !
Thanks :)


When will $cb work again?

Any updates on this?

Hi, sorry for the late reply.
I finally got around to reaching out to the coinmarketcal team to learn more about their new API plans and I've decided that I'm dropping the coinmarketcal feature from the bot. Without funding, the free tier service doesn't provide great features to make the implementation worthwhile for the bot. Nevertheless, I'm open to any PR's if someone is willing to go ahead and make an implementation for it, but I'm not going to bother with it.


Add notifications for pushes from BioC-sync

Set up email notification upon push to repo (see https://help.github.com/articles/receiving-email-notifications-for-pushes-to-a-repository/)


0.0.2.dev6

Current coverage is 100% (diff: 100%)

Merging #59 into master will not change coverage

@@           master   #59   diff @@
===================================
  Files           7     7
  Lines         131   131
  Methods         0     0
  Messages        0     0
  Branches        0     0
===================================
  Hits          131   131
  Misses          0     0
  Partials        0     0

Powered by Codecov. Last update ceec47a...2f724ed

Enjoy your bike riding!


Investigate loading of draco morph targets

We currently disable morph targets when loading draco compressed meshes here because they were found to not work.
However the standard gltf morph test model found here loads fine when this restriction is removed.
We should investigate whether we can re-enable draco morph targets in general.

Hi @slimbuck - you've done bunch of work on draco recently .. any insights on this issue?


Preact + Express can't render server-side

If I call
app.renderToHTML

it will use react-dom's renderToStaticMarkUp or renderToString. However, Preact doesn't have those, so aliasing doesn't do anything. Is there anything to make it work? (error: Error: Cannot find module 'preact-compat\lib\ReactComponentTreeHook')
But there is the module preact-render-to-string - how could I implement that?

Is this based on https://github.com/zeit/next.js/tree/master/examples/using-preact?

Yes, but I wanted to combine it with SSR caching (https://github.com/zeit/next.js/blob/master/examples/ssr-caching/server.js). Hence the app.renderToHTML.

When I don't switch to Express / renderToHTML everything works fine.

@kurtextrem Would love to see a gist/example if you managed to resolve this issue? I am looking to do something similar

Sure, that would be great :)
Am 15.04.2017 04:21 schrieb "Rem"<EMAIL_ADDRESS>@kurtextrem https://github.com/kurtextrem Would love to see a
gist/example if you managed to resolve this issue? I am looking to do
something similar
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/zeit/next.js/issues/1375#issuecomment-294266385, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAepgjXXescf9V59QJ-nbq2IO8jw8xqpks5rwCmpgaJpZM4MXx7i
.

Sorry, I've misread your comment - no, I haven't solved it. However, there
are other boilerplates out there for Preact only SSR.
Am 15.04.2017 10:25 schrieb "Jacob G"<EMAIL_ADDRESS>
Sure, that would be great :)
Am 15.04.2017 04:21 schrieb "Rem"<EMAIL_ADDRESS>@kurtextrem https://github.com/kurtextrem Would love to see a
gist/example if you managed to resolve this issue? I am looking to do
something similar
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/zeit/next.js/issues/1375#issuecomment-294266385, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAepgjXXescf9V59QJ-nbq2IO8jw8xqpks5rwCmpgaJpZM4MXx7i
.


Duplicate Values for Structure ID are not flagged when importing Structure Inventory

When the values under the Structure ID's attribute are duplicated in the source structure inventory shapefile, the import process does not catch and prevent the import process because of the duplicated Structure ID values.
Steps to reproduce:

I created a copy of LowerCacheCreekStructures.shp and labeled it LowerCacheCreekStructuresDEZIQA.shp.
In QGIS, I edited the attribute table of the shapefile so that the first two rows have the same value under the APN column.

In the Import Structure Inventory panel, I mapped the APN column as the attribute for the Structure ID.
I completed the rest of the required attributes for the Structure Inventory and committed the new Structure Inventory.
The Structure Inventory was created without any errors produced regarding duplicate values for the Structure ID.

Additionally, I tried assigning HT_Found which had duplicating values in the column. I then mapped this as the Structure ID attribute and I was still able to create the Structure Inventory without any error produced for duplicate values.

Looks like i just forgot to include that validation check. It is fixed now.

Tested and verified fixed in HEC-FDA-2.0-Snapshot 12_22_21


Move vega dependencies to devDependencies?

@bilalshaikh42 should the Vega dependencies be moved to devDependencies?
"canvas": "^2.7.0",
"vega": "^5.19.1",
"vega-embed": "^6.15.2",
"vega-lite": "^5.0.0",

Done. Needed to avoid installing dependencies needed to compile canvas into the production Docker image. This would have added a few hundred MB to the Docker image.


Support IPython help in REPL

This doesn't display anything:
?list

Compared to:

Reported at https://developercommunity.visualstudio.com/content/problem/261625/python-interactive-window-does-not-support-input-r.html

? at the start or end produces the same result:

In our REPL, they both do nothing.


Automatic Python 3 compatibility via 2to3

In some cases, Python 2 may not be available.

Thanks for putting this in, @nikolay - I've actually got a new branch with simultaneous support that I'll be pushing as soon as I work out some internal issues


[Question] Possibility to relay topics to another MQTT broker?

Component

Generic

Description
Is it possible to configure some kind of relay/bridge from one broker to the other?
We have a test and prod environment. On the prod environment we have several devices
that are publishing data to the topic edge/# - it would be nice to have the possibility to
"mirror" those messages from the prod environment to the test environment.
The mosquitto MQTT broker supports this and calls that a bridge.
See http://www.steves-internet-guide.com/mosquitto-bridge-configuration/
Environment

OS:  SLES 15
TBMQ: 1.4.0

Hi @apachler,
Thank you for your feature request!
Unfortunately, the TBMQ does not currently support a relay or bridge functionality like the one described for the Mosquitto broker. While we are not actively working on this feature at the moment, it is planned to enter development soon and will be available in one of the upcoming releases.
We appreciate your patience and will keep you updated as we make progress!

nice, thanks for you response!


JoinHandle.await in another thread has chance to block forever.

Version
monoio v0.2.2
Platform
uname -a
Darwin drdrxps-MBP 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:38:37 PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T6000 arm64

ulimit -l
unlimited

Description
Calling JoinHandle.await in another thread(and also in another monoio runtime), it has odd to return the expected value and odd to block forever.
The code to reproduce this issue is in the following repo:
https://github.com/drmingdrmer/t-monoio/blob/ec306d33e9f581eacd06c51034de7d0d4698f9dd/src/bin/cross-rt-join.rs#L1
It is quite simple. Create a monoio runtime to run a future, send the JoinHandle to another thread and await it in another monoio runtime:
use std::sync::mpsc;
use std::time::Duration;

fn main() {
    let (tx, rx) = mpsc::channel();

    std::thread::spawn(move || {
        let mut rt = monoio::RuntimeBuilder::<monoio::FusionDriver>::new()
            .enable_all()
            .build()
            .expect("Failed building the Runtime");

        rt.block_on(async move {
            let fu = async move {
                // println!("inner-fu: ready");
                1u64
            };

            let handle = monoio::spawn(fu);
            tx.send(handle).unwrap();

            monoio::time::sleep(Duration::from_millis(1_000)).await;
            println!("outer-fu: after sending handle and sleep");
        });
    });

    let handle = rx.recv().unwrap();

    let mut rt = monoio::RuntimeBuilder::<monoio::FusionDriver>::new()
        .enable_all()
        .build()
        .expect("Failed building the Runtime");

    rt.block_on(async move {
        println!("joiner: before handle.await");
        let got = handle.await;
        println!("joiner: after handle.await: {:?}", got);
    });
}

Clone it and run it with:
cargo run --bin cross-rt-join

Sometimes it blocks forever:
joiner: before handle.await
outer-fu: after sending handle and sleep

Sometime it returns the expected value:
joiner: before handle.await
joiner: after handle.await: 1

The main issue here is that you run it on mac os, so you do not have any guarantee on where the task is going to be ran (on which core it's going to be executed). That why you have an erratic behavior.
For this part, you can activate the sync feature flag on monoio and you'll be able to do cross-thread async communication documented here (but it would be better in term of performance if you avoided cross-thread async communication as it's way more efficient when you avoid it)

@Miaxos
Thank you so much! it looks like the sync feature flag is the cause. With sync enabled, the program outputs consistent result.
The doc does not provide any clue about how to use JoinHandle correctly in another thread.
Is it possible to remove the Send and Sync bound from JoinHandle if sync feature flag is not enabled?
Another issue is that if the task panics, the JoinHandle.await in another thread blocks forever.

Hey @drmingdrmer I took some time to provide you an example of a working solution without sync. If you change the pinned core, the solution won't work anymore without sync.
(To be able to run those tests & io-uring using macos, I suggest you run those inside a VM backed by tart, it's what I'm using and it's working very well).
A working solution without `sync` (Linux only)
use std::sync::mpsc;
use std::time::Duration;

fn main() {
    let (tx, rx) = mpsc::channel();

    let t1 = std::thread::spawn(move || {
        // If you want to force a thread to be pinned on a core
        // (Only available on linux)
        monoio::utils::bind_to_cpu_set(Some(0)).unwrap();
        let mut rt = monoio::RuntimeBuilder::<monoio::FusionDriver>::new()
            .enable_all()
            .build()
            .unwrap();

        rt.block_on(async move {
            let fu = async move {
                println!("inner-fu: ready");
                1u64
            };

            let handle = monoio::spawn(fu);
            tx.send(handle).unwrap();

            monoio::time::sleep(Duration::from_millis(1_000)).await;
            println!("outer-fu: after sending handle and sleep");
        });
    });

    let t2 = std::thread::spawn(move || {
        // If you want to force a thread to be pinned on a core
        // (Only available on linux)
        monoio::utils::bind_to_cpu_set(Some(0)).unwrap();

        let mut rt = monoio::RuntimeBuilder::<monoio::FusionDriver>::new()
            .enable_all()
            .build()
            .unwrap();

        rt.block_on(async move {
            println!("joiner: before handle.await");
            loop {
                if let Ok(handle) = rx.try_recv() {
                    let got = handle.await;
                    println!("joiner: after handle.await: {:?}", got);
                    break;
                }
            }
        });
    });

    t1.join().unwrap();
    t2.join().unwrap();
}

Is it possible to remove the Send and Sync bound from JoinHandle if sync feature flag is not enabled?

Based on the implementation, you'll need to have T: !Send for JoinHandle<T> to not be Send.

Another issue is that if the task panics, the JoinHandle.await in another thread blocks forever.

Yeah, right now you would need to wrap future spawned into a catch_unwind with a manual Result


Parse step input url correctly

If the step input url is http://{$$.env.todosHost}/todos, it won't be parsed correctly. Solution, don't parse the url until the variables are replaced.

Fixed, releasing soon.


Add forwarded port in Vagrantfile

Describe the bug
Port 9876 should be added as a forwarded port in the Vagrantfile so that hot reloading can done as mentioned in the readme .
Port 9900 is present as a forwarded port which I think is not used anywhere and should be replaced with 9876.
Affects

[ ] Designer
[ ] Blocks editor
[ ] Companion
[ ] Compiled apps
[x] Buildserver
[ ] Other... (please describe)

Expected behaviour
http://localhost:9876 should be reachable from the host machine and should redirect the request to VM at the port 9876.
Steps to reproduce
Try reaching http://localhost:9876 using any browser after starting the server from the VM.

Created duplicate issue by mistake, hence closed this one.


Failed to compile and no traffic

Hello,
I would like to report two issues that I found yesterday:

The latest GitHub version fails to compile. It says error on line 152, "badgeFlatGit" is undefined. I reverted to commit 4d8960a in order for it to compile.
I tried using "ga-beacon.appspot.com", but no real-time traffic is registered in my Analytics Dashboard. I then pushed the code to my own GAE account and attempted to use it there. I tried Version 16 (4d8960a), 15 (f0dca65) and 12 (81f2a17). Unfortunately none of them showed any traffic in my dashboard as well. In other words the code is not working at all. Can you please check? By the way, the other "nojsstats.appspot.com" service works and traffic is shown in my dashboard.

Thank you for the awesome work!

Fixed the compile error in https://github.com/igrigorik/ga-beacon/commit/43a9c22a26dc8974db8efbbe5f03fc46ec6b04bd - thanks for the heads up.
I am seeing hits from ga-beacon.appspot.com in my own GA account. Are you sure you're setting the right property ID in your tests? Also, make sure you're using real-time analytics when checking for the hits.

Thanks for the quick reply. I double checked my test configuration and they are fine. And I am indeed using the real-time analytics.
I just ran the test again and observed something interesting: If I open this page directly in IE: https://ga-beacon.appspot.com/UA-XXXXXXXX-Y/ (the string is my Analytics property ID), I do see a hit registered in real-time dashboard. And the "active page" is shown as "/UA-XXXXXXXX-Y/". However, if I append anything to that URL, such as https://ga-beacon.appspot.com/UA-XXXXXXXX-Y/my-site/test-page , then no hit is registered in my real-time dashboard at all.

I checked the Google Analytics forums and realized that because my GA property is brand new, there may be a delay of 24~72 hours before any data from the measurement protocol show up in my dashboard.
Thanks again for your help - I'll report back if it still does not work after a few days. Otherwise I plan to close this issue.

@hwdsl2 the real-time tab in GA should show data in... real-time.

+1, I don't see any traffic in my project

@monolithed please check your real-time analytics.
Closing this bug since original issue was fixed.

Finally found out the reason why ga-beacon was not working in my Analytics account. It boils down to changing a single setting in GA.
FYI for other people having this issue: You MUST UN-check the option "Exclude all hits from known bots and spiders" in the "View Settings" of your GA account. Otherwise no hits will be registered and the beacon won't work.

@hwdsl2 ah, that makes sense -- thanks for the heads up!

Hi - I have just discovered GA-Beacon - thank you for creating it.
I'd like to use it to track traffic to my eBay product listings where I can't use js
I have installed the following line:
img src="https://ga-beacon.appspot.com/UA-7687050-3/ebay/test"
But when I visit the page, it is not showing up in my real-time stats in GA
Have I missed something?
Thanks for your help

@Ragrant57 first, make sure the image is actually included in the ebay page — you might have to inspect it. Per discussion above, uncheck the options mentioned above in GA and also see https://github.com/igrigorik/ga-beacon/issues/62


misc: create list of underconstrained areas in AES circuit

circuits/aes is underconstrained, make a list of constraints to enforce.

Closed


A Nearest Neighbor Search Question

Good job!
But could you please tell me where the code for the nearest neighbor matching methods (such as PQ/IVF/PQIVF/HNSW, etc.) is located?

Hi, we didn't include that code because it is trivial to change it. You can just change the line of code with the faiss index to include one of the faiss indexes from this list. It should be as simple as adding 1 to 3 lines of code


loading font from command line doesn't copy set#2 to location used by scanline interrupt to display font

The second set should always be copied from $1000-$17ff to $3800-$3fff on program start, regardless of whether already initialized, because user can only load font from command prompt currently.

workaround:
key / twice which will cause copy


WebView is incompatible with CSP?

With Content Security Policy (CSP) you can protect Web pages from being infected by inline scripts.  Using postMessage() you can still communicate with an embedded IFRAME which is quite useful.  Using such a scheme with WebView does not seem to be possible, since WebView requires injected JavaScript in order to communicate.  I guess WebView doesn't care about CSP either?

CSP is just a hint, it's up to the browser / webview component to honor it.


FUSEDOC-2074 - update contents

The contents have been given an overhaul. I have corrected grammar and clarified instructions. The original requirements were:
broken link to Exchange in deleteOutputFile option
these sentences should probably we reworked a bit:
    "Then in Camel 2.10.3 and older, then Camel will use "classpath:" as the default prefix, and load the resource from the classpath."
    "You can still though explicit configure a prefix, and then mix and match."

I have updated a fair bit more than that, though.

Thanks