Remove BeautifulSoup warning, force it to use lxml

I have explicitly set the code to use lxml as the HTML processor for BeautifulSoup. In addition, I have refactored most of the inspectors to replace the call to utils.download followed by a BeautifulSoup(body) call with single utils.beautifulsoup_from_url lines. This reduced the number of places in the code with BeautifulSoup was instantiated, but there are some exceptions, usually because the scraper is using a post request instead. In the case of HHS, they do a bit more with the response as well.
I have noticed a few scraping errors when testing this change out in the following scrapers:

CFTC
DOT
EPA
FCA
HHS
NCUA
SEC
Treasury
USPS

Many of these look to be issues with changes in the page structure/URLs of the IG sites, but until I can investigate, this PR should be considered a WORK IN PROGRESS and probably not be merged quite yet. Thank you.

Okay, I think all of the remaining issues I am seeing are scraper problems (sites changing page structure, new reports without dates) and not the result of these changes. I think it's clear for your review if you are ready. Thank you.

Looks good to me! :+1:

What an excellent and helpful contribution, @harrisj! Like pruning a bonsai tree.
And @divergentdave, thanks for :eyes:-ing it. You can feel free to merge PRs like this too after reviewing them, I don't mean to be the bottleneck for stuff like this.

Thank you and sorry for the typo


Avoid running pager if not set

This PR avoids starting a pager if $PAGER is not set or the, newly introduced,
config option nopager is true.
Fixes #167

LGTM, although it makes the code a little less readable that the flag is already inverted with the no prefix. Wouldn't it be possible to introduce pager default true?

Right now I'd like to keep it as is, but for #35 we'll need to clean up the kinda messy configs anyway. Would probably make sense to takle this there for all existing options.


cwltoil: bring up to date with CWL v1.0 release

Synchronizes Toil CWL support with the latest cwltool supporting v1.0 of
the CWL specification:

Handles back compatibility of path/location specifications for files
(path is old, location is new).
Adjusts imports to match cwltool reorganization changes
Some functions use kwargs instead of default arguments
(single_job_executor, _init_job).
Changes from output to outputSource in job outputs.

cc @tetron for sanity checks

Can one of the admins verify this patch?

+1!

Jenkins, test this please.

It looks like I must have broke some of the CWL tests, although can't see what specifically failed. Apologies, I'm trying to learn how to run the tests locally to see if I can fix the issues.

Apologies

No need. That's what tests are for ;-)
Note that the tests check out a specific commit of the compliance suite. I'm not surprised that they break when upgrading CWL.

Messed up rebase so closing to continue discussion in #1056


Support for .NET 5

Hello,
Is it possible to add support for .NET 5 in the published NuGet package? Right now, the library is compiled for Full Framework and, AFAIK it cannot be used on .NET 5 projects.
Thanks!

I am not a .NET authority, by any means, perhaps @emgre can weigh in since he's looked at this for our new libraries that are based on PInvoke instead of C++/CLI.  In those projects we are targeting .NET Standard which gives broad framework interop.
Here's a guide a about targeting different frameworks with C++/CLI:
https://docs.microsoft.com/en-us/dotnet/core/porting/cpp-cli
It seems that we cannot target .NET Standard, so we might have to build a multi-target package in the next release.

For me, it would be enough if next release multi-targets Full Framework/.NET 5. Thanks!

@andrepf I quickly tried it this morning. With this feature that was merged in CMake 3.17, it is possible to define the target property DOTNET_TARGET_FRAMEWORK to net5.0 and it should work. The changes would be here
https://github.com/dnp3/opendnp3/blob/132f2fdb62ad9325efa8293fb122800ee2476aa2/dotnet/CLRInterface/CMakeLists.txt#L138
and here
https://github.com/dnp3/opendnp3/blob/132f2fdb62ad9325efa8293fb122800ee2476aa2/dotnet/CLRAdapter/CMakeLists.txt#L55
and properly setting the DNP3_DOTNET_FRAMEWORK_VERSION CMake variable to net5.0.
I tried compiling it that way but I ran on issues with my installation of Visual Studio (I had to update it to get .NET 5.0 and it failed).


Created an example.

Refactored factories to return Query interface.
todo: docs.

[Current coverage][cc-pull] is 90.31%

Merging [#8][cc-pull] into [master][cc-base-branch] will decrease coverage by 0.47%

@@             master         #8   diff @@
==========================================
  Files            61         61
  Lines          1151       1157     +6
  Methods           0          0
  Messages          0          0
  Branches         58         58
==========================================
  Hits           1045       1045
- Misses           80         86     +6
  Partials         26         26

Powered by Codecov. Last updated by [80bcf09...c49b339][cc-compare]
[cc-base-branch]: https://codecov.io/gh/SpineEventEngine/jdbc-storage/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/SpineEventEngine/jdbc-storage/compare/80bcf091956220fffa71f7b8fe51a5bb91f40f1a...c49b339a74b871774feb1a2e9792e074994ad8c5?src=pr
[cc-pull]: https://codecov.io/gh/SpineEventEngine/jdbc-storage/pull/8?src=pr


Conversion from OpenAPI 3.0 to Swagger 2.0

Hi, guys. Does this project support the conversion from OpenAPI 3.0 to Swagger 2.0? Too many tools only support Swagger 2.0 and we hope it can do it.

Yes, we can down-convert to 2.0. But be aware that the conversion is lossy - some things won't be converted.


Hot deployment stream-rocketmq MQClientException: The producer group[mygroup] has been created before

Which Component
spring-cloud-starter-stream-rocketmq
Describe the bug
When hot deployment in eclipse throw MQClientException: The producer group[mygroup] has been created before
To Reproduce
Steps to reproduce the behavior:

Spring boot application with devtools in Eclipse
Send message using org.springframework.cloud.stream.function.StreamBridge.send
Modify any code trigger spring devtools hot deployment
When send message using org.springframework.cloud.stream.function.StreamBridge.send,Then throw exception

Expected behavior
Not throw exception and send message successfully.
Screenshots
streamBridge.send("mytopic-out-0","hello");

org.apache.rocketmq.client.exception.MQClientException: The producer group[mygroup] has been created before, specify another name please.
See http://rocketmq.apache.org/docs/faq/ for further details.
	at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.start(DefaultMQProducerImpl.java:195)
	at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.start(DefaultMQProducerImpl.java:175)
	at org.apache.rocketmq.client.producer.DefaultMQProducer.start(DefaultMQProducer.java:271)
	at com.alibaba.cloud.stream.binder.rocketmq.integration.outbound.RocketMQProducerMessageHandler.start(RocketMQProducerMessageHandler.java:112)
	at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder.doBindProducer(AbstractMessageChannelBinder.java:248)
	at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder.doBindProducer(AbstractMessageChannelBinder.java:91)
	at org.springframework.cloud.stream.binder.AbstractBinder.bindProducer(AbstractBinder.java:152)
	at org.springframework.cloud.stream.binding.BindingService.doBindProducer(BindingService.java:313)
	at org.springframework.cloud.stream.binding.BindingService.bindProducer(BindingService.java:282)
	at org.springframework.cloud.stream.function.StreamBridge.resolveDestination(StreamBridge.java:200)
	at org.springframework.cloud.stream.function.StreamBridge.send(StreamBridge.java:152)
	at org.springframework.cloud.stream.function.StreamBridge.send(StreamBridge.java:130)

Additional context
Linux  、Java8 、spring-cloud-starter-stream-rocketmq:2.2.7.RELEASE 、spring-cloud-stream:3.1.2
Reason
Not call MQClientInstance.unregisterProducer when stop application.

Please assign to me

@zkzlx I am sorry I have no time to finish it now, I give up it.

@zkzlx , @HHoflittlefish777 , @steverao,
I have this issue reproduced in spring-cloud-starter-stream-rocketmq:2.2.7.RELEASE and 2.2.9.Release.  This issue is due to below piece of code in com.alibaba.cloud.stream.binder.rocketmq.integration.outbound.RocketMQProduceFactory.initRocketMQProducer():
		if (RocketMQProducerProperties.ProducerType.Trans
				.equalsName(producerProperties.getProducerType())) {
                        ......
                }
		else {
			producer = new DefaultMQProducer(producerProperties.getNamespace(),
					producerProperties.getGroup(), rpcHook,
					producerProperties.getEnableMsgTrace(),
					producerProperties.getCustomizedTraceTopic());
		}

In initRocketMQProducer(), the RocketMQ Producer be created by newing a DefaultMQProducer. When some code change cause StreamBridge.send() rerun, the DefaultMQProducer will be new again, which has already been created. So, the Exception raised inside of DefaultMQProducer().
In the current code branch 2.2.x which version now is 2.2.10-RC1, I found this commit https://github.com/alibaba/spring-cloud-alibaba/commit/5e80d7c2c4e048f9eda6a624dadaf5c15c8c6e76  which already replaced the issue code by new ReusableMQProducer(...).  The ReusableMQProducer() avoids the RocketMQ Producer be created twice. Finally, I tested it   with devtools and the Exception does not occur.
So, I believe this issue has been fixed in 2.2.10-RC1.
Thanks.
JiaJun


Feature/head and sensorring

integrates head and sensorring on one bus
adapts diagnostics

extends #568

tested on cob4-10: OK


Fix for WaitDriverDownOnNode and WaitDriverUpOnNodein Reboot test

Now, test is more robust, as it checks all needed statuses on the node which was reboot:
18:21:13 STEP: reboot node: torpedo-tp-nextpx-basic-vol-driver-158-2
18:21:29 STEP: wait for node: torpedo-tp-nextpx-basic-vol-driver-158-2 to be back up
18:21:29 STEP: wait for volume driver to stop on node: torpedo-tp-nextpx-basic-vol-driver-158-2
18:21:29 INFO[2020-06-26 01:21:29] px on node torpedo-tp-nextpx-basic-vol-driver-158-2 is now down.
18:21:39 2020/06/26 01:21:39 Failed to wait for px status on: torpedo-tp-nextpx-basic-vol-driver-158-2 due to err: failed to get node info [torpedo-tp-nextpx-basic-vol-driver-158-2]. Err: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:9020: connect: connection refused" Next retry in: 10s
18:21:49 DEBU[2020-06-26 01:21:49] Getting node info: torpedo-tp-nextpx-basic-vol-driver-158-2
18:21:59 DEBU[2020-06-26 01:21:59] checking PX status on node: torpedo-tp-nextpx-basic-vol-driver-158-2
18:21:59 2020/06/26 01:21:59 Failed to wait for px status on: torpedo-tp-nextpx-basic-vol-driver-158-2 due to err: px cluster is usable but node torpedo-tp-nextpx-basic-vol-driver-158-2 status is not ok. Expected: STATUS_OK Actual: STATUS_NOT_IN_QUORUM Next retry in: 10s
18:22:19 2020/06/26 01:22:19 Failed to wait for px status on: torpedo-tp-nextpx-basic-vol-driver-158-2 due to err: px cluster is usable but node torpedo-tp-nextpx-basic-vol-driver-158-2 status is not ok. Expected: STATUS_OK Actual: STATUS_INIT Next retry in: 10s
18:22:29 INFO[2020-06-26 01:22:29] px on node: torpedo-tp-nextpx-basic-vol-driver-158-2 is now up. status: STATUS_OK

@mborodin-px did you run dev-test? let's run before merging

I ran directly https://jenkins.portworx.co/job/Torpedo/view/Torpedo NextPX/job/tp-nextpx-basic-vol-driver/158/ on my docker image


FORWARD-TSN for unreceived chunk results in ABORT

This is similar to issue #592, which was found to be a dcSCTP issue. In this case, I don't exactly know why usrsctp sends an ABORT. I have a packetdrill script that reproduces it (I haven't rebuilt the FreeBSD kernel in 21 years, so that was fun - and it doesn't seem to have changed a bit!)
So: The scenario is as follows:
Peer A is sending a lot of data. Some of these packets are lost (representing TSN ranges [284, 297], and some of these packets had a limited retransmission timeout, which expired before they could be retransmitted. So when the chunks were supposed to be resent, a FORWARD-TSN was sent to "skip" some of these chunks, while also bundling some chunks that hasn't yet expired but that were scheduled for retransmission.
The packetdrill script:
--tolerance_usecs=25000
 0.000 `sysctl -w net.inet.sctp.ecn_enable=0`
+0.000 `sysctl -w net.inet.sctp.pr_enable=1`
+0.000 `sysctl -w net.inet.sctp.asconf_enable=0`
+0.000 `sysctl -w net.inet.sctp.auth_enable=0`
+0.000 `sysctl -w net.inet.sctp.reconfig_enable=1`
+0.000 `sysctl -w net.inet.sctp.nrsack_enable=0`
+0.000 `sysctl -w net.inet.sctp.pktdrop_enable=0`
+0.000 socket(..., SOCK_STREAM, IPPROTO_SCTP) = 3
+0.000 bind(3, ..., ...) = 0
+0.000 listen(3, 1) = 0
+0.000 < sctp: INIT[flgs=0, tag=1, a_rwnd=5242880, os=65535, is=65535, tsn=281,
                    FORWARD_TSN_SUPPORTED[],
                    SUPPORTED_EXTENSIONS[types=[RECONFIG, FORWARD_TSN]]]
+0.000 > sctp: INIT_ACK[flgs=0, tag=2, a_rwnd=..., os=..., is=..., tsn=10,
                        FORWARD_TSN_SUPPORTED[],
                        SUPPORTED_EXTENSIONS[types=[FORWARD_TSN, RECONFIG]],
                        STATE_COOKIE[len=..., val=...]]
+0.010 < sctp: COOKIE_ECHO[flgs=0, len=..., val=...]
+0.000 > sctp: COOKIE_ACK[flgs=0]
+0.000 accept(3, ..., ...) = 4
+0.000 close(3) = 0
+0.000 < sctp: DATA[flgs=UBE, len=356, tsn=281, sid=1, ssn=0, ppid=0];
               DATA[flgs=UBE, len=354, tsn=282, sid=1, ssn=0, ppid=0];
               DATA[flgs=UB, len=172, tsn=283, sid=1, ssn=0, ppid=0]

+0.000 read(4, ..., 2000) = 340

+0.000 > sctp: SACK[flgs=0, cum_tsn=283, a_rwnd=..., gaps=[], dups=[]]

// There are four packets lost, containing TSNs [284..297]:
// 284 (UE), 285 (UBE), 286 (UBE), 287 (UB), 288 (UE), 289 (UBE),
// 290 (UBE), 291 (UB), 292 (UE), 293 (UBE), 294 (UBE), 295 (UB),
// 296 (UE), 297 (UB)

// The next packet that is actually received, contains TSNs 298 and forwards.

+0.040 < sctp: DATA[flgs=UE, len=135, tsn=298, sid=1, ssn=0, ppid=0];
               DATA[flgs=UBE, len=355, tsn=299, sid=1, ssn=0, ppid=0];
               DATA[flgs=UBE, len=353, tsn=300, sid=1, ssn=0, ppid=0];
               DATA[flgs=UB, len=264, tsn=301, sid=1, ssn=0, ppid=0]

+0.000 > sctp: SACK[flgs=0x00, cum_tsn=283, a_rwnd=1861474, gaps=[15:18], dups=[]]

// Some of these chunks that were never received, expire, which results
// in a FORWARD-TSN sent. That chunk is bundled with some DATA chunks that
// represent these being retransmitted. In this simple packetdrill, it's not
// clear why these would be retransmitted, but the script is made to be simple.
// In reality, numerous SACKs resulted in these being retransmitted.

+0.050 < sctp: FORWARD_TSN[cum_tsn=288, ids=[]];
               DATA[flgs=UBE, len=355, tsn=289, sid=1, ssn=0, ppid=0];
               DATA[flgs=UBE, len=356, tsn=290, sid=1, ssn=0, ppid=0];
               DATA[flgs=UB, len=260, tsn=291, sid=1, ssn=0, ppid=0];
               DATA[flgs=UE, len=96, tsn=292, sid=1, ssn=0, ppid=0]

+0.000 > sctp: ABORT[flgs=0, PROTOCOL_VIOLATION[info="Illegal message sequence, missing end for MID: 00000120"]]

+0.200 close(4) = 0

0x120 is 288, which is an end chunk, but the receiver doesn't even know that, as it has never seen that chunk.
I have the full PCAP of the sender side, if it would be of any help. And I have the most highest regard of usrsctp when it comes to protocol conformity, so I look forward to learning where my understanding of RFC3758 is lacking.
Thanks,
Victor

By looking at the packetdrill script, I would say that it is a bug in the FreeBSD stack... At least by looking at it at this point of time... Will look into in more detail...

Thanks Michael.
To help you pinpoint the issue, I have created a minimal reproducing sample, and learned some things while doing it. I'm essentially trying to find out if there is any way I can ensure that the packets I sent out will not provoke this issue. It's not a super-common issue, but if I can avoid it (as upgrading all clients will take some time), I'd prefer to do that.
--tolerance_usecs=25000
 0.000 `sysctl -w net.inet.sctp.ecn_enable=0`
+0.000 `sysctl -w net.inet.sctp.pr_enable=1`
+0.000 `sysctl -w net.inet.sctp.asconf_enable=0`
+0.000 `sysctl -w net.inet.sctp.auth_enable=0`
+0.000 `sysctl -w net.inet.sctp.reconfig_enable=1`
+0.000 `sysctl -w net.inet.sctp.nrsack_enable=0`
+0.000 `sysctl -w net.inet.sctp.pktdrop_enable=0`
+0.000 socket(..., SOCK_STREAM, IPPROTO_SCTP) = 3
+0.000 bind(3, ..., ...) = 0
+0.000 listen(3, 1) = 0
+0.000 < sctp: INIT[flgs=0, tag=1, a_rwnd=5242880, os=65535, is=65535, tsn=281,
                    FORWARD_TSN_SUPPORTED[],
                    SUPPORTED_EXTENSIONS[types=[RECONFIG, FORWARD_TSN]]]
+0.000 > sctp: INIT_ACK[flgs=0, tag=2, a_rwnd=..., os=..., is=..., tsn=10,
                        FORWARD_TSN_SUPPORTED[],
                        SUPPORTED_EXTENSIONS[types=[FORWARD_TSN, RECONFIG]],
                        STATE_COOKIE[len=..., val=...]]
+0.010 < sctp: COOKIE_ECHO[flgs=0, len=..., val=...]
+0.000 > sctp: COOKIE_ACK[flgs=0]
+0.000 accept(3, ..., ...) = 4
+0.000 close(3) = 0
+0.000 < sctp: DATA[flgs=UB, len=356, tsn=281, sid=1, ssn=0, ppid=0]

+0.000 > sctp: SACK[flgs=0, cum_tsn=281, a_rwnd=..., gaps=[], dups=[]]

// There are a few packets lost, TSNs [282..300],

+0.040 < sctp: DATA[flgs=UE, len=135, tsn=301, sid=1, ssn=0, ppid=0];
               DATA[flgs=UB, len=135, tsn=302, sid=1, ssn=0, ppid=0]

+0.000 > sctp: SACK[flgs=0x00, cum_tsn=281, a_rwnd=..., gaps=[20:21], dups=[]]

+0.050 < sctp: FORWARD_TSN[cum_tsn=288, ids=[]];
               DATA[flgs=UBE, len=355, tsn=289, sid=1, ssn=0, ppid=0]

+0.000 > sctp: ABORT[flgs=0, PROTOCOL_VIOLATION[info="Illegal message sequence, missing end for MID: 00000120"]]

+0.200 close(4) = 0

For this to trigger, the second packet (with a TSN much larger than what was previously seen) has to have at least two DATA chunks in it, with the first having the END bit set, and the last having the BEGINNING bit set. Any other combinations, and the ABORT will not trigger, and a normal SACK will be sent instead.

Hi again,
I'm wondering if you have any idea if it's possible to work around the problem - to ensure that usrsctp doesn't abort? With the packetdrill and the sequence above, I tried to always send FORWARD-TSN as the only chunk in its packet, as bundling it with a DATA chunk seemed to reproduce the issue more frequently, but when that workaround was evaluated in production, it still happened.
With the popularity of usrsctp as SCTP library in most browsers - and which will remain so for a long time, I really need to consider working around the issue even if it would be corrected.
Thanks!

Let me see if I can figure out what the bug is. Then it might be able to give a way to work around it.
I'm a bit surprised by the plan to work around the bug, even if it is fixed. I'm not familiar with the browser world, but when talking at the beginning about the usrsctp integration into browsers, the statement was that not much effort needs to go into testing, since the browser vendors can roll out bug  fixes world wide very fast. So has this changed and code stays much longer in use?

While it is possible to release emergency fixes quickly or use our experiment framework to disable codepaths, I don't think that a fix for this issue would qualify, so it would have to go through the regular pipeline which takes a few months at least between the time it takes to merge the fix to the fix being promoted to stable. Some other browsers are also only updated with OS releases, which might be postponed indefinitely by the users.
There are also a lot of users with fixed older versions (typically enterprise users) and other releases integrating Chromium (such as Electron or CEF) that might take a while to be updated. You also have a lot more users using SaaS products based on older versions of libwebrtc (and thus usrsctp) trying to communicate with all kinds of browser versions.
Hence, a workaround is probably needed in dcSCTP and usrsctp might benefit from one as well.

Thanks Michael - looking forward to your findings. Let me know if I can be of any assistance.
Adding to what @Orphis said, while Chrome and Firefox on the consumer desktop are often up-to-date, mobile browsers and also mobile apps that use usrsctp, that are not strictly browsers. also have a lower uptake of new versions. Some mobile phones simply don't have enough storage to update, or are stuck on old OS versions that the new versions of the apps are incompatible with and will never be updated. Firefox says that it takes 4 weeks to reach 70% with the new version, and I don't know if the remaining 30% are ever updated...

While it is possible to release emergency fixes quickly or use our experiment framework to disable codepaths, I don't think that a fix for this issue would qualify, so it would have to go through the regular pipeline which takes a few months at least between the time it takes to merge the fix to the fix being promoted to stable. Some other browsers are also only updated with OS releases, which might be postponed indefinitely by the users.
There are also a lot of users with fixed older versions (typically enterprise users) and other releases integrating Chromium (such as Electron or CEF) that might take a while to be updated. You also have a lot more users using SaaS products based on older versions of libwebrtc (and thus usrsctp) trying to communicate with all kinds of browser versions.
Hence, a workaround is probably needed in dcSCTP and usrsctp might benefit from one as well.

Thanks for the explanation!

Thanks Michael - looking forward to your findings. Let me know if I can be of any assistance.

Will report here by the end of this week.

Adding to what @Orphis said, while Chrome and Firefox on the consumer desktop are often up-to-date, mobile browsers and also mobile apps that use usrsctp, that are not strictly browsers. also have a lower uptake of new versions. Some mobile phones simply don't have enough storage to update, or are stuck on old OS versions that the new versions of the apps are incompatible with and will never be updated. Firefox says that it takes 4 weeks to reach 70% with the new version, and I don't know if the remaining 30% are ever updated...

Thanks for the data point.

Just for reference, and perhaps it helps during troubleshooting, this is a packetdrill that reproduced the issue with FORWARD-TSN in its own packet, not bundled with any DATA. Another difference here (from previous example), is that there is only a single fragmented message in-flight, and the FORWARD-TSN is not moved beyond it - just skipping a few unfragmented messages that was sent prior.
+0.000 `sysctl -w net.inet.sctp.pktdrop_enable=0`
+0.000 socket(..., SOCK_STREAM, IPPROTO_SCTP) = 3
+0.000 bind(3, ..., ...) = 0
+0.000 listen(3, 1) = 0
+0.000 < sctp: INIT[flgs=0, tag=1, a_rwnd=5242880, os=65535, is=65535, tsn=0,
                    FORWARD_TSN_SUPPORTED[],
                    SUPPORTED_EXTENSIONS[types=[RECONFIG, FORWARD_TSN]]]
+0.000 > sctp: INIT_ACK[flgs=0, tag=2, a_rwnd=..., os=..., is=..., tsn=10,
                        FORWARD_TSN_SUPPORTED[],
                        SUPPORTED_EXTENSIONS[types=[FORWARD_TSN, RECONFIG]],
                        STATE_COOKIE[len=..., val=...]]
+0.010 < sctp: COOKIE_ECHO[flgs=0, len=..., val=...]
+0.000 > sctp: COOKIE_ACK[flgs=0]
+0.000 accept(3, ..., ...) = 4
+0.000 close(3) = 0
+0.000 < sctp: DATA[flgs=UBE, len=100, tsn=0, sid=1, ssn=0, ppid=0]

+0.000 > sctp: SACK[flgs=0, cum_tsn=0, a_rwnd=..., gaps=[], dups=[]]

// There are a few packets lost, TSNs [1 (UBE), 2 (UBE), 3 (UB)],

+0.040 < sctp: DATA[flgs=U, len=100, tsn=4, sid=1, ssn=0, ppid=0]

+0.000 > sctp: SACK[flgs=0x00, cum_tsn=0, a_rwnd=..., gaps=[4:4], dups=[]]

+0.000 < sctp: DATA[flgs=UE, len=100, tsn=5, sid=1, ssn=0, ppid=0]

+0.000 > sctp: SACK[flgs=0x00, cum_tsn=0, a_rwnd=..., gaps=[4:5], dups=[]]

// The SACK will abandone TSN 1, as it was sent with limited retransmissions.
+0.050 < sctp: FORWARD_TSN[cum_tsn=1, ids=[]]

+0.000 > sctp: SACK[flgs=0x00, cum_tsn=1, a_rwnd=..., gaps=[3:4], dups=[]]

// Retransmits TSN 2 and 3
+0.000 < sctp: DATA[flgs=UBE, len=100, tsn=2, sid=1, ssn=0, ppid=0]
+0.000 < sctp: DATA[flgs=UB, len=100, tsn=3, sid=1, ssn=0, ppid=0]

+0.000 > sctp: ABORT[flgs=0, PROTOCOL_VIOLATION[info="Illegal message sequence, missing end for MID: 00000001"]]

+0.200 close(4) = 0

Here is a minimal (at least according to my current thinking) reproducer of the issue:
--tolerance_usecs=25000
 0.000 `kldload -n sctp`
+0.000 `sysctl -w net.inet.sctp.ecn_enable=0`
+0.000 `sysctl -w net.inet.sctp.pr_enable=1`
+0.000 `sysctl -w net.inet.sctp.asconf_enable=0`
+0.000 `sysctl -w net.inet.sctp.auth_enable=0`
+0.000 `sysctl -w net.inet.sctp.reconfig_enable=0`
+0.000 `sysctl -w net.inet.sctp.nrsack_enable=0`
+0.000 `sysctl -w net.inet.sctp.pktdrop_enable=0`
+0.000 socket(..., SOCK_STREAM, IPPROTO_SCTP) = 3
+0.000 bind(3, ..., ...) = 0
+0.000 listen(3, 1) = 0
+0.000 < sctp: INIT[flgs=0, tag=1, a_rwnd=5242880, os=65535, is=65535, tsn=0,
                    FORWARD_TSN_SUPPORTED[],
                    SUPPORTED_EXTENSIONS[types=[FORWARD_TSN]]]
+0.000 > sctp: INIT_ACK[flgs=0, tag=2, a_rwnd=..., os=..., is=..., tsn=10,
                        FORWARD_TSN_SUPPORTED[],
                        SUPPORTED_EXTENSIONS[types=[FORWARD_TSN]],
                        STATE_COOKIE[len=..., val=...]]
+0.010 < sctp: COOKIE_ECHO[flgs=0, len=..., val=...]
+0.000 > sctp: COOKIE_ACK[flgs=0]
+0.000 accept(3, ..., ...) = 4
+0.000 close(3) = 0

//    <----- DATA[UBE, TSN=0] --------
//    ------ SACK[cumTSN=0]---------->
//     |-----DATA[UBE, TSN=1] --------  To be skipped
//     |-----DATA[UBE, TSN=2] --------  Trigger the problem on retransmission
//     |-----DATA[UB, TSN=3] ---------
//    <------DATA[U,  TSN=4] ---------
//    ------ SACK[cumTSN=0,[4]]------>
//    <------FORWARD_TSN[cumTSN=1]----
//    ------ SACK[cumTSN=1,[3]]------>
//    <------DATA[UBE, TSN=2] -------- Bug!

+0.000 < sctp: DATA[flgs=UBE, len=116, tsn=0, sid=0, ssn=0, ppid=0]
+0.000 > sctp: SACK[flgs=0, cum_tsn=0, a_rwnd=..., gaps=[], dups=[]]
+0.000 recv(4, ..., 1024, 0) = 100
+0.040 < sctp: DATA[flgs=U, len=120, tsn=4, sid=0, ssn=0, ppid=0]
+0.000 > sctp: SACK[flgs=0x00, cum_tsn=0, a_rwnd=..., gaps=[4:4], dups=[]]
+0.050 < sctp: FORWARD_TSN[cum_tsn=1, ids=[]]
+0.000 > sctp: SACK[flgs=0x00, cum_tsn=1, a_rwnd=..., gaps=[3:3], dups=[]]
+0.000 < sctp: DATA[flgs=UBE, len=118, tsn=2, sid=0, ssn=0, ppid=0]
+0.000 > sctp: ABORT[flgs=0, PROTOCOL_VIOLATION[info="Illegal message sequence, missing end for MID: 00000001"]]
+0.200 close(4) = 0

This contemplating about the best way to fix this...

Thanks for the update, Michale.
I agree - it's very similar to my previous comment which was an actual capture "from the wild", but even shorter.

Hi again,
Just wondering if you have made any progress on this issue?
Thanks, and have a great weekend!

I'm interrupted by other bugs. Need to look into more detail. Planned for next week.

Thanks, Michael.

Hi Michael,
I'm just checking in to see if you have had time to look more into this.
Thanks!

Hi Michael,
I'm just checking in to see if you have had time to look more into this.

Haven't nailed down it yet. The semester just started and the IETF meeting is next week... Will look into it.
Best regards
Michael

Thanks!

@tuexen Any chance of restarting work on this?


If USERNAME is set by default, it breaks make shell.

I generally test on Arch Linux. On Arch, the variable USERNAME is not set by default. When using any of the make shell commands, you can set an optional USERNAME var to change the default username of the container.
On Fedora, I found out that USERNAME is set, by default to the same value as USER on the host, which seems non-standard to my experience, but it is probably totally normal to someone used to Fedora.
If USERNAME is set by default in the host OS, it will break the default username of the container, and it will attempt to use your system username instead of the container default one.
The fix is for you to unset the USERNAME variable beforehand,  by adding unset USERNAME to your own ~/.bashrc.
There should be a warning about this in the README and when the user first runs make config.

Fixed in #192


nothing happens after start

After executing the .exe nothing happens. I can press CTRL+D ingame to get the overlay but cannot close it or anything.
The only thing I can see is this (bottom left corner): https://imgur.com/a/4gnrjhn
It opens a 32 bit background process and 4 other poe-overlay instances.
Windows 10  pro x64 v1909

I am seeing the exact same issue on alpha 0.4 version.

Since I have a icon in the system tray where I can open the settings now with right click my issue solved I guess.


dx serve panic: Could not receive a socket - the devtools could not boot - the port is likely already in use

Problem
$ RUST_BACKTRACE=1 dx serve
2024-09-04T21:05:36.662764Z  INFO dx::builder::cargo: 🚅 Running build [Desktop] command...
2024-09-04T21:05:36.663218Z  INFO dx::builder::cargo: 🚅 Running build [Desktop] command...
thread 'main' panicked at /home/deb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dioxus-cli-0.6.0-alpha.2/src/serve/server.rs:267:21:
Could not receive a socket - the devtools could not boot - the port is likely already in use
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: dx::serve::serve_all::{{closure}}::{{closure}}
   3: dx::serve::serve_all::{{closure}}
   4: tokio::runtime::park::CachedParkThread::block_on
   5: dx::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
$ dx --version
dioxus 0.6.0-alpha.2 (was built without git repository)

Steps To Reproduce
Steps to reproduce the behavior:

Unsure; I've run into this twice now, my system somehow seems to get stuck in this state. It's worked fine before.

Expected behavior
No panic.
Environment:

Dioxus version:  0.6.0-alpha.2
Rust version:    Probably 1.80.1? Not sure what it got installed with.
OS info:         Ubuntu 20.04.6 LTS, 5.15.0-119-generic
App platform:    Fullstack

I believe rebooting my machine fixed it. It may have also been reinstalling the CLI.

Hmm, I did get this error when I had another instance of dx serve running for a different project. Is there some other fixed port that it's referencing?


Added light and dark mode toggle

Light mode dark mode toggle added to the website.

Hey please send the link also

Hey please send the link also

https://suman599.github.io/light-and-dark-mode/

Hey please send the complete PR on updated main branch as you have done earlier not just heading.

Ya i have sent the PR in updated main branch itself.

Hey please send the complete PR on updated main branch as you have done earlier not just heading.

Ok check I have sent the complete PR in the updated main branch


Order master playlist by bitrate and resolution

Order master playlist by descending bitrate or resolution if bitrate is the same.
Still need to confirm that the bandwidth is correctly sourced.
Closes #94

Will wait for https://github.com/livepeer/catalyst-api/pull/135 to land before merging.


[av1d] Add handling UMC_ERR_INVALID_PARAMS in SubmitFrame

Also, please, provide issue description you are working on. It's unclear to the audience why you are giving such PR.

Also, please, provide issue description you are working on. It's unclear to the audience why you are giving such PR.

Not all errors from driver are handled properly in this piece of code. If we ignore an error it can lead to memory corruption on msdk or sample side. This PR is intended to fix these kind of errors.

I believe we can merge in this change.


More detail for qinfer installation instructions

This should help address issue #37 by adding some more instructions for Anaconda Python, "regular"-Python, as well as pointing to using the PyPI package for QInfer instead of having to use Git.

Coverage remained the same at 39.073% when pulling bf0aa798a2fbdc91bbe15f1e244f49aabfc4437d on readme_improvements into c15497d67ed43f809d76512fcb2555f4138a2c6f on master.

Thanks, those improvements are quite useful! Merging now.


Fix: Text Visibility

Closes #94

Please consider this for KWoC`23

@AaadityaG

@AaadityaG Are you working on the issue?

Yes but I might raise pr till tomorrow, if someone does before tommorow let them do it!

@AaadityaG It's still black text, make it white.


GitHub action to auto update all links that point to cookbook-template GH repo to point to real cookbook

This looks for the cookbook-template and the repo ID number and replaces them with the new repo name and ID number. This has affects in a few files reducing the burden of the cookbook creator to track down all of these places that need to be changed. This workflow must be manually triggered and should only need to be called once per cookbook.
See this work on a dummy cookbook here: https://github.com/jukent/draft-cookbook/commit/d4c08cdd1ed3ca2589644de7086673948fcde622#diff-ecec67b0e1d7e17a83587c6d27b6baaaa133f42482b07bd3685c77f34b62d883

The title in the README.md and in CITATION.cff still need to be manually edited.

Yes that's correct. I will work on writing up the steps for both this and the Zenodo infrastructure today.


Fail fast in cytomining if no write permissions exist

Simpler than #61 but similar idea
Closes #23
(but just for cytomining workflow)

Actually the best way to do this would be to split the cytomining workflow into three tasks:

produce the SQLite via cytominer-databse ingest
optionally delocalize the SQLite to a bucket of interest
optionally perform the pycytominer aggregation step

That way, if write permission does not exist for (2), the whole workflow can be re-run, and (1) will call-cache, meaning it will not actually need to re-run.
I don't know if we want to do this right now... what do you think @carmendv ?

It still doesn't "fail-fast". Here

Yay! Working great! 👍


New Hygiene Tests - T2 Test Candidate

T2 test candidate suggests that all FIBO classes should be rooted either in FND, BE or FBC.
The query below finds all cases where a top-level class has its location in a different domain:
prefix rdf:   <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
prefix rdfs:  <http://www.w3.org/2000/01/rdf-schema#>
prefix owl:   <http://www.w3.org/2002/07/owl#>

SELECT DISTINCT ?warning ?class ?domainIdentifier
WHERE
{
    ?class a owl:Class .
    FILTER NOT EXISTS {?class rdfs:subClassOf ?classParent}
    FILTER (CONTAINS(str(?class), "edmcouncil"))
    BIND (STRBEFORE(STRAFTER(str(?class),"https://spec.edmcouncil.org/fibo/ontology/"),"/") As ?domainIdentifier)
    FILTER (?domainIdentifier NOT IN ('FND', 'BE', 'FBC'))
    BIND (concat ("WARN: Class ", str(?class), " is a top-level class outside FIBO foundations.") AS ?warning)
}
ORDER BY ?domainIdentifier

In DEV the query found 50+ such disconnected classes: disconnected_classes_in_fibo_dev.xlsx
I would propose adding this as a new hygiene test or dropping its description from the ontology guide.
We could also have a similar query for objects properties - although the guide does not seem to consider this option.

@merelog - Ultimately I would like every class to be rooted in FND (or LCC), but we are not quite there yet.  In Prod there are 4 classes rooted in FBC, none in BE, so maybe test for LCC, FND, and FBC at the moment.  The 4 classes rooted in FBC could be moved to FND, but they are widely used so I'll have to bring it up in the FBC/SEC FCT to see what kind of reaction I get to that before we do so.  They are fairly fundamental - Principal, Exposure, Offering, and most importantly, Account.  If we move them, we should do so before we pull the trigger on the OMG FIBO specification.  I think moving Principal, Exposure, and Offering to FND would be straightforward since there are not a lot of dependencies on them.  Account is more of a challenge.
For properties it will take more work to get there.  We have one in DER, three in IND, and quite a few in SEC.  Many but not all of these should be subproperties of something in FND, BE, or FBC, though, and there are a gazillion in Dev that are disconnected.

@merelog - Ultimately I would like every class to be rooted in FND (or LCC), but we are not quite there yet.  In Prod there are 4 classes rooted in FBC, none in BE, so maybe test for LCC, FND, and FBC at the moment.  The 4 classes rooted in FBC could be moved to FND, but they are widely used so I'll have to bring it up in the FBC/SEC FCT to see what kind of reaction I get to that before we do so.  They are fairly fundamental - Principal, Exposure, Offering, and most importantly, Account.  If we move them, we should do so before we pull the trigger on the OMG FIBO specification.  I think moving Principal, Exposure, and Offering to FND would be straightforward since there are not a lot of dependencies on them.  Account is more of a challenge.
For properties it will take more work to get there.  We have one in DER, three in IND, and quite a few in SEC.  Many but not all of these should be subproperties of something in FND, BE, or FBC, though, and there are a gazillion in Dev that are disconnected.


(New sheet) Macchiato Monsters

Character sheet for Macchiato
Monsters,
the Black and White Hack. Macchiato Monsters is a rules-light, old
school, slightly collaborative roleplaying game. Brewing two hacks to
get the perfect cup of DIY dungeonverse! Macchiato Monsters features
simple roll-under resolution, omnipresent risk dice, streamlined combat,
freeform characters, and no spell list.

First, thank you for all of the dual language sheets you've made. They are amazing. One quick tweak, that will save me some time. For the 2 language files, instead can you do:
/translations/en.json => translation.json (make the en.json file the default translation.json in root of the game folder)
/translations/fr.json (this is perfect, keep it right where it is)

@pjbehr87 Oh, I didn't realize that! Dully noted.
Thank you.


DOC: update README.md and theme config

Updates README.md to reflect ERAD2024 content.
closes #1

Going to merge this. The webpage will be better reviewed if the things in this PR are in place. We can iterate later.


Clarify how page faults are handled on vector table reads

Not clear how to recover if EPC written with table entry location.

One possibility is that we don't write anything except xcause[30] on a page fault (or other exception on the table lookup).  Xcause[30] says there's a fault on the table lookup.  From that, and the interrupt number in xcause, software can determine the location of the fault as well as how to restart.
Another possibility is that xtval contains the location of the fault (like it would for other faults).
Using EPC seems odd as well as seeming like recovery isn't possible.  If it does work, some non-normative text is needed to say why recovery is possible and why it makes sense to use EPC.

In general this is a difficult problem - we're all building RISC cores, and this is a CISCish sort thing - as an example in my (highly speculative) core an interrupt forces a 'trap' uop into the pipeline - that uop has the future EPC - and for most interrupts acks the highest pending interrupt and then traps - but in the vectored case that uop when executed has to become 2 more uops - a load followed by a branch - the load might in turn trigger a page fault at the supervisor and/or hypervisor level - if that happens you want to leave the CPU in a state when returning from that page fault restarts the interrupt processing again
Unlike most almost every other case in a RISCy core these have to involve hidden state across uops, the CPU can't ack the interrupt with the CLIC until the 'branch' is executed (otherwise the interrupt can't be restarted after the page fault is resolved), equally the interrupt state has to be frozen at the start of the process to make sure you're acking the same interrupt you fetched the vector for .... lots of potential evil error cases (hidden state is bad)
I don't actually expect a supervisor interrupt vector fault to be able to be usefully handled with a supervisor page fault (how are you going to do disk IO to pull in a page containing the vectors if you can't take interrupts because you don't have vectors?). What's more useful (and important) is being able to handle user mode interrupts in supervisor mode or supervisor interrupts in hypervisor mode.
However there are some somewhat spurious cases that should be avoided, in particular systems software people should be cautioned to set the ACCESSED bit in every PTE traversed when fetching an interrupt vector .....

A horizontal trap on vector table access (e.g., interrupt taken in M-mode with table not accessible to M-mode) is a fatal error, for which recovery is not expected (similar to avoiding exceptions in the early part of any ISR).  Vertical traps to a higher-privilege mode (e.g., from S-mode interrupt handler to HS-mode fault handler) write hsepc, preserving sepc, so the S-mode handler can be restarted after table is paged in for example.
This explanatory text needs to be added to spec.
Separately, there is a concern about why xepc is written and not mtval (#111).

Closed by #131


Thank you for providing such an excellent project. May I ask how to test my dataset

Hi,
You can organize the record video sequence as the data structure of Replica or Scannet, and run our slam in monocular, rgbd, or stereo mode.
The simplest way, for example in monocular mode, is to extract frames from the video sequence and put them in a folder in order, and also, you need to know the intrinsic.  For the bounding box, you can set a quite large value, such as [[-10, 10], [-10, 10], [-10, 10]].

Best,
Youmin

Hi,
Yes, for the pose, it's used only for evaluation. You can just set this flag to False and set all poses to the identity matrix.
Best,
Youmin

Hello, may I ask you to solve the problem of positioning error in running custom data? Could you tell me how to solve it?

Hi,
You can organize the record video sequence as the data structure of Replica or Scannet, and run our slam in monocular, rgbd, or stereo mode.
The simplest way, for example in monocular mode, is to extract frames from the video sequence and put them in a folder in order, and also, you need to know the intrinsic. For the bounding box, you can set a quite large value, such as [[-10, 10], [-10, 10], [-10, 10]].
As our slam reports the real bounding box value of the scene when running, you will get a more accurate result if you change the bounding box accordingly and give a second run.
Best, Youmin

Hi，Youmin
I have a set of image sequences as my dataset, how can I obtain the corresponding mesh file (. ply) of my dataset like the replica dataset？
Looking forward to your reply！


Fixed kdcHost set to None

When requesting for the ticket, the sendReceive() function was called with the kdcHost argument set to None, preventing the script to work when not having DNS resolution set properly on the host.
python3 getnthash.py -key "2ddb [...] 9969" -dc-ip "<IP_ADDRESS>" "domain.local/user2"
Impacket v0.9.24.dev1+20210706.140217.6da655ca - Copyright 2021 SecureAuth Corporation

[*] Using TGT from cache
[*] Requesting ticket to self with PAC
[-] [Errno Connection error (DOMAIN.LOCAL:88)] [Errno -2] Name or service not known

good catch, thanks!


adding project version description

Coverage decreased (-0.07%) to 14.172% when pulling d5d42ee1ec07201d0194958e67e9dc4738f703d4 on project-version-description into f872b5ac4ad78ca2afadf53dd1132a822d1d78e1 on master.


Do not default to LTR for empty editors

Do you want to request a feature or report a bug?
Reporting a bug, I guess.
What is the current behavior?
On an empty Draft.js field, the first block has a public-DraftStyleDefault-ltr class and the caret is on the left side. This is before the first character is typed. This default does not make sense for an RTL environment, where everything else is RTL except for the caret.
What is the expected behavior?
Not setting a direction when no content is available. It would default to the parent direction, which is a sensible default.
Which versions of Draft.js, and which browser / OS are affected by this issue? Did this work in previous versions of Draft.js?
Draft 0.8.1, Chrome 52 and Safari 9.1.3, OS X 10.11.16. Didn’t check the older versions.
I’d be happy to work on a PR if you agree with the suggestion.

Hey! This would be awesome. Feel free to pop in a PR @mostafah

It seems like fbjs has a default direction of LTR. This default direction can not be neutral, must be either LTR or RTL.
I can fix it in my app by setting the global direction using UnicodeBidiDirection.setGlobalDir(UnicodeBidiDirection.RTL), but since importing fbjs is not generally a good idea, I don’t think this is a proper solution.
What is the proper solution for this? Automatically setting global direction based on the parent or document direction, or exposing a function for setting the global direction?

On a second thought, both of the above solutions have a global side effect, which is not good. What about having a defaultDirection prop on Editor component? That can then be sent as a defaultDir to UnicodeBidiService.

Found a problem with that too!
It’s EditorState that creates and maintains direction map, not the Editor component. EditorState creates direction map in its create function. In other words, the initial direction map is calculated before the component is even created. So, having default direction as a prop does not work. (Unless we want to force a recalculation, but we don’t.)
So, it’s EditorState that can have a default direction property. Does that make sense to you?
PS: Sorry for the noise. I was not familiar with the internal design and code of Draft and am trying to find a proper design and implementation.

We need to manually control RTL as well. Is that what you are ultimately looking for @mostafah? If so, I won't create a new issue. I think this would be a property that would be set on the EditorState during initial state creation. Then it could be used to override EditorBidiService.getDirectionMap if set.

Yes, that’s what I’m after here. My original idea was to somehow remove the default direction, but after looking into the code I found out that that’s not possible without big changes to fbjs and Draft.js.
Since this would add a new property to EditorState I was waiting for a confirmation from the team, but I might as well go ahead and send a PR anyway.


Memory problems that occur when the wasm class is destroyed

When the emscripten class over wasm is no longer needed, you need to call the delete method as recommended in the documentation to free up occupied memory and prevent memory leaks.
https://emscripten.org/docs/porting/connecting_cpp_and_javascript/embind.html#memory-management
Memory errors often occur when calling the delete method, here are a few error stacks:
RuntimeError: memory access out of bounds
    at thorvg-wasm.wasm.tvg::Scene::Impl::clear(bool) (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[619]:0x199d5)
    at thorvg-wasm.wasm.tvg::Scene::clear(bool) (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[618]:0x1992e)
    at thorvg-wasm.wasm.LottieBuilder::update(LottieComposition*, float) (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[1785]:0x65dfb)
    at thorvg-wasm.wasm.LottieLoader::run(unsigned int) (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[1933]:0x6ec0f)
    at thorvg-wasm.wasm.non-virtual thunk to LottieLoader::run(unsigned int) (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[1952]:0x6f56e)
    at thorvg-wasm.wasm.tvg::TaskSchedulerImpl::request(tvg::Task*) (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[728]:0x1d348)
    at thorvg-wasm.wasm.tvg::TaskScheduler::request(tvg::Task*) (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[727]:0x1d2e4)
    at thorvg-wasm.wasm.LottieLoader::frame(float) (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[1970]:0x704f6)
    at thorvg-wasm.wasm.tvg::Animation::frame(float) (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[135]:0x8d50)
    at thorvg-wasm.wasm.TvgLottieAnimation::frame(float) (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[3197]:0xcce0e)

RuntimeError: null function or function signature mismatch
    at thorvg-wasm.wasm.tvg::Picture::Impl::~Impl() (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[554]:0x17918)
    at thorvg-wasm.wasm.tvg::Picture::~Picture() (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[125]:0x899f)
    at thorvg-wasm.wasm.tvg::Picture::~Picture().1 (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[553]:0x17851)
    at thorvg-wasm.wasm.tvg::Canvas::Impl::clearPaints() (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[194]:0xae82)
    at thorvg-wasm.wasm.tvg::Canvas::Impl::~Impl() (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[251]:0xc116)
    at thorvg-wasm.wasm.tvg::Canvas::~Canvas().1 (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[161]:0x967f)
    at thorvg-wasm.wasm.tvg::SwCanvas::~SwCanvas().1 (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[711]:0x1cb5d)
    at thorvg-wasm.wasm.std::__2::default_delete<tvg::SwCanvas>::operator()[abi:un170004](tvg::SwCanvas*) const (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[3154]:0xcb598)
    at thorvg-wasm.wasm.std::__2::unique_ptr<tvg::SwCanvas, std::__2::default_delete<tvg::SwCanvas>>::reset[abi:un170004](tvg::SwCanvas*) (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[3151]:0xcb48a)
    at thorvg-wasm.wasm.std::__2::unique_ptr<tvg::SwCanvas, std::__2::default_delete<tvg::SwCanvas>>::~unique_ptr[abi:un170004]() (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[3149]:0xcb3a4)

RuntimeError: Aborted(native code called abort())
    at abort (https://manager_index_worker_js.js:489:51)
    at _abort (https://ex_worker_js.js:5475:13)
    at thorvg-wasm.wasm.dlfree (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[59]:0x4a5b)
    at thorvg-wasm.wasm.operator delete(void*) (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[126]:0x89d4)
    at thorvg-wasm.wasm.tvg::SwRenderer::~SwRenderer() (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[985]:0x3575d)
    at thorvg-wasm.wasm.tvg::SwRenderer::~SwRenderer().1 (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[995]:0x35b23)
    at thorvg-wasm.wasm.tvg::Canvas::Impl::~Impl() (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[251]:0xc14e)
    at thorvg-wasm.wasm.tvg::Canvas::~Canvas().1 (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[161]:0x967f)
    at thorvg-wasm.wasm.tvg::SwCanvas::~SwCanvas().1 (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[711]:0x1cb5d)
    at thorvg-wasm.wasm.std::__2::default_delete<tvg::SwCanvas>::operator()[abi:un170004](tvg::SwCanvas*) const (wasm://wasm/thorvg-wasm.wasm-013439fa:wasm-function[3154]:0xcb598)

The order of calling methods in JS:
const thorvg = new TvgLottieAnimation();

thorvg.load();
thorvg.totalFrame();
thorvg.resize();
thorvg.frame();
thorvg.update();
thorvg.render();
thorvg.delete();

For each new lottie animation, a new instance of the TvgLottieAnimation class is created, after all animation frames have been rendered, the class is deleted using the delete method:
const createAnimation = () => {
   const thorvg = new TvgLottieAnimation();

   thorvg.load();
   ...

   return () => thorvg.delete(); // destroy
};

createAnimation();
createAnimation();
...

Hello, @MrZillaGold.
Thanks for sharing the memory management.
We'll consider the explicit delete in official js player as emscripten strongly recommends it.
However I tried to call thorvg.delete() in our JS side, seemed to delete well, and there was no any runtime error.

So I wonder which code base you tried specifically. If it's logical error in your js binding side, you can review our js code with delete().

The problem occurs after several consecutive calls to emscripten methods, as I described above. The problem occurs randomly, I guess it depends on the number of calls to methods in total.
I use my own player, not the version from thorvg.

@tinyjin hey!
https://github.com/MrZillaGold/thorvg-bug
I have created a minimal example with a reproduction of the problem, to run it you need to run:

yarn
yarn dev

after opening the browser tab on http://localhost:3000/ after 3-4 seconds, the players start drawing empty frames, at which point errors appear in the console and the CPU load on the browser tab increases to 100%.

https://github.com/thorvg/thorvg/assets/42776347/94a7d1d8-2eea-4a1b-b90e-08bb2e070dd4

Hello, @MrZillaGold
We also found that bug from calling ThorVG binding function after wasm has been deleted.
Will check the example and CPU load as well soon.
Thank you.

I've found memory leak between deletion and creation of TvgLottieAnimation.
It leaks 96bytes in total for each recreation with given example, despite program called destructor and delete() function.
The out of bounds is also caused because the stack is going to be full. (Also that's the why it caused randomly)
The error could be delayed if we give Emscripten a parameter like -sSTACK_SIZE=10MB, but this isn't valid way to solve it.
Still unidentified issue, I'm not sure which part is exactly the point, will take a look into them.

SUMMARY: AddressSanitizer: 96 byte(s) leaked in 12 allocation(s).

@tinyjin I think you can firstly identify the constructor/destructor pair calls for TvgLottieAnimation / Animation and SwCanvas by placing a TVG log message in each of them.

The memory leak I found seems to come from EMSCRIPTEN_BINDING.
I've tested an example with empty function bindings.
#include <emscripten/bind.h>
#include <sanitizer/lsan_interface.h>

using namespace emscripten;
using namespace std;

static const char* NoError = "None";

class __attribute__((visibility("default"))) TvgLottieAnimation
{
public:
    ~TvgLottieAnimation()
    {
    }

    static unique_ptr<TvgLottieAnimation> create()
    {
        return unique_ptr<TvgLottieAnimation>(new TvgLottieAnimation());
    }

    void render()
    {
    }

    bool update()
    {
        return true;
    }

    bool frame(float no)
    {
        return true;
    }

    void resize(int width, int height)
    {
    }

    TvgLottieAnimation()
    {
    }
};

EMSCRIPTEN_BINDINGS(thorvg_bindings)
{
    class_<TvgLottieAnimation>("TvgLottieAnimation")
        .constructor(&TvgLottieAnimation ::create) // No leak
        .function("render", &TvgLottieAnimation::render) // Leak 8 byte
        .function("update", &TvgLottieAnimation ::update) // Leak 8 byte
        .function("frame", &TvgLottieAnimation ::frame) // Leak 8 byte
        .function("resize", &TvgLottieAnimation ::resize); // Leak 8 byte

    emscripten::function("doLeakCheck", &__lsan_do_recoverable_leak_check); // No leak
}

In JS, I create the module and delete immediately.
const tvg = new thorvg.TvgLottieAnimation();
tvg.delete();

// Check leak
thorvg.doLeakCheck();

Although I didn't import any thorvg API and I just create them and delete, the sanitizer detects memory leak.
The binding seems to have 8 bytes leak per each class function.
I assume those leakings are from emscripten class function binding in unknown reason.
I'm going to take a look at similar issue on emscripten side and find workaround for this.

I've noticed that those leaks are not actual memory issue.
Although this somehow indicates a direct leak, it seems not to affect the WASM memory area.
see: https://github.com/emscripten-core/emscripten/issues/21709#issuecomment-2056799457
In addition, I've luckily found that v0.13.0 doesn't have same issue at all.
According to the my looking, the https://github.com/thorvg/thorvg/commit/efc7199e514cb40bfb1cff489b3043d363646738 eliminates the memory increment in calling delete().
Hence, I want to check v0.13.0 by your side as well. @MrZillaGold
Could you please give them a try and check whether it still happens?

The problems in the test example that was attached above stopped being reproduced using the wasm build from the master branch. It seems the problem has been solved. We will additionally check the new version in the production environment, I will return with information later. Thanks!

Appreciate for that. I'll be still investigating for:

How the commit eventually stops this issue.
Way to avoid the fake memory leaks in Meson.

Once you can test, we can close this issue.
For sure, take your time on testing and let me know :)

@tinyjin
We have tested the new version in the production environment and everything works without problems, thanks again!


Style guide: provide links into framework documentation

There are a couple of places in the style guide that I'm not quite sure what the recommendation means, and it would be really helpful to link back to the appropriate place in the main API documentation (when appropriate).
For example, keyed v-for is marked as essential, but I wasn't sure what the :key attrib actually did. However the API entry for key was pretty clear.
So a general style-guide for the style-guide would be to link to the API or guide docs wherever possible!

Sounds great! I've already been trying to do this, but will make a dedicated pass to add more of these wherever I see a specific API referenced. 🙂

Closing old ticket


Testnet 6.6

Staging for testnet 6.6.

Codecov Report

Merging #247 (9612665) into develop (33a42ab) will decrease coverage by 0.10%.
The diff coverage is 0.00%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
@@             Coverage Diff             @@
##           develop     #247      +/-   ##
===========================================
- Coverage    33.54%   33.45%   -0.10%
===========================================
  Files           24       24
  Lines         8216     8239      +23
===========================================
  Hits          2756     2756
- Misses        5460     5483      +23

Files
Coverage Δ

src/bin/nomic.rs
0.00% <ø> (ø)

src/app.rs
3.98% <0.00%> (ø)

src/app/migrations.rs
0.00% <0.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


bug: Transliteration fails with exception for some strings

Description
Transliteration raises an IndexError exception.
Expected results
String is transliterated.
Current results
Exception is raised:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.11/site-packages/pythainlp/transliterate/core.py", line 59, in romanize
    return romanize(text)
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pythainlp/transliterate/royin.py", line 216, in romanize
    romanized_words = [_romanize(word) for word in words]
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pythainlp/transliterate/royin.py", line 216, in <listcomp>
    romanized_words = [_romanize(word) for word in words]
                       ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pythainlp/transliterate/royin.py", line 200, in _romanize
    word = _replace_consonants(word, consonants)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pythainlp/transliterate/royin.py", line 184, in _replace_consonants
    mod_chars.append(_CONSONANTS[consonants[j]][1])
                                 ~~~~~~~~~~^^^
IndexError: list index out of range

Steps to reproduce
Try to transliterate string "โรงเรียนฤทธิยะวรรณาลัย":
$ docker run --rm -it --entrypoint /bin/sh python:3.11
# pip install pythainlp==3.1.1
# python
>>> from pythainlp.transliterate import romanize
>>> romanize('โรงเรียนฤทธิยะวรรณาลัย')

PyThaiNLP version
3.1.1
Python version
3.11
Operating system and version
Debian 11
More info
No response
Possible solution
No response
Files
No response

@roshats Thank you for reporting! I checked the bug. The royin engine (default engine of romanize function) of romanize doesn't supporting word that have many syllables. I recommend you should use thai2rom engine instead royin engine.
romanize('โรงเรียนฤทธิยะวรรณาลัย',engine="thai2rom")
# output: 'rongrianritthiyawannalai'

You can read the pythainlp.transliterate documentation at https://pythainlp.github.io/docs/3.1/api/transliterate.html.

I will close the issues. If you has new update, you can reopen this issues.


Move retrieval of track path data to a Cloudflare function.

Right now the track data is hard coded into the front end.  Consider moving it to a Cloudflare function and pulling it in on page load.

Decided not to do this as the track paths are pretty static, so they can stay in the front end app.js code.


Update hdinsight-hadoop-use-hive-ssh.md

Changed "Putty" to "PuTTY."

Hi @ShawnJackson, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

It looks like you're working at Microsoft (v-shawja). If you're full-time, we DON'T require a contribution license agreement.

If you are a vendor, or work for Microsoft Open Technologies, DO please sign the electronic contribution license agreement. It will take 2 minutes and there's no faxing! https://cla.azure.com.

TTYL, AZPRBOT;


Conventional 0-255 colors values doesn't accept floats

Rounding the inputs for RGB as conventionally 0-255 doesn't take floats.
Closes https://github.com/mattrobenolt/colors.py/issues/8

Is there some reference that says this is strictly true? I feel like typically they are integers, but not sure why a float wouldn’t be allowed.

If it wasn't strictly true and we did included decimals it would be a lot more than 16777216 colors (24-bit). Which is why I thought it made sense :)
Also most (all I tried) software's doesn't accept floats for 0-255 color values.

Oh, I guess that makes a lot of logical sense, you'd be increasing the number of colors.
I guess from my perspective, I was trying to maintain lossless conversions, but technically you can't do that.
I wonder if it makes sense to store as a floating point internally, but use the r/g/b values as integers for output only.

I wonder if it makes sense to store as a floating point internally, but use the r/g/b values as integers for output only.

I think this would make a lot of sense. Wouldn't make it as "lossy" to convert to RGB.

But then data loss would be expected if you go from 32 bit -> 16 bit -> 32 bit


What caused the spike in wellcomelibrary users on 6/10/21?

What is the problem to solve?
What caused this spike?

It appears to originate from this article published 5/10/21:
https://www.iflscience.com/health-and-medicine/james-morrison-the-doctor-who-made-his-patients-poop-themselves-to-death/
which contained a link to https://catalogue.wellcomelibrary.org/search~S12?/YMorison{u2019}s+pills&searchscope=12&SORT=D/YMorison{u2019}s+pills&searchscope=12&SORT=D&SUBKEY=Morison’s+pills/1%2C24%2C24%2CB/frameset&FF=YMorison{u2019}s+pills&searchscope=12&SORT=D&2%2C2%2C

996 users who came to wl.org from this article 5-10/10/21. The bounce rate was 85% but excluding these bounces, the average number of pages viewed per user was 1.38.
IFL Science is not a consistent referrer to wellcomelibrary.org (1,104 users have been referred from this site since 1/1/21)

There are also links to wellcomecollection.org in this article:
https://wellcomecollection.org/images?query=t78chphk
https://wellcomecollection.org/works/andrfpy6?wellcomeImagesUrl=/indexplus/image/V0011125.html
However, the level of users referred to wellcomecollection.org from this site is relatively low on 6/10/21 and since 1/1/21:


[alertmanager] support to auto set replicas by node num

What this PR does / why we need it
Which issue this PR fixes
(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

fixes #

Special notes for your reviewer
Checklist

[ ] DCO signed
[ ] Chart Version bumped
[ ] Title of the PR starts with chart name (e.g. [prometheus-couchdb-exporter])


added snapchat "add user" URI

this would work perfectly :+1:

awesome, let's wait for the verdict, and then :arrow_forward: :arrow_backward:


inconsistency in Section <IP_ADDRESS>

the title of the section says "Example: local level model" however what is described in equations (29.76), (29.77) and (29.78) is a local linear trend model.

fixed


fix(perlpls): turn on syntax checking by default

Turns on a newly documented feature. Brings to parity with vscode defaults.


[Snyk] Security upgrade pillow from 9.5.0 to 10.0.1

User description

Snyk has created this PR to fix 1 vulnerabilities in the pip dependencies of this project.
Snyk changed the following file(s):

examples/speed_estimation/requirements.txt

⚠ Warning
data-gradients 0.3.2 requires scipy, which is not installed.
scikit-learn 1.0.2 requires scipy, which is not installed.
super-gradients 3.5.0 requires scipy, which is not installed.
xhtml2pdf 0.2.11 requires python-bidi, which is not installed.
botocore 1.33.13 has requirement urllib3<1.27,>=1.25.4; python_version < "3.10", but you have urllib3 2.0.7.
onnxruntime 1.13.1 has requirement numpy>=1.21.6, but you have numpy 1.21.3.
pyhanko 0.20.1 has requirement pyhanko-certvalidator==0.24.*, but you have pyhanko-certvalidator 0.26.5.
sphinx-rtd-theme 2.0.0 has requirement sphinx<8,>=5, but you have sphinx 4.0.3.
tensorboard 2.11.2 has requirement setuptools>=41.0.0, but you have setuptools 40.5.0.

[!IMPORTANT]

Check the changes in this PR to ensure they won't cause issues with your project.
Max score is 1000. Note that the real score may have changed since the PR was raised.
This PR was automatically created by Snyk using the credentials of a real user.
Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
📜 Customise PR templates
🛠 Adjust project settings
📚 Read about Snyk's upgrade logic

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Learn about vulnerability in an interactive lesson of Snyk Learn.

PR Type
enhancement, dependencies

Description

Updated the examples/speed_estimation/requirements.txt file to pin the pillow package to version 10.0.1 or higher, addressing a known security vulnerability.
This change was made to reduce vulnerabilities in the project's dependencies as identified by Snyk.

Changes walkthrough 📝
Relevant filesDependencies

requirements.txtPin `pillow` dependency to mitigate security vulnerability

examples/speed_estimation/requirements.txt
Added a version constraint for the pillow package.  Pinned pillow to version 10.0.1 or higher to address a security vulnerability.

+1/-0

💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information

:tada: Snyk checks have passed. No issues have been found so far.
:white_check_mark: security/snyk check is complete. No issues have been found. (View Details)


Interface ARIMA from pmdarima/statsmodels

Describe the solution you'd like
We already have AutoARIMA, but it would be nice to also interface ARIMA, either from statsmodels or pmdarima.

@mloning what exactly needs to be implemented here?

Hi @MixKlim take a look at the AutoARIMA wrapper, we'd like to have something similar for the simple ARIMA from pmdarima.

I'd like to work on this issue.

Hi @MixKlim take a look at the AutoARIMA wrapper, we'd like to have something similar for the simple ARIMA from pmdarima.

Hi @mloning , would this mean that we'll be interfacing from pmdarima instead of statsmodels?

Hey @HYang1996, pmdarima itself interfaces statsmodels, so whatever is easier. We already have a wrapper for statsmodels, so perhaps that's more straightforward.


Error page when designing guides

I wanted to alert you to a problem with the GUIDES tool. I input the gene "HPRT1" with all of the default settings, and was taken to this page:
http://guides.sanjanalab.org/#/designer/87e65e3e-4bd5-4699-917c-2df04b1f20c4
This error occurred in both Chrome and Safari, and also occurred when I tried a few other genes. Occasionally, the tool didn't reach that error page at all, but just hung on a blank screen indefinitely.

Hi @ruth-hanna, I tried running GUIDES with "HPRT1" with the default settings and don't get any errors:
http://guides.sanjanalab.org/#/designer/23663850-c7ed-4124-b77a-f3590b879db0
It's possible that you ran this while GUIDES was down for maintenance. Please let us know if there's anything we can help with!


Does the peek() function provide any timeout ?

Imagine following scenario:
m += a1 >> c["in1"];
m += a2 >> c["in2"];
c loops ports in1 and in2 to consume the corresponding messages from a1 and a2. Now imagine producer a1 does not produce anything. Even though a1 delivers raft::proceed, c gets stuck waiting for a message from it.
I searched in the documentation but could not find any timeout flag for peek().
How would you recommend proceeding ?
Here a little poc:
/**
 *
 * Proof of concept Raftlib
 *
 * Want to have 1 consumer and 2 producers. The consumer should timeout and continue with next
 * producer if first producer does not produce..
 *
 */
#include <raft>
#include <raftio>

struct big_t
{
    int i;
    std::uintptr_t start;
#ifdef ZEROCPY
    char padding[ 100 ];
#else
    char padding[ 32 ];
#endif
};

/**
 * Producer: sends down the stream numbers from 1 to 10
 */
class A1 : public raft::kernel
{
private:
    int i   = 0;
    int cnt = 0;

public:
    A1() : raft::kernel()
    {
        output.addPort< big_t >("out");
    }

    virtual raft::kstatus run()
    {
        i++;

        if ( i <= 10 )
        {
            auto &c( output["out"].allocate< big_t >() );
            c.i = i;
            c.start = reinterpret_cast< std::uintptr_t >( &(c.i) );
            output["out"].send();
        }
        else
        {
            return (raft::stop);
        }

        return (raft::proceed);
    };
};

class A2 : public raft::kernel
{
private:
    int i   = 0;
    int cnt = 0;

public:
    A2() : raft::kernel()
    {
        output.addPort< big_t >("out");
    }

    virtual raft::kstatus run()
    {
        i++;

        if ( i <= 10 )
        {
            NULL; //
            sleep(1);
        }
        else
        {
            return (raft::stop);
        }

        return (raft::proceed);
    };
};

/**
 * Consumer: takes the number from input and dumps it to the console
 */
class C : public raft::kernel
{
private:
    int cnt = 0;
public:
    C() : raft::kernel()
    {
        input.addPort< big_t >("in1");
        input.addPort< big_t >("in2");
    }

    virtual raft::kstatus run()
    {
        if (cnt % 2 == 0)
        {
            try {
                std::cout << "in1: " << cnt << "\n";
                auto &a( input[ "in1" ].peek< big_t >() );
                std::cout << std::dec << a.i << " - " << std::hex << a.start << " - " << std::hex <<
                    reinterpret_cast< std::uintptr_t >( &a.i ) << "\n";
                input[ "in1" ].recycle(1);
            }
            catch(ClosedPortAccessException& cpae)
            {
                NULL; // continue
            }
        }
        else
        {
            try {
                std::cout << "in2: " << cnt << "\n";
                auto &a( input[ "in2" ].peek< big_t >() );
                std::cout << "yeap\n";
                std::cout << std::dec << a.i << " - " << std::hex << a.start << " - " << std::hex <<
                    reinterpret_cast< std::uintptr_t >( &a.i ) << "\n";
                input[ "in2" ].recycle(1);
            }
            catch(ClosedPortAccessException& cpae)
            {
                NULL; // continue
            }
        }
        ++cnt;
        return (raft::proceed);
    }
};

int main()
{
    A1 a1;
    A2 a2;
    C c;

    raft::map m;

    m += a1 >> c["in1"];
    m += a2 >> c["in2"];

    m.exe();

    return( EXIT_SUCCESS );
}

The first thing that is apparent is that RaftLib is in need of a  raft::sleep that corresponds to the underlying runtime implementation. For instance, if I'm using qthreads, I  definitely don't want to call sleep as it would sleep the entire kernel visible thread (with my own stack swapping  routine the same). So, that's the first thing. That abstraction needs to be provided. Likely with some form of the  std::duration spec for time vs. the clock_t or some other struct.
RaftLib currently doesn't provide a timeout for the peek function.  I could easily add one, however, lets see if the current API  supports what you need. Looking at the structure of C I see that you've caught the closed port exception that is called once the producer attached to in1 shuts down. That's the right thing to do. There's also a size() function for each port that will return the number of elements currently on that input  stream. If it is zero then you could skip the call. That will also not trigger the closed port exception, which could be  faster.
The peek() implementation itself will call the runtime  specific yield for this kernel if no data is available, so in that respect, it's not wasting compute cycles spinning  on a slow producer. The issue as you point out, is that the consumer C would spin waiting on in1 vs going to in2.  A fix using the current API would be by checking for size() of the stream before calling the peek so that the number of  items could be found (I've added a peek timeout example below if it would be more useful/intuitive than size() ).
I can see a timeout being  better for some code (cleaner, easier to fit the paradigm). I propose having a function call variant that mimics the  current peek() but has two parameters, one a reference to a flag, and two a std::chrono::duration of time.
using peek_timeout_t = std::chrono< /** some templates go here **/ >;
/** current definition **/
template< class T,
             typename std::enable_if< inline_alloc< T >::value >::type* = nullptr >
   T& peek( raft::signal *signal = nullptr );

/**
 * NOTE: there would of course be three of these, one for
 * each allocation type, perhaps more.
 */
template< class T,
             typename std::enable_if< inline_alloc< T >::value >::type* = nullptr >
   T& peek( bool &timeout, const peek_timeout_t &&timeout, raft::signal *signal = nullptr );

To modify the given example for the size() function I mentioned,  it would look something like this (also, I'm now entertaining the  idea of a simple dataAvail function given that it would only take two ops to determine if data is avail).
auto &port_in1( input[ "in1" ] );
if( port_in1.size() > 0 )
{
std::cout << "in1: " << cnt << "\n";
                auto &a( port_in1.peek< big_t >() );
                std::cout << std::dec << a.i << " - " << std::hex << a.start << " - " << std::hex <<
                    reinterpret_cast< std::uintptr_t >( &a.i ) << "\n";
                port_in1.recycle(1);

} /** end size check **/

Hi Jonathan,
Thanks for the reply. Firstly the raft::sleep would be something really good to have, I was wondering how much the sleep() system call was affecting the streaming flow.
Secondly particularly in my case the size() option is exactly what I need, it works like a charm. I don't think I would need a function with a timeout. Having the size() method referenced in the documentation would have been enough for me.
For my question I consider this matter close. Not sure if you want to keep it open for the raft::sleep idea?
Thanks again..

Will update the documentation. closing issue.


Log correct build job counts

Sorry, didn't mean to disable auto merge. Tapped the wrong button in the app!


Executable bit not set for fsc.exe and fsi.exe in FSharp.Compiler.Tools package

Ideally the executable bit should be set for .EXEs in the tools directory of nuget packages so they can be run directly on Mono.  I don't know how to make this happen

@forki Any idea how to do this?

Urgs not sure if paket/NuGet could do it. We need real mono experts.
/cc @7sharp9 @nosami @dnauck

Yeah, this search doesn't reveal much

/cc @emgarten @yishaigalatzer

I normally just use chmod if I need to set executable bits /shrug
On 5 November 2016 at 17:35, Steffen Forkmann<EMAIL_ADDRESS>wrote:

/cc @emgarten https://github.com/emgarten @yishaigalatzer
https://github.com/yishaigalatzer
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fsharp/fsharp/issues/638#issuecomment-258628559, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAj7yruQOztPP5c43vR93seUM_jrqDG5ks5q7L51gaJpZM4KqUvK
.

Yes, that's the workaround here: https://github.com/dsyme/visualfsharp/blob/ae5b3ec722b64b9d44a28050d128876bfb259aa3/before_install.sh#L18
But it's wrong :)

Cc @migueldeicaza

@yishaigalatzer I suspect that this is a nuget issue.  Is it feasible for nuget to just set the executable bit on all .EXEs when it unzips?

We could look at that, but I'd like to hear feedback from the mono guys first

We are already "chmoding" some stuff in the shared lock folder path

Yes, this is similar to the (still open) issue in NuGet where restoring from a different user on Linux fails:
https://github.com/NuGet/Home/issues/3897
The workaround is to manually chmod the file or directory in question. In this particular case, I would:
$ cd to/package/directory
$ chmod +x tools/fsc.exe
$ ./tools/fsc.exe

This is equivalent to chmod +x on Mono
(rwx user, r group, r other)
open Mono.Unix.Native
let perms = FilePermissions.S_IRWXU ||| FilePermissions.S_IRGRP ||| FilePermissions.S_IROTH // 0x744
Syscall.chmod(FILENAME, perms) |> ignore

@joelverhagen The nuget issue you mentioned about is now closed and fixed.  So is it now possible to set the executable bit - or does it get done automatically/  What happens if you're building a nuget package on Windows - is it possible to set the executable bit for when the package is unzipped on linux?  thanks

The fix here has been integrated

@dsyme yeah packaging on windows is still a problem.
Best is probably if nuget would support settings an executable flag in the nuspec and than properly saving that in the zip (http://stackoverflow.com/a/13633772/1269722). Paket probably needs to support that as well.
As a workaround for now we can try to edit the zip to have the required info. Would be interesting to see if nuget/paket properly respect the flag in the zip while unzipping (if not that would be another bug)


sitemap: Hide pages from sitemap if .Sitemap.Hidden is true

This commit changes the default sitemap.xml to exclude pages from the
sitemap if a page author has set .Sitemap.Hidden to true in the
front-matter of a page.
The motivation for this change is for cases where a site maintainer
wants to use content from a special category to embed onto another page.
In this case, the content in the special category is already present on
the site, but the pages in the special category are created anyways.
These pages are included in the sitemap and will be indexed by search
engines, but this may not be the desired behavior. So, a site maintainer
is now able to exclude pages from the sitemap without making custom
modifications to the sitemap.xml and losing out on any changes made to
the default sitemap in Hugo in the future.
This change was inspired from a support thread from May 2015:
https://discourse.gohugo.io/t/how-to-omit-page-from-sitemap/1126?u=jwf
Documentation is also updated so designers can also take note of this
new change when reading about Hugo's sitemap template.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

A couple of comments...
First, adding a field to the documentation does not add it to the data structure. Running your patch:

WARN 2021/11/21 08:36:26 Unknown Sitemap field: hidden

Second, there's an open PR that already addresses this: https://github.com/gohugoio/hugo/pull/6370

Hi @jmooring, thanks for the quick review! Given that @felicianotech already sent a more polished version of this change, I will close this PR in favor of #6370.


Fluid grid calculation off

While using Semantic.gs, I found what i believe to be a bug, or a miscalculation in Grid.less. these were the settings i used:
@column-width: 54;
@gutter-width: 30;
@columns: 12;
this gave me 978. i then set @total-width: 100% for a fluid layout. it was a few pixels off from my design, but ok when i switched back to px.
Heres where i found the problem:
@_gridsystem-width: (@column-width*@columns) + (@gutter-width*@columns) * 1px;
i did the math: (5412) + (3012) = 1008px.
i took 1008 - 978 = 30 (which is the gutter size!)
so i fixed the formula for my usage:
@_gridsystem-width: (((@column-width*@columns) + (@gutter-width*@columns)) - @gutter-width ) * 1px;
and everything came out great in the fluid grid.  I never really contributed to the web community before, but i hope this helps anyone who runs into a similar problem :D I really dig the semantic grid system

Thanks for the detailed explanation! Those calculations look correct to me (that's what's supposed to happen with the 'leftover' gutter). But I've also experienced slight discrepancies between browsers when working with percentages. I think you're probably experience the sub-pixel rounding issue http://ejohn.org/blog/sub-pixel-problems-in-css/ for which there is currently no solution known to man. :-/ It's frustrating, I know.


docs: Fix ssh test command

The Issue
The current command generates an error:
❯ ssh<EMAIL_ADDRESS><EMAIL_ADDRESS>Invalid command<EMAIL_ADDRESS>You appear to be using ssh to clone a git:// URL.

Confirmed issues in the following terminals:

PowerShell 7.4.6
Windows Powershell 5.1.22621.4391
Git bash 2.47.0.windows.2

How This PR Solves The Issue
Update the command.
Now generates the following:
❯  ssh<EMAIL_ADDRESS>Enter passphrase for key '/c/Users/XXXXXX/.ssh/id_ed25519':
Hi XXXXXX! You've successfully authenticated, but GitHub does not provide shell access.

Manual Testing Instructions
Automated Testing Overview

Related Issue Link(s)
Release/Deployment Notes

Looking at https://docs.github.com/en/authentication/connecting-to-github-with-ssh/testing-your-ssh-connection, they recommend:
ssh -T<EMAIL_ADDRESS>

Thanks, will pull when you're ready, thanks for -T ! Great to have you back.

Ready when you are.

Would appreciate any other improvements on that blog!


Add attribute scrubbing to documentation for blazor:elementreference

I am trying to render Blazor components, but Blazor adds the "blazor:elementreference" attribute which changes every time we run the tests using bUnit.
Generated code:
Test value

In this case, this result is rendered by the following code:
<Blazorise.Text Style="<EMAIL_ADDRESS>
Before submitting a potential PR to update the docs, would you be interested in this?
HtmlPrettyPrint.All(nodes => nodes.ScrubAttributes("blazor:elementreference"));

shouldnt this be scrubbed by default?

I would vote yes

Given i faced the same problem there are 2 scrubbers i found helpful:
VerifierSettings.ScrubInlineGuids();

will replace the blazor:elementreference value. Good enough for most stuff.
second one is remove noise when you don't care about event's or those special blazor attributes at all (but just passing down some parameters to a child component and make sure they got propagated:
Thanks ChatGPT for the solution:

        // Add custom scrubber for blazor:xxx attributes
        VerifierSettings.ScrubLinesWithReplace(s =>
        {
            if(!s.Contains("blazor:"))
            {
                return s;
            }

            // Use regex to match and remove any blazor:oninput attribute
            // For example, it will match 'blazor:oninput="some value"' and remove it.
            // Remove all blazor event attributes (e.g., blazor:onclick, blazor:oninput, etc.)
            var scrubbed = Regex.Replace(s, @"(\s*)blazor:[a-zA-Z]+\s*=\s*""[^""]*""", "");

            return scrubbed;
        });

which would be equivalent to
        HtmlPrettyPrint.All(nodes => nodes.ScrubAttributes(a => a.Name.StartsWith("blazor:")));

Because it depends on your testing scenario i would vote for several extension methods:

ScrubBlazorElementreference
ScrubBlazorEvent("blazor:oninput") or ScrubBlazorEvent("oninput") (second just prepends the blazor:
`ScrubBlazorEvents()


Need to boot test images in travis, not just build them

This is going to become more important now that we have 2 architectures. We'll need to create a test image and boot it. This is hard but doable; we did it on NIX.

Spring cleaning - closing old issues


Problem sending tweet

As Laravel is using this package underneath Socialite, I thought I might as well keep using it for interacting with an OAuth server (that of Twitter, but more are on the roadmap) after authenticating. So what I did is the following:

I am using Laravel 5 its default implementation of Socialite

After authenticating, I save the User its token and tokenSecret (Twitter is OAuth 1)

Because the default server implementation for Twitter in this package does not leverage any functionality beyond authenticating, I extended the Server\Twitter class

Created a method for retrieving the statuses/update resource URL as already demonstrated in the base class, very straight forward:
  class Twitter extends \League\OAuth1\Client\Server\Twitter
  {

      public function urlStatusUpdate()
      {
          return 'https://api.twitter.com/1.1/statuses/update.json';
      }

      // ...

Created a method for sending a Tweet. I copied all the code from Server\Server->fetchUserDetails() to my extended Twitter class (see above) its postStatusUpdate method and made the following changes:

$url = $this->urlStatusUpdate();
$header = $this->protocolHeader('POST', $url, $tokenCredentials); (POST instead of GET)
$response = $client->post($url, $headers, $postBody)->send(); (->post instead of ->get and appended $postBody which is an argument of postStatusUpdate (for setting the status body param, see below))

The server is tested as follows
  $credentials = new TokenCredentials();
  $credentials->setIdentifier($this->auth->getUser()->oauth_token);
  $credentials->setSecret($this->auth->getUser()->oauth_token_secret);

  $twitterServer = new Twitter($twitterConfig);
  dd($twitterServer->postStatusUpdate($credentials, ['status' => 'test']));

where the oath_token and oauth_token_secret were earlier saved

However, I get a Authorization Required status response from Twitter. Upon dumping the headers of the internal Guzzle client, all required authorization headers are properly set. My Twitter application has the correct amount of permissions (read  & write) to write to the statuses/update resource.

Am I missing something obvious?
Is there an easier way to do this? (other OAuth requests than those already defined in the package)
Is this library even intended to be extended in such a way that requests can be made beyond authorization?

Thanks in advance.

@EliasZ I would recommend locating a separate library to use when communicating with Twitters API for non-authenticating tasks. While this package does include some basic user account access, you should consider limiting its responsibility to handling the authentication tasks and providing you with the resulting access tokens that will be required for a proper Twitter API client.

@stevenmaguire Thanks for your help.
For something which is called an oauth1-client I would have expected a bit more functionality, especially sending arbitrary requests? If I knew a bit more about oauth, I would have done it myself by simply extracting the authorization and header logic required from the abstract Server its fetchUserDetails method. If the building of a Guzzle client with the mandatory authorization headers/parameters is separated from fetchUserDetails, any method can use that client as a stub and additionally define its own HTTP verb, endpoint and parameter(s), it would be a lot more flexible.
But it's okay, I checked out thujohn/twitter (2.0) and it does seem to work out-of-the-box with the tokens earlier retrieved. So I now have two oauth libraries as a dependency; one which can only authorize against a variety of servers and one specifically for the Twitter server which can authorize and do a bit more.
Using authorization from this package "and a bit more" from the other, feels counter intuitive, especially because with some minor modifications this package could also do a bit more.

This package never was intended to be, and never will be, an API package for every conceivable OAuth provider.
This package is designed for authentication and retrieval of tokens. Do with the tokens what you will, be it store them, use them against an API or you could build a collage of tokens to use as your wallpaper. Point is, this package fills a specific need by design.
For what it's worth, picture something like GitHub'a API. Last I checked (I remember reading about phasing this out), you could provide an OAuth 2 access token or plain HTTP auth to make requests. A GitHub API SDK is a perfect contender to deal with its API. How you get your tokens, or indeed username / password is up to you.
Sent from my iPhone
Please excuse my brevity

On 22 Mar 2015, at 8:32 pm, EliasZ<EMAIL_ADDRESS>wrote:
@stevenmaguire Thanks for your help.
For something which is called an oauth1-client I would have expected a bit more functionality, especially sending arbitrary requests? If I knew a bit more about oauth, I would have done it myself by simply extracting the authorization and header logic required from the abstract Server its fetchUserDetails method. If the building of a Guzzle client with the mandatory authorization headers/parameters is separated from fetchUserDetails, any method can use that client as a stub and additionally define its own HTTP verb, endpoint and parameter(s), it would be a lot more flexible.
But it's okay, I checked out thujohn/twitter (2.0) and it does seem to work out-of-the-box with the tokens earlier retrieved. So I now have two oauth libraries as a dependency; one which can only authorize against a variety of servers and one specifically for the Twitter server which can authorize and do a bit more.
Using authorization from this package "and a bit more" from the other, feels counter intuitive, especially because with some minor modifications this package could also do a bit more.
—
Reply to this email directly or view it on GitHub.

I am not saying this package should feature a full-fledged SDK for every provider, just to make it possible to send arbitrary requests. So other packages can use this client as a base and extend it for platform-specific functionality.
It currently allows to step through the oauth1 authorization process and send a few provider-based hard-coded resource-requests (one of them is fetching user data, others are needed for the authorization process).
Wouldn't it be better to split the common authorization-process and the querying of resources? So there is one element/component which does the oauth1-authorization-process and another component which can send authenticated requests to resources. The latter was what I expected an oauth1-client can do.
If this package is only intended for quickly doing the authorization-dance and have an easy time being used for other oauth1-providers, then this decoupling is not needed and indeed you will then have to use this library and an additional specific package which you can then feed the retrieved tokens and send other platform-specific requests with. That specific package will often also feature an oauth-client and because it is specific, it will already contain functionality for the authorization process.
Does any of this make sense?

I get what you're saying. I'd need to talk to the others in the League to see what the direction of the OAuth-2 client is, as I'd like to move the packages along together. @shadowhand, thoughts?

This is exactly what #18 made possible. Here's some sample code:
protected function getCredentials(array $token)
{
    $credentials = new TokenCredentials;
    $credentials->setIdentifier($token['identifier']);
    $credentials->setSecret($token['secret']);
    return $credentials;
}

public function post($url, array $body = [])
{
    $credentials = $this->getCredentials($this->getToken());

    $headers  = $this->client->getHeaders($credentials, 'POST', $url, $body);
    $request  = $this->client->createHttpClient()->post($url, $headers, $body)->send();
    $response = $request->getBody();

    return json_decode($response);
}

The rest of the code in the class is specific to our implementation of token storage, but this should be enough to illustrate how you can build a generic "middleware" for API requests.

I should also say that oauth2-client has a much more advanced interface for this in v1.0:
$request = $provider->getAuthenticatedRequest($method, $url, $token);
// any modifications you need to make, set POST body, etc
$response = $provider->getResponse($request);

A similar interface could be developed for oauth1-client, but it would require using a PSR-7 compatible HTTP client, which would require a major version bump, as the interfaces would be incompatible.


k2 Hausdorff

This completes everything in #428 except RC-not-US.

More references for $k_2$-Hausdorff as a concept is requested here: https://mathoverflow.net/questions/453848/reference-for-k-hausdorff-in-terms-of-compact-t2-images

trivial comment: should the aliases for P170, P171 be "k-Hausdorff" without the dollar signs around the "k" ?

The following reference looks closely related.  It would be very interesting to take a look.  Unfortunately I don't have access, but maybe you can get it from your university.
https://zbmath.org/0463.54016
https://doi.org/10.1007/BF01238530

https://doi.org/10.1007/BF01238530

No dice - our subscription only starts with stuff from the 90s.


Fix some warnings

It seems #64 has not been merged properly, it already contains the fixes for those warnings.

Indeed, sorry


kubespray issue fix

If the answer is "Y" and the network plugin is not set in config, the script will report an error:

https://github.com/ansible/ansible/pull/9839 @hzy46


Avg Column Aggregation

Right now aggFunc can be sum, first, last, min, max. Could there be avg added?
If not maybe we could pass a callback which would compute custom to whatever we would like?

latest release allows custom agg functions for the col. i also give an example of doing average (which gets tricky btw if you are grouping multiple levels, as the non-leaf levels have to be weighted averages as the child groups will have differing numbers of rows)


Simple Animations in MBB

The ViewExtensions class provides extension methods that can be used to construct simple animations. However, this doesn't seem to work with MBB components. Is there any way to make this work ATM, or guidance on how it could be made to work?

Actually, you should be able to use those! Because they work on the Xamarin.Forms controls and not the Blazor components, you'll need to access native controls. Once you have the native control (such as Xamarin.Forms.Image), you should be able to use any of those extensions.