Config/Settings to ApplicationRunner (openHandshakeTimeout)

Hi,
How can I override default openHandshakeTimeout param?
I am going run ApplicationRunner with greater handshake timout in python code.
I have tried, but nothing:
subscriber = ApplicationRunner(url=u"wss://api.....", realm=u"realm1",extra={'openHandshakeTimeout':10})

+1 struggling to find where to change this as well.

Some here some developers to answer?

This isn't currently exposed at the ApplicationRunner level (only on the underlying WebSocketFactory level).

To clarify, there is currently no access via ApplicationSession or ApplicationRunner? Is there a single place to hard-patch at the module level?

@michael-wise monkey patching will not work, as the underlying factory is a locally scoped variable: https://github.com/crossbario/autobahn-python/blob/master/autobahn/twisted/wamp.py#L256

I'm kinda stuck waiting for this fix to be able to work with a server under heavy load that usually takes longer to complete handshake.
The hard-coded values in https://github.com/crossbario/autobahn-python/blob/master/autobahn/twisted/wamp.py#L256 would still override the fix in #844? Would love it if you would correct that before doing the next release.
I tried using the head of master after #844, and passing the options to the extra kwarg in the ApplicationRunner, which I assume should get passed as these new component options, but to no avail.

@myth curious, what server exactly? could you describe your setup?

Hi and thanks for the fast reply!
I'm attempting to connect to wss://api.poloniex.com, a cryptocurrency exchange, to get the ticker and orderbook feeds. I assume they are regularly DDoSed, but they are behind cloudflare.
I'm just prototyping the application for now, hence using the ApplicationRunner convenience, but my setup is just:
class Poloniex(ApplicationSession):
    def __init__(self, *args, **kwargs):
        ApplicationSession.__init__(self, *args, **kwargs)
        self.logger = getLogger(__name__)

    def on_event(self, *args):
        self.logger.info('{0}: {1} A:{2} B:{3} {4}% V:{5} H:{8} L:{9}'.format(*args))

    @inlineCallbacks
    def onJoin(self, details):
        yield self.subscribe(self.on_event, 'ticker')

And i start using the runner:
component_runner = ApplicationRunner(
    'wss://api.poloniex.com',
    'realm1',
    extra={'openHandshakeTimeout': 15, 'closeHandshakeTimeout': 15}
)
component_runner.run(Poloniex)

To figure out what I needed to set the timeout to to make sure the connection would finish before being preemtively terminated, I patched the hardcoded numbers in https://github.com/crossbario/autobahn-python/blob/master/autobahn/twisted/wamp.py#L256 and set both to 60 just to test. I also had to increase the ping timeout, as that also happened a couple of times.
This is the debug output from the runner, where my attempt at passing in an option dict to the extra kwarg with timeouts of 15s respectively, has no effect.
2017-06-08T14:45:50+0200
[('logOctets', False, 'WampWebSocketClientFactory'),
 ('logFrames', False, 'WampWebSocketClientFactory'),
 ('trackTimings', False, 'WampWebSocketClientFactory'),
 ('utf8validateIncoming', True, 'WampWebSocketClientFactory'),
 ('applyMask', True, 'WampWebSocketClientFactory'),
 ('maxFramePayloadSize', 1048576, 'WampWebSocketClientFactory'),
 ('maxMessagePayloadSize', 1048576, 'WampWebSocketClientFactory'),
 ('autoFragmentSize', 65536, 'WampWebSocketClientFactory'),
 ('failByDrop', False, 'WampWebSocketClientFactory'),
 ('echoCloseCodeReason', False, 'WampWebSocketClientFactory'),
 ('openHandshakeTimeout', 60.0, 'WampWebSocketClientFactory'),
 ('closeHandshakeTimeout', 60.0, 'WampWebSocketClientFactory'),
 ('tcpNoDelay', True, 'WampWebSocketClientFactory'),
 ('autoPingInterval', 5., 'WampWebSocketClientFactory'),
 ('autoPingTimeout', 10.0, 'WampWebSocketClientFactory'),
 ('autoPingSize', 4, 'WampWebSocketClientFactory'),
 ('version', 18, 'WampWebSocketClientFactory'),
 ('acceptMaskedServerFrames', False, 'WampWebSocketClientFactory'),
 ('maskClientFrames', True, 'WampWebSocketClientFactory'),
 ('serverConnectionDropTimeout', 1, 'WampWebSocketClientFactory'),
 ('perMessageCompressionOffers',
  [PerMessageDeflateOffer(accept_no_context_takeover = True, accept_max_window_bits = True, request_no_context_takeover = False, request_max_window_bits = 0)],
  'WampWebSocketClientFactory'),
 ('perMessageCompressionAccept',
  <function ApplicationRunner.run.<locals>.accept at 0x7f8f07996d08>,
  'WampWebSocketClientFactory')]
2017-06-08T14:45:50+0200 connection to tcp4:<IP_ADDRESS>:443 established
2017-06-08T14:45:50+0200 GET / HTTP/1.1
User-Agent: AutobahnPython/17.5.1
Host: api.poloniex.com:443
Upgrade: WebSocket
Connection: Upgrade
Pragma: no-cache
Cache-Control: no-cache
Sec-WebSocket-Key: *removed*
Sec-WebSocket-Protocol: wamp.2.json.batched,wamp.2.json
Sec-WebSocket-Extensions: permessage-deflate; client_no_context_takeover; client_max_window_bits
Sec-WebSocket-Version: 13

2017-06-08T14:45:50+0200 Connection made to tcp4:<IP_ADDRESS>:443
2017-06-08T14:46:18+0200 received HTTP response:

b'HTTP/1.1 101 Switching Protocols\r\nDate: Thu, 08 Jun 2017 12:46:18 GMT\r\nConnection: upgrade\r\nSet-Cookie: __cfduid=*removed*; expires=Fri, 08-Jun-18 12:45:50 GMT; path=/; domain=.poloniex.com; HttpOnly\r\nX-Powered-By: AutobahnPython/0.13.0\r\nUpgrade: WebSocket\r\nSec-WebSocket-Protocol: wamp.2.json.batched\r\nSec-WebSocket-Accept: *removed*=\r\nServer: cloudflare-nginx\r\nCF-RAY: *removed*\r\n\r\n'

2017-06-08T14:46:18+0200 received HTTP status line in opening handshake : HTTP/1.1 101 Switching Protocols
2017-06-08T14:46:18+0200 received HTTP headers in opening handshake : {'set-cookie': '__cfduid=*removed*; expires=Fri, 08-Jun-18 12:45:50 GMT; path=/; domain=.poloniex.com; HttpOnly', 'upgrade': 'WebSocket', 'sec-websocket-accept': '*removed*', 'server': 'cloudflare-nginx', 'date': 'Thu, 08 Jun 2017 12:46:18 GMT', 'connection': 'upgrade', 'cf-ray': '*removed*', 'x-powered-by': 'AutobahnPython/0.13.0', 'sec-websocket-protocol': 'wamp.2.json.batched'}
2017-06-08T14:46:18+0200 openHandshakeTimeoutCall.cancel
2017-06-08T14:46:28+0200 Auto ping/pong: sending ping auto-ping/pong
2017-06-08T14:46:28+0200 Expecting ping in 10.0 seconds for auto-ping/pong
2017-06-08T14:46:28+0200 WebSocketProtocol.onPing(payload=<4 bytes>)
2017-06-08 14:46:28,165     INFO     MAIN BTC_BTS: 0.00005360 A:0.00005360 B:0.00005337 0.32738979% V:6480.87132205 H:0.00005499 L:0.00003708

As you can see, both the RTT for the handshaking took a very long time, 28s in this case. It varies greatly, and sometimes it will connect just after a few seconds. I'm unaware of what is the root cause of these huge swings in delay, but a means to control the protocol parameters here is required to make it work...

Interesting! They must be running a quite old version of Crossbar.io, which you can tell from the WebSocket upgrade header: 'x-powered-by': 'AutobahnPython/0.13.0'.
We (Crossbar.io GmbH) are offering commercial support for Crossbar.io and all Autobahn client libraries, and as they are running a commercial venture, we'd be happy to address these issues under a support contract. We currently are working with a couple of companies already, and because of limited resources, we prioritize paying customers.

That is understandable!
I'll just override the setting manually for the time being then. Thank you
for your time!
Den tor. 8. jun. 2017, 16.01 skrev Tobias Oberstein <
<EMAIL_ADDRESS>
Interesting! They must be running a quite old version of Crossbar.io,
which you can tell from the WebSocket upgrade header: 'x-powered-by':
'AutobahnPython/0.13.0'.
We (Crossbar.io GmbH) are offering commercial support for Crossbar.io and
all Autobahn client libraries, and as they are running a commercial
venture, we'd be happy to address these issues under a support contract. We
currently are working with a couple of companies already, and because of
limited resources, we prioritize paying customers.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/crossbario/autobahn-python/issues/838#issuecomment-307112729,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACTdBqptBT7zNAqDw_6Ro3SuUqvR6_Ohks5sB_68gaJpZM4NanuX
.

If I may add an unsolicited opinion from someone who's knowledge of this library extends only to what was necessary to handle the poloniex feed:
Groking the code to change a timeout parameter should not be this hard. The heavy use of OOP here makes it very difficult to understand what the code base is doing if you didn't write it. Generally speaking if you're writing automated trading systems you can RTFM. I've rarely encountered a code base that was as difficult to parse as this one.

Have you looked at the (in-progress) Component API? On master this allows changing the handshake timeout, too. It only currently works with Twisted (not yet asyncio) but of course that is planned.
For your use-case that would look something like this:
from autobahn.twisted.component import Component, run

component = Component(
    transports=[
        "type": "websocket",
        "url": u"wss://api.poloniex.com",
        "options": {
            "open_handshake_timeout": 60.0,
        }
    ],
    realm=u'realm1',
    # authentication= {...} if you need too
)

def on_event(*args):
    print('{0}: {1} A:{2} B:{3} {4}% V:{5} H:{8} L:{9}'.format(*args))

@component.on_join
@inlineCallbacks
def join(session, details):
    print("Session {} joined: {}".format(details.session, details))
    yield self.subscribe(on_event, 'ticker')

if __name__ == '__main__':
    run(component)

Is this API easier to understand? Although you can still use an ApplicationSession subclass if you want, there's no need to subclass to use it...

Note you can also do @component.subscribe(...) on the on_event function if you prefer.

@meejah Thanks, i'll look into that!
I was probably going to try to replicate the basics of what ApplicationRunner does, but was unsure of how much was needed. That example looks simple enough to get me started working with the Component API!

@jonathanstrong there are 3 main reasons for the relative complexity of the code base:

we run on pretty much any combination of python version X networking framework (Twisted/asyncio)
you can write app code that will run unmodified on any of 1.
the library is able to "digest" broken WebSocket peers, because it is the basis for Autobahn testsuite

The simple answer is that you shouldn't need to delve into the code just to change an option.
This is addressed by the "component" API, but is a shortcoming of the ApplicationRunner API. It should have a way to set the websocket options, but doesn't currently. This runner was originally intended as a quick way to try things out or otherwise run pieces without configuring them inside Crossbar and has slowly acquired more functionality -- but doesn't do everything.
If you are interested in making ApplicationRunner better, pull-requests are of course welcome! The "run" function from the component API can hopefully give some hints as to where to tie in to set the websocket options if you want to try this :)
Ping me in IRC (#autobahn) if you need some help

@meejah I tried your example using the following setup, but keep getting some errors:
_push_api = Component(
    realm='poloniex',
    transports=[
        {
            'url': 'wss://api.poloniex.com',
            'type': 'websocket',
            'options': {
                'auto_ping_interval': 10.0,
                'auto_ping_timeout': 10.0,
                'open_handshake_timeout': 30.0,
                'close_handshake_timeout': 30.0,
            }
        }
    ],
)

@_push_api.subscribe('ticker')
@inlineCallbacks
def on_ticker(self, *args):
    self.logger.info('{0}: {1} A:{2} B:{3} {4}% V:{5} H:{8} L:{9}'.format(*args))

if __name__ == '__main__':
    run(_push_api)

Errors:
2017-06-08T23:45:40+0200 connecting once using transport type "websocket" over endpoint "tcp"
2017-06-08T23:45:40+0200 component failed: RuntimeError: unknown type <class 'dict'> for "tls" configuration in transport
2017-06-08T23:45:40+0200 Connection failed: RuntimeError: unknown type <class 'dict'> for "tls" configuration in transport

I've tried looking at the source for the Component class, but can't seem to find exactly where this hiccup is coming from.

Ah, thanks! See https://github.com/crossbario/autobahn-python/pull/848
Meantime, you should be able to just put "tls": true alongside "type" to work around this bug.

@meejah Ty for the fast response, really appreciate it!

The 'tls': True option did not work, just spits out:
 File "*removed*/env/lib/python3.5/site-packages/autobahn/twisted/component.py", line 180, in _create_transport_factory
    "Unknown {} transport option: {}={}".format(transport.type, k, v)
builtins.ValueError: Unknown websocket transport option: tls=True

I also tried constructing the endpoint dict manually, like it is done here: https://github.com/crossbario/autobahn-python/blob/master/autobahn/wamp/component.py#L148 but that just triggered the same error as in #848

@meejah is there an aproximate date when we can expect the component API to work with asyncio? I am suffering the same problem and my program is written all in asyncio.

@subiol pls see my comment here https://github.com/crossbario/autobahn-python/issues/838#issuecomment-307112729

@oberstet Sure, I understand, but as far as I can see this is also an open source project, so I dont see how asking for an approximate date when an API in development will be ready is bad or disrespectful. If the answer is you dont know, so be it, but I dont see how it hurts to ask.

Nothing disrespectful about asking!
The answer is: unknown. What @oberstet is getting at is that having a paying customer who wants this would speed it up dramatically :)
The reason I've been doing "Twisted only" side first is to make sure the API is right so that if it's not, we only have to change one framework's code. Also I know Twisted better than asyncio.
Of course, if you're interesting in helping by making a autobahn/asyncio/component.py that mirrors what's in autobahn/twisted/component.py currently, that would be very welcome! Ping us in #autobahn on Freenode if you need some guidance.
All that said, this API has decent consensus I think so I don't expect any major changes -- more like "possibly some tweaks".

Yeah, IMO the component API is definitely the way forward. The ApplicationRunner was basically a quick hack to collect the low-level boilerplate on top of ApplicationSession and the raw transport factory classes.
On a sidenote, the new AutobahnJava is following some crucial design improvements that the component API has, eg the session lifecycle can be tapped into from user code using the observer pattern.

Hey, i'm having the exact same problem. Adding 'tls': True to the transport config just produces this error:
ValueError: 'tls' is not a valid configuration item

@ashaffer
I just tried that with a fresh install, getting this error:
2017-06-20T19:37:17+1000 connecting once using transport type "websocket" over endpoint "tcp"
2017-06-20T19:37:17+1000 component failed: TypeError: optionsForClientTLS requires text for host names, not str
2017-06-20T19:37:17+1000 Connection failed: TypeError: optionsForClientTLS requires text for host names, not str
2017-06-20T19:37:19+1000 connecting once using transport type "websocket" over endpoint "tcp"
2017-06-20T19:37:19+1000 component failed: TypeError: optionsForClientTLS requires text for host names, not str
2017-06-20T19:37:19+1000 Connection failed: TypeError: optionsForClientTLS requires text for host names, not str

any chance you ran into the same thing?
Appreciate the help!

Nice! Ya, using CertificateOptions() is probably more right. On an unrelated note, do you happen to know if there's a recommended way of running one of these things in another thread? What i'd like to do is have a separate thread running my websocket connection, and then be able to subscribe/unsubscribe to different channels from the main thread, and have the websocket thread push messages over as they come in. It's not super obvious to me how to do that, though.

'tls': True should work fine, though so there's a problem here somewhere.. is this with Python2 or Python3?
You almost certainly do not want threads for what you're asking for. Are you doing some very CPU-intensive things somewhere?

Nope, no CPU-intensive work. What I want is to be able to have multiple websocket clients running simultaneously, connected to different servers. And then I want code running 'above' them in the logical sense to be able to manage them (i.e. subscribe/unsubscribe to different channels, add event listeners for messages, etc..).

@ashaffer there is no point on using threads since the python GIL does not allow for two threads to run at the same time and the library already uses asyncio/twisted to deal with blocking calls. So using threads will (most likely) slightly decrease performance.
But if you still wanted to do it just because, you just need to pass messages between the threads, for example using a asyncio thread queue like janus https://github.com/aio-libs/janus .

@ashaffer you don't need threads for that when using Twisted (or asyncio)

@meejah Ya, I kind of suspected that. It didn't feel right, but it's not clear to me how else to do it, since the Component mechanism is all sort of encapsulated in itself, it can't easily be called from the outside. Would you mind pointing me towards some docs or in the right direction there?

@ashaffer you have two options:

make all your code asynchronous, so you use asyncio or twisted style coding in one thread together with the websocket.

Put all the websocket code in its own thread and communicate with your threads using a push/pull queue like the already mentioned Janus (and I'm sure you can find something equivalent for twisted). In the Janus readme you can find a code example on how to communicate an asyncio thread (in your case the websocket thread) with other threads (in your case the rest of your program).

Note that in case 2 your program will still use only one core since the python GIL does not allow two threads to run at the same time. The difference is that in case 1 you decide in which order your code executes, while in case 2 the python thread scheduler will switch between the threads as its see fit, so it should be slightly less efficient.
This should solve your doubts, since I think we have already dirtied the issue with enough of topic.

api.poloniex.com does not provide up to date info, however slightly altering the above script to api2.poloniex.com provides updates
Now the question is, why does it come in chunks with delays? This is supposed to be asyncio but it seems this might be a flush problem? Anyone have any ideas?
`from autobahn.twisted.component import Component, run
from twisted.internet.defer import inlineCallbacks
from twisted.internet.ssl import CertificateOptions
component = Component(
realm=u'realm1',
transports=[{
'endpoint': {
'type': 'tcp',
'host': u'api2.poloniex.com',
'port': 443,
#'tls': True
'tls': CertificateOptions()
},
'type': 'websocket',
'url': u'wss://api.poloniex.com',
'options': {
'open_handshake_timeout': 60.0,
'tcpNoDelay': True
}
}]
)
def on_event(*args, seq):
print(seq, ":", len(args), ":", args, *args)
#print('{0}: {1} A:{2} B:{3} {4}% V:{5} H:{8} L:{9}'.format(*args))
@component.on_join
@inlineCallbacks
def join(session, details):
print("Session {} joined: {}".format(details.session, details))
yield session.subscribe(on_event, 'BTC_ETH')
if name == 'main':
run(component)`


weechat sasl_mechanism needs to be given lowercase

(Else there's an error.)

Yes, tested in Weechat 3.8 (which downcases, 4.0.1 doesn't.)


Bot going too fast?

Watching in Coinbase, it appears there is a lag between moving orders in which it does not appear there is an open order.
The bot moves very fast.
Is this a lag on Coinbase GUI part, or is there really a lag between? I would estimate it only appears that 10-25% of the time, there is actually an open order. The rest is the delay between canceling and reopening a new order.
However, this is only how it shows in Coinbase GUI. Thoughts?

Hi indeed Tribeca is very fast.
It can send up to 2000 orders a minute to the exchange, so there is no way to see it in the GUI or follow it with your naked eyes.
At the time of writing it is sending around 450 orders a minute for me.

OK, good to know. In that case, this is pretty much a non-issue.

reopening as I talked to my coder and he says the bot CANCELS and REPLACES orders and doesn't use a MOVE function. This means there is downtime between orders.
Has anyone calculated what % of the time Tribeca has no order in the system?

@renkcub What do you mean by MOVE? GDAX doesn't support replace/move (https://docs.gdax.com/#orders). There is a small period with no order in the exchange, but it sends both the cancel and the new order at the same time asynchronously.

Is there a way to limit how quickly tribeca makes and cancels orders? Like, limit how quick it accesses the API per second?


With #6194 , we can filter sst by commit ts. Then, we can scan much less data then before.
This PR does not add unit tests but test_output_delete_and_after_ts still covers it.
What is the type of the changes?

Improvement (a change which is an improvement to an existing feature)

How is the PR tested?

Unit test

Does this PR affect documentation (docs) or should it be mentioned in the release notes?
No
Does this PR affect tidb-ansible?
No
Benchmark result if necessary
Dataset: sysbench 100,000,000 rows x 1 table
Incremental data: sysbench update_index 60s (about 120k rows, 28MB)
Before: 57.8s
After: 1.0s

/release

/release

/release

@5kbpers PTAL again

/merge

/merge

/merge


Trim white space around shell input

These have leading spaces

I was going to add a test case, but I wasn't positive on how the syntax of QuoteParserTest works.
Would this work?
test(Case.of("  trimming_spaces  ", "trimming_spaces"));

@PseudoSky check the test case, you have one failure (leading whitespace after the trim).

From what I can tell it looks like the error is
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:2.13:check (validate) on project heroic-shell: Failed during checkstyle execution: There are 1 checkstyle errors. -> [Help 1]

@udoprog does that look right? I don't have too much experience with the testing framework or the style guidelines, do you know where I would look to find out how to fix it?

The line you want to look at is this:
[INFO] --- maven-checkstyle-plugin:2.13:check (validate) @ heroic-shell ---
[INFO] Starting audit...
/home/travis/build/spotify/heroic/heroic-shell/src/main/java/com/spotify/heroic/shell/QuoteParser.java:63: error: Line has trailing spaces.
Audit done.

It indicates that your line has trailing spaces.

Ha sorry the error was too meta for me to see - Trailing spaces within a trailing spaces fix.
I think I fixed it.


N19: Add wgpu/gfx/naga news

Part of #496

Thank you for the review! How about now?

wgpu and gfx sections look great but naga's nested list still doesn't fit the new simplified guidelines

Removed the nesting now. I find this limitation unfortunate :/

I admit this PR does look rather boring with these lists and no screenshots. But that's what we have right now, so better than nothing anyway.

Merged, thanks for the PR!
And, yeah, maybe we should relax some of these formatting limitations. :thinking:


Use equals() for String comparisons instead of '==' and Remove unused imports

Corrected some String comparisons to use equals() instead of '=='

I've reverted the commit for unused imports

Thanks for your contribution.


Touchsy HAT Display Stuck on Boot

I am experiencing an issue where the Touchsy HAT will turn on and display the boot sequence with the starting of services, but stops there and does not display the desktop.  I can then connect through VNC or connect a monitor to the micro HDMI and the Raspberry Pi 4B is working perfectly, but the Touchsy HAT is stuck only displaying the boot sequence.  Any advise would be much appreciated.  Thank you!

The problem is with the 64-bit version of Raspberry Pi OS.  I installed the 32-bit version of Bullseye and everything worked great.


Add FgenSignalVoltageGeneration and DCPowerPPS readme's

What does this Pull Request accomplish?
Adds readme's for the FgenSignalVoltageGeneration and DCPowerPPS measurements
Why should this Pull Request be merged?
Each measurement needs a readme and these weren't added in the original PR that added the measurement code.
What testing has been done?
N/A

Looks good！


multi-language

Predictions->Identify text
Does it support multilingual recognition?
Does it support multilingual recognition for the same image?

@dota2AndTI The AWS Rekognition service (which backs remote calls to detect text in images) basically just identifies strings of Latin characters. The Rekognition service docs specify:

A word is one or more ISO basic Latin script characters that aren't separated by spaces. DetectText can detect up to 50 words in an image, and GetTextDetection can detect up to 50 words per frame of video. Amazon Rekognition can also detect numbers and common symbols such as @, /, $, %. -, _, +, *, and #.

Based on that, my guess is that the service is going to return areas of an image that contain strings of characters, and it would be up to you to identify the language of the returned string, such as with a Predictions.interpret() call.
Hope this helps.


UDLs For <string_view> ("meow"sv, etc.)

ISO paper for this feature: P0403R1
First implemented in: VS 2017 15.3
existing doc
Remaining work: Should   mention /std:c++17

moved to quest


[BACKEND] Fix inline asm bug for multiple packed <32bit output

Resolves #5272

Fixes logic for walking result struct from LLVM InlineAsm in case of multiple sub-32bit results
Adds lit test

oops - just did. I keep on forgetting. will configure this to run automatically


Docs: Update 'Indexing across models' example to include all necessary steps

There are currently a couple of required steps missing from this example, which might get in the way of developers experimenting with the app. I've made quite a few more customisations in my version, so have tried to pull out the bare minimum.
@jacobtoppm Would you mind giving this a scan to make sure I haven't missed anything?

Thanks @ababic! I'm going to merge this so that RTD docs aren't misleading, but would still be good to hear your feedback @jacobtoppm


Dockerfiles missing dependencies

I was trying to get LDAP working with alpine and found a couple of problems.

Even though perl-ldap module is installed, perl itself isn't a dependency so it isn't installed
perl-io-socket-ssl also is not installed so we can't use ldaps

The ubuntu image has a similar issue.  apt-get will pull in the perl dependency so that is no problem, but the SSL module isn't installed in the running image.  It is present in the "build" stage so it might be an oversight.

Sorry for the delay. I've updated the images with the packages you needed. You can pull the latest of either one or use any of the "-202104181633" tags. It was an oversight. I didn't think about the packages being needed during runtime.
Give them a test and let me know if it works. I don't use the LDAP feature so I can't test myself.


Add download and documentation buttons in header

Add a download & documentation buttons in the header below the logo and text that says "Download" & "Documentation". The buttons' color is preferably white. Try to use one of the animation buttons we have, but please refrain from using a bootstrap button. For the download link, use one of the animation buttons, but as for the documentation just use outline button. Currently, we don't have a white color for the buttons, so you can just use any button color and override it in the assets/css/index.css for now.
If you have a good idea for the design of a button you can make a new one too. It would be nice if you can add it into the list of buttons but it's not required.
The link of the download should lead to the same download link as the one in the navbar. The link for the documentation should lead to the README.
Don't use the font we're using for the buttons, use the font we're using for the text in the header

I would like to work on this issue !

sure go ahead!

I would like to work on this please.

@yash2189 someone is already assigned to this one. Please find other unassigned issues.

Hi @kanavpruthi are you having any trouble with this? Do you need any assistance?

I'd like to work on this one if it's still open!


Add a special gulp xfatest command, to limit the ref-tests to only XFA-documents (issue 13744)

The new command is a variation of the standard gulp test command and will run all unit/font/integration-tests just as normal, while only running ref-tests for XFA-documents to speed up development.
Given that we currently have (some) unit-tests for XFA-documents, and that we may also (in the future) want to add integration-tests, it thus makes sense to run all test-suites in my opinion.
Please note: Once this patch has landed, I'll submit a follow-up patch to https://github.com/mozilla/botio-files-pdfjs such that we can also run the new command on the bots.

Nice work!


Add Appveyor configuration for MSVC 64-bit

For #457. Currently just runs cargo test.
To set up CI checks and for non-MSVC or 32-bit builds, please perform the necessary adjustments in a future PR.

Do I need to turn on Appveyor for the repo for this to work?


feat: support export metrics as prometheus format

close #756

metrics are exported under the admin port with path /metrics/prometheus

Coverage increased (+0.005%) to 39.079% when pulling ed3bf0d4d5b9296adacd0a314e8ef83a821e3ce8 on elonzh:feat/prometheus-metrics into 970ec21e9929bac5277dde52b9bc8a1cabe14df8 on kermitt2:master.

It works fine! thank you @elonzh

I added a note in the documentation about this, cdc33fff2d90cb6d8189ef60f6b99f69ca1e418f


File google-services.json is missing.

The Google Services Plugin cannot function without it.
Please update the file to the project.

Hi, Donut does not use Google Services Gradle Plugin.

Okay let me check why i am facing this Issue in master branch build.


Update vehicle journey types

in a StopTime struct, there is DepartureTime/UTCDepartureTime and UTCArrivalTime, but not ArrivalTime


BULK-Updates-2024-06-13

Updates to crossings incorrectly assigned as 'Passable'

Thanks.
I know there are definitely instances in BULK and perhaps other WCRPs where PSCIS barrier status fixes were applied based on the PSCIS comments and were a bit too generous in the interpretation.
But I don't see these two records in the fix table already, that is not the issue here.
PSCIS data indicates these are both habitat confirmations and PSCIS scoring assigns these as passable. PSCIS comments indicate there are other issues going on that are not captured by the scoring - the crossings do need attention/fixes.
I'm fine with switching these to POTENTIAL but for users comparing PSCIS data to bcfishpass data we need to be able to tell why and when the status was changed from what is assigned by PSCIS. Can you adjust the notes values for these?
I'll have to make adjustments to ensure that the notes that describe this change are included in various output tables - crossings_vw, crossings_wcrp_vw and the FPTWG extract - so the casual user can tell why barrier status in PSCIS and bcfishpass do not match. In hindsight, this should already have been done - we already have an extensive list of barrier status adjustments.
@NewGraphEnvironment @captainmarmot @nickw-CWF - any comments welcome. Barrier status for a given crossing in bcfishpass may not match PSCIS - even for a habitat confirmation. But going forward something in the crossings views will be added to note why this is the case.

seems odd to me to make these changes. The PSCIS metrics were collected somewhat recently using standard protocols and the scores came out passable. There is detailed reporting here https://www.newgraphenvironment.com/fish_passage_bulkley_2020_reporting/appendix-197665-197664-3042-barren-creek.html
crossings were considered passable when assessed. Impacts on fish populations were related to highway maintenance dredging vs passability at that time. We did not have any indication that the railway was being dredged as per the photos or we would have noted that...
Might make sense to reassess and re-enter the data as collected but doing custom alterations in the office to override detailed field assessment data seems like a slippery slope to me...
The detail needed to communicate what is being communicated with these edits (as Simon mentioned - more detail is needed to clarify) - unless I am missing something - IMO belongs outside of bcfishpass.  Options could be standalone reporting or perhaps a note in an issue of the repo that builds the report of potential edits that would help clear up some uncertainty.
also just noticed this within this same PR:

197017,PASSABLE,LNIC,LB,2024-04-09,"This is a ford, PSCIS barrier_status = UNKNOWN, edit existing record in fix table to PASSABLE"

There are many thousands of fords and I believe 99.9% of them are all barrier_status = UNKNOWN.  I wouldn't think we need to use [data/user_pscis_barrier_status.csv](https://github.com/smnorris/bcfishpass/pull/519/commits/1ed77f0701aed45b443a03f73488a3fc3835bb0c#diff-1679957ccb61a5f7b077ceebe77c7bae545b94145cd4b532dfa805d2b887c848) to adjust them..... This is already happening in the modelling.
Also - of importance related to the last note - symbolizing fords as "passable" (equivalent to bridges) can be confusing in the field because when we role up on a ford that looks like a bridge on the maps surveyors may think they should reassess it so that bcfishpass and PSCIS has the latest data. This common "unknown" and "passable" symbology used to happen in the past but we rectified that so that now we have a different symbol for the crossing while the modelling recognizes "unknown" as equivalent to "passable"....

I agree that barrier status divergence between bcfishpass and PSCIS should be minimal if any.
This is more of a question for @nickw-CWF @oakoppel-1 - can adjustments to PSCIS status be made in the reporting rather than directly in the bcfishpass crossings table? Or maybe we could apply these changes only in crossings_wcrp_vw for CWF planning use?
There are currently ~1300 PSCIS barrier status adjustments but mostly in WCRP watersheds.

I'm going to merge, this isn't an issue with the PR but with how bcfishpass handles this data generally.

Hi,
Thank you for your feedback - I am still getting familiar with everything, so I appreciate these explanations. As of right now, when I make changes to the fix table, I am mostly getting them through different emails or excel files on SharePoint so there may have been miscommunication regarding the edits that should be made to these crossings.
Updates that were sent to me regarding 197664 and 197665 crossings:

Update Barren Creek structures (197664 and 197665) to POTENTIAL barrier status using fix table

When I initially did the PSCIS review I read all the comments of associated PSCIS points to determine passability of the crossings, then applied those fixes to the table to make these crossings passable based on their PSCIS assessment. I am not entirely positive, but I think the WCRP team made the decision to classify any CBS as 'Potential' even if the PSCIS assessment indicates that its passable, so that's why these changes were made.  Here is a note from one of the WCRP meetings: The team concluded that barriers should not be called passable if they need more information (Data Deficient structures) and their PSCIS score / barrier status should remain until further information is gathered.  The team also agrees that keeping a record of this information would be helpful so the site can be revisited in the future.  Based on that note, I assume these structures are considered data deficient and the team wants to get more information about the crossings before classifying them as passable.
Updates that were sent to me regarding 197017 and 197018 crossings:

1011302218 Mine Creek and<PHONE_NUMBER> Mine Creek are visibly fords on satellite imagery - check satellite imagery and update to OBS in fix table if confirmed. Check satellite imagery and update to OBS in fix table if confirmed. Ensure there is not already an entry in the fix table for this crossing.

Above, you can see that fix doesn't make sense because the reviewer mentions that the structure is clearly a ford in satellite imagery (this is confirmed), but to update the structure type to OBS. Because this doesn't make sense, I just updated the passability status to passable.  I will ask Nick M-W if this update is incorrect when he returns.
In the meantime, I will make notes of your comments and forward them to the team.
Thank you for your help and let me know if you have any questions.
Laura Bernardi
GIS Technician
@.***
350, promenade Michael Cowpland Drive, Kanata, ON K2M 2W1
1.877.599.5777 |<PHONE_NUMBER>
CanadianWildlifeFederation.cahttp://www.canadianwildlifefederation.ca/?src=signature | Federationcanadiennedelafaune.cahttp://www.federationcanadiennedelafaune.ca/?src=signature
[seeds]http://donate.cwf-fcf.org/site/Donation2?df_id=7440&mfc_pref=T&7440.donation=form1&utm_source=signature&utm_medium=referral&utm_campaign=biod
Let's celebrate Canada's biodiversity by planting pollinator-friendly seeds.
Request your free seeds >http://donate.cwf-fcf.org/site/Donation2?df_id=7440&mfc_pref=T&7440.donation=form1?utm_source=signature&utm_medium=email&utm_campaign=mmg
Célébrons la biodiversité du Canada en plantant des semences favorables aux pollinisateurs.
Demandez vos semences gratuites >http://donate.cwf-fcf.org/site/Donation2?df_id=7440&mfc_pref=T&7440.donation=form1?utm_source=signature&utm_medium=referral&utm_campaign=biod&s_locale=fr_CA
From: Allan Irvine @.>
Sent: Thursday, June 13, 2024 2:16 PM
To: smnorris/bcfishpass @.>
Cc: Laura Bernardi @.>; Author @.>
Subject: Re: [smnorris/bcfishpass] BULK-Updates-2024-06-13 (PR #519)
CAUTION: EXTERNAL EMAIL / ATTENTION: COURRIEL EXTERNE
seems odd to me to make these changes. The PSCIS metrics were collected somewhat recently using standard protocols and the scores came out passable. There is detailed reporting here https://www.newgraphenvironment.com/fish_passage_bulkley_2020_reporting/appendix-197665-197664-3042-barren-creek.htmlhttps://linkprotect.cudasvc.com/url?a=https%3A%2F%2Fwww.newgraphenvironment.com%2Ffish_passage_bulkley_2020_reporting%2Fappendix-197665-197664-3042-barren-creek.html&c=E,1,1xtJZLveCgGj5Y2qveplENmlYpm4rXGFEobe8eAKlMR642ne35gKZjb0xETLcfWpCfAVlrwWjLARN6OroDEKhvYLXLWOFWxCzTT44YJRCNrOMBg6Pz25ISQ,&typo=1
crossings were considered passable when assessed. Impacts on fish populations were related to highway maintenance dredging vs passability at that time. We did not have any indication that the railway was being dredged as per the photos or we would have noted that...
Might make sense to reassess and re-enter the data as collected but doing custom alterations in the office to override detailed field assessment data seems like a slippery slope to me...
The detail needed to communicate what is being communicated with these edits (as Simon mentioned - more detail is needed to clarify) - unless I am missing something - IMO belongs outside of bcfishpass. Options could be standalone reporting or perhaps a note in an issue of the repohttps://github.com/NewGraphEnvironment/fish_passage_bulkley_2020_reporting/issues that builds the report of potential edits that would help clear up some uncertainty.
also just noticed this within this same PR:
197017,PASSABLE,LNIC,LB,2024-04-09,"This is a ford, PSCIS barrier_status = UNKNOWN, edit existing record in fix table to PASSABLE"
There are many thousands of fords and I believe 99.9% of them are all barrier_status = UNKNOWN. I wouldn't think we need to use data/user_pscis_barrier_status.csv to adjust them..... This is already happening in the modelling.
Also - of importance related to the last note - symbolizing fords as "passable" (equivalent to bridges) can be confusing in the field because when we role up on a ford that looks like a bridge on the maps surveyors may think they should reassess it so that bcfishpass and PSCIS has the latest data. This common "unknown" and "passable" symbology used to happen in the past but we rectified that so that now we have a different symbol for the crossing while the modelling recognizes "unknown" as equivalent to "passable"....

Reply to this email directly, view it on GitHubhttps://github.com/smnorris/bcfishpass/pull/519#issuecomment-2166489046, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BCMSYWCV4K7RNS7VVT65L2TZHHOXNAVCNFSM6AAAAABJIID5Y6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRWGQ4DSMBUGY.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>

Thanks @LauraB-CWF ! That all sounds good and your updates work well for what the WCRP team needs.
The remaining issue is more of a general bcfishpass usage/architecture question that is not hard to resolve - but will take a bit of work on my end. Discussion on that can move over to https://github.com/smnorris/bcfishpass/issues/521


linter-eslint triggers "Cannot read property 'match' of undefined"

linter-eslint causes this error inside TokenizedBuffer:
/Users/alex/.atom/packages/linter-eslint/lib/linter-eslint.coffee:136 [Linter-ESLint] error while linting file
/Users/alex/.atom/packages/linter-eslint/lib/linter-eslint.coffee:137 Cannot read property 'match' of undefined
/Users/alex/.atom/packages/linter-eslint/lib/linter-eslint.coffee:138 TypeError: Cannot read property 'match' of undefined
  at TokenizedBuffer.module.exports.TokenizedBuffer.indentLevelForLine (/Applications/Atom.app/Contents/Resources/app.asar/src/tokenized-buffer.js:647:23)
  at DisplayBuffer.module.exports.DisplayBuffer.indentLevelForLine (/Applications/Atom.app/Contents/Resources/app.asar/src/display-buffer.js:818:35)
  at TextEditor.module.exports.TextEditor.indentLevelForLine (/Applications/Atom.app/Contents/Resources/app.asar/src/text-editor.js:2220:33)
  at TextEditor.module.exports.TextEditor.indentationForBufferRow (/Applications/Atom.app/Contents/Resources/app.asar/src/text-editor.js:2192:19)
  at /Users/alex/.atom/packages/linter-eslint/lib/linter-eslint.coffee:108:44
  at Array.map (native)
  at /Users/alex/.atom/packages/linter-eslint/lib/linter-eslint.coffee:104:18
  at exports.allowUnsafeNewFunction (/Users/alex/.atom/packages/linter-eslint/node_modules/loophole/lib/loophole.js:25:14)
  at Object.provider.lint (/Users/alex/.atom/packages/linter-eslint/lib/linter-eslint.coffee:101:13)
  at /Users/alex/.atom/packages/linter/lib/editor-linter.coffee:79:24
  at /Users/alex/.atom/packages/linter/lib/editor-linter.coffee:78:25
  at Set.forEach (native)
  at EditorLinter._lint (/Users/alex/.atom/packages/linter/lib/editor-linter.coffee:72:26)
  at EditorLinter.lint (/Users/alex/.atom/packages/linter/lib/editor-linter.coffee:66:18)
  at /Users/alex/.atom/packages/linter/lib/editor-linter.coffee:22:37
  at Emitter.module.exports.Emitter.emit (/Applications/Atom.app/Contents/Resources/app.asar/node_modules/event-kit/lib/emitter.js:82:11)
  at /Applications/Atom.app/Contents/Resources/app.asar/node_modules/text-buffer/lib/text-buffer.js:1183:25

Please reopen this on the linter-eslint repo.

Actually @thomasjo this is an issue with internal functions called by TextEditor.indentationForBufferRow not properly checking return values.
Specifically TextEditor.indentationForBufferRow() calls DisplayBuffer.indentLevelForLine() which calls TokenizedBuffer.indentLevelForLine() which attempts to match the line against a regex and assumes that it gets a match (Which it looks like it would fail if the line has no indentation).
Do you want a separate issue filed that specifically mentions that?


Minimal scroll size

Hi, I've noticed than when you will put huge contrnt - let's say 100.000px height into scrollable are then calculated size of the scroll indicator is nothing - and so it does not get displayed. There shoul be some option to enable setting that let say it can be smaller than 5px so you will be still able to catch it with mouse cursor.
Regards

Update to version v5.0.1, all your problems should be solved :)


No support for custom datasets in ogb format?

I want to use custom dataset in ogb format

I am trying this too.
And here is my dataset
Is this what u meant? Although I am facing some problem with pretrained model, this dataset can be downloaded and preprocessed successfully.
If this is what u r asking about, I think that maybe I can help u.


Create a PR template

Create a template for submitting PRs to help with #105 style and formatting

Went ahead and added an issue template in ca0a7fc1864c42162236b4e68af6f44d07f0a164


Codec-Dump-PNG Jack Mic External connected\disconnected

I extracted codec dumps from linux with external mic connector (jack) connected and disconnected and this is the result:
1 internal microphone

2 External microphone

As you can see, the dump highlights the connection nodes of the internal\external Mic, furthermore it is visible on connector 27 (0x1b) something changes when connecting the Jack, but at the moment the 27 OUT node is unused

My guess is that audio is not forwarded to the speaker if a external MIC is connected to avoid feedback loops.


Relaxable TLSDESC only requires that R_RISCV_TLSDESC_HI20 to be paired with R_RISCV_RELAX

Requiring 4 R_RISCV_RELAX relocations impose a large size increase to the relocatable file. To mitigate this size increase, we can say that the whole TLSDESC code sequence is relaxable if the first instruction (R_RISCV_TLSDESC_HI20) is paired with R_RISCV_RELAX.
A statically linked executable typically has a simple relocation resolver that handles just RELATIVE/IFUNC. For a -fpic -mtls-dialect=desc relocatable file, the linker is required to perform TLS optimization to local-exec for a statically linked executable. Therefore, instruction rewriting is essentially inevitable if the relocation resolver is kept simple. Ths general-dynamic TLS model, without a defined optimization scheme, actually has the same issue.

@ilovepi @ishitatsuyuki
My lld/ELF implementation checks whether the first instruction (R_RISCV_TLSDESC_HI20) has R_RISCV_RELAX
https://github.com/llvm/llvm-project/pull/79239
On the LLVM side, I want to push for emitting one R_RISCV_RELAX as well.
If people say "this is not conforming", I think I'd like a mode to remove the redundant R_RISCV_RELAX.
We don't need to do the same as R_RISCV_TPREL_HI20.

Looks sensible. I think mold works without the RELAX annotations on paired relocs too, but I'll need to double check.

I think that this issue is red herring because the number of RELAX relocations for TLSDESC is probably fewer than 0.1% of those for R_RISCV_HI20 and R_RISCV_LO12_I. To solve it, I believe we should really consider something like https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/401. I'm not opposed to this proposal, but I'm skeptical about whether this change would make any meaningful difference.

I think that this issue is red herring because the number of RELAX relocations for TLSDESC is probably fewer than 0.1% of those for R_RISCV_HI20 and R_RISCV_LO12_I. To solve it, I believe we should really consider something like #401. I'm not opposed to this proposal, but I'm skeptical about whether this change would make any meaningful difference.

I made a reply: https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/401#issuecomment-1912548898
I think we can ensure that R_RISCV_RELAX causes less harm (size increase) for newer relaxable code sequences like TLSDESC.

Can this be merged? :)

LGTM


1a. Create new custom auth guard for checkout

At the moment the checkout process is blocked by a simple auth guard that checks if the user is signed in. The auth guard should be kept but a new guard should be created for the checkout.
This guard should redirect to login but should also pass some data to the login page (using the data field https://angular.io/api/router/Route#data). The data field is a key value pair. It should be the key should be "forced" (forced login) and the value true. The login page will be updated to read this in #1387.
This guard should also check if the cart contains a user to see if the user is a "guest". If he is a guest, he should be able to checkout.
This will be the only guard for checkout so it should also allow logged in users through. Therefore, it should reuse some of the logic from auth guard.
Acceptance criteria

Checkout only accessible as user or guest user
If a user has no role and tries to checkout he is redirected to the "guest login" page

I have pushed the newly created guard for checkout to the feature branch feature/GH-1182.
@LTiger14, @mszczepaniak  please note that we are not going to use data for passing the forced param. Instead we use the queryParam (?forced=true). The reason is that data is not mutable as well it will be lost in case of page refresh (we do not keep that in the store also).
As for now I also left the cart user part as we have to wait for #1179, RAY-229


Error in key derivation

The same mnemonic in smcli wallet create and smapp is now producing two different keys:
./smcli wallet read ~/.config/Spacemesh/my_wallet_2023-04-18T17-14-24.445Z.json -p
Enter wallet password:
2023/04/26 17:05:08 Warning: wallet file iterations count lower than recommended
+-------------------------------------------------------------------------------------------------------------+
| Wallet Contents                                                                                |
+----------------------+----------------------+---------------------+--------------+--------------------------+
| PUBKEY               | PRIVKEY              | PATH                | NAME         | CREATED                  |
+----------------------+----------------------+---------------------+--------------+--------------------------+
| a6c95349c8224..b7d82 | afa0ffdb92ef5..5228a | m/44'/540'/0'/0'/0' | Main Account | 2023-04-18T17-14-24.445Z |
+----------------------+----------------------+---------------------+--------------+--------------------------+
Mnemonic: abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon

> ./smcli wallet read ~/.spacemesh/wallet_2023-04-26T20-55-55.891Z.json -p
Enter wallet password:
+------------------------------------------------------------------------------------------------------------+
| Wallet Contents                                                                                |
+----------------------+----------------------+---------------------+-------------+--------------------------+
| PUBKEY               | PRIVKEY              | PATH                | NAME        | CREATED                  |
+----------------------+----------------------+---------------------+-------------+--------------------------+
| 06ea119a4bf7a..46ecc | a35668e1ce0eb..46ecc | m/44'/540'/0'/0'    | Master Key  | 2023-04-26T20-55-53.806Z |
| d4c1a605f1468..cb905 | 74c16d3e7148d..cb905 | m/44'/540'/0'/0'/0' | Child Key 0 | 2023-04-26T20-55-53.806Z |
+----------------------+----------------------+---------------------+-------------+--------------------------+
Mnemonic: abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon

I've verified that the behavior of smcli is correct as it matches the behavior of the solana keygen tool and the ledger hardware wallet, so something is wrong in smapp. I suspect the error is here:
https://github.com/spacemeshos/smapp/blob/e250ff64aa591e07b9a7ce2633c52ee9e620cbad/desktop/main/bip32-key-derivation.ts#L16-L23
Note how smapp uses the first 32 bytes as the secret key. Here's the corresponding go code:
https://github.com/spacemeshos/smcli/blob/669ec023f8478e3f22090c9d4f1b090ed94bf25d/wallet/bip32.go#L71-L81
If you look at what ed25519.PrivateKey.Public() is doing, you can see it's using the first 32 bytes as the public key:
https://cs.opensource.google/go/go/+/master:src/crypto/ed25519/ed25519.go;l=55-60;drc=84609d874e19e9d2419e07b72e8c8e2d24dcfc3a
We need to reconcile this behavior.

Is it possible to use a similar, battle-tested ed25519 library on the smapp side to avoid errors here?

@lrettig do I need to store the Master key in the wallet file as well?
Or do you leave it here just for debugging?

@lrettig do I need to store the Master key in the wallet file as well? Or do you leave it here just for debugging?

No, that's not required - we may remove it on the smcli side as well. Just the m/44'/540'/0'/0'/0' "child" key is sufficient. Thanks!


Upgrade from 1.5.3 -> 2.0.2 breaks Flask-JWT-Extended

Description
I've been using Connexion 1.X with Flask-JWT-Extended for multiple years without issue. I recently started a new project, and decided to start with Connexion 2.0.2. Unfortunately, it appears that Connexion 2.0.2 breaks Flask-JWT-Extended. Every unit test I have that tests against a protected endpoint fails with 2.0.2, but works with 1.5.3 and below.
Expected behaviour
Flask-JWT-Extended decorators should work for Connexion endpoints.
Actual behaviour
Any HTTP requests sent to Flask-JWT-Extended protected endpoints fail with "No authentication token provided."
Steps to reproduce
I have a Flask app that eposes a small set of REST APIs, and the project I'm building them in has a set of unit tests. The test process is to start an instance of the API server in a Kubernetes container, then run a set of tests that make requests to the APIs. In my local environment, the tests run from an OS X shell, and make http requests to the server running in a minikube container. In my Jenkins environment, the tests run from one container, and make http requests to the server in another container.
The issue is that the tests all pass in my local environment, but any test that runs against a flask-jwt-extended endpoint in my Jenkins environment fail due to missing access token errors from the flask-jwt-extended decorators.
Here is the request handler method and decorator for the first test that fails with this symptom.
# The /login enpoint with the GET verb is intended to be used by client apps
# to reload the user's application data when needed (e.g. after an application
# refresh, or browser restart)
@jwt_refresh_token_required
def search():
    """Handles GET verb for /login endpoint"""
    return jsonify(hydrate(g.user)), 200

I have before_request and after_request logging enabled with the following code:
@FAPP.before_request
def before_request():
    """Method to do work before the request"""
    # Ensure there is a database session available for the request
    g.db_session = DBSESSION()

    # Log request headers & body if FLASK_DEBUG > 1
    if 'FLASK_DEBUG' in os.environ and int(os.environ['FLASK_DEBUG']) >= 2:
        LOGGER.debug('Starting request for path %s', request.url)
        LOGGER.debug('Request Headers: %s', request.headers)
        LOGGER.debug('Request Body: %s', request.get_data())
        LOGGER.debug('Request Cookies: ' + str(request.cookies))

and
@FAPP.after_request
def after_request(resp):
    """Method to do work after the request"""
    # Log request headers & body if FLASK_DEBUG > 1
    if 'FLASK_DEBUG' in os.environ and int(os.environ['FLASK_DEBUG']) >= 2:
        LOGGER.debug('Response Headers: %s', resp.headers)
        LOGGER.debug('Response Body: %s', resp.get_data())

Here is a snippet of the logs from the server in the Jenkins test that is failing
[2018-12-01 11:59:52,854] DEBUG in server: Starting request for path http://rftw-jenkins-test-server.wittlesouth:8280/api/v1/login
[2018-12-01 11:59:52,854] DEBUG in server: Request Headers: Accept: */*
X-Csrf-Token: 510f5491-8e2c-4c49-add9-b7bcf21de353
Cookie: refresh_token_cookie=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJuYmYiOjE1NDM2NjU1OTIsImV4cCI6MTU0NjI1NzU5MiwiaWRlbnRpdHkiOiI4ZjkyNGQ4OC0zNzUyLTQzYzYtODhiOC1mNjA0Yzc4MzAyZGMiLCJ0eXBlIjoicmVmcmVzaCIsImp0aSI6IjJjZjYxMTY1LTkxYzQtNDJmYS1iYTk1LWU0Njc0YjY1N2Y1MiIsImNzcmYiOiI1MTBmNTQ5MS04ZTJjLTRjNDktYWRkOS1iN2JjZjIxZGUzNTMiLCJpYXQiOjE1NDM2NjU1OTJ9.Zmaw-Zar1psqzVbGyKnttbQZW9zwbIuOytEkrKpNuYU; access_token_cookie=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJuYmYiOjE1NDM2NjU1OTIsImV4cCI6MTU0MzY2NzM5MiwidHlwZSI6ImFjY2VzcyIsImp0aSI6ImViMWFmNTA5LTRjOWUtNGQ1MS04Yzg1LWRlMzJmODNlOGU0NCIsImNzcmYiOiJmZjYyOTMyNC02OWI5LTQ0MGMtYTM5ZC1kMWQwNDdiMzFkNzMiLCJmcmVzaCI6ZmFsc2UsImlkZW50aXR5IjoiOGY5MjRkODgtMzc1Mi00M2M2LTg4YjgtZjYwNGM3ODMwMmRjIiwiaWF0IjoxNTQzNjY1NTkyfQ.p1_RVP0rhnzcRvhVxqjtMMa7wpCeM7Ymcvii5A3HN2w; csrf_refresh_token=510f5491-8e2c-4c49-add9-b7bcf21de353; csrf_access_token=ff629324-69b9-440c-a39d-d1d047b31d73
Host: rftw-jenkins-test-server.wittlesouth:8280
User-Agent: python-requests/2.20.1
Connection: keep-alive
Accept-Encoding: gzip, deflate

[2018-12-01 11:59:52,854] DEBUG in server: Request Body: b''
[2018-12-01 11:59:52,854] DEBUG in server: Request Cookies: {'csrf_refresh_token': '510f5491-8e2c-4c49-add9-b7bcf21de353', 'refresh_token_cookie': 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJuYmYiOjE1NDM2NjU1OTIsImV4cCI6MTU0NjI1NzU5MiwiaWRlbnRpdHkiOiI4ZjkyNGQ4OC0zNzUyLTQzYzYtODhiOC1mNjA0Yzc4MzAyZGMiLCJ0eXBlIjoicmVmcmVzaCIsImp0aSI6IjJjZjYxMTY1LTkxYzQtNDJmYS1iYTk1LWU0Njc0YjY1N2Y1MiIsImNzcmYiOiI1MTBmNTQ5MS04ZTJjLTRjNDktYWRkOS1iN2JjZjIxZGUzNTMiLCJpYXQiOjE1NDM2NjU1OTJ9.Zmaw-Zar1psqzVbGyKnttbQZW9zwbIuOytEkrKpNuYU', 'access_token_cookie': 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJuYmYiOjE1NDM2NjU1OTIsImV4cCI6MTU0MzY2NzM5MiwidHlwZSI6ImFjY2VzcyIsImp0aSI6ImViMWFmNTA5LTRjOWUtNGQ1MS04Yzg1LWRlMzJmODNlOGU0NCIsImNzcmYiOiJmZjYyOTMyNC02OWI5LTQ0MGMtYTM5ZC1kMWQwNDdiMzFkNzMiLCJmcmVzaCI6ZmFsc2UsImlkZW50aXR5IjoiOGY5MjRkODgtMzc1Mi00M2M2LTg4YjgtZjYwNGM3ODMwMmRjIiwiaWF0IjoxNTQzNjY1NTkyfQ.p1_RVP0rhnzcRvhVxqjtMMa7wpCeM7Ymcvii5A3HN2w', 'csrf_access_token': 'ff629324-69b9-440c-a39d-d1d047b31d73'}
[2018-12-01 11:59:52,855] DEBUG in server: Response Headers: Content-Type: application/problem+json
Content-Length: 119

[2018-12-01 11:59:52,855] DEBUG in server: Response Body: b'{\n  "detail": "No authorization token provided",\n  "status": 401,\n  "title": "Unauthorized",\n  "type": "about:blank"\n}\n'
<IP_ADDRESS> - - [01/Dec/2018 11:59:52] "GET /api/v1/login HTTP/1.1" 401 -

Although the refresh-token-cookie required by the @jwt_refresh_token_required decorator is present in the request, the operation still fails with the error "No authorization token provided".
Here is the section of the server code where I set the flask JWT configuration.
FAPP.config['JWT_TOKEN_LOCATION'] = ['cookies']
if 'RFTW' in os.environ and os.environ['RFTW'] == 'PROD':
    FAPP.config['JWT_COOKIE_SECURE'] = True
else:
    FAPP.config['JWT_COOKIE_SECURE'] = False
FAPP.config['JWT_ACCESS_COOKIE_PATH'] = '/'
FAPP.config['JWT_ACCESS_TOKEN_EXPIRES'] = datetime.timedelta(minutes=30)
FAPP.config['JWT_REFRESH_COOKIE_PATH'] = os.environ['API_PATH']
FAPP.config['JWT_REFRESH_TOKEN_EXPIRES'] = datetime.timedelta(days=30)
FAPP.config['JWT_SESSION_COOKIE'] = False
FAPP.config['JWT_COOKIE_CSRF_PROTECT'] = True
FAPP.config['JWT_CSRF_METHODS'] = ['POST', 'PUT', 'PATCH', 'DELETE', 'GET']

The container specs for the server container are not identical for the local environment vs. the Jenkins environment, but base image and python-related packages installed are the same. Here are the key parts of the local test server container spec for each.
Local test environment (where the server application works correctly):
FROM ubuntu:16.04
LABEL<EMAIL_ADDRESS>project="ourlifestories.net"
RUN apt-get update -y
RUN apt-get install -y python3-pip python3-dev build-essential mysql-client libmysqlclient-dev curl sudo
ADD requirements.txt /
RUN pip3 install -r /requirements.txt
CMD ["bash", "-c", "${SERVER_COMMAND}"]

Jenkins test environment (where the above issue happens):
FROM ubuntu:16.04
LABEL<EMAIL_ADDRESS>project="Roadmaps FTW"
RUN apt-get update -y
RUN apt-get install -y python3-pip python3-dev build-essential mysql-client libmysqlclient-dev curl sudo
# Below doesn't seem to work to get time correct on server
# RUN sudo echo "America/New_York" > /etc/timezone
# RUN sudo dpkg-reconfigure -f noninteractive tzdata
ADD requirements.txt /
RUN pip3 install -r /requirements.txt
RUN mkdir /app
ADD src /app/
CMD ["bash", "-c", "${SERVER_COMMAND}"]

If I change the Connexion version to 1.5.3, everything works fine. If I use 2.0.2, all endpoints with Flask-JWT-Extended decorators return errors. Endpoints without decorators work fine.
Additional info:
Output of the commands:

python --version = 3.5.3
pip show connexion | grep "^Version\:" = 2.0.2
Flask-JWT-Extended version is 3.13.1

Let me know if any additional information would be beneficial.

Hey @wittlesouth - I'm not super familiar with the auth parts in Connexion 2.0, but @krise3k may be able to help.
I think your issue might be from applying the decorator to the handler function, which would apply auth much too late (after the entire connexion layer).
There are docs on how to set your auth function here : https://github.com/zalando/connexion/blob/master/docs/security.rst
Hope that helps!

Facing the same issue.

@dtkav - Maybe I wasn't clear what I'm trying to do. I don't want to use any part of the Connexion 2.0 authorization right now; I have already implemented authorization using the flask-JWT-extended package. My authorization works fine with Connexion 1.5.3, and fails with 2.0.2. Therefore, I'm pretty sure that there is a conflict between the two packages. It hasn't irritated me enough yet to figure out what it is on my own, but it's getting close.

hey @wittlesouth - I understand what you're trying to do. Sorry that it's frustrating.
One thing to note is that applying a security decorator to the handler function is fundamentally insecure.
In addition, Connexion uses semantic versioning, so 2.0.x is a major revision bump, and includes some backwards-incompatible changes. You can always pin connexion to 1.5.3.
If you still want to upgrade to connexion 2.0, and are ok with applying the security decorator in an insecure way, then I'm happy to help debug.
In the readme, under 'new in connexion 2.0', there's a section that reads:

If unsupported security requirements are defined or x-tokenInfoFunc/x-tokenInfoUrl is missing, connexion now denies requests instead of allowing access without security-check.

Based on this, I think you may need to set x-tokenInfoFunc and bypass connexion auth.
It would help me to tease apart the two issues you're having (environment, version) - can you confirm that connexion 2.0 doesn't work for you in either environment?

One more update. I went and read through the 2.0 security doc quickly, and checked out the sample app even more quickly. I would guess based on that quick read-through, the "unsupported security requirements are defined" statement is likely to refer to the API specification. I frankly forgot that my API specification does include some attempt to define the security approach for the API. As far as I know, in connexion versions prior to 2.0, this specification had no effect, and at best was useful for generated documentation.
Do you know if I remove the security specification from the API spec entirely, will connexion 2.0 revert to 1.X behavior, namely avoiding any attempt at authentication / security?
I took a quick look a the JWT example, from that example it doesn't appear that the JWT implementation in Connexion is nearly as rich as what Flask-JWT-Extended provides. Specifically, I'm using both authentication and refresh tokens (with different time to live values), as well as including the tokens in form posts for (hopefully) CSRF protection. It wasn't clear from my admittedly quick read that the Connexion JWT implementation supports either of these capabilities.

I frankly forgot that my API specification does include some attempt to define the security approach for the API. As far as I know, in connexion versions prior to 2.0, this specification had no effect, and at best was useful for generated documentation.

The purpose of connexion is to automatically do authentication, validation, serialization/de-serialization based on this specification. The idea is that the spec defines the interface (which connection handles), and then you define the business logic in the handler function.
In connexion 1.5.3, the only supported method of auth was oauth2 (IIRC). It would pass through any other auth methods (without doing any auth). This behavior was changed in connexion 2.0 to require the user to provide an auth handler if they had specified an auth method. You can set this auth handler to a pass-through if you like.

Can you elaborate, or provide a reference, as to how "applying a security decorator is fundamentally insecure"? That is the fundamental methodology of Flask-JWT-Extended and numerous other Flask extensions for authentication I reviewed before I selected a package.

The way connexion is implemented is basically by applying several decorators to each handler function. The connexion security decorator is applied first, so that the minimal amount of connexion code runs before the security decorator is applied.
If you apply the decorator to the handler function, what's basically happening is (pseudocode):
@connexion_security
@connexion_array_handling
@connexion_validation
@connexion_deserialization
@connexion_etc...
@jwt_refresh_token_required
def search():
    """Handles GET verb for /login endpoint"""
    return jsonify(hydrate(g.user)), 200

And all of that code is running before your jwt_refresh_token_requried decorator, so your attack surface is really large, and your users will only get an auth error if they provide a perfectly formatted payload (otherwise they'll get a validation error).
What you're doing in connexion 1.5.3 is this:
@connexion_security  # pass through - doesn't support JWT auth!
@connexion_array_handling
@connexion_validation
@connexion_deserialization
@connexion_etc...
@jwt_refresh_token_required
def search():
    """Handles GET verb for /login endpoint"""
    return jsonify(hydrate(g.user)), 200

what's happening in 2.0.x is this:
@connexion_security  # AUTH ERROR! Connexion 2.0.x is more strict!
@connexion_array_handling
@connexion_validation
@connexion_deserialization
@connexion_etc...
@jwt_refresh_token_required
def search():
    """Handles GET verb for /login endpoint"""
    return jsonify(hydrate(g.user)), 200

What does "If unsupported security requirements are defined" mean? Are you talking about security requirements in the API specification, or somewhere else? I certainly don't think I intended to configure any security validation by Connexion, so I assumed that this statement would not apply to me.

If you have security definitions in your spec, then they are "defined" for the purpose of connexion.

I took a quick look a the JWT example, from that example it doesn't appear that the JWT implementation in Connexion is nearly as rich as what Flask-JWT-Extended provides
...
It wasn't clear from my admittedly quick read that the Connexion JWT implementation supports either of these capabilities.

Connexion doesn't implement JWT - but it does currently pull the Bearer token from the header and then passes it to the function that you've registered with x-BearerInfoFunc in the spec.
AFAICT for your use case there is a missing features required before you could register an equivalent jwt function with x-BearerInfoFunc:
a) ability to use cookies with JWT auth
There is some work going on in #791 to support cookies, but I'm not sure it will work with JWT.
After that, you should be able to write a x-BearerInfoFunc that just calls flask_jwt_extended.verify_jwt_refresh_token_in_request().

@krise3k please jump in if you have some ideas for how to implement cookies for JWT and/or passing multiple tokens, or if you have any other ideas on how to help @wittlesouth

@cziebuhr do you have any thoughts / ideas on supporting JWT refresh tokens and cookies?

Do you have security defined in your swagger.spec? If you remove that, connexion won't do any authentication/authorization and you should be able to use your own decorators.

The answers above provide an extensive set of options for further investigation on my part. I need to do some more reading and an experiment or two. Unfortunately with this busy time of year, it is likely to be several days before I come back to this.

OK, I was wrong about how long it would take me to come back. Some more reading has lead me to a somewhat different conclusion. According to the OpenAPI 3.0 specification, the only valid authentication types in the spec are defined ("apiKey", "http", "oauth2", "openIdConnect"). Most of these (except openIdConnect) seem to be directly supported by one of the methods defined in the Connexion 2.0 security section. It doesn't appear that the authentication schemes provided by Flask-JWT-Extended comply with any of the authentication protocols defined in the OpenAPI 3.0 spec. And that would lead me to the conclusion that if I want a OpenAPI 3.0 compliant specification, I need to omit the security definition component of the API spec.
Am I reading the OpenAPI docs correctly?
Flask-JWT-Extended is pretty customizable. In my implementation, there are four distinct authentication components, the first two of which are HTTP only:

access_token_cookie - This essentially represents a session; identifies a user and has a ~30 minute timeout
refresh_token_cookie - This identifies a user and has a much longer timeout, but requires a reload before making data changes
csrf_access_token - required to be passed back to the server in a custom HTTP header for session-oriented requests
csrf_refresh_token - required to be passed back to the server in a custom HTTP header for identity-oriented requests (that are accessible after an individual session has expired)

Successful back-end authentication varies by which decorator you use, but can be based on either of the two HTTP only cookies as well as a request header with one of the two csrf tokens.
Correctly documenting the security requirements of any of my API endpoints would require documenting which of the two HTTP only cookies is required, as well as which of the two CSRF tokens, if any, is required in the header. Since Flask-JWT-Extended allows customization of the cookie names, and HTTP header name, those would need to be able to be documented.
If the schemes provided by Flask-JWT-Extended are formalized in a IETF document, I'd be surprised.
If I've read the OpenAPI 3.0 docs correctly, please let me know, and I'll close this issue. I will then need to decide whether to continue with the authentication schema I have (with the security requirements documented outside the formal API docs), or convert my application to an IETF standard authentication approach supported by the OpenAPI 3.0 specification.

I removed the security specification from my APIs, and I'm now able to run Connexion 2.1 without an error. Thank you for the detailed responses, that pointed me in the right direction. I don't know if this problem is going to trip up others, but it would have helped me if there was a debug log message or something in the error message response that showed that the missing token detection error was generated from Connexion. The error text was similar enough to what Flask-JWT-Extended generates when its decorators detect a missing token that I looked there first.

@wittlesouth - Excellent summary. I have been stumped on this issue for a while and was stalling the upgrade to 2.x. I also concur that if I remove security specs, I can still apply whatever method I need to authorize the token.


Build(deps-dev): Bump @testing-library/user-event from 14.4.2 to 14.4.3 (#3854)

Pull request opened by the merge tool on behalf of #3854

Merged in 6d0243e058aadc3ddf724307955f7ede1acfc0c5


autorest generated does not throw the exception for a failed azure async operation

We have a PUT call that will be tracked by Azure async operation and the final response would be 200 with the following body:

We expect the client report the error, but seems client called another GET to the original uri. We looked into the log and code and found that client did notice the provisioning state to be Failed:

While seems the final GET call happens in onOk:

Root cause:
In the logic of polling inside generated code, we are handling the "failed" and "cancelled" provisioning states the same way as "succeeded". They should throw error.
                                    switch( state?.ToString()?.ToLower() )
                                    {
                                      case "succeeded":
                                      case "failed":
                                      case "canceled":
                                        // we're done polling.
                                        break;

                                      default:
                                        // need to keep polling!
                                        _response.StatusCode = global::System.Net.HttpStatusCode.Created;
                                        continue;
                                    }


Setting minutes from the result of .diff(moment, 'minutes') does not work correctly

Apologies but I am busy and do not have time for the whole shebang here, but there seems to be an issue with either the result of .diff(moment, 'minutes') or the .minutes(someDiffResultInMinutes). I guess it might be because I was going from PM back to AM or something, like .minutes(-735) but it lost the quarter of an hour.
Getting the diff as milliseconds and using .add(dif, 'milliseconds') workwed fine.

This doesn't sound like a bug.
It sounds like you expected .minutes(x) to behave the same as add(x, 'minutes'), but it's more like .minutes(0).add(x, 'minutes')
There's nothing in this report to suggest a problem with .diff(moment, 'minutes')

This is a great question for Stack Overflow.


Use prefab from build.gradle to publish a precompiled library.

Describe the bug
Hello.
I have an android library that has a precompiled dependency, I include it in CMakeLists.txt as follows:
add_library(externalLib SHARED IMPORTED)
set_target_properties(externalLib PROPERTIES IMPORTED_LOCATION
    ${PROJECT_FOLDER}/externalDependency/lib/${ANDROID_ABI}/libexternalLibrary.so
)

But later if in build.gradle I try to expose it in the aar using prefab like this:
buildFeatures.prefabPublishing true
prefab{
    externalLib{
        headers "externalDependency/include"
    }
}

I get an error:
[CXX1404] did not find implicitly required targets: externalLib
For CMake targets that define the code from where they are compiled if it works:
add_library(internalLib SHARED
    myInternalLibrary/src/internalLibrary.cpp)

buildFeatures.prefabPublishing true
prefab{
    internalLib{
        headers "src/main/cpp/myInternalLibrary/include"
    }
}

To Reproduce
Steps to reproduce the behavior:

Download code from https://github.com/ximo1234/PrefabPrebuildLibrary
Make sure it compiles
Remove comments from the file: mylib/build.gradle, line 43 and 47
Try to compile again, it fails

Expected behavior
I expected the precompiled library to be added to the .aar but it doesn't work.
Environment:
Prefab version: Don't know, the one that comes with Android Studio built on May 18, 2022
Host OS: Ubuntu 22.04
Target platform: Android
Target ABI: arm64-v8a  armeabi-v7a  x86  x86_64
Target OS version: API 32
Build system: clang
Additional context
I tried putting GLOBAL in the CMake add_library command but that didn't work either.
Can any changes be made to the CMakeLists.txt or build.gradle to expose this library in the .aar output?
Using command line prefab I sucess done what I want, but be cool that this can be done via build.gradle.
Thank you so much.

You want the AGP bug tracker. Prefab doesn't create archives. https://issuetracker.google.com/173551290


Frontend coding guidelines

The first draft.
close #531
closes #626
Basics

[ ] I added a line to /doc/CHANGELOG.md
[ ] The PR is rebased with current master.
[ ] Details of what you changed are in commit messages.
[ ] References to issues, e.g. close #X, are in the commit messages.
[ ] The buildserver is happy.

Checklist

[ ] I have installed and I am using pre-commit hooks
[ ] I fully described what my PR does in the documentation
(not in the PR description)
[ ] I fixed all affected documentation
[ ] I fixed all affected decisions
[ ] I added unit tests for my code
[ ] I added code comments, logging, and assertions as appropriate
[ ] I translated all strings visible to the user
[ ] I mentioned every code or binary not directly written or done by me in reuse syntax
[ ] I created left-over issues for things that are still to be done
[ ] Code is conforming to our Architecture
[ ] Code is conforming to our Guidelines
(exceptions are documented)
[ ] Code is consistent to our Design Decisions

Review

[ ] I've tested the code
[ ] I've read through the whole code
[ ] Examples are well chosen and understandable

Not sure why the map cronjob stuff is in this?

Closed by accident.

Jenkins build please

Can I basically take out the coding part from /e2e/README.md and create a guideline from it?

Can I basically take out the coding part from /e2e/README.md and create a guideline from it?
Oh that would be a e2e-coding guidelines.

Yes, guidelines should be at one place (doc/guidelines), introduction at another place (README.md).


Add brower build targets to travis

[X] By placing an X in the preceding checkbox, I verify that I am a PayPal employee or
I have signed the Contributor License Agreement

Add additional build targets to travis CI to also run our
chrome and firefox suites locally.
Also moved the "deploy" step to phantom target.
Also reduced the number of threads in some suites
Also added support for passing chrome options in Config:
Add "--no-sandbox" for chrome execution to work around
chrome on OpenVZ.

I have updated this PR to not depend on sudo to allow us to continue using travis container based builds.

One additional improvement could be to move all these additional browser/driver files out of the root into the "target" folder?

One additional improvement could be to move all these additional browser/driver files out of the root into the "target" folder?

sounds like a good idea.. if it is not a trivial change, I'd suggest  -- not super important on a throw away travis instance. :)


[Suggestion] Configurable anvils

As in the title, abilities to:

Disable anvil 'too expensive' note
Disable anvil breaking
Make renaming free

(These I have got in my head for now)

Maybe a configurable cost factor ?
Ex.: Mending cost 2lvl on a new item
--> Make it cost 4lvl with a cost factor = 2
I love your idea, it looks great !

It's impossible to remove the "too expensive" note without setting a hard limit of 39 on every anvil repair. The note is given client side and you can only remove it via client side mod.

https://github.com/Jikoo/AnvilUnlocker ; what i have been using. It's not a perfect solution.

https://github.com/pl3xgaming/Purpur/issues/25

Anvils are a tricky thing, tbh. I've already added as much API to anvils as I can, including ways to allow plugins to bypass max cost. But as mentioned, its a bit tricky to pull it off correctly. You can take a look at my Hats plugin to see how I made a crown bypass max cost when worn. The rest of your request is already possible with the existing API. Just have to make yourself a tiny plugin to handle it with the specifics you want.


i likwe this alot

great optic good exampel

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7a1503ba-55b3-177f-fb47-204bd5b19a64
Version Independent ID: 241d90b0-13b9-ae00-276a-3f8b31db7fbe
Content: Command-line build with csc.exe
Content Source: docs/csharp/language-reference/compiler-options/command-line-building-with-csc-exe.md
Product: dotnet-csharp
Technology: csharp-language-reference
GitHub Login: @BillWagner
Microsoft Alias: wiwagn

Thank you. Closing as kudos.


[Bug] Persistent connection error thrown when using Unity 2022 on Mac

Description
When setting up or triggering ValueChanged callbacks for a DatabaseReference, errors are being thrown by the SDK in Unity Editor.
These errors only occur to our Mac machines, Windows machines and Android devices are unaffected.
The ValueChanged callbacks can be triggered, but they can then cause the same error that occurred when setting up the listener.
Originally observed this on Firebase version 10.7, upgraded to 11.9.0 and issue is still present.
Reproducing the issue

DatabaseReference is obtained for a particular node in the Firebase Realtime Database
A ValueChanged callback is added to the DatabaseReference object
Error is triggered referencing persistent_connection.cc
If a corresponding value is changed in the Firebase Realtime Database, the callback is triggered and so is the error. The callback is executed successfully.

Firebase Unity SDK Version
11.9.0
Unity editor version
2022.3.23f1
Installation Method
.unitypackage
Problematic Firebase Component(s)
Database
Other Firebase Component(s) in use
Analytics, Authentication, Storage
Additional SDKs you are using
Azure Speech, AWS
Targeted Platform(s)
Android
Unity editor platform
Mac
Scripting Runtime
IL2CPP
Release Distribution Type
Pre-built SDK from https://firebase.google.com/download/unity
Relevant Log Output
/Users/runner/work/firebase-unity-sdk/firebase-unity-sdk/firebase-cpp-sdk/database/src/desktop/connection/persistent_connection.cc(367): listens_.find(query_spec) == listens_.end()
UnityEngine.Debug:LogError (object)
Firebase.Platform.FirebaseLogger:LogMessage (Firebase.Platform.PlatformLogLevel,string) (at /Users/runner/work/firebase-unity-sdk/firebase-unity-sdk/app/platform/Unity/FirebaseLogger.cs:95)
Firebase.LogUtil:LogMessage (Firebase.LogLevel,string) (at /Users/runner/work/firebase-unity-sdk/firebase-unity-sdk/app/src/LogUtil.cs:69)
Firebase.LogUtil:LogMessageFromCallback (Firebase.LogLevel,string) (at /Users/runner/work/firebase-unity-sdk/firebase-unity-sdk/app/src/LogUtil.cs:77)
Firebase.AppUtil:PollCallbacks () (at /Users/runner/work/firebase-unity-sdk/firebase-unity-sdk/android_unity/armeabi-v7a/app/swig/Firebase.App_fixed.cs:4869)
Firebase.Platform.FirebaseAppUtils:PollCallbacks () (at /Users/runner/work/firebase-unity-sdk/firebase-unity-sdk/app/platform/FirebaseAppUtils.cs:33)
Firebase.Platform.FirebaseHandler:Update () (at /Users/runner/work/firebase-unity-sdk/firebase-unity-sdk/app/platform/Unity/FirebaseHandler.cs:208)
Firebase.Platform.FirebaseMonoBehaviour:Update () (at /Users/runner/work/firebase-unity-sdk/firebase-unity-sdk/app/platform/Unity/FirebaseMonoBehaviour.cs:45)

If using CocoaPods for Apple platforms, the project's Podfile.lock

Expand Podfile.lock snippet

👀 Replace this line with the contents of your Podfile.lock!

Hey @lmt385, thanks for reaching out. I'm trying to test your scenario using our quickstart repo but it seems to work just fine. I'm testing with version Firebase Realtime Database 11.9.0 and Unity Editor 2022.3.24f1. That said, could you answer the following:

You mentioned in [3] that "Error is triggered referencing persistent_connection.cc". Are you saying after adding ValueChanged callback, the error is already triggered?
Can you provide a MCVE so that we check if there's a difference in setup that's causing this issue"

Thanks in advance!

Hello,
The error triggers upon the callback being registered, yes.
I'll see what I can do in terms of providing a MCVE.


csplit: adjust the error message to match GNU's

Should fix tests/csplit/csplit

Congrats! The gnu test tests/csplit/csplit is no longer failing!

Cool :)


Vagrantfile: added ability to install Docker EE (specified version) on centos

envvar name is copied from https://docs.docker.com/engine/installation/linux/docker-ee/centos/
driveby: fixed a bug where node_os was an empty string instead of the default value because ENV["CONTIV_NODE_OS"] was being set to an empty string in the make target
Signed-off-by: Bill Robinson<EMAIL_ADDRESS>

@dseevr: Can you provide some steps to know how to test this, please?

build pr

build PR

build pr


[otbn] Support for unconditional jumps in otbn-rig

There are 3 commits in this PR.

"Split snippet generators into a separate directory": This just moves the snippet generators defined so far out of the way to avoid snippet_gen.py growing without bound(!). No functional change.
"Avoid getting stuck in otbn-rig": This avoids a bug where we'd have a region with space for (say) 10 instructions before we hit another piece of code. We have to make sure that if we've generated 9 of them, the 10th is either a jump or an ecall.
"Add support for unconditional jumps in otbn-rig": This is the interesting one!

Once this is merged, we should be generating every instruction that OTBN inherits from RISC-V, except for the CSR instructions. Those are a bit harder, because we have to track which CSRs should be defined and have values, which I haven't done yet.
The message for the 3rd commit is:
[otbn] Add support for unconditional jumps in otbn-rig
For a tiny example, run:
hw/ip/otbn/util/otbn-rig --asm-output gen.S --size 10 -o gen.json

and gen.S will contain:
/* Snippet 0 (2 instructions) */
.offset 0x0
addi          x30, x0, 1269
jal           x6, 3320

/* Snippet 1 (4 instructions) */
.offset 0x2d4
lui           x29, 29447
srl           x18, x30, x30
slli          x17, x24, 5
ecall

/* Snippet 2 (2 instructions) */
.offset 0xcfc
andi          x24, x0, 4
jal           x3, -2604

Note that the .offset directives in the generated assembly don't
actually work for generating different sections: that, together with
the rather ugly command line, will be fixed in a separate commit.

Force-push rebases onto master (there have been various PRs that touch these files, so the rebasing is a bit hairy)

I've pushed up a version that renames get_blank_insns_above and avoids high/low/top in that function's code. There are quite a lot of other places that use that terminology, though, and I honestly think it's the best one to use. A high address is a big number: this doesn't seem particularly confusing to me, at least.
About snippets and basic blocks: they are sort of orthogonal concepts. At the moment, there are just three types of snippets, each of which is a single instruction. So a basic block in the code will contain multiple snippets. Eventually, we'll have snippets for things like "conditional branch, with two different paths, then a phi node" or "call a function". These more complex snippets will contain smaller snippets in a sort of tree structure. (I've not implemented any of that yet, but that's the plan).
I think the right way to think of a snippet is "a parametrizable template, containing one or more instructions". However, I do intend to keep a certain "basic-block-ness" about them: each snippet will have one entry point and one exit point.

At the moment, there are just three types of snippets, each of which is a single instruction.

In the generated assembly, each "snippet" is a series of instructions. That's what motivated this comment.

In the generated assembly, each "snippet" is a series of instructions. That's what motivated this comment.

Ah, yes, you're right! That is a mistake (which I've got a fix for in a follow-up commit that also replaces the .offset directives with something that actually works). But it was lazy of me not to fix it properly - I'll do than now.

Close & reopen to run CI with correct verilator (I hope).


Autoexposure issue with DAVIS346

Hello,
I was using dvs_renderer to record ROS bags with a DAVIS346 device. Autoexposure was enabled by setting autoexposure_enabled: true in the configuration .yaml file. At first everything seemed to work fine, but then exposure started to oscillate between two values and never recovered. Is there anything that can be done to avoid this?
Thank you,
Antea

Just for info, the DAVIS ROS driver uses its own implementation of auto-exposure, distinct from the libcaer one, code here:
https://github.com/uzh-rpg/rpg_dvs_ros/blob/master/davis_ros_driver/src/driver.cpp#L857
Looks like both suffer from potential oscillations in edge cases due to relatively simple control loops.


collection format

Hi,
Nice work with Resgate.
I am having fun with it since a few days.
Not I want to complicate a little bit my model.
But, on a GET request, complicated JSON struct don't seems to pass.
For instance I return this Json structure on a Get request:
{
"result": {
"collection": [
{
"description": "Room1",
"device.description": "Room1",
"device.icon_name": "audio-card",
"device.intended_roles": "music",
"device.model": "WX-010",
"device.string": "[<IP_ADDRESS>]:5000",
"index": 0,
"name": "raop_output.Room1.local",
"path": "system.audio.raop_output.Room1.local",
"subscribe": [
{
"format": "pa_sample_spec",
"methode": "createStream"
},
{
"format": "sample",
"methode": "writeStream"
}
],
"uuid": "raop_output.Room1.local"
}
]
}
}
But, the Get request return this:
{
code: "system.internalError",
message: "Internal error: Internal error: Invalid value"
}
Could you take a look at it please?
Thanks,
Michael

Hi Michel,
Glad you are trying it out!
When doing nested/complex structures, you need to work with resource references.
The RES protocol only allows collections and models to have values that are either primitives or references to other resources. That means, a model may only contain a reference to a collection, but not the collection itself.
See: Res Protocol#Values
There are multiple reasons why the protocol doesn't allow resources with complex nested structures, but I won't go into design decisions. Instead I can tell you how it is meant to be done using resource references :)
I can see the following 5 resources in your structure:
service.rooms
[ { "rid": "service.room.1" } ]

service.room.1
{
	"description":"Room1",
	"device.description":"Room1",
	"device.icon_name":"audio-card",
	"device.intended_roles":"music",
	"device.model":"WX-010",
	"device.string":"[<IP_ADDRESS>]:5000",
	"index":0,
	"name":"raop_output.Room1.local",
	"path":"system.audio.raop_output.Room1.local",
	"subscribe": { "rid": "service.room.1.subscriptions" },
	"uuid":"raop_output.Room1.local"
}

service.room.1.subscriptions
[
	{ "rid": "service.subscription.1" },
	{ "rid": "service.subscription.2" },
]

service.subscription.1
{
	"format":"pa_sample_spec",
	"methode":"createStream"
}

service.subscription.2
{
	"format":"sample",
	"methode":"writeStream"
}

I've made up all the resource ID's . You obviously should name them differently. Especially the service.subscription.x, since I don't know if the subscription objects might be shared, or if they are unique for each room. In such a case, they might more suitably be service.room.1.subscription.x.
When you send a Get request for the service.rooms resource, Resgate will follow the references and return all these resources in one single response to the client.
Did that make it clearer?
If you still have questions, don't hesitate to reply! I like explaining these things :) .
/Samuel

Hi Michel,
Glad you are trying it out!
When doing nested/complex structures, you need to work with resource references.
The RES protocol only allows collections and models to have values that are either primitives or references to other resources. That means, a model may only contain a reference to a collection, but not the collection itself.
See: Res Protocol#Values
There are multiple reasons why the protocol doesn't allow resources with complex nested structures, but I won't go into design decisions. Instead I can tell you how it is meant to be done using resource references :)
I can see the following 5 resources in your structure:
service.rooms
[ { "rid": "service.room.1" } ]

service.room.1
{
	"description":"Room1",
	"device.description":"Room1",
	"device.icon_name":"audio-card",
	"device.intended_roles":"music",
	"device.model":"WX-010",
	"device.string":"[<IP_ADDRESS>]:5000",
	"index":0,
	"name":"raop_output.Room1.local",
	"path":"system.audio.raop_output.Room1.local",
	"subscribe": { "rid": "service.room.1.subscriptions" },
	"uuid":"raop_output.Room1.local"
}

service.room.1.subscriptions
[
	{ "rid": "service.subscription.1" },
	{ "rid": "service.subscription.2" },
]

service.subscription.1
{
	"format":"pa_sample_spec",
	"methode":"createStream"
}

service.subscription.2
{
	"format":"sample",
	"methode":"writeStream"
}

I've made up all the resource ID's . You obviously should name them differently. Especially the service.subscription.x, since I don't know if the subscription objects might be shared, or if they are unique for each room. In such a case, they might more suitably be service.room.1.subscription.x.
When you send a Get request for the service.rooms resource, Resgate will follow the references and return all these resources in one single response to the client.
Did that make it clearer?
If you still have questions, don't hesitate to reply! I like explaining these things :) .
/Samuel

I can also add that, I am currently working on a proper website for Resgate, resgate.io , containing all the guides and examples one need to learn these things. The section on Nested Resources does explain this eact part.
Too bad the site is not public yet. But I plan on having it made public before the end of the year!

Closing topic


TypeError: Cannot read property 'time' of undefined

This appears in the js console the moment the video ends playback for the first time using videojs 5. Restarting the video again makes these messages accumulate on a continuous basis. Full output in the console (Chrome):
Uncaught TypeError: Cannot read property 'time' of undefined in videojs-markers.min.js:3

I'll take a look. Thanks for reporting :)

fixed it in commit 702727

Thank you!

Hmmm... After updating to your latest commit it seems this problem still persists.

Can you show me an example code so I can repro?
Sent from my iPhone

On Nov 4, 2015, at 2:51 AM, Michael Dzjaparidze<EMAIL_ADDRESS>wrote:
Hmmm... After updating to your latest commit it seems this problem still persists.
―
Reply to this email directly or view it on GitHub.

I am getting this same error if I use
    markers: [       {time: 0}     ]
I am getting this same error if I use below.  I personally want to start with no markers.
    markers: [     ]
No error with this
    markers: [       {time: 1}     ]
No error with this
    markers: [       {time: 0},{time: 1}     ]

Facing the same problem, getting that error on console every second after re-playing de video

markers: [
{time: 2, text: "Marker text"}
]

Hi all!
Is this bug fixed?
Cause I'm facing the same problem: when the video reach the end at the first time, the errors below are throw to console:

I've already double checked and no mark is crossing the video duration.
I would appreciate if someone could share a fix or workaround.
Thanks! :+1:

Hi guys, sorry to get back so late. I can't seem to reproduce the problem when replaying video.

Can you try it with the latest commit? If the problem still exists, it'll be great if you can provide a short demo that repros the issue! Thanks

Still facing the same error on console after replaying the video with the lastest commit:
VIDEOJS: ERROR: TypeError: Cannot read property 'time' of undefined(…)
@ video.js:6359
f.apply?f[Array.isArray(b)?"apply":"call"](d,b):f(b)};c.logByType=i,/**

@marianoarga  can you show me the code so I can try to repro?

Does anyone found a fix (or at least a workaround) for this problem.
I don't see a pattern on this error, happening with some cases.
Thanks!

@spchuang I think I found the error pattern:
Here, in my project (which unfortunately I can't share any code), the error happens every time the last marker doesn't come after the video duration. When the last marker has it's time set to after total video duration, the error doesn't happen.
Please consider that I'm creating my markers in a crescent order of times.
So, for example:
Video duration = 60s
Last marker time = 55s -> The error happens
Last marker time = 65s -> The error doesn't happens

@mauricioklein it makes sense that you can't share any real code :) But can you create a small and simple demo that reproduces the issue for me? So I can debug and fix it?

@spchuang I'll try to do so in a JSBin, but I'm not sure if I'll be able to reproduce the error. I'll let you know.
Anyway, I'm now putting an extra marker on my setup, with time = Number.MAX_SAFE_INTEGER, which will be always greater than the video duration, and the error is gone.
Just to clarify: I'm intercepting the markers with the onMarkerReached callback.
Thanks!


Smart merging not working properly with function in rules options

I am trying to use smart merging in order to merge options of a loader (file loader in my case).
const mergedConfig = webpackMerge.smart(
  {
    module: {
      rules: [
        {
          test: /\.ttf|eot|jpg|png)(\?v=\d+\.\d+\.\d+)?$/,
          use: [
            {
              loader: 'file-loader',
              options: {
                limit: 8192,
                outputPath: (url) => {
                  const extension = url.substr(url.lastIndexOf('.') + 1);
                  switch (extension) {
                    case 'jpg':
                    case 'png':
                      return `assets/images/${url}`;
                    case 'ttf':
                    case 'eot':
                      return `assets/fonts/${url}`;
                    default:
                      return `assets/${url}`;
                  }
                }
              },
            }
          ]
        }
      ]
    }
  },
  {
    module: {
      rules: [
        {
          test: /\.ttf|eot|jpg|png)(\?v=\d+\.\d+\.\d+)?$/,
          use: [
            {
              loader: 'file-loader',
              options: {
                name: '[name].[ext]',
              },
            }
          ]
        }
      ]
    }
  }
);

The results is:
{
  "module": {
    "rules": [
      {
        "test": {
        },
        "use": [
          {
            "loader": "file-loader",
            "options": {
              "limit": 8192,
              "name": "[name].[ext]"
            }
          }
        ]
      }
    ]
  }
}

As you can see, it merges the limit and name option which are strings but it does not merge the outputPath which is a function.
Is it a normal behavior ? How could I fix it ?

It works fine for me. The outputPath function is there.
How did you print the merged result? For JSON.stringify, it can't print function fields.

How can I check then ?

@PierBJX Print it directly, like mergedConfig.module.rules[0].use[0].options.outputPath. Or use node --inspect-brk to debug your Node.js program and check the object with dev tools.

Thanks


Convert README to Build API syntax

Convert the README doc (./packages/breadboard/README.md) to the Build API syntax.

@dglazkov Hi Dimitri! This file has since been deleted from the repo, and I was wondering if you would still want it converted to the Build API or if this issue should be closed now? Thank you :heart_hands:

Ready to close!