[SD-463] Add linkset processor for site menus

Issue: https://digital-vic.atlassian.net/browse/SD-463
What I did

Added processor to hydrate API response from drupal linkset endpoint
Added unit test
Refactored original menu_items processor

How to test

Unit tests
nuxt-app

Checklist

For all PR's

[ ] I've added relevant changes to the project Readme if needed
[ ] I've updated the documentation site as needed
[ ] I have added tests to cover my changes (if not applicable, please state why in a comment)

For new UI components only

[ ] I have added a storybook story covering all variants
[ ] I have checked a11y tab in storybook passes
[ ] I have added cypress component tests (if the component is interactive)
[ ] Any events are emitted on the event bus using emitRplEvent

Oops didn't leave any instructions to test this:

Backend with linkset enabled https://nginx-php.pr-1740.content-vic.sdp4.sdp.vic.gov.au
Endpoint is (e.g. vic main) https://nginx-php.pr-1740.content-vic.sdp4.sdp.vic.gov.au/system/menu/site-main-menu-vicgovau/linkset
Set in nuxt.config.ts

  runtimeConfig: {
    public: {
      tide: {
        menuEndpoint: 'linkset'
        ...


Hide layers

Version
6.1.4
Browser and OS info
Chrome 102
Steps to reproduce
Go to timeline, click on "hide a layer"
What is expected?
The entry to stay in the list but be hidden in the timeline chart
What is actually happening?
There is no way to unhide a layer, the entry in the list is gone forever

Add the layer back from the layers list in the top right toolbar.

@Akryum I just want to give extra feedback that this "layers list" is very hard to find. I was confused as well and spend 10 minutes looking how to bring back the hidden layer. : D
there's so much room BELOW the actual layers, maybe you could add a button there as well that says "toggle layers" ?
something like this:

I too wasted time trying to get my "layers" back. It was only by Googling and finding this very bug report that I discovered from the OP how to get them back!
Please follow his suggestion and put something in the layers panel. Maybe just put the layer icon there instead of way up in upper right corner. It sure seems to belong to the layers panel
Thanks in advance


All uppercase properties are not serialized properly

A schema with a property whose name is all-uppercase will not get serialized correctly with jackson
Role:
  type: object
  properties:
    TC:
      type: string

In the resulting JSON it will be {"tc": "value"} instead of {"TC":"value"}.

This bug was already fixed with version 3.x

This is not fixed completely. Properties do no get serialized correctly if they are required and not nullable. This also includes properties which start with a single lowercase letter:
Role:
  type: object
  required:
    - TC
    - dTC
  properties:
    TC:
      type: string
    dTC:
      type: string


AVRO-3033: Fix flaky file descriptor test

Retry the test for leaked file descriptors in case there was a change due to a running background process.
Make sure you have checked all steps below.
Jira

[X] My PR addresses the following Avro Jira issues and references them in the PR title. For example, "AVRO-3033: My Avro PR"

https://issues.apache.org/jira/browse/AVRO-XXX
In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Tests

[ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[X] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[ ] In case of new functionality, my PR adds documentation that describes how to use it.

All the public functions and the classes in the PR contain Javadoc that explain what it does

Hello!  When I repeatedly ran the test, there were 5-10 failures over 100K runs in 20 seconds.  That's not really rigorous, because I wasn't running in any sort of isolation and my machine was doing other stuff.  There were never two failures in a row.
It looks like before fixing the bug AVRO-2286, the number of openFilesBeforeOperation was always greater than openFilesAfterOperation.  In practice, now they're usually the same value, which is promising evidence that there isn't a resource leak there.
The flakiness of the test comes from the fact that "something" might be fuzzing either of them off by one.  I used to see this in travis and local builds too (but infrequently).    I don't think it's instability of the machine, it's just not as deterministic as we'd like.
I'll add this info to the commit message and we can watch for further failures, I guess!  If we find evidence for either resource leaks or we see the flaky test continue, we can revisit it.


Entity edit packets.

Basic functionality for sending erase, edit and add packets, including the common and shape properties. Missing: other entity type properties, unit tests, integration tests and documentation.

Should also handle receiving entity add, edit, and erase packets.


Outlook Add-in Compose - Dynamic Control Button Text Updating

Note: This repo is only for questions related to its samples. If you have questions about how to use office.js or the Office developer platform, please post your question on https://stackoverflow.com. Tag your question with office-js or outlook-web-addins

Question
I developed an add-in built with yo generator (Node.js) that works as a configurator which enables the outlook compose to enable/disable sending a copy of the new email to a custom url on-send.
There's a control button added that says 'Toggle Send to cCRM' but I would prefer a toggle text dynamically from 'Enable send to cCRM' to 'Disable send to cCRM' and vice-versa when clicking the control button to improve UI experience. Is it possible to accomplish?

Closing as a duplicate of https://github.com/OfficeDev/office-js-docs-pr/issues/3282.


mitigate OOM errors on reservoir

[ ] increase swap space
[ ] limit reservoir container memory
[ ] limit user memory via systemd
[ ] limit R process memory via systemd

Each of this should start with a manual change on the servers as a test before adding to the Ansible deploy scripts.

Starting with increasing swap space.
The process is a little different on the two servers because aegypti is running Ubuntu 20.04 and prospero is running Ubuntu 18.04.
For prospero the existing swap space is on a partition and not in a file, so we'll have to remove the partition before adding the file.  To remove the existing swap partition we'll do:

Turn off the existing swap device: sudo swapoff /dev/nvme0n1p3
comment out existing swap entry from /etc/fstab

Now adding or modifying the swapfile is largely the same:

turn off all swap: sudo swapoff -a
make/resize swap file: sudo dd if=/dev/zero of=/swap bs=1G count=50
change swap file permissions: sudo chmod 600 /swapfile
make the file usable as swap: sudo mkswap /swapfile
activate the swap file: sudo swapon /swapfile
add to /etc/fstab: /swapfile none swap sw 0 0
Check: grep SwapTotal /proc/meminfo and swapon --show

Done with manual update of swap space on aegypti and prospero

I've manually run a script on aegypti and prospero that creates systemd files to limit user memory usage to 95% of RAM.


Compatibility with "City Original Owner" mod (or similar functionality)

Information about city original owner is always good one before you conquer some city, so I think good to have installed mod with that name (author is Imhotep: https://steamcommunity.com/sharedfiles/filedetails/?id=2021476879).
There is only one small glitch: both mods aren't totally compatible (but COO is compatible with Sukritact's Simple UI-A).
So is there any chance to make it compatible or to include that nice functionality in CQUI?

It would probably easier to just add this info to one of city tooltips. City’s original owner is directly available via Lua, so it is like couple of lines of code.

That would be good addition...

It is only implemented for expansions. Base game doesn't have the "owner info" functionality in the first place, so there is nothing to work on.


Explicitly default seccomp profile for gardenlet

How to categorize this PR?

/area robustness quality
/kind bug
What this PR does / why we need it:
Similar to https://github.com/gardener/gardener/pull/6175
A bug has been fixed which could prevent gardenlet pods from coming up in case the seccomp-profile webhook served by gardener-resource-manager is unavailable or broken. This is easily reproducible if a seed is hibernated and then woken up.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Thanks to @rfranzke for reporting the issue! @timuthy this can also be relevant for the pod-topology-spread-constraints webhook.
/cc @rfranzke
Release note:

A bug has been fixed which could prevent `gardenlet` pods from coming up in case the `seccomp-profile` webhook served by `gardener-resource-manager` is unavailable or broken.

@timuthy this can also be relevant for the pod-topology-spread-constraints webhook.

Thanks @dimityrmirchev, I'll take care.


Make monorepo more friendly for forking/contribution

Describe the bug
Currently the build task fails on a fork due to NPM integration.  The error is Usage Error: Environment variable not found (NPM_TOKEN) and it's because there isn't a proper NPM_TOKEN in the .yarnrc.yml.  Is there a way around this for contributors other than removing the npm entries in .yarnrc.yml?
To Reproduce
Steps to reproduce the behavior:

Fork this monorepo
clone to the fork
run yarn build
See error Usage Error: Environment variable not found (NPM_TOKEN)

Expected behavior
I expect to be able to build the fork on my local machine.
Desktop (please complete the following information):

OS: Windows 11 Home

Hey Tom
All you need is to export an NPM_TOKEN to your local env, it doesn't have to be a valid one, it's only used to publish to NPM
export NPM_TOKEN=xxxxxxxxxxxxxxxx
Let me know ;-)

Ok that worked with the NPM_TOKEN env variable!  Now the issue I'm having is I can't use the local monorepo libs as a local dependencies in another project due to how workspaces work.  The workspaces don't resolve.  Do you know a workaround for this?

I don't... I guess that's the idea of the mono repo though, to have all libs inside the same mono-repo project. You can add your lib to the /libs folder and use the same patterns I'm using there..., but you will have to tweak the publishing workflow if you want to publish your lib to another npm account.
Let me know if you find a better way to collaborate with mono-repos and we can plan the upgrade.

Thanks Roger, I did figure out I can use the monorepo libs folder as a workspace in my other project.  Now I should be able to easily develop my eventually-firestore lib and contribute it back!


Problem with onnxruntime-gpu when run webui on ubuntu

After completing install step, I run with this command
 python src/webui.py --listen
Everything is ok but when I click to Generate button on webui, with Lisa model and a url of music video on youtube, on my terminal, it throws this message:
[E:onnxruntime:Default, provider_bridge_ort.cc:1745 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1426 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory [W:onnxruntime:Default, onnxruntime_pybind_state.cc:895 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirementsto ensure all dependencies are met.
And after that, it keep running but it's very slow because, I think it's running on my CPU, but I want it run on my GPU. How can I fix it?
Thank you so much for supporting!

@huyctdev I have experienced the same issue. This seems to be due to using an invalid combination of onnx-runtime and CUDA.  I have a fork of this repo here: https://github.com/JackismyShephard/ultimate-rvc. It should fix that issue, in addition to adding many new features. You can have a look at it, if you feel like it  :blush:


proxy: revert custom changes

Revert this code for now as I think it was responsible for the most recent incident.

Not true :'(


Create aggregate_spatial_window

I created this pull request to discuss about the new process I would like to add as discussed here https://github.com/Open-EO/openeo-processes/issues/187

Thanks for the changes. I tried to parse it, but the JSON is invalid. Could you make it valid JSON and run it through https://jsonformatter.org/ so that it's better indented and easier understandable? Must be made before merging anyway.

Closing in favor of #192


Not finding any channels

Hi. Just before the last update I was getting errors when searching for channels. Since the update the errors are gone but I still don't see any channels appear.
[2021-02-24 21:15:55,064] INFO - Checking for tvheadend Channel information stored in the database.
[2021-02-24 21:15:55,066] INFO - Performing Channel Scan for tvheadend.
[2021-02-24 21:15:55,072] INFO - Found 0 channels for tvheadend.
[2021-02-24 21:15:55,072] INFO - Performing tvheadend Channel Import, This can take some time, Please wait.
[2021-02-24 21:15:55,072] INFO - tvheadend Channel Import took just now
[2021-02-24 21:15:55,073] INFO - Found no NEW channels for tvheadend.
[2021-02-24 21:15:55,073] INFO - Total tvheadend Channel Count: 0
[2021-02-24 21:15:55,073] DEBUG - Saving tvheadend channels to database.

My ini for TVH
[tvheadend]
address = <IP_ADDRESS>
tuners = 1
port = 9981
ssl = None
weight = 300
stream_method = direct

Any ideas?

Username/password?

Just as I posted this I thought I'd better try with a user and pass. TVH doesn't require a user so I never one set up. I added one for the purpose of this and sure enough it works now.


[FE]: Bulk Reset/Edit Attributes

Description: (details of what and how should work)
Some entrypoint to basically list a lot of items (potentially different types) and reset / change the values of some or all of their properties at once
Additional Context & Ad Hoc Info:

[ ] Support for multiple types at once?
[ ] How to get all the Items?
[ ] Reset only, or some way to edit as well?

More context in Club Channel (31st Aug 23')

SPIKE:

[ ] Whether GUI change is required? (aka By-Terminal only?)

This is possible but only via Terminal and will stay that way


Add random number to echo example using DRBG

Add a random number to echo requests and responses using CHIP's DRBG_get_bytes() API. The request will contain the random number generated by client. The response message will contain client's original random number, and also include a new random number generated by the server. The intent is to test DRBG_get_bytes() API, ensuring each request/response contains a unique identifier.

echo does not exists anymore. Should we close this ? @pan-apple

Yes, I think we should close it

Closing. Feel free to reopen if needed.


Project management workflows

In Artemis we're also using GH actions to automate our project management workflows (https://github.com/DiamondLightSource/python-artemis/issues/87). Would it be useful to have these as a standard thing too? Maybe as optional

Does it need an additional token added as a secret? What scope does it need?

I have it using a separate token yes, the token I have it on has project, public_repo, repo:status, write:org. It is possible that's more than it requires exactly as I was fiddling a bit.

As discussed in person with @coretl, happy for us to not have it in this repo but separate and linked in the DLS developer guidelines so closing this out


Decryption succeeds with invalid passphrase, but returns gibberish

I was trying out your library, and I noticed that decrypting using an invalid passphrase returns gibberish instead of failing with an error.
For example, if the message="test", passphrase="a", the ciphertext=bfzGx4hFvMbLqX/O6yIkYQbv59TKSiyYyhJ54gSVKVI=.
If I try to decrypt with the passphrase "a", I get the message "test". However, if I try with anything else like "ab", I will get the following: Únç֋ïiR°îâ. This is also the case in your demo.
Is this the expected behaviour? If so, under which circumstances will Crypto.strings.decrypt return an error?

There are two ways for Crypto.Strings.decrypt to return an error.

A failure of Crypto.Hash.sha256, in hashing the passphrase, which happens if it isn't a UTF-8 string.
An error in Crypto.Crypt.decrypt, using Base64.decode to convert the Ciphertext from Base64 to binary.

If you pass the wrong passphrase, you'll get no error, just a gibberish return.
If it's important for your application to recognize wrong passphrases, you could prepend some known text before encryption, and look for, and remove it after decryption. This slightly decreases the security of your encryption, since an attacker will also be able to tell that the wrong passphrase was wrong, but my intuition is that it probably won't matter enough to care about (for good passphrases). (There is a tiny possibility for the prefix to be in the decrypted text, with gibberish following it, when the key is wrong, but that is likely so rare that you needn't worry about it; monkeys typing Shakespeare).
Note that billstclair/elm-crypto-aes, on which billstclair/elm-crypto-string is based, was mostly just an exercise, to see what Crypto would look like in Elm. I optimized for code readability and beauty (in the eyes of this beholder). It is very slow (but may be fast enough for your application). Elm and JavaScript are rotten languages for doing crypto (but Turing machines, so it's possible, just bloody slow); the JavaScript Crypto interface likely uses native code inside, so is likely hugely faster (I say "likely", because I haven't timed it).
Closing this issue, since I consider it to be a feature, not a bug.


Ford: seperate logging-only request in FW query config

Why does the query timing increase for this? Isn't it the same number of queries*ecus?

Why do we need this? The time increases because we query sequentially, and don't optimize this into one query again.

Why do we need this? These ECUs being in extra_ecus means they are already marked as logging.

Oh I see


Status bar does not stick to the bottom of the browser window.

Not sure why:

http://katydecorah.com/css-ruler/

Looks like it is caused by -webkit-transform: translate3d(0, 0, 0) on the page’s body. I will have time to set-up a test-case late tonight (CET).

http://stackoverflow.com/questions/15194313/webkit-css-transform3d-position-fixed-issue

I am working on a fix. All that has to be figured out is the exact height of the statusbar (or cave in and set a height). Then the position can be top: calc(100vh - $height), where $height is the statusbar’s height, to force the statusbar to display depending on the viewport height rather than the containing block.
The following (really bad) CSS has been added to the bottom of div#com-fortnight-status-bar on my machine and seems to display right:
div#com-fortnight-status-bar {
  /* We are setting padding and height, make sure box-sizing is right. */
  box-sizing: content-box !important;
  /* font-size: 8pt = 10.666666px really, but Safari’s computed value is 11px. */
  font-size: 11px !important;
  /* No variable sizing, everything pixels. (Probably, maybe, not necessary?) */
  padding: 2.2px 12px 2.75px !important;
  /* Helvetica Neue size 11px seems to get a 1.1818181818181819 line-height?!?!?!
     http://meyerweb.com/eric/css/tests/line-height/inspect-multi.html */
  line-height: 13px !important;
  height: 13px !important;
  /* Now we can calculate at how many pixels from the viewport top to display. */
  top: calc(100vh - 2.2px - 2.75px - 13px) !important;
}

This seems to be slightly off compared to the current height of the statusbar, and I would love to get it closer to what we currently have before shipping. Should probably start a branch for this.

I was playing with something similar. Could also try using ems in the calculation too.

Could also try using ems in the calculation too.

Oh definitely. I just have an easier time visualising with px, and there is no real difference either way as modern browser zooms effect pixels just as much. The real problem is it seems to me that the current statusbar’s height is mostly dependent on the text’s line-height, and a line-height of normal keeps me guessing as to the actual height.
I’ll push a new branch if I figure it out today.

I have been going with the viewport height + calc() method for the last 3 days now, and it seems to work except when a horizontal scrollbar is displayed. In that case it draws the statusbar behind the scrollbar. Is this a bug or is vh supposed to ignore browser UI?


Form semantics change for

Current behaviour:
>>> minify_html.minify('<select><option value="">select a value</option></select>')
'<select><option>select a value</select>'

The value="" is dropped. This changes the submitted value for that option. As per MDN docs for value of <option>:

The content of this attribute represents the value to be submitted with the form, should this option be selected. If this attribute is omitted, the value is taken from the text content of the option element.

So dropping value="" here causes the form to submit "select a value" instead of "".
(Originally reported downstream in https://github.com/adamchainz/django-minify-html/issues/21)

I think this will have been fixed by https://github.com/wilsonzlin/html-data/commit/8451b7dd1d3c551c744092b5ac6aeaf4f4427b62

Thanks for raising the issue, and yes this will be fixed in the next release cut.

Hey, I don't mean to be a pain, as I know open source is rarely the priority, but would you be able to do a release soon? The lack of this bugfix is stopping me from advocating minify-html for use in Django projects, since I think <option value=""> is pretty common.

I need this fix too. Django complains "Select a valid choice. That choice is not one of the available choices." when empty value selected.

I've just released a new version 0.9.0 with the fix for value attributes on <option>. Let me know if it works on your end. It's a minor version change so there may be other changes that might be worth viewing in the changelog.


Migrate remaining packages to TypeScript

These packages are not yet migrated to TypeScript:

[x] proppy-preact
[x] proppy-frint-react

Once done, we can remove Babel and ESLint as dependencies from the monorepo.

Closed via #23


[Bug] Adding agents affects spring AOP

Search before asking

[X] I had searched in the issues and found no similar issues.

Apache SkyWalking Component
Java Agent (apache/skywalking-java)
What happened
After I add an agent, a class defaultaopproxyfactory in spring AOP will have one more field by default. This class is a serialized class, resulting in a mismatch in the deserialization suid

What you expected to happen
Adding agent does not affect the existing structure 。
How to reproduce
。。。
Anything else
No response
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

All issues should have good title and reported in English.

Yes, this field is planned to be added, and a lot of plugins require this field to provide high performance. That is one of many reasons of SkyWalking agent having such good performance

At present, spring security should serialize this class to redis. We directly join skywalking through javaagent without other special operations. At present, because the additional field of agent affects suid calculation, we can only deal with spring security?

Sorry, it's translated in English by software. I want to explain that this defaultaopproxyfactory is saved by spring security after the user logs in. I didn't do this. When I request an address, it will be intercepted by spring security and go to redis to obtain the user information. Can it be solved only through spring security 。
（抱歉，英文使用软件翻译的，我的意思想说明，这个DefaultAopProxyFactory是spring security在用户登录之后保存的，我没有做这个事情，当我请求一个地址的时候，会被spring security拦截，前往redis获取用户信息反序列化出现的，是不是只能通过spring security解决）

OK, I may add something enhancing this later. But, generally, Spring should not do like this.

Thank you. Can you release a version after modification, and then close the issue so that I can receive notifications

Thank you. Can you release a version after modification, and then close the issue so that I can receive notifications

Official release is controlled by the PMC, not me. So, I don't know when they call for release.

stream classdesc serialVersionUID =<PHONE_NUMBER>282325166, local class serialVersionUID =<PHONE_NUMBER>487042907

@457775987 I read this error again, it makes me feeling, a dynamic field is not the root cause. The instrumentation itself is.
I am not sure when serialVersionUID gets changed, does rebooting make local class serialVersionUID changed every time?
Also, another question is, does the same application(agent installed) put the AopProxyFactory into the cache? From my understanding, there is one JVM process without agent installed puts the class instance(AopProxyFactory) into the cache, and another JVM process with agent installed read it out, or vice versa. Then, this class structure mismatching happens.

I'm trying to change serialization to Jackson. Compared with jdkserializationredisserializer, I have encountered some problems. OAuth and spring security may need to do a lot of work, which has not been solved yet

After many attempts, there are many problems with JSON serialization and its support is not very good. The current solution is to add the skywalking agent system to declare that defaultaopproxyfactory overrides the one provided by spring AOP, and declare a fixed private static final long serialVersionUID =<PHONE_NUMBER>282325166l;


Update README.md

Coverage remained the same at 100.0% when pulling 99a39267ab0d55229a25cd2ffdacc6918e96b3f5 on Ekman-patch-1 into 6d66a5cfae91619be5be81936a6d8e2537f3ff1e on master.

Coverage remained the same at 100.0% when pulling 99a39267ab0d55229a25cd2ffdacc6918e96b3f5 on Ekman-patch-1 into 6d66a5cfae91619be5be81936a6d8e2537f3ff1e on master.


asyncio support

Hi Joohwan,
We really like your arango library. I am curious what your plans are in regards to asyncio support. I saw you started another repo for it. I'd be interested in helping out where I could to move this forward.
Thanks!

Hi @wshayes,
Thank you for liking python-arango. I am glad to hear that people are finding it useful.
Unfortunately I am too busy right now to start a new project and my knowledge on asyncio (and async programming in general) is novice at best. I'm not sure when I will have time to invest in it. I will keep this issue open for any updates in the future.
Best,
Joohwan

I was about to put this request in when I found it was already in.
I thought about it a bit and here is what I know.
Most of the API would not have to change,  in particular,  you wouldn't have to make every def an async def.  That is because you can make it so that in "async mode" the ordinary functions return a coroutine that returns the result instead of returning a result.  Somebody can await that result in async code and it will be all good.
The first step is making an HttpClient that uses aiohttp instead of requests;  that should be easy,  and the key is that we return a coroutine that returns the result.
Another thing that needs to change are the Executors;  I think we need to build a parallel set of
executors for the four different implementations of executor.
Then there is the plumbing to make sure that when you using the async HttpClient you also get
the async executors.  Then I think we would be in good shape.
The main annoyance I see is that the async/await syntax is Python 3.5+ only.  I think the clean solution is make an interface for the sync/asyncio implementations and then have a second package which
is Py3.5 that contains the asyncio implementation.
I might work up the motivation to make a pull request.

This will also be a very useful feature for me. @joowani, is this still something you plan to invest time in?

Hi!
I was on the need of an async driver for arangodb so I took yours @joowani and did a fork on async style! Feel free to comment it.
https://github.com/bloodbare/aioarangodb
Thanks for your amazing work, its been so easy to adapt to async world

Hi @bloodbare,
Wow this is amazing! I'll make sure to mention your work in the project readme. You should also send an email to ArangoDB team so they can have your driver on their website. Awesome work.

@bloodbare's package looks amazing!
Is there still a plan to support async io in the formal package?
Seems like this repo keeps updating every few months while the asyncio's last commit is from 1y ago and less popular.

Hi!
I was also looking for the asynchronous version and noticed that the version from @bloodbare had not been updated for over a year, so I made a fork (not a copy) and made it fully asynchronous. The current version (1.0.0) fully complient to python-arango 7.2.0.
https://github.com/mirrorrim/aioarango

Hi!
I was also looking for the asynchronous version and noticed that the version from @bloodbare had not been updated for over a year, so I made a fork (not a copy) and made it fully asynchronous. The current version (1.0.0) fully complient to python-arango 7.2.0.
https://github.com/mirrorrim/aioarango
@rennenc please take a look, I'll try to keep it up to date :)

Thanks. I still think it would be great to have to adopted and maintained as part of the same repo.
Nevertheless, will take a look.

Hey I have no problem to keep it on my repo, I've not been updating because
we did not had the need, PRs are more than welcome.
R
Missatge de Rennen @.***> del dia dl., 5 de jul. 2021 a
les 15:18:

Thanks. I still think it would be great to have to adopted and maintained
as part of the same repo.
Nevertheless, will take a look.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/joowani/python-arango/issues/95#issuecomment-874107768,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADW4AKUW6UCNE4JEAJVJMTTWGWMDANCNFSM4GZY6DFA
.

--
Ramon a.k.a bloodbare

Sorry, I didn't see any activity in your repository: 4 open issues and 1 pull request :(

I see .. the PR notification where lost. I'll update it thanks for the ping!
Missatge de mirrorrim @.***> del dia dl., 5 de jul.
2021 a les 17:09:

Sorry, I didn't see any activity in your repository: 4 open issues and 1
pull request :(
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/joowani/python-arango/issues/95#issuecomment-874182590,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADW4APGHJBXR3ESJ6Q3MS3TWHDL5ANCNFSM4GZY6DFA
.

--
Ramon a.k.a bloodbare

First thanks @joowani for all the efforts on the python-arango library it's really a well-written project.
I was looking for an async solution and judging from the comments it seems like having an async version of this library will be beneficial. I see @mirrorrim and @bloodbare has already done some work to make this possible.
@mirrorrim or @bloodbare have you reached out or would you consider making either of your repositories part of the ArangoDB Community?
I think it would be cool to have this part of an official community package just to get more eyes on it and would be happy to contribute in terms of keeping it up to date with the python-arango package. Also would like to avoid creating another fork if possible.

It would be great if @mirrorrim's work could be merged back into the main project as an optional API or it was somehow mad an official community package as @alexvanzyl suggested.

I agree with @incorvia, it would be really nice if we could get the async versions merged into the official community package and maintained as part of it in the future. I also want to highlight that at the moment this issue is closed and I think based on the discussion here it would be worth considering to reopen it and try to get forward with incorporating the work into the main community package, or what do you think?

We would gladly welcome the aioarango package as a part of the arangodb-community org! I think merging it as part of the python-arango package might be beneficial but would need @joowani and @aMahanna to weigh in on that.
If you want to bring the project over please ping me on Slack to discuss the details: https://join.slack.com/t/arangodb-community/shared_invite/zt-1b66mygms-j8TmOdXE7FojR5yA2Yg8kg (Chris.ArangoDB)

I'm open to merging into python-arango, but it will nearly double the codebase (and double the effort to maintain). I'm not sure how much code reuse there could be between sync and async yet. It will probably require a non-trivial amount of work to refactor.

This seems to be the main commit that was written to convert python-arango to use async.. perhaps a PR could be opened that uses this as a starting point. It doesn't look like it would double the code base even if async was conditionalized..
@mirrorrim any suggestions here since you did the work?

In case you want some ideas for how to easily maintain the codebase with both the sync and async version, here's what we did for firedantic (shares many ideas with arangodantic that we also created, but for which we don't have a sync version).

We created an async version of the library (this commit) in addition to the sync version.
A colleague set up a tool that from the async version automatically generates the sync version of the code (this commit) The main things here is the unasync.py script that generates the sync version from the async one and the pre-commit hook that ensures the sync version is generated before committing.
Now any further work is just a matter of updating the async version and ensure the sync version (which is generated automatically) works (mainly reviewing the generated code and run the tests) and possibly do minor adjustments to the unasync.py script.

I should highlight that the idea is from https://github.com/python-trio/unasync and the unasync.py we use is a modified version of https://github.com/encode/httpcore/blob/master/unasync.py.
A suggestion for how to incorporate this same into python-arango would be to restructure the code a into directories for the sync/async versions, then ensure the async version is up to date and then create a modified version of the unasync.py so it's able to generate the sync version from the async one. Some special treatment likely needed for the client (httpx/requests), but that should be possible to do as well. Then after that, the main work for maintaining the library should be just a matter of updating the async version and ensure there's no issues with the generated sync version, so not a lot more work than maintaining just one code base.
There might be better approaches (if you know any, please let me know as well), but this has served us really well for firedantic.

What would be nice is to call a async version of db from the regular. Then start with a limited set of operations that benefit the most of async operations such as reads and insertions.

@joakimnordling unasync.py looks interesting but feels a little hacky. I'm leaning towards @davidschrooten's suggestion a little more currently, but will explore both.

I'm open to merging into python-arango, but it will nearly double the codebase (and double the effort to maintain). I'm not sure how much code reuse there could be between sync and async yet. It will probably require a non-trivial amount of work to refactor.

Why not keep the async definitions truly async?

Today I am using
https://aioarango.readthedocs.io/en/latest/
I also see there is
https://aioarangodb.readthedocs.io/en/latest/
which I have not done a real comparison so I don't feel a lot of reason
to push for changes.
My guess is that you could,  however,  use code generation to make both
sync and async stubs for the all the functions in the library and have a
sync-async library without undue duplication in a library.
------ Original Message ------
From "Slava Shor" @.>
To "ArangoDB-Community/python-arango" @.>
Cc "Paul Houle" @.>; "Comment"
@.>
Date 1/31/2023 4:20:21 PM
Subject Re: [ArangoDB-Community/python-arango] asyncio support (#95)

I'm open to merging into python-arango, but it will nearly double the
codebase (and double the effort to maintain). I'm not sure how much
code reuse there could be between sync and async yet. It will probably
require a non-trivial amount of work to refactor.

Why not keep the async definitions truly async?
—
Reply to this email directly, view it on GitHub
https://github.com/ArangoDB-Community/python-arango/issues/95#issuecomment-1411085005,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAUUCINV2VZ7ZRDFPKQQRGDWVF6ZLANCNFSM4GZY6DFA.
You are receiving this because you commented.Message ID:
@.***>

Today I am using
https://aioarango.readthedocs.io/en/latest/
I also see there is
https://aioarangodb.readthedocs.io/en/latest/
which I have not done an actual comparison so I don't feel a lot of reason
to push for changes.

As of today, neither libraries are actively developed nor maintained.
It's better to keep synchronous and async libraries separately, as it's doubtful that both versions will be used in the same project. As well one may find that in other DB drivers blocking and async versions are being developed separately.

If I run into some problem with aioarango I'll consider patching it.
Very big API wrappers like boto3 use code generation to build sync and
async apis from the same source.  API wrapper code is highly structured
and much of the complexity is embodied in functions that marshall and
unmarshall arguments.  Those functions are nice synchronous functions
that can be used in either environment.
Personally I code almost everything adb-related async even if it doesn't
need it just to keep the async API on my fingertips and for the
possibility I might want to move a bit or two of code into the async
world.  Having almost the same API would make it be convenient for a
programmer to switch and with one code base the problem of maintaining
the sharable functions on both sides is solved.
------ Original Message ------
From "Slava Shor" @.>
To "ArangoDB-Community/python-arango" @.>
Cc "Paul Houle" @.>; "Comment"
@.>
Date 2/5/2023 4:20:17 AM
Subject Re: [ArangoDB-Community/python-arango] asyncio support (#95)

Today I am using
https://aioarango.readthedocs.io/en/latest/
I also see there is
https://aioarangodb.readthedocs.io/en/latest/
which I have not done an actual comparison so I don't feel a lot of
reason
to push for changes.

As of today, neither libraries are actively developed nor maintained.
It's better to keep synchronous and async libraries separately, as it's
doubtful that both versions will be used in the same project. As well
one may find that in other DB drivers blocking and async versions are
being developed separately.
—
Reply to this email directly, view it on GitHub
https://github.com/ArangoDB-Community/python-arango/issues/95#issuecomment-1417186589,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAUUCIOE7YTXIMCZNCLPVOLWV5WFDANCNFSM4GZY6DFA.
You are receiving this because you commented.Message ID:
@.***>

I started a PR at Arangodantic. Maybe you want to have a look at it.
I have chosen (Asyncer)[https://asyncer.tiangolo.com/] as AsyncLib and python-arango as client.
With Asyncer i can call sync code from async etc. This Package is made by tiangolo. The Author from FastApi.
I would be very happy about comments and remarks
Arangodantic V2 PR

Hi @dasTholo,
It's great to see proactive steps like yours, especially on such an important and highly requested feature. We recognize the increasing need for asyncio support and are actively pushing to add it to our roadmap.
I like your asyncify approach, it seems to simplify the implementation and avoid code duplication.

Welll... I have created a simple class, to help with this....
import asyncio
from asyncio import Future
from concurrent.futures import ThreadPoolExecutor
from arango import ArangoClient

class ArangoDBRepository:
    def __init__(self, database_url, database_name, username, password):
        self._executor = ThreadPoolExecutor()
        self._client = ArangoClient(hosts=database_url)
        self._db = self._client.db(database_name, username=username, password=password)

    async def run_in_executor(self, func, *args):
        loop = asyncio.get_event_loop()
        try:
            result = await loop.run_in_executor(self._executor, func, *args)
            return result
        except Exception as e:
            print(f"An error occurred: {e}")
            raise

    async def has_collection(self, collection_name):
        return await self.run_in_executor(lambda: self._db.has_collection(collection_name))

    async def create_collection(self, collection_name, edge=False):
        return await self.run_in_executor(lambda: self._db.create_collection(collection_name, edge))

    async def delete_collection(self, collection_name):
        return await self.run_in_executor(lambda: self._db.delete_collection(collection_name))

    async def create_persistent_index(self,
                                      collection_name,
                                      index_name,
                                      index_fields,
                                      cache_enabled):
        return await self.run_in_executor(
            lambda: self._db.collection(collection_name).add_persistent_index(name=index_name,
                                                                              fields=index_fields,
                                                                              cacheEnabled=cache_enabled))

    async def execute_aql_query(self, query, bind_vars=None):
        def aql_query():
            cursor = self._db.aql.execute(query, bind_vars=bind_vars)
            return [doc for doc in cursor]

        return await self.run_in_executor(aql_query)

    async def insert(self, collection_name,
                     document,
                     overwrite,
                     silent,
                     return_new):
        def insert():
            return self._db.collection(collection_name).insert(document,
                                                               overwrite=overwrite,
                                                               silent=silent,
                                                               return_new=return_new)

        return await self.run_in_executor(insert)

    async def insert_many(self,
                          collection_name,
                          documents,
                          overwrite,
                          silent):
        def insert_many():
            return self._db.collection(collection_name).insert_many(documents, overwrite=overwrite, silent=silent)

        return await self.run_in_executor(insert_many)

    async def batch_find_by_key(self,
                                collection_name,
                                key):
        return await self.run_in_executor(lambda: self._db.collection(collection_name).find({'_key': key}).batch())

Hey @bencz, I like your class. The ThreadPoolExecutor approach is great for ensuring some asynchronous functionality, while keeping the rest of the codebase flexible.
I would like to point out that we're about to start working on an asynchronous driver: python-arango-async. The time allocated for this is limited, but yes, it's finally happening. I can't give a clear estimation of when it would be ready, but I expect about 3 months. We plan to release it gradually, starting with a minimal release once we've covered the basic functionality, and then adding more incrementally. The new driver is going to follow the same python-arango interface for most classes.
In the meantime, workarounds such as yours are a great way to "asynchronize" already existing code. Thanks for providing that example!

I don't see change for last 3 months in python-arango-async. Buit his repo is being updated regulalry. can we have additional apis in thise repo to support async to avoid missing updates in async repo?

any news?

Hi! The async  repo is still work in progress. It has not been updated in a while because the time I can dedicate to that is unfortunately limited, and first priority has to be the currently existing driver, for which we had to work on a major release.
I totally get that the situation is not great in terms of timing, but we'll be resuming work on the async driver very soon (this week or next one). I can't give you an ETA, but if you're looking for a ballpark estimate, we should release a function first version of the async driver sometime this year.
Now @arunzone, regarding the additional APIs for this repo, I'm afraid that's currently not possible on our side. The main reason being that I would prefer to put the limited time that I got into developing the new async driver and maintaining the currently existing code in the synchronous one. Adding code, even temporary, would mean additional maintenance efforts on our side, which is not sustainable in the long run. So please bear with me, and if you really need some async wrapper, I know some of our community members have created some in the past (although I'm not sure how up-to-date these are).
Rest assured that I'm aware of that fact the most of our user base is already using the synchronous driver, therefore the async implementation won't totally deviate from the standard that we're all used to. An easy integration is one of our goals.


Change Ant(s) to Neo

It seems to me the use of Ants- in the code is outdated; we are NEO now. I changed all instances of references to Antshares in the code to NEO, except for the Chinese comments.
I don't know how to test the code; is there someone who could run a test to see if I got every file right?

@jhepkema this compiler folder is a fork of the official NEO compiler, and they still keep the Ants there.
I suggest you first submit this PR to the official compiler repo, as this code here should keep as similar as possible to the original. Once they accept this change, they its fine to accept it here too.


Add documentation XML comments on IDistributedCache and associated classes/types.

At the very least a high level summary for IDistributedCache + descriptions for the behaviour of the main operations (Get/Set/Refresh). E.g. I'm assuming Refresh() is mainly to 'touch' entries with a sliding expiry, etc.

This is added. Thanks Seb!


feat: manage state with zustand

Less re-renders wohoo!
Before.
https://github.com/user-attachments/assets/e8d46a6b-87cb-499d-8df3-488b26291823
After.
https://github.com/user-attachments/assets/1a5eb075-9fa6-439e-9ba5-9e7dc56ba2c0

cc @BrandonRomano

This looks great; I'm going to give a quick spin for this locally.


⚠ Jenkins Plugins Artifactory has degraded performance

In 4c1c49e, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1513 ms

Resolved: Jenkins Plugins Artifactory performance has improved in 16f2020 after 37 minutes.


Merging people (new)

This is @gvwilson's work from #366 (rebased and enhanced).
WIP: missing tests.
This fixes #193.
This suppresses #366.

Will merge as soon as Travis-CI confirms my tests.


Tabs incorrect behavior (tab bar filling widget space) when combined with window menus

This issue is based on a discussion thread. I was asked to open it.

Hello
I combined the menu example with the tabs example, but the result seems to be faulty somehow. The tabs bar fills the whole screen no matter what. I tried to change everything possible (even removing the icons altogether), but nothing helped.
Please find a minimal reproducible example in this repo: https://github.com/TheQuantumPhysicist/IcedTabsTest
build and run with cargo run. On some of my machines I needed to install libfontconfig-dev.
Thank you and have a great day.

Can someone please confirm that this is a bug? I was told to put this in here but never got that confirmation.

seems changing the view to be
pub fn view(
&self,
backend_controller: &NodeBackendController,
) -> Element<'_, Message, iced::Renderer> {
let c = iced::widget::column![
iced::widget::row!(self.main_menu
.view(backend_controller)
.map(Message::MenuMessage)),
iced::widget::row!(self.main_widget
.view(backend_controller)
.map(Message::MainWidgetMessage))
];
    c.into()
}

fixes the issue. This might be because the Menu and other Views cant be within a column together if they are Fill or what not. Please give this a try and let me know what else might need to be looked at.

Yep. This fixes the issue. Thank you!
My opinion: This should be disallowed at compile-time with type-safety. There should be a class of types allowed in column! and row! that only lead to correct results.
Feel free to close this issue, or keep it open to address this problem with type-safety. Cheers!


[Improve] Improve documentation related to platform installation and basic usage

Improve documentation related to platform installation and basic usage
Areas that need to be improved have been highlighted in red in the document

Maybe we can also update the English document, but currently our Chinese document is being reconstructed. I suggest that we update the document together after the Chinese document is updated


modules/systemd: don't install home.conf example

We don't want /srv on NixOS, and /home is already created by
users-groups.nix.
Furthermore, systemd tmpfiles are set up post-activation, and so
it’s extremely difficult for a user to override them.  They can't
even set their own rules in systemd.tmpfiles, because "home.conf"
comes before "nixos.conf" lexicographically, and so systemd always
picks the "home.conf" ones.

Checking nixos modules:
lighthttpd: defaults to /srv/www in the document-root option.
spacecookie: defaults to /srv/gopher in the root option.
tftpd: defaults to /srv/tftp in the path option.
atftpd: defaults to /srv/tftp in the root option.
gitweb: defaults to /srv/git in the projectroot option.
vdr: defaults to /srv/vdr/video in the videoDir option. Also conveniently adds a tmpfiles entry.
IMO this change is fine, but subsequently we should note this in the docs and fix the modules with defaults pointing to /srv/.

Do we also chown the permission of existing directories?
Also the tmpfiles variant creates a subvolume on btrfs?
In doubt we might remove the same functionality from users-groups.nix.
Regarding /srv is there a reason we should not stick to upstream's default of choosing their location. In my experience this makes it easier for users (relying on upstream documentation) and third-party tools that make assumption where data is stored.

Regarding being able to override entries. Should we prefix nixos.conf with a suffix to give it higher priority?

See also https://github.com/NixOS/nixpkgs/pull/85371

Rouven Czerwinski<EMAIL_ADDRESS>writes:

Checking nixos modules:
lighthttpd: defaults to /srv/www in the document-root option.
spacecookie: defaults to /srv/gopher in the root option.
tftpd: defaults to /srv/tftp in the path option.
atftpd: defaults to /srv/tftp in the root option.
gitweb: defaults to /srv/git in the projectroot option.
vdr: defaults to /srv/vdr/video in the videoDir option. Also conveniently adds a tmpfiles entry.
IMO this change is fine, but subsequently we should note this in the docs and fix the modules with defaults pointing to /srv/.

As far as I can tell none of those modules have been modified since the
/srv rule was introduced.  Which presumably means they worked fine
before /srv was added in January, and will continue to work fine after.

Do we also chown the permission of existing directories?

Yes, in users-groups.

Regarding /srv is there a reason we should not stick to upstream's
default of choosing their location. In my experience this makes it
easier for users (relying on upstream documentation) and third-party
tools that make assumption where data is stored.

I just don't think it needs to be globally created by systemd.  It's
trivial for individual users (or modules!) to ensure /srv exists if they
want it, but I don't want a useless empty top-level directory on my
system if nothing is using it.  We got along fine without it until
January of this year.

Jörg Thalheim<EMAIL_ADDRESS>writes:

See also https://github.com/NixOS/nixpkgs/pull/85371

This works, but produces a warning.  If we can do it without the
warning, I think that would be fine for /home.  I still don't think we
need /srv by default.

Jörg Thalheim<EMAIL_ADDRESS>writes:
See also #85371
This works, but produces a warning. If we can do it without the warning, I think that would be fine for /home. I still don't think we need /srv by default.

Maybe not, but it also does not really hurt to have it and I don't want to miss future changes that are applied to this file.

I still don't think we need /srv by default.

Maybe not, but it also does not really hurt to have it and I don't
want to miss future changes that are applied to this file.

I doesn't feel right to me that the systemd project should be deciding
what directories exist on NixOS.  What if they decide to start
automatically creating /usr/lib?

Not only that, but home.conf creates subvolumes on btrfs, which goes against the whole NixOS concept of completely deterministically setting up your machine.

Not only that, but home.conf creates subvolumes on btrfs, which goes against the whole NixOS concept of completely deterministically setting up your machine.

How is this not deterministic?

I still don't think we need /srv by default. Maybe not, but it also does not really hurt to have it and I don't want to miss future changes that are applied to this file.
I doesn't feel right to me that the systemd project should be deciding what directories exist on NixOS. What if they decide to start automatically creating /usr/lib?

We would patch this out for sure, but such a directory is likely in a new file rather than home.conf.

How is this not deterministic?

Awkward wording on my end (I meant something closer to "unexpected", as nowhere in my config do I say "please put /home on a subvolume"), but if I set up my system in a certain way I don't want systemd to go ahead and say "nope that's wrong, here, do it my way". There's no need for a /srv on my non-server machines, and there's no reason for /home to be a subvolume. I'd have set it up that way if I desired it to be so.

such a directory is likely in a new file rather than home.conf.

Dumb question maybe, but what makes /srv being in home.conf different from, say, /usr/lib being in home.conf?

How is this not deterministic?

Awkward wording on my end (I meant something closer to "unexpected", as nowhere in my config do I say "please put /home on a subvolume"), but if I set up my system in a certain way I don't want systemd to go ahead and say "nope that's wrong, here, do it my way". There's no need for a /srv on my non-server machines, and there's no reason for /home to be a subvolume. I'd have set it up that way if I desired it to be so.

That's fair.  Otherwise it is handy if those directories are created with the right permissions/owner for majority of users.

cc @flokli @arianvp

Hm, I think #85371 should allow overriding things and be merged first.
We should check back with upstream why /srv was added to home.conf.
This probably should be in a distro-specific file, or a fhs.conf file, but certainly not in home.conf.

/home shouldn't be automatically created by systemd either.  We have
never even enforced that home directories live in /home, and we've been
creating it fine for ourselves when it actually is needed forever.
If we really need to create a btrfs subvolume for the three users using
it, we can do that in a way that doesn't break existing setups like this
broke mine.  (/home is my home directory, and it's read-only to avoid
untracked state being written there by applications.  With this change,
it was silently being set to read-write every boot, and even finding out
why this was happening wasted hours of my time bisecting Nixpkgs.)
The systemd files are even called "example"s.  They're clearly supposed
to be something that can be adjusted by the distribution.  And since our
distribution has always let home directories be placed wherever the user
wants, we should not be applying policy to /home.
Applying the NixOS overrides first is a good start, but it prints a
warning during the activation process, which has otherwise always been
completely silent (and just as well -- can you imagine how annoying a
noisy activation process would be, given how often it runs?).

FYI the example directory and what is in there and what not is something we came up with; not something that is present in systemd's upstream itself.
https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/linux/systemd/default.nix#L238-L240
But yeh  I agree that the systemd package should be treated as a "pick n place" source; not a definite source of truth.  E.g. we do not ship sysusers.d either because our user-groups activation script implements the same behaviour.
I'm personally in favor of not shipping home.conf at all as I don't see it solving a particular problem at the moment (/home is already created anyway is it?)

The reasoning behind /home being a subvolume is explained in https://github.com/systemd/systemd/commit/fed2b07ebc9e8694b5b326923356028f464381ce:

tmpfiles: make /home and /var btrfs subvolumes by default when booted up with them missing
This way the root subvolume can be left read-only easily, and variable
and user data writable with explicit quota set.
(as per the tmpfiles.d docs, on non-btrfs filesystems, it'll just be a directory.)

This doesn't sound too wrong for me. I personally like to be able to use the btrfs send/receive/snapshot functionality for /home.
/home being the home directory for a specific user sound a bit exotic to me  - assuming overriding these directives since https://github.com/NixOS/nixpkgs/pull/85371 is possible, would using this to make it non-writable again work for you, too?

You can also use this to override the systemd provided file with an empty one:
environment.etc."tmpfiles.d/home.conf".source = lib.mkForce (pkgs.writeText "dummy" "");

The snippet above should make it easy enough to remove home.conf in case it is not wanted.


Update custom styles md

Fixed some typos (very few!), and modified the language a bit to (hopefully) make things a bit clearer

@jdiddydave thanks! Finally starting to get back to all of this 😅


MCP report is blank on ACS commons version 5.2.0 on ACS 2022.4.7138

Required Information

[x] ACS AEM Commons Version: 2022.4.7138, same works on feature works on ACS version 2022.1.6198 with ACS commons version 5.2.0/5.3.0
MCP_report_error_log.txt

[x] Reproducible on Latest? Yes

Expected Behavior
After running the MCP program, when generating the report in HTML/excel it should generate/print report in/to excel/page.
Actual Behavior
After running the MCP program, when generating the report in HTML we see a blank page and when generating the report in excel, we get a 500 error.
Steps to Reproduce
This issue can be seen in "Asset Report" feature as well.

Click on "Start Process" button
Select "Asset Report" process
Accept the defaults and click Start
once finished, navigate to the list of processes screen "/apps/acs-commons/content/manage-controlled-processes.html"
Click on the "Asset Report" instance and select "view" or "Downlod (Excel)"

Installed ACS AEM SDK version "2022.4.7138" and installed ACS commons version 5.2.0. Run any utility which generates reports based on "/apps/acs-commons/components/utilities/process-instance/process-generic-report" component. We can either view the report in HTML or download the report in excel. Currently, the HTML is blank and throws 500 error while downloading the excel. Attached the error log "MCP_report_error_log.txt" for reference.
The same features are working fine in ACS AEM version "2022.1.6198".
MCP_report_error_log.txt
Links
/apps/acs-commons/content/manage-controlled-processes.html
/apps/acs-commons/components/utilities/process-instance/process-generic-report

I can't reproduce it on my local "2022.4.7138" + ACS Commons v5.2.0. I also tried with 5.3.1-SNAPSHOT which is also working fine.
Are you running the report for the entire /content/dam or for a sub-folder? Is there anything special in the data ?
Going by the exception, the report has no columns. Can you please confirm it? The exception message contains the path of report node,e.g. /var/acs-commons/mcp/instances/6A64C1698EE3B81B/_jcr_content/report

Does it have the columns property ?  Please provide the full json dump of the MCP instance.
Also, are you running the report as an admin ?I wonder if it matters.

Thanks Yegor for looking into this issue. Please find below the info related to your questions and more details.

I am running the "Tag Report" MCP process with defaults:

Yes, I see the columns:

Attached the json dump of the instance, please refer to "json_dump.txt"
Yes, I am running the report as admin

After raising the bug, I did more analysis and found :

The report works fine with vanilla instance of AEM "2022.4.7138" with just ACS commons v5.2.0
But, it fails to load when I install AEM forms add on. I am installing AEM form add-on v"aem-forms-addon-2<IP_ADDRESS>-220400.far". To install AEM Forms add on, I stopped the AEM instance, created "install" folder under "crx-quickstart" and placed the ".far" file and started the instance.

After this the report stops loading. It seems to me that the sling model "com.adobe.acs.commons.mcp.model.GenericBlobReport" in "/apps/acs-commons/components/utilities/process-instance/process-blob-report/process-blob-report.html" doesn't load. on the rendered HTML, thead or tbody is blank.

json_dump.txt

It's a bug in the Forms SDK. I traced it down to SlingModelEnumInjector in the AEM Forms Output Batch API Impl bundle.
SlingModelEnumInjector is a custom Sling Models injector which breaks the contract for multi-valued fields. The impact is that injecting multi-valued fields is broken AEM-wide. I won't be surprised the issue will pop up in other parts of AEM.
The simplest test case that demonstrates the issue:
@Model(adaptables = Resource.class)
public class UserModel  {

    @Inject
    private List<String> columns; // BUG: multi-valued fields won't be injected

    @Inject
    private String name; // single-value fields are injected fine

}

and this is exactly what's happening with the ACS Commons report models: they can't initialize columns which are stored in a String[] field.
Injectors in Sling Models are pluggable and when injecting a value into a model, AEM iterates over all registered injectors until one returns a non-null value. SlingModelEnumInjector breaks this contract and returns an empty collection. It should return null instead and let other injectors,  until ValueMapInjector will read the data.

We can switch to injector-specific annotations in the ACS AEM Commons models to prevent the wrong injector from kicking in.

Thanks Yegor and kwin for quick reply.
@kwin, could you please provide an example on how to switch to "Injector-specific annotations"? This issue is breaking many features in our project. I will try and see if this can be fixed with this approach.

I raised a ticket with Adobe regarding this issue. Thanks.

https://sling.apache.org/documentation/bundles/models.html#injector-specific-annotations

@kwin thanks for the tip. Changing @Inject to @ValueMapValue did the trick and MCP Reports started to work with the Forms SDK. The PR with the fix is https://github.com/Adobe-Consulting-Services/acs-aem-commons/pull/2846

Thanks Yegor and Kevin. Any idea on the date when this would be released?


added formFactor "TABLET" and WGS84

formFactor "TABLET" can be seen in newer Records.json files
added WGS84 to coordinates for clarification

Great, thanks!


Fix memory leak in Simulator::recomputeNavMesh

Motivation and Context
This patch aims to fix a memory leak in Simulator::recomputeNavMesh as pointed out here: https://github.com/facebookresearch/habitat-sim/pull/333#discussion_r360183036
How Has This Been Tested
Open to thoughts on memory leak testing. 🤔
Types of changes

[ ] Docs change / refactoring / dependency upgrade
[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have completed my CLA (see CONTRIBUTING)
[ ] I have added tests to cover my changes.
[x] All new and existing tests passed.

@erikwijmans Think we may also have a problem here with navData?
if (!dtCreateNavMeshData(&params, &navData, &navDataSize)) {

Might be simpler to create a wrapper class.

There's this also: https://doc.magnum.graphics/corrade/classCorrade_1_1Containers_1_1ScopeGuard.html :)

A std::unique_ptr with a custom deleter would probably be best: https://en.cppreference.com/w/cpp/memory/unique_ptr

@erikwijmans Think we may also have a problem here with navData?
if (!dtCreateNavMeshData(&params, &navData, &navDataSize)) {

@aclegg3 No, we don't, the navMesh will take ownership of navData later on.

Thanks for all the discussion, I've implemented @msbaines solution. Thoughts?

Codecov Report

Merging #398 into master will decrease coverage by 7.66%.
The diff coverage is 79.8%.

@@            Coverage Diff             @@
##           master     #398      +/-   ##
==========================================
- Coverage   55.59%   47.92%   -7.67%
==========================================
  Files         167      122      -45
  Lines        7866     5794    -2072
  Branches       82       84       +2
==========================================
- Hits         4373     2777    -1596
+ Misses       3493     3017     -476

Flag
Coverage Δ

#CPP
52.39% <75%> (+0.42%)
:arrow_up:

#JavaScript
10% <90.62%> (+4.46%)
:arrow_up:

#Python
?

Impacted Files
Coverage Δ

src/esp/scene/SemanticScene.h
14.28% <ø> (+10.2%)
:arrow_up:

src/esp/gfx/Simulator.cpp
63.92% <33.33%> (-0.79%)
:arrow_down:

src/esp/scene/GibsonSemanticScene.h
62.5% <62.5%> (ø)

src/esp/scene/test/GibsonSceneTest.cpp
72% <72%> (ø)

src/esp/scene/GibsonSemanticScene.cpp
87.87% <87.87%> (ø)

src/esp/bindings_js/tests/test_utils.js
90.62% <90.62%> (ø)

tests/test_physics.py

tests/test_simulator.py

...bitat_sim/agent/controls/pyrobot_noisy_controls.py

habitat_sim/agent/controls/default_controls.py

... and 48 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 47f0262...160f5ac. Read the comment docs.

Thank you @aclegg3. Just one suggestion :)

No problem, thanks for catching this. 👀


Cafe

Thanks for the cleanup Dave. I was wondering if we still wanted to support
cafe though?
sent from my mobile device
On 15 Jun 2015 1:38 am, "David Horwitz"<EMAIL_ADDRESS>wrote:

You can view, comment on, or merge this pull request online at:
https://github.com/sakaiproject/sakai/pull/662
Commit Summary

Merge branch 'master' of https://github.com/sakaiproject/sakai
Merge branch 'master' of https://github.com/sakaiproject/sakai
Merge branch 'master' of https://github.com/sakaiproject/sakai
Merge branch 'master' of https://github.com/sakaiproject/sakai
Merge branch 'master' of https://github.com/sakaiproject/sakai
Merge branch 'master' of https://github.com/sakaiproject/sakai
Merge branch 'master' of https://github.com/sakaiproject/sakai
Merge branch 'master' of https://github.com/sakaiproject/sakai
Merge branch 'master' of https://github.com/sakaiproject/sakai
Merge branch 'master' of https://github.com/sakaiproject/sakai
Merge branch 'master' of https://github.com/sakaiproject/sakai
Merge branch 'master' of https://github.com/sakaiproject/sakai
Merge branch 'master' of https://github.com/sakaiproject/sakai
Merge branch 'master' of https://github.com/sakaiproject/sakai
Merge branch 'master' of https://github.com/sakaiproject/sakai
Merge branch 'master' of https://github.com/sakaiproject/sakai
Merge branch 'master' of https://github.com/sakaiproject/sakai
Merge branch 'master' of https://github.com/sakaiproject/sakai
Cafe

File Changes

M common/pom.xml
https://github.com/sakaiproject/sakai/pull/662/files#diff-0 (80)
M pom.xml
https://github.com/sakaiproject/sakai/pull/662/files#diff-1 (75)
M profile2/pom.xml
https://github.com/sakaiproject/sakai/pull/662/files#diff-2 (4)

Patch Links:

https://github.com/sakaiproject/sakai/pull/662.patch
https://github.com/sakaiproject/sakai/pull/662.diff

—
Reply to this email directly or view it on GitHub
https://github.com/sakaiproject/sakai/pull/662.

-1

I recall that cafe should be removed as it caused some confusion. Also with the removal of the indies and the move to a simpler build I think cafe has out-lived its usefulness?

If cafe works again and David's keeping it up to date I think it's worthwhile having.
It's useful to have a cutdown build that you can both deploy with and import into Eclipse so you don't see everything at once. It also helps us keep Sakai modular and think clearly about where responsibilities and functionality lies.

I might get shot for this but if we had working sets in Eclipse we could manage this a bit better without cafe. Eclipse doesnt grind to a halt anymore (hooray for Moore's law). That would require committing the .project files though I think.
Probably needs to go to core team if we want the cafe build back in.

:gun: :smile:
How are you generating you .classpath/.project files @steveswinsburg?
m2eclipse plugin or mvn eclipse:eclipse?

I use mvn eclipse: eclipse. Sets it up really well.
sent from my mobile device
On 16 Jun 2015 10:55 pm, "Matthew Buckett"<EMAIL_ADDRESS>wrote:

[image: :gun:] [image: :smile:]
How are you generating you .classpath/.project files @steveswinsburg
https://github.com/steveswinsburg?
m2eclipse plugin or mvn eclipse:eclipse?
—
Reply to this email directly or view it on GitHub
https://github.com/sakaiproject/sakai/pull/662#issuecomment-112418208.

Yeah we should never add IDE specific files to the code base!
I also use mvn eclipse:eclipse and import everything no issues.


proxy path doesn't work in dbless mode with ingress controller

NOTE: GitHub issues are reserved for bug reports only.
For anything else, please join the conversation
in Kong Nation https://discuss.konghq.com/c/kubernetes.

Summary
SUMMARY_GOES_HERE
Kong Ingress controller version
0.5.0
Kong or Kong Enterprise version
1.2.1
Kubernetes version
1.11.7
paste kubectl version output
k version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.7", GitCommit:"65ecaf0671341311ce6aea0edab46ee69f65d59e", GitTreeState:"clean", BuildDate:"2019-01-24T19:32:00Z", GoVersion:"go1.10.7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"7a578febe155a7366767abce40d8a16795a96371", GitTreeState:"clean", BuildDate:"2019-05-01T04:05:01Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Environment

Cloud provider or hardware configuration: AWS
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Install tools: helm
Others:
installed with stable/kong chart in db less mode

What happened
I've installed kong in dbless mode with ingress. I try to use KongIngress to redirect all traffic from /echo/ to /bar/echo as described in example "with Service resource" from: https://github.com/Kong/kubernetes-ingress-controller/blob/master/docs/guides/using-kongingress-resource.md
I use Ingress instead of Service.
Ingress and KongIngress config:
---
apiVersion: configuration.konghq.com/v1
kind: KongIngress
metadata:
  name: kongingress-echo
proxy:
  path: /bar/
route:
  strip_path: false
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: echo
  annotations:
    configuration.konghq.com: kongingress-echo
spec:
  rules:
  - http:
      paths:
      - path: /echo
        backend:
          serviceName: echo
          servicePort: 80

When i hit my Loadbalancer for kong-proxy http://myip/echo/  i got following response from echo service:
Hostname: echo-d778ffcd8-r8z4p

Pod Information:
	node name:	ip-10-10-1-43
	pod name:	echo-d778ffcd8-r8z4p
	pod namespace:	default
	pod IP:	<IP_ADDRESS>

Server values:
	server_version=nginx: 1.12.2 - lua: 10010

Request Information:
	client_address=<IP_ADDRESS>
	method=GET
	real path=/echo/
	query=
	request_version=1.1
	request_scheme=http
	request_uri=http://my_ELB:8080/echo/

Based on real_path i see that strip_path parameter in KongIngress works just fine, but proxy: path:  doesn't.
I saw suggestions to use https://docs.konghq.com/hub/kong-inc/serverless-functions/
but not sure what is the difference between the plugin and proxy: path from KongIngress.
Expected behvaior
Echo client return real_path that contains "/bar/"
Steps To Reproduce

Install kong in dbless mode with ingress enabled
Install echo service as described in https://github.com/Kong/kubernetes-ingress-controller/blob/master/docs/guides/using-kongingress-resource.md
Configure Ingress and KongIngress as i described

Did you patch your service to add configuration.konghq.com annotation to your Kubernetes echo service as described below?
kubectl patch service echo -p '{"metadata":{"annotations":{"configuration.konghq.com":"demo-customization"}}}'

The proxy and upstream sections are associated with a Kubernetes Service, while the route part need to be associated with Ingress resource.

Hi @hbagdi ,
I haven't seen in the documentation that proxy and upstream are associated with Service and route with Ingress, i though i can use either Ingress or Service.
Hence proxy path is associated with services how can i achieve below configuration:
http://load_balancer/url1/  --->> http://my_service/path_A/
http://load_balancer/url2/  --->> http://my_service/path_B/
I could create 2 separate services associated with the same pod, but it doesn't seems like following DRY methodology.

I could create 2 separate services associated with the same pod, but it doesn't seems like following DRY methodology.

Although this solution works, I think it's important to take a step back and define the separate of service and Ingress.
What you are really looking for a way to change your path before you proxy the request to your service.
You should use URL rewrite plugin or use serverless plugin in Kong to change the URL before it is sent upstream.
Usually, you want to do the following:
If you have the following:
http://load_balancer/url1/ --->> http://my_service/some/path/path_A/
http://load_balancer/url2/ --->> http://my_service/some/path/path_B/

http://my_service/some/path/ belongs to your service configuration and then the customization of path_A or path_B belongs to your Ingress or Route definition.
Hope this helps.
If your question is answered, please close this issue. Thanks!

Hi @hbagdi,
Thanks for your response. I've checked both plugins and i see both of them are barely documented plus rewrite plugin is a 3rd party maintained with last commit a year ago.
Is this kong ingress behavior subject to a change in a future or there are great reasons why it was designed and implemented like this? Don't get me wrong, but trying the go from simple kong config like below:
 - name: my-service-path1
        host: my-service
        port: 8080
        path: /path1
        routes:
        - name: loadbalancer-url1
          paths: [/url1]
          strip_path: true
 - name: my-service-path2
        host: my-service
        port: 8080
        path: /path2
        routes:
        - name: loadbalancer-url2
          paths: [/url2]
          strip_path: true

I need to add Ingress, KongIngress and external plugin to make it work which introduces additional complexity.

You have a fair point about the complexity and we would like to simplify it.
We don't have a working solution yet on how to work around the limitation of narrow Ingress API, which doesn't offer a lot.
In the example you have, you had to create two services in Kong, and with Kong Ingress, you will have to exactly the same. Kubernetes services map to Kong Services, hence you need to create two Kubernetes services.
If this is one off, I'd recommend that you do duplicate the Kubernetes service.

Harry, thanks for the clarification it's clear now! :)


colcon build package should default to building package and dependencies

With catkin_tools, it was simple to build a package with its dependencies, which is one of the most common use cases when building packages. With colcon, this can be done using the flag --packages-up-to, which is a bit cumbersome. I suggest to add colcon build package as a shorthand for colcon build --packages-up-to. Maybe also add colcon build --no-deps package for colcon build --packages-select package.

Please take a look at the colcon-alias extension. You should be able to add an alias to do something like what you're asking.
Setup: colcon alias build-up-to build --packages-up-to
Execution: colcon build-up-to foo would translate to colcon build --packages-up-to foo.

Hi @timonegk, were you able to give colcon-alias a try?

Yes, thanks, that works. (With colcon alias --command build --packages-up-to -- build-up-to.)


Is it possible to render to a view rather than winodw?

Hi,
Looking into using Veldrid for rendering on iOS/Android/Mac/Win but one thing I noticed is that it seems to need a window to be created. However my current use case is to embed 3d rendering inside an existing window/view. Is it currently possible?
As a side note been having problems running the mobile samples as always says can't find x embedded asset for example on iOS on texture cube demo crashes with "Cube-vertex.ios.metallib" not existing.
I also had to re-setup Veldrid NuGet dependancies in the TextureCube.iOS / other projects as they were not found.

it seems to need a window to be created

Generally speaking, you don’t need a window at all to use Veldrid (with the exception of the OpenGL backend). You can create a device and render entirely outside of any visible window if you want. To present images to the user (e.g. to a window or other application view), you do need a “Swapchain”, a special kind of resource. You can create Swapchains for several native UI views, including UIViews on iOS, SurfaceViews on Android, and custom controls in WPF. However, making Swapchains is platform-specific (because it relies on the specific operating system and UI framework being used) and can be a bit confusing. If I knew what you were trying to accomplish I could provide some more specific help and guidance.

As a side note been having problems running the mobile samples as always says can't find x embedded asset for example on iOS on texture cube demo crashes with "Cube-vertex.ios.metallib" not existing.

What version of macOS and Xcode do you have installed? I recently found that the newer versions of Xcode changed some parameters in the metal compiler, which could have affected the sample projects since they invoke the compiler during the build.

Hi Thanks for getting back so quickly.
To try and summarize, I have a native application where user goes thru many steps. But at various points they can do a 3d preview. The 3d preview comes with a native side panel and native controls as an overlay. The app in question is also multi platform. I currently created the 3d part using OpenTK but since OpenGL is no longer officially supported on Mac. I thought it's time to switch to a more cross backend platform, hence why I think Veldrid is a perfect solution.
If i'm understanding correctly it is mostly case of setting up the viewing layer (i.e for metal a CAMetalLayer ) as you would normally do natively on MacOS then using render cycle code as per OffScreen example.
However i'm not quite sure what to do about this part, and if any extra steps needed...
            GraphicsDeviceOptions options = new GraphicsDeviceOptions(
                debug: false,
                swapchainDepthFormat: PixelFormat.R16_UNorm,
                syncToVerticalBlank: true,
                resourceBindingModel: ResourceBindingModel.Improved);
#if DEBUG
            options.Debug = true;
#endif
            _gd = VeldridStartup.CreateGraphicsDevice(_window, options);
            _factory = new DisposeCollectorResourceFactory(_gd.ResourceFactory);
            GraphicsDeviceCreated?.Invoke(_gd, _factory, _gd.MainSwapchain);

Any example code to get me started on MacOS would be greatly appreciated.
Im currently running Mac OS 10.14.1 (Mojave) with latest stable VisualStudio for Mac, with Xcode 10.1

To try and summarize, I have a native application where user goes thru many steps. But at various points they can do a 3d preview. The 3d preview comes with a native side panel and native controls as an overlay. The app in question is also multi platform.

I see -- so the 3D viewer (written in C#) interoperates in some way with the native application and serves as a visualization of sorts? I think what would help most would be to understand how you are currently creating the OpenGL widget. If you indeed have a CAMetalLayer (or can create one), then it should be very easy to integrate with Veldrid. Currently, there isn't a way to create a Swapchain from an already-created CAMetalLayer, but a subset of the NSWindow Swapchain code can be pulled out to support that directly very easily.

since OpenGL is no longer officially supported on Mac. I thought it's time to switch to a more cross backend platform, hence why I think Veldrid is a perfect solution.

Yes, Veldrid will be a very good replacement -- performance on macOS will be significantly better using the Metal backend, and you'll have more consistent behavior all around compared to OpenGL (on all platforms).

Im currently running Mac OS 10.14.1 (Mojave) with latest stable VisualStudio for Mac, with Xcode 10.1

I suspect you are hitting the same problem I just dealt with after my own Xcode upgrade, then. I will try to implement a fix for ShaderGen sometime soon and update the samples repository.

@equinox2k After looking into this some more over the weekend, I added support for creating a Metal Swapchain directly from an NSView. I will upload a usage example sometime soon.

Awesome I can’t thank you enough, I’m looking forward to be able to contribute to this project once I get my bearings :)

@mellinoe thanks, I've now created an android example (using Vulkan) and deployed to device but getting...
"SamplerFilter.Anisotropic cannot be used unless GraphicsDeviceFeatures.SamplerAnisotropy is supported."
When CreateVulkan is called...
        GraphicsDeviceOptions options = new GraphicsDeviceOptions(
                debug: false,
                swapchainDepthFormat: null,
                syncToVerticalBlank: false,
                resourceBindingModel: ResourceBindingModel.Improved,
                preferDepthRangeZeroToOne: true,
                preferStandardClipSpaceYDirection: true);

            Device = GraphicsDevice.CreateVulkan(options);

Is there a way to disable filter?

Interesting, your device doesn’t support anisotropic filtering. I’ll put up a fix for that tonight as well. What device are you using, out of curiosity?

Samsung S8, btw I did pr for shadergen which solves fact my Xcode is not Xcode.app but updating samples to use new version I still have problems but at least fixes that part :)

I've fixed the Metal backend on iOS in the prerelease packages, and also allowed the Vulkan backend to work when anisotropic filtering is unavailable.

@mellinoe I can confirm fixes are working. I have one issue with my android test VeldridActivityExample located in https://github.com/equinox2k/VeldridNSViewExample
I have forced app to use OpenGLES and works in emulator, however I get error on device. Wondering if works for you?

@mellinoe as to doing a windows example I have got a bit stuck...
I am assuming for windows I create a custom user control and do a similar setup to android. However im not sure what to pass into...
public static SwapchainSource CreateWin32(IntPtr hwnd, IntPtr hinstance)

Heres a WPF example that I shared when this came up previously:
https://gist.github.com/mellinoe/2ba885476f51ef867715e5aa64b390de#file-veldridcomponent-cs

@melinoe thanks so much. Spent most of day wondering why I get problem above with OpenGLES on android. Wondering if for some reason velgrid thinks i want to use compressed textures or is doing checks and crashing causing device to return null.

Wondering if for some reason velgrid thinks i want to use compressed textures or is doing checks and failing causing device to return null.

I'm not sure that's a fatal error message, since it appears twice, and seems to keep doing a bunch of stuff after the first occurrence. I can't really identify what is actually going wrong other than that -- the log seems normal for the most part. I see some hits on Google from other folks with Samsung devices complaining about that error message, but it's not clear to me whether it's actually problematic, or just a weird driver message that is harmless.

<uses-feature android:glEsVersion="0x00020000" android:required="true" />

Veldrid actually requires GLES 3.0, so you might try changing it to 0x00030000
Feel free to open a different issue if you keep running up against that. The message overall suggests you should add something to the app's permissions, but I'm not sure what.


Update README.md

Just a little typo, cryto -> crypto.

You will always have a typo in crypto ;) (kidding )

@luisincrespo  Thanks for the crypto typo fix. Keep it up. Cheers.


No new lines on wuxiaworld

When downloading from wuxiaword all newlines get deleted.
Dunno if that is wanted…

Try new version


Small fixed in doxygen

Signed-off-by: Alejandro Hernández<EMAIL_ADDRESS>🦟 Bug fix
Summary
Minor doxygen fixes
Checklist

[x] Signed all commits for DCO
[ ] Added tests
[ ] Updated documentation (as needed)
[ ] Updated migration guide (as needed)
[x] codecheck passed (See contributing)
[x] All tests passed (See test coverage)
[x] While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

This pull request has been mentioned on Gazebo Community. There might be relevant details there:
https://community.gazebosim.org/t/new-ignition-releases-2022-03-01-citadel-edifice-fortress/1313/1


Problems with running on Linux

On Linux (and probably) Mac, it does not simply work to enter the Godot application path as program name in launch.json. Instead, the debugger (or the C# debugger extension?) enforces a dotnet- prefix on the path when no .exe suffix was found. This means, when entering "godot" as "program" name, it will look for dotnet-godot in the PATH and subsequently, the Godot application requires a GodotSharp directory in its working directory (where it is called). Both can be linked to avoid copying the files over to a new location.
What I tested so far: it will not work to rename the Godot application to an .exe suffix. Because ultimately, it will fail with an BadImageFormatException-like Exception.
It runs when the dotnet-godot command is faked to point to the godot application but pausing and breakpoints won't work at all.
I really would like to know if anyone managed to make Godot 4+ C# debugging running on Linux and how.

@krisutofu
Its working fine for me.
For convenient, I just symlink to Gotdot excutable like so
ln -s '/path/to/your/godot/Godot_v4.2.2-stable_mono_linux.x86_64' /usr/local/bin/godot

Then in the settings.json
{
  ...
  "godot-dotnet-tools.executablePath": "godot"
}

Make sure /usr/local/bin is in your PATH, or do an absolute path /usr/local/bin/godot on the above
When a new version of Godot comes out, we just need to relink

@krisutofu Its working fine for me. For convenient, I just symlink to Gotdot excutable like so
ln -s '/path/to/your/godot/Godot_v4.2.2-stable_mono_linux.x86_64' /usr/local/bin/godot

Then in the settings.json
{
  ...
  "godot-dotnet-tools.executablePath": "godot"
}

Make sure /usr/local/bin is in your PATH, or do an absolute path /usr/local/bin/godot on the above When a new version of Godot comes out, we just need to relink

Thank you for the reminder. I "fixed" the issue by using Microsoft's proprietary version of VSCode. I originally installed just any version which was the non-proprietary one.


Windows overlay backend depends on external HNS network

Expected Behavior
The Windows overlay backend assumes you have created an external HNS network, and fails if it can't find one. This isn't very clearly documented and we had to dig into Microsoft's SDN code to find reference to what this backend is expecting. This also has the unfortunate consequence of requiring users to import Microsoft's hns.psm1 library to add the Add-HNSNetwork function to Powershell. If it were just part of Flannel it could use hcsshim to do the same thing.
It would be better and more intuitive if the overlay backend would just create this itself. I don't know if there's a reason it can't do that.
Possible Solution
Create the external network if it doesn't already exist.
@ksubrmnn

Hi Ben,
In Overlay, the vxlan0 network is created by FlannelD  here
However, you need to create a separate Overlay Network for vxlan0 to be layered on top of as is done here
Note that this is also done for L2Bridge (this is so that you don't lose connectivity as the cbr0 network is created/torn down). For L2Bridge, Flannel will still work properly without that External network.
The reason that you need the External for Overlay is because in Windows, the DR NIC is created when an Overlay network is present. This NIC's MAC address is needed in the subnet lease attributes. So we create an Overlay network called "External" to get that DR MAC. Then when we have the subnet lease, we can create the vxlan0 network.
I agree with you that it is preferable to not have to depend on that external network.
Can you edit this issue to reflect what I've explained above? Regarding the actual backend network, both implementations will either reuse an existing network or create a new one. The only difference is that Overlay has a hard dependency on that external network.

Thanks Kalya, I started digging in more and realized it is an external network and not the container subnet as it is on L2Bridge. I updated my issue to try to describe the problem better.

@madhanrm @daschott

@zawachte-msft


Smoke tests: Do we require AZURE_AUTHORITY_HOST (or whatever the final name is)

Right now 3 of the 4 smoke tests fall back to a default authority host associated with the public cloud. Should we require that an authority host be provided for the smoke tests or use the default behavior?
Continuing discussion from: https://github.com/Azure/azure-sdk-for-net/pull/10417#pullrequestreview-374611099

You've changed the world a bunch since this issue was opened.  Not sure if this matters or still applies any longer.

Closing this as it doesn't relevant, or at least, a priority since it hasn't been touched. In general the azure authority host config is settled with a best practice for now, at least in live tests. When we onboard samples for net smoke tests, we may open this issue again.


Scanner abruptly exits with "error: can't start new thread" after certain extent

Here is an Error I keep getting for larger websites , and the scanner abruptly shuts down after scpidering the URL's to a certain extent.
_Exception in thread Thread-2467:
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 801, in __bootstrap_inner
self.run()
File "C:\Python27\lib\site-packages\nyawc\CrawlerThread.py", line 116, in run
self.__callback(self.__queue_item, new_requests, failed)
File "C:\Python27\lib\site-packages\nyawc\Crawler.py", line 252, in __request_finish
self.__spawn_new_requests()
File "C:\Python27\lib\site-packages\nyawc\Crawler.py", line 105, in __spawn_new_requests
if self.__spawn_new_request():
File "C:\Python27\lib\site-packages\nyawc\Crawler.py", line 125, in __spawn_new_request
self.__request_start(first_in_line)
File "C:\Python27\lib\site-packages\nyawc\Crawler.py", line 217, in __request_start
thread.start()
File "C:\Python27\lib\threading.py", line 736, in start
_start_new_thread(self._bootstrap, ())
error: can't start new thread

@jovyn Thank you for reporting this issue. I think this is due to a design issue in the crawler.
Currently, the crawler starts new threads in the thread that finished. I'm going to change it so that new threads will be started from the main thread.

@jovyn Do you know after how much requests this happens?

Referring to my Burp logs (I chained the scanner via Burp)  I can about 8492 requests.
This may not be the exact number everytime this happens. Maeanwhile let me run it against some other site and share the feedback with you.

@jovyn I think I fixed this issue on the develop branch, however, I can't test it since I could not reproduce the issue after ~8500 requests. Could you test if it works for you?

Hey @tijme I installed the new version from the develop branch and ran the scanner, but I am getting the below errors for larger scanning requests (about 7000 requests) :
[ERROR]
error : Memory allocation failed : growing buffer
error : Memory allocation failed : growing buffer
followed by :
_Traceback (most recent call last):
File ".\extended.develop.py", line 177, in
main()
File ".\extended.develop.py", line 84, in main
driver.start()
File "C:\Python27\lib\site-packages\acstis\Driver.py", line 136, in start
crawler.start_with(startpoint)
File "C:\Python27\lib\site-packages\nyawc\Crawler.py", line 95, in start_with
self.__crawler_start()
File "C:\Python27\lib\site-packages\nyawc\Crawler.py", line 166, in __crawler_start
self.__spawn_new_requests()
File "C:\Python27\lib\site-packages\nyawc\Crawler.py", line 111, in __spawn_new_requests
if self.__spawn_new_request():
File "C:\Python27\lib\site-packages\nyawc\Crawler.py", line 131, in __spawn_new_request
self.__request_start(first_in_line)
File "C:\Python27\lib\site-packages\nyawc\Crawler.py", line 230, in __request_start
thread.start()
File "C:\Python27\lib\threading.py", line 736, in start
_start_new_thread(self._bootstrap, ())
thread.error: can't start new thread

I did not explicitly specify the number of threads (-mt option) and I the default number of threads the scanner takes is 8.
I think the scanner is trying to create too many threads and is going beyond the  default value of 8.
Reference : https://johnsofteng.wordpress.com/2010/03/05/python-thread-error-cant-start-new-thread/

Tried specifying the number of threads using the -mt option. I still get the same error.

@jovyn Did you update the dependencies? If you run pip freeze the nyawc dependency must be on version 1.7.9.

Yes I did . The nyawc dependency is on 1.7.9
`PS C:\Angular-CSTI-Scanner\Develop_Branch> pip freeze
acstis==3.0.2
alabaster==0.7.10
Babel==2.5.1
beautifulsoup4==4.6.0
certifi==2<IP_ADDRESS>
chardet==3.0.4
colorama==0.3.9
colorlog==2.10.0
docutils==0.14
idna==2.5
imagesize==0.7.1
Jinja2==2.9.6
lxml==4.0.0
MarkupSafe==1.0
nyawc==1.7.9
pockets==0.5.1
Pygments==2.2.0
pytz==2017.2
requests==2.18.1
requests-toolbelt==0.8.0
selenium==3.4.3
six==1.11.0
snowballstemmer==1.2.1
Sphinx==1.5.5
sphinx-better-theme==0.13
sphinxcontrib-napoleon==0.6.1
urllib3==1.21.1`

Hi @jovyn, I tried to reproduce the issue again but unfortunately I did not succeed yet.
The thread count of the process is always 9 at my machine (8 + the main thread). Besides that the memory usage always stays at ~ 80MB (also if I'm scanning thousands of URLs).
Would it be possible for you to provide the information below?

What OS (including version) do you use?
What version of Python are you using?
What version of ACSTIS are you using?
How did you install ACSTIS?
Are you still using a proxy?
Could you sent me the exact command (including arguments) you are executing (e.g. python extended.py -d http://example.com)?
Would it be possible for you to share the URL you are scanning?
How much memory is the process using?
How many thread is the process using?

Hey @tijme ,
Below are my responses >
What OS (including version) do you use?  --  Windows 10 (64-bit)
What version of Python are you using?  -- Python 2.7.13
What version of ACSTIS are you using? -- Version 3.0.2
How did you install ACSTIS? --  I downloaded the .zip from (https://github.com/tijme/angularjs-csti-scanner/tree/develop) and then did a pip install --upgrade --force-reinstall .\angularjs-csti-scanner-develop.zip
Are you still using a proxy?  -- Yes. however i tried acstis w/o the proxy settings as well.
Could you sent me the exact command (including arguments) you are executing (e.g. python extended.py -d http://example.com)?   -- python .\extended.py -c -d "https://www.example.com/" -tc "Burp_Cert.pem" -mt 12  also tried acstis -c -d "https://www.example.com/" -tc "Burp_Cert.pem" -mt 12 . I have also tried without the -mt option as well as tried lesser threads (8 or 9)
Would it be possible for you to share the URL you are scanning?  -- Sorry @tijme I wont be able to share the target URL.
How much memory is the process using?  -- Not sure, will get back to you on this
How many thread is the process using?   -- Not sure, will get back to you on this

Hi @jovyn,
Did you already have the time to look at the last two questions?
Thanks in advance.

@jovyn I was able to reproduce the issue and I found out it occurs on Windows only.
The scanner used a lot of memory per request since it cached the lxml tree of every response. When it reached the ~8000 requests it used an average of 2GB of memory. 2GB is the limit for a 32-bit application running on the 64-bit Windows 10 OS (source), which is why it crashed on your machine.
By removing the lxml tree caching you can now scan up to 60000 requests with 2GB of memory. I will continue to improve this in the future. If you want to scan more requests already you could try to install the 64-bit version of Python.
I just released this fix in version 3.0.5 of ACSTIS.

Thanks Time :)
On 1 Nov 2017 4:44 a.m., "Tijme Gommers"<EMAIL_ADDRESS>wrote:

Closed #6 https://github.com/tijme/angularjs-csti-scanner/issues/6.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tijme/angularjs-csti-scanner/issues/6#event-1319905492,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADXbBCpQqhTAzn6JSis-9y_Oc3nsYf1mks5sx6ntgaJpZM4Pb6QY
.